# index.html.md

# Dask

*Dask is a Python library for parallel and distributed computing.*  Dask is:

- **Easy** to use and set up (it’s just a Python library)
- **Powerful** at providing scale, and unlocking complex algorithms
- and **Fun** 🎉

<script src="https://fast.wistia.com/embed/medias/l9sgt2saht.jsonp" async></script><script src="https://fast.wistia.com/assets/external/E-v1.js" async></script><div class="wistia_responsive_padding" style="padding:75.0% 0 0 0;position:relative;"><div class="wistia_responsive_wrapper" style="height:100%;left:0;position:absolute;top:0;width:100%;"><div class="wistia_embed wistia_async_l9sgt2saht seo=true videoFoam=true" style="height:100%;position:relative;width:100%">&nbsp;</div></div></div>

## How to Use Dask

Dask provides several APIs.  Choose one that works best for you:

### Tasks

Dask Futures parallelize arbitrary for-loop style Python code,
providing:

- **Flexible** tooling allowing you to construct custom
  pipelines and workflows
- **Powerful** scaling techniques, processing several thousand
  tasks per second
- **Responsive** feedback allowing for intuitive execution,
  and helpful dashboards

Dask futures form the foundation for other Dask work

Learn more at [Futures Documentation](futures.html)
or see an example at [Futures Example](https://examples.dask.org/futures.html)

```python
from dask.distributed import LocalCluster
client = LocalCluster().get_client()

# Submit work to happen in parallel
results = []
for filename in filenames:
    data = client.submit(load, filename)
    result = client.submit(process, data)
    results.append(result)

# Gather results back to local computer
results = client.gather(results)
```

![image](images/futures-graph.png)

### DataFrames

Dask Dataframes parallelize the popular pandas library, providing:

- **Larger-than-memory** execution for single machines, allowing you
  to process data that is larger than your available RAM
- **Parallel** execution for faster processing
- **Distributed** computation for terabyte-sized datasets

Dask Dataframes are similar in this regard to Apache Spark, but use the
familiar pandas API and memory model.  One Dask dataframe is simply a
collection of pandas dataframes on different computers.

Learn more at [DataFrame Documentation](dataframe.html)
or see an example at [DataFrame Example](https://examples.dask.org/dataframe.html)

```python
import dask.dataframe as dd

# Read large datasets in parallel
df = dd.read_parquet("s3://mybucket/data.*.parquet")
df = df[df.value < 0]
result = df.groupby(df.name).amount.mean()

result = result.compute()  # Compute to get pandas result
result.plot()
```

![image](images/dask-dataframe.svg)

### Arrays

Dask Arrays parallelize the popular NumPy library, providing:

- **Larger-than-memory** execution for single machines, allowing you
  to process data that is larger than your available RAM
- **Parallel** execution for faster processing
- **Distributed** computation for terabyte-sized datasets

Dask Arrays allow scientists and researchers to perform intuitive and
sophisticated operations on large datasets but use the
familiar NumPy API and memory model.  One Dask array is simply a
collection of NumPy arrays on different computers.

Learn more at [Array Documentation](array.html)
or see an example at [Array Example](https://examples.dask.org/array.html)

```python
import dask.array as da

x = da.random.random((10000, 10000))
y = (x + x.T) - x.mean(axis=1)

z = y.var(axis=0).compute()
```

![image](images/dask-array.svg)

Xarray wraps Dask array and is a popular downstream project, providing
labeled axes and simultaneously tracking many Dask arrays together,
resulting in more intuitive analyses.  Xarray is popular and accounts
for the majority of Dask array use today especially within geospatial
and imaging communities.

Learn more at [Xarray Documentation](https://docs.xarray.dev/en/stable/)
or see an example at [Xarray Example](https://examples.dask.org/xarray.html)

```python
import xarray as xr

ds = xr.open_mfdataset("data/*.nc")
da.groupby('time.month').mean('time').compute()
```

![image](https://docs.xarray.dev/en/stable/_static/logos/Xarray_Logo_RGB_Final.png)

### Bags

Dask Bags are simple parallel Python lists, commonly used to process
text or raw Python objects.  They are …

- **Simple** offering easy map and reduce functionality
- **Low-memory** processing data in a streaming way that minimizes memory use
- **Good for preprocessing** especially for text or JSON data prior
  ingestion into dataframes

Dask bags are similar in this regard to Spark RDDs or vanilla
Python data structures and iterators.  One Dask bag is simply a
collection of Python iterators processing in parallel on different computers.

Learn more at [Bag Documentation](bag.html)
or see an example at [Bag Example](https://examples.dask.org/bag.html)

```python
import dask.bag as db

# Read large datasets in parallel
lines = db.read_text("s3://mybucket/data.*.json")
records = (lines
    .map(json.loads)
    .filter(lambda d: d["value"] > 0)
)
df = records.to_dask_dataframe()
```

## How to Install Dask

Installing Dask is easy with `pip` or `conda`.

Learn more at [Install Documentation](install.html)

### pip

```default
python -m pip install "dask[complete]"
```

### conda

```default
conda install dask
```

## How to Deploy Dask

You can use Dask on a single machine, or deploy it on distributed hardware.

Learn more at [Deploy Documentation](deploying.html)

### Local

Dask can set itself up easily in your Python session if you create a
`LocalCluster` object, which sets everything up for you.

```python
from dask.distributed import LocalCluster
cluster = LocalCluster()
client = cluster.get_client()

# Normal Dask work ...
```

Alternatively, you can skip this part, and Dask will operate within a
thread pool contained entirely with your local process.

### Cloud

[Coiled](https://docs.coiled.io/user_guide/index.html?utm_source=dask-docs&utm_medium=homepage)
is a commercial SaaS product that deploys Dask clusters on cloud platforms like AWS, GCP, and Azure.

```python
import coiled
cluster = coiled.Cluster(
   n_workers=100,
   region="us-east-2",
   worker_memory="16 GiB",
   spot_policy="spot_with_fallback",
)
client = cluster.get_client()
```

Learn more at [Coiled Documentation](https://docs.coiled.io/user_guide/index.html?utm_source=dask-docs&utm_medium=homepage)

### HPC

The [Dask-Jobqueue project](https://jobqueue.dask.org) deploys
Dask clusters on popular HPC job submission systems like SLURM, PBS, SGE, LSF,
Torque, Condor, and others.

```python
from dask_jobqueue import PBSCluster
cluster = PBSCluster(
   cores=24,
   memory="100GB",
   queue="regular",
   account="my-account",
)
cluster.scale(jobs=100)
client = cluster.get_client()
```

Learn more at [Dask-Jobqueue Documentation](https://jobqueue.dask.org)

### Kubernetes

The [Dask Kubernetes project](https://kubernetes.dask.org) provides
a Dask Kubernetes Operator for deploying Dask on Kubernetes clusters.

```python
from dask_kubernetes.operator import KubeCluster
cluster = KubeCluster(
   name="my-dask-cluster",
   image="ghcr.io/dask/dask:latest",
   resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "64Gi"}},
)
cluster.scale(10)
client = cluster.get_client()
```

Learn more at [Dask Kubernetes Documentation](https://kubernetes.dask.org)

## Learn with Examples

Dask is used widely across all industries and scales.  Dask is used
anywhere Python is used and people experience pain due to large scale data, or
intense computing.

You can learn more about Dask applications at the following sources:

- [Dask Examples](https://examples.dask.org)
- [Dask YouTube Channel](https://youtube.com/@dask-dev)

Additionally, we encourage you to look through the reference documentation on
this website related to the API that most closely matches your application.

Dask was designed to be **easy to use** and **powerful**.  We hope that it’s
able to help you have fun with your work.


# index.html.md

# How To…

This section contains snippets and suggestions about how to perform different actions
using Dask. If you have an idea of a how-to that we should add, please
[make a suggestion](https://github.com/dask/dask/tree/main/docs/source/how-to)!

<!-- Articles in this section should be short and not contain much explanation. -->

# How To...

* [Connect to remote data](connect-to-remote-data.md)
* [Debug](debug.md)
* [Extend sizeof](extend-sizeof.md)
* [Selecting the collection backend](selecting-the-collection-backend.md)
* [Use GPUs](../gpu.md)


# 10-minutes-to-dask.html.md

# 10 Minutes to Dask

This is a short overview of Dask geared towards new users.
There is much more information contained in the rest of the documentation.

![Dask overview. Dask is composed of three parts: collections, task graphs, and schedulers.](images/dask-overview.svg)

We normally import Dask as follows:

```python
>>> import numpy as np
>>> import pandas as pd

>>> import dask.dataframe as dd
>>> import dask.array as da
>>> import dask.bag as db
```

Based on the type of data you are working with, you might not need all of these.

## Creating a Dask Object

You can create a Dask object from scratch by supplying existing data and optionally
including information about how the chunks should be structured.

### DataFrame

See [Dask DataFrame](dataframe.md).

```python
>>> index = pd.date_range("2021-09-01", periods=2400, freq="1h")
... df = pd.DataFrame({"a": np.arange(2400), "b": list("abcaddbe" * 300)}, index=index)
... ddf = dd.from_pandas(df, npartitions=10)
... ddf
Dask DataFrame Structure:
                        a       b
npartitions=10
2021-09-01 00:00:00  int64  object
2021-09-11 00:00:00    ...     ...
...                    ...     ...
2021-11-30 00:00:00    ...     ...
2021-12-09 23:00:00    ...     ...
Dask Name: from_pandas, 10 tasks
```

Now we have a Dask DataFrame with 2 columns and 2400 rows composed of 10 partitions where
each partition has 240 rows. Each partition represents a piece of the data.

Here are some key properties of a DataFrame:

```python
>>> # check the index values covered by each partition
... ddf.divisions
(Timestamp('2021-09-01 00:00:00', freq='H'),
Timestamp('2021-09-11 00:00:00', freq='H'),
Timestamp('2021-09-21 00:00:00', freq='H'),
Timestamp('2021-10-01 00:00:00', freq='H'),
Timestamp('2021-10-11 00:00:00', freq='H'),
Timestamp('2021-10-21 00:00:00', freq='H'),
Timestamp('2021-10-31 00:00:00', freq='H'),
Timestamp('2021-11-10 00:00:00', freq='H'),
Timestamp('2021-11-20 00:00:00', freq='H'),
Timestamp('2021-11-30 00:00:00', freq='H'),
Timestamp('2021-12-09 23:00:00', freq='H'))

>>> # access a particular partition
... ddf.partitions[1]
Dask DataFrame Structure:
                  a       b
npartitions=1
2021-09-11     int64  object
2021-09-21       ...     ...
Dask Name: blocks, 11 tasks
```

### Array

See [Array](array.md).

```ipython3
import numpy as np
import dask.array as da

data = np.arange(100_000).reshape(200, 500)
a = da.from_array(data, chunks=(100, 100))
a
```

```none
dask.array<array, shape=(200, 500), dtype=int64, chunksize=(100, 100), chunktype=numpy.ndarray>
```

<table>
    <tr>
        <td>
            <table style="border-collapse: collapse;">
                <thead>
                    <tr>
                        <td> </td>
                        <th> Array </th>
                        <th> Chunk </th>
                    </tr>
                </thead>
                <tbody>

                    <tr>
                        <th> Bytes </th>
                        <td> 781.25 kiB </td>
                        <td> 78.12 kiB </td>
                    </tr>

                    <tr>
                        <th> Shape </th>
                        <td> (200, 500) </td>
                        <td> (100, 100) </td>
                    </tr>
                    <tr>
                        <th> Dask graph </th>
                        <td colspan="2"> 10 chunks in 1 graph layer </td>
                    </tr>
                    <tr>
                        <th> Data type </th>
                        <td colspan="2"> int64 numpy.ndarray </td>
                    </tr>
                </tbody>
            </table>
        </td>
        <td>
        <svg width="170" height="98" style="stroke:rgb(0,0,0);stroke-width:1" >

  <!-- Horizontal lines -->
  <line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
  <line x1="0" y1="24" x2="120" y2="24" />
  <line x1="0" y1="48" x2="120" y2="48" style="stroke-width:2" />

  <!-- Vertical lines -->
  <line x1="0" y1="0" x2="0" y2="48" style="stroke-width:2" />
  <line x1="24" y1="0" x2="24" y2="48" />
  <line x1="48" y1="0" x2="48" y2="48" />
  <line x1="72" y1="0" x2="72" y2="48" />
  <line x1="96" y1="0" x2="96" y2="48" />
  <line x1="120" y1="0" x2="120" y2="48" style="stroke-width:2" />

  <!-- Colored Rectangle -->
  <polygon points="0.0,0.0 120.0,0.0 120.0,48.0 0.0,48.0" style="fill:#ECB172A0;stroke-width:0"/>

  <!-- Text -->
  <text x="60.0" y="68.0" font-size="1.0rem" font-weight="100" text-anchor="middle" >500</text>
  <text x="140.0" y="24.0" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,140.0,24.0)">200</text>
</svg>
        </td>
    </tr>
</table>

Now we have a 2D array with the shape (200, 500) composed of 10 chunks where
each chunk has the shape (100, 100). Each chunk represents a piece of the data.

Here are some key properties of a Dask Array:

```ipython3
# inspect the chunks
a.chunks
```

```none
((100, 100), (100, 100, 100, 100, 100))
```

```ipython3
# access a particular block of data
a.blocks[1, 3]
```

```none
dask.array<blocks, shape=(100, 100), dtype=int64, chunksize=(100, 100), chunktype=numpy.ndarray>
```

<table>
    <tr>
        <td>
            <table style="border-collapse: collapse;">
                <thead>
                    <tr>
                        <td> </td>
                        <th> Array </th>
                        <th> Chunk </th>
                    </tr>
                </thead>
                <tbody>

                    <tr>
                        <th> Bytes </th>
                        <td> 78.12 kiB </td>
                        <td> 78.12 kiB </td>
                    </tr>

                    <tr>
                        <th> Shape </th>
                        <td> (100, 100) </td>
                        <td> (100, 100) </td>
                    </tr>
                    <tr>
                        <th> Dask graph </th>
                        <td colspan="2"> 1 chunks in 2 graph layers </td>
                    </tr>
                    <tr>
                        <th> Data type </th>
                        <td colspan="2"> int64 numpy.ndarray </td>
                    </tr>
                </tbody>
            </table>
        </td>
        <td>
        <svg width="170" height="170" style="stroke:rgb(0,0,0);stroke-width:1" >

  <!-- Horizontal lines -->
  <line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
  <line x1="0" y1="120" x2="120" y2="120" style="stroke-width:2" />

  <!-- Vertical lines -->
  <line x1="0" y1="0" x2="0" y2="120" style="stroke-width:2" />
  <line x1="120" y1="0" x2="120" y2="120" style="stroke-width:2" />

  <!-- Colored Rectangle -->
  <polygon points="0.0,0.0 120.0,0.0 120.0,120.0 0.0,120.0" style="fill:#ECB172A0;stroke-width:0"/>

  <!-- Text -->
  <text x="60.0" y="140.0" font-size="1.0rem" font-weight="100" text-anchor="middle" >100</text>
  <text x="140.0" y="60.0" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,140.0,60.0)">100</text>
</svg>
        </td>
    </tr>
</table>

### Bag

See [Bag](bag.md).

```python
>>> b = db.from_sequence([1, 2, 3, 4, 5, 6, 2, 1], npartitions=2)
... b
dask.bag<from_sequence, npartitions=2>
```

Now we have a sequence with 8 items composed of 2 partitions where each partition
has 4 items in it. Each partition represents a piece of the data.

## Indexing

Indexing Dask collections feels just like slicing NumPy arrays or pandas DataFrame.

### DataFrame

```python
>>> ddf.b
Dask Series Structure:
npartitions=10
2021-09-01 00:00:00    object
2021-09-11 00:00:00       ...
                        ...
2021-11-30 00:00:00       ...
2021-12-09 23:00:00       ...
Name: b, dtype: object
Dask Name: getitem, 20 tasks

>>> ddf["2021-10-01": "2021-10-09 5:00"]
Dask DataFrame Structure:
                                 a       b
npartitions=1
2021-10-01 00:00:00.000000000  int64  object
2021-10-09 05:00:59.999999999    ...     ...
Dask Name: loc, 11 tasks
```

### Array

```ipython3
a[:50, 200]
```

```none
dask.array<getitem, shape=(50,), dtype=int64, chunksize=(50,), chunktype=numpy.ndarray>
```

<table>
    <tr>
        <td>
            <table style="border-collapse: collapse;">
                <thead>
                    <tr>
                        <td> </td>
                        <th> Array </th>
                        <th> Chunk </th>
                    </tr>
                </thead>
                <tbody>

                    <tr>
                        <th> Bytes </th>
                        <td> 400 B </td>
                        <td> 400 B </td>
                    </tr>

                    <tr>
                        <th> Shape </th>
                        <td> (50,) </td>
                        <td> (50,) </td>
                    </tr>
                    <tr>
                        <th> Dask graph </th>
                        <td colspan="2"> 1 chunks in 2 graph layers </td>
                    </tr>
                    <tr>
                        <th> Data type </th>
                        <td colspan="2"> int64 numpy.ndarray </td>
                    </tr>
                </tbody>
            </table>
        </td>
        <td>
        <svg width="170" height="79" style="stroke:rgb(0,0,0);stroke-width:1" >

  <!-- Horizontal lines -->
  <line x1="0" y1="0" x2="120" y2="0" style="stroke-width:2" />
  <line x1="0" y1="29" x2="120" y2="29" style="stroke-width:2" />

  <!-- Vertical lines -->
  <line x1="0" y1="0" x2="0" y2="29" style="stroke-width:2" />
  <line x1="120" y1="0" x2="120" y2="29" style="stroke-width:2" />

  <!-- Colored Rectangle -->
  <polygon points="0.0,0.0 120.0,0.0 120.0,29.030629010473877 0.0,29.030629010473877" style="fill:#ECB172A0;stroke-width:0"/>

  <!-- Text -->
  <text x="60.0" y="49.030629010473874" font-size="1.0rem" font-weight="100" text-anchor="middle" >50</text>
  <text x="140.0" y="14.515314505236939" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,140.0,14.515314505236939)">1</text>
</svg>
        </td>
    </tr>
</table>

### Bag

A Bag is an unordered collection allowing repeats. So it is like a list, but it doesn’t
guarantee an ordering among elements. There is no way to index Bags since they are
not ordered.

## Computation

Dask is lazily evaluated. The result from a computation isn’t computed until
you ask for it. Instead, a Dask task graph for the computation is produced.

Anytime you have a Dask object and you want to get the result, call `compute`:

### DataFrame

```python
>>> ddf["2021-10-01": "2021-10-09 5:00"].compute()
                     a  b
2021-10-01 00:00:00  720  a
2021-10-01 01:00:00  721  b
2021-10-01 02:00:00  722  c
2021-10-01 03:00:00  723  a
2021-10-01 04:00:00  724  d
...                  ... ..
2021-10-09 01:00:00  913  b
2021-10-09 02:00:00  914  c
2021-10-09 03:00:00  915  a
2021-10-09 04:00:00  916  d
2021-10-09 05:00:00  917  d

[198 rows x 2 columns]
```

### Array

```python
>>> a[:50, 200].compute()
array([  200,   700,  1200,  1700,  2200,  2700,  3200,  3700,  4200,
      4700,  5200,  5700,  6200,  6700,  7200,  7700,  8200,  8700,
      9200,  9700, 10200, 10700, 11200, 11700, 12200, 12700, 13200,
      13700, 14200, 14700, 15200, 15700, 16200, 16700, 17200, 17700,
      18200, 18700, 19200, 19700, 20200, 20700, 21200, 21700, 22200,
      22700, 23200, 23700, 24200, 24700])
```

### Bag

```python
>>> b.compute()
[1, 2, 3, 4, 5, 6, 2, 1]
```

## Methods

Dask collections match existing numpy and pandas methods, so they should feel familiar.
Call the method to set up the task graph, and then call `compute` to get the result.

### DataFrame

```python
>>> ddf.a.mean()
dd.Scalar<series-..., dtype=float64>

>>> ddf.a.mean().compute()
1199.5

>>> ddf.b.unique()
Dask Series Structure:
npartitions=1
   object
      ...
Name: b, dtype: object
Dask Name: unique-agg, 33 tasks

>>> ddf.b.unique().compute()
0    a
1    b
2    c
3    d
4    e
Name: b, dtype: object
```

Methods can be chained together just like in pandas

```python
>>> result = ddf["2021-10-01": "2021-10-09 5:00"].a.cumsum() - 100
... result
Dask Series Structure:
npartitions=1
2021-10-01 00:00:00.000000000    int64
2021-10-09 05:00:59.999999999      ...
Name: a, dtype: int64
Dask Name: sub, 16 tasks

>>> result.compute()
2021-10-01 00:00:00       620
2021-10-01 01:00:00      1341
2021-10-01 02:00:00      2063
2021-10-01 03:00:00      2786
2021-10-01 04:00:00      3510
                        ...
2021-10-09 01:00:00    158301
2021-10-09 02:00:00    159215
2021-10-09 03:00:00    160130
2021-10-09 04:00:00    161046
2021-10-09 05:00:00    161963
Freq: H, Name: a, Length: 198, dtype: int64
```

### Array

```python
>>> a.mean()
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=(), chunktype=numpy.ndarray>

>>> a.mean().compute()
49999.5

>>> np.sin(a)
dask.array<sin, shape=(200, 500), dtype=float64, chunksize=(100, 100), chunktype=numpy.ndarray>

>>> np.sin(a).compute()
array([[ 0.        ,  0.84147098,  0.90929743, ...,  0.58781939,
         0.99834363,  0.49099533],
      [-0.46777181, -0.9964717 , -0.60902011, ..., -0.89796748,
      -0.85547315, -0.02646075],
      [ 0.82687954,  0.9199906 ,  0.16726654, ...,  0.99951642,
         0.51387502, -0.4442207 ],
      ...,
      [-0.99720859, -0.47596473,  0.48287891, ..., -0.76284376,
         0.13191447,  0.90539115],
      [ 0.84645538,  0.00929244, -0.83641393, ...,  0.37178568,
      -0.5802765 , -0.99883514],
      [-0.49906936,  0.45953849,  0.99564877, ...,  0.10563876,
         0.89383946,  0.86024828]])

>>> a.T
dask.array<transpose, shape=(500, 200), dtype=int64, chunksize=(100, 100), chunktype=numpy.ndarray>

>>> a.T.compute()
array([[    0,   500,  1000, ..., 98500, 99000, 99500],
      [    1,   501,  1001, ..., 98501, 99001, 99501],
      [    2,   502,  1002, ..., 98502, 99002, 99502],
      ...,
      [  497,   997,  1497, ..., 98997, 99497, 99997],
      [  498,   998,  1498, ..., 98998, 99498, 99998],
      [  499,   999,  1499, ..., 98999, 99499, 99999]])
```

Methods can be chained together just like in NumPy

```python
>>> b = a.max(axis=1)[::-1] + 10
... b
dask.array<add, shape=(200,), dtype=int64, chunksize=(100,), chunktype=numpy.ndarray>

>>> b[:10].compute()
array([100009,  99509,  99009,  98509,  98009,  97509,  97009,  96509,
      96009,  95509])
```

### Bag

Dask Bag implements operations like `map`, `filter`, `fold`, and
`groupby` on collections of generic Python objects.

```python
>>> b.filter(lambda x: x % 2)
dask.bag<filter-lambda, npartitions=2>

>>> b.filter(lambda x: x % 2).compute()
[1, 3, 5, 1]

>>> b.distinct()
dask.bag<distinct-aggregate, npartitions=1>

>>> b.distinct().compute()
[1, 2, 3, 4, 5, 6]
```

Methods can be chained together.

```python
>>> c = db.zip(b, b.map(lambda x: x * 10))
... c
dask.bag<zip, npartitions=2>

>>> c.compute()
[(1, 10), (2, 20), (3, 30), (4, 40), (5, 50), (6, 60), (2, 20), (1, 10)]
```

## Visualize the Task Graph

So far we’ve been setting up computations and calling `compute`. In addition to
triggering computation, we can inspect the task graph to figure out what’s going on.

### DataFrame

```python
>>> result.dask
HighLevelGraph with 7 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7f129df7a9d0>
1. from_pandas-0b850a81e4dfe2d272df4dc718065116
2. loc-fb7ada1e5ba8f343678fdc54a36e9b3e
3. getitem-55d10498f88fc709e600e2c6054a0625
4. series-cumsum-map-131dc242aeba09a82fea94e5442f3da9
5. series-cumsum-take-last-9ebf1cce482a441d819d8199eac0f721
6. series-cumsum-d51d7003e20bd5d2f767cd554bdd5299
7. sub-fed3e4af52ad0bd9c3cc3bf800544f57

>>> result.visualize()
```

![Dask task graph for the Dask dataframe computation. The task graph shows a "loc" and "getitem" operations selecting a small section of the dataframe values, before applying a cumulative sum "cumsum" operation, then finally subtracting a value from the result.](images/10_minutes_dataframe_graph.png)

### Array

```python
>>> b.dask
HighLevelGraph with 6 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fd33a4aa400>
1. array-ef3148ecc2e8957c6abe629e08306680
2. amax-b9b637c165d9bf139f7b93458cd68ec3
3. amax-partial-aaf8028d4a4785f579b8d03ffc1ec615
4. amax-aggregate-07b2f92aee59691afaf1680569ee4a63
5. getitem-f9e225a2fd32b3d2f5681070d2c3d767
6. add-f54f3a929c7efca76a23d6c42cdbbe84

>>> b.visualize()
```

![Dask task graph for the Dask array computation. The task graph shows many "amax" operations on each chunk of the Dask array, that are then aggregated to find "amax" along the first array axis, then reversing the order of the array values with a "getitem" slicing operation, before an "add" operation to get the final result.](images/10_minutes_array_graph.png)

### Bag

```python
>>> c.dask
HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7f96d0814fd0>
1. from_sequence-cca2a33ba6e12645a0c9bc0fd3fe6c88
2. lambda-93a7a982c4231fea874e07f71b4bcd7d
3. zip-474300792cc4f502f1c1f632d50e0272

>>> c.visualize()
```

![Dask task graph for the Dask bag computation. The task graph shows a "lambda" operation, and then a "zip" operation is applied to the partitions of the Dask bag. There is no communication needed between the bag partitions, this is an embarrassingly parallel computation.](images/10_minutes_bag_graph.png)

## Low-Level Interfaces

Often when parallelizing existing code bases or building custom algorithms, you
run into code that is parallelizable, but isn’t just a big DataFrame or array.

### Delayed: Lazy

[Dask Delayed](delayed.md) lets you to wrap individual function calls into a lazily constructed task graph:

```python
import dask

@dask.delayed
def inc(x):
   return x + 1

@dask.delayed
def add(x, y):
   return x + y

a = inc(1)       # no work has happened yet
b = inc(2)       # no work has happened yet
c = add(a, b)    # no work has happened yet

c = c.compute()  # This triggers all of the above computations
```

### Futures: Immediate

Unlike the interfaces described so far, Futures are eager. Computation starts as soon
as the function is submitted (see [Futures](futures.md)).

```python
from dask.distributed import Client

client = Client()

def inc(x):
   return x + 1

def add(x, y):
   return x + y

a = client.submit(inc, 1)     # work starts immediately
b = client.submit(inc, 2)     # work starts immediately
c = client.submit(add, a, b)  # work starts immediately

c = c.result()                # block until work finishes, then gather result
```

#### NOTE
Futures can only be used with distributed cluster. See the section below for more
information.

## Scheduling

After you have generated a task graph, it is the scheduler’s job to execute it
(see [Scheduling](scheduling.md)).

By default, for the majority of Dask APIs, when you call `compute` on a Dask object,
Dask uses the thread pool on your computer (a.k.a threaded scheduler) to run computations in parallel.
This is true for [Dask Array](array.md), [Dask DataFrame](dataframe.md),
and [Dask Delayed](delayed.md). The exception being [Dask Bag](bag.md)
which uses the multiprocessing scheduler by default.

If you want more control, use the distributed scheduler instead. Despite having
“distributed” in it’s name, the distributed scheduler works well
on both single and multiple machines. Think of it as the “advanced scheduler”.

### Local

This is how you set up a cluster that uses only your own computer.

```python
>>> from dask.distributed import Client
...
... client = Client()
... client
<Client: 'tcp://127.0.0.1:41703' processes=4 threads=12, memory=31.08 GiB>
```

### Remote

This is how you connect to a cluster that is already running.

```python
>>> from dask.distributed import Client
...
... client = Client("<url-of-scheduler>")
... client
<Client: 'tcp://127.0.0.1:41703' processes=4 threads=12, memory=31.08 GiB>
```

There are a variety of ways to set up a remote cluster. Refer to
[how to deploy dask clusters](deploying.md) for more
information.

Once you create a client, any computation will run on the cluster that it points to.

## Diagnostics

When using a distributed cluster, Dask provides a diagnostics dashboard where you can
see your tasks as they are processed.

```python
>>> client.dashboard_link
'http://127.0.0.1:8787/status'
```

To learn more about those graphs take a look at [Dashboard Diagnostics](dashboard.md).


# adaptive.html.md

# Adaptive deployments

## Motivation

Most Dask deployments are static with a single scheduler and a fixed number of
workers.  This results in predictable behavior, but is wasteful of resources in
two situations:

1. The user may not be using the cluster, or perhaps they are busy
   interpreting a recent result or plot, and so the workers sit idly,
   taking up valuable shared resources from other potential users
2. The user may be very active, and is limited by their original allocation.

Particularly efficient users may learn to manually add and remove workers
during their session, but this is rare.  Instead, we would like the size of a
Dask cluster to match the computational needs at any given time.  This is the
goal of the *adaptive deployments* discussed in this document.

<br/>
![Dask adaptive scaling](images/dask-adaptive.svg)
<br/>

These are particularly helpful for interactive workloads, which are characterized by long
periods of inactivity interrupted with short bursts of heavy activity.
Adaptive deployments can result in both faster analyses that give users much
more power, but with much less pressure on computational resources.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/dViyEqOMA8U"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

## Adaptive

To make setting up adaptive deployments easy, some Dask deployment solutions
offer an `.adapt()` method.  Here is an example with
[dask_kubernetes.KubeCluster](https://kubernetes.dask.org/en/latest/kubecluster.html).

```python
from dask_kubernetes import KubeCluster

cluster = KubeCluster()
cluster.adapt(minimum=0, maximum=100)  # scale between 0 and 100 workers
```

For more keyword options, see the Adaptive class below:

| [`Adaptive`](#distributed.deploy.Adaptive)(cluster[, interval, minimum, ...])   | Adaptively allocate workers based on scheduler load.   |
|---------------------------------------------------------------------------------|--------------------------------------------------------|

## Dependence on a Resource Manager

The Dask scheduler does not know how to launch workers on its own. Instead, it
relies on an external resource scheduler like Kubernetes above, or
Yarn, SGE, SLURM, Mesos, or some other in-house system (see [how to deploy Dask
clusters](deploying.md) for options).  In order to use adaptive deployments, you
must provide some mechanism for the scheduler to launch new workers.  Typically,
this is done by using one of the solutions listed in the [how to deploy Dask
clusters](deploying.md), or by subclassing from the Cluster superclass and
implementing that API.

| [`Cluster`](#distributed.deploy.Cluster)([asynchronous, loop, quiet, name, ...])   | Superclass for cluster objects   |
|------------------------------------------------------------------------------------|----------------------------------|

## Scaling Heuristics

The Dask scheduler tracks a variety of information that is useful to correctly
allocate the number of workers:

1. The historical runtime of every function and task that it has seen,
   and all of the functions that it is currently able to run for users
2. The amount of memory used and available on each worker
3. Which workers are idle or saturated for various reasons, like the presence
   of specialized hardware

From these, it is able to determine a target number of workers by dividing the
cumulative expected runtime of all pending tasks by the `target_duration`
parameter (defaults to five seconds).  This number of workers serves as a
baseline request for the resource manager.  This number can be altered for a
variety of reasons:

1. If the cluster needs more memory, then it will choose either the target
   number of workers or twice the current number of workers (whichever is
   larger)
2. If the target is outside of the range of the minimum and maximum values,
   then it is clipped to fit within that range

Additionally, when scaling down, Dask preferentially chooses those workers that
are idle and have the least data in memory.  It moves that data to other
machines before retiring the worker.  To avoid rapid cycling of the cluster up
and down in size, we only retire a worker after a few cycles have gone by where
it has consistently been a good idea to retire it (controlled by the
`wait_count` and `interval` parameters).

## API

### *class* distributed.deploy.Adaptive(cluster: [Cluster](#distributed.deploy.Cluster), interval: [str](https://docs.python.org/3/library/stdtypes.html#str) | [float](https://docs.python.org/3/library/functions.html#float) | timedelta | [None](https://docs.python.org/3/library/constants.html#None) = None, minimum: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, maximum: [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, wait_count: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, target_duration: [str](https://docs.python.org/3/library/stdtypes.html#str) | [float](https://docs.python.org/3/library/functions.html#float) | timedelta | [None](https://docs.python.org/3/library/constants.html#None) = None, worker_key: Callable[[[distributed.scheduler.WorkerState](https://distributed.dask.org/en/latest/scheduling-state.html#distributed.scheduler.WorkerState)], Hashable] | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs: Any)

Adaptively allocate workers based on scheduler load.  A superclass.

Contains logic to dynamically resize a Dask cluster based on current use.
This class needs to be paired with a system that can create and destroy
Dask workers using a cluster resource manager.  Typically it is built into
already existing solutions, rather than used directly by users.
It is most commonly used from the `.adapt(...)` method of various Dask
cluster classes.

* **Parameters:**
  **cluster: object**
  : Must have scale and scale_down methods/coroutines

  **interval**
  : Milliseconds between checks

  **wait_count: int, default 3**
  : Number of consecutive times that a worker should be suggested for
    removal before we remove it.

  **target_duration: timedelta or str, default “5s”**
  : Amount of time we want a computation to take.
    This affects how aggressively we scale up.

  **worker_key: Callable[WorkerState]**
  : Function to group workers together when scaling down
    See Scheduler.workers_to_close for more information

  **minimum: int**
  : Minimum number of workers to keep around

  **maximum: int**
  : Maximum number of workers to keep around

  **\*\*kwargs:**
  : Extra parameters to pass to Scheduler.workers_to_close

### Notes

Subclasses can override `Adaptive.target()` and
`Adaptive.workers_to_close()` to control when the cluster should be
resized. The default implementation checks if there are too many tasks
per worker or too little memory available (see
[`distributed.Scheduler.adaptive_target()`](deploying-python-advanced.md#distributed.Scheduler.adaptive_target)).
The values for interval, min, max, wait_count and target_duration can be
specified in the dask config under the distributed.adaptive key.

### Examples

This is commonly used from existing Dask classes, like KubeCluster

```pycon
>>> from dask_kubernetes import KubeCluster
>>> cluster = KubeCluster()
>>> cluster.adapt(minimum=10, maximum=100)
```

Alternatively you can use it from your own Cluster class by subclassing
from Dask’s Cluster superclass

```pycon
>>> from distributed.deploy import Cluster
>>> class MyCluster(Cluster):
...     def scale_up(self, n):
...         """ Bring worker count up to n """
...     def scale_down(self, workers):
...        """ Remove worker addresses from cluster """
```

```pycon
>>> cluster = MyCluster()
>>> cluster.adapt(minimum=10, maximum=100)
```

<!-- !! processed by numpydoc !! -->

### *class* distributed.deploy.Cluster(asynchronous=False, loop=None, quiet=False, name=None, scheduler_sync_interval=1)

Superclass for cluster objects

This class contains common functionality for Dask Cluster manager classes.

To implement this class, you must provide

1. A `scheduler_comm` attribute, which is a connection to the scheduler
   following the `distributed.core.rpc` API.
2. Implement `scale`, which takes an integer and scales the cluster to
   that many workers, or else set `_supports_scaling` to False

For that, you should get the following:

1. A standard `__repr__`
2. A live IPython widget
3. Adaptive scaling
4. Integration with dask-labextension
5. A `scheduler_info` attribute which contains an up-to-date copy of
   `Scheduler.identity()`, which is used for much of the above
6. Methods to gather logs

<!-- !! processed by numpydoc !! -->


# api.html.md

# API Reference

Dask APIs generally follow from upstream APIs:

- [Arrays](array-api.md) follows NumPy
- [DataFrames](dataframe-api.md) follows Pandas
- [Bag](bag-api.md) follows map/filter/groupby/reduce common in Spark and Python iterators
- [Delayed](delayed-api.md) wraps general Python code
- [Futures](futures.md) follows [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) from the standard library for real-time computation.

Additionally, Dask has its own functions to start computations, persist data in
memory, check progress, and so forth that complement the APIs above.
These more general Dask functions are described below:

| [`compute`](#dask.compute)(\*args[, traverse, optimize_graph, ...])   | Compute several dask collections at once.     |
|-----------------------------------------------------------------------|-----------------------------------------------|
| [`is_dask_collection`](#dask.is_dask_collection)(x)                   | Returns `True` if `x` is a dask collection.   |
| [`optimize`](#dask.optimize)(\*args[, traverse])                      | Optimize several dask collections at once.    |
| [`persist`](#dask.persist)(\*args[, traverse, optimize_graph, ...])   | Persist multiple Dask collections into memory |
| [`visualize`](#dask.visualize)(\*args[, filename, traverse, ...])     | Visualize several dask graphs simultaneously. |

| [`tokenize`](generated/dask.tokenize.tokenize.md#dask.tokenize.tokenize)                            | Deterministic token   |
|-----------------------------------------------------------------------------------------------------|-----------------------|
| [`TokenizationError`](generated/dask.tokenize.TokenizationError.md#dask.tokenize.TokenizationError) |                       |

These functions work with any scheduler.  More advanced operations are
available when using the newer scheduler and starting a
`dask.distributed.Client` (which, despite its name, runs nicely on a
single machine).  This API provides the ability to submit, cancel, and track
work asynchronously, and includes many functions for complex inter-task
workflows.  These are not necessary for normal operation, but can be useful for
real-time or advanced operation.

This more advanced API is available in the [Dask distributed documentation](https://distributed.dask.org/en/latest/api.html)

### dask.annotate(\*\*annotations: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [Iterator](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[None](https://docs.python.org/3/library/constants.html#None)]

Context Manager for setting HighLevelGraph Layer annotations.

Annotations are metadata or soft constraints associated with
tasks that dask schedulers may choose to respect: They signal intent
without enforcing hard constraints. As such, they are
primarily designed for use with the distributed scheduler.

Almost any object can serve as an annotation, but small Python objects
are preferred, while large objects such as NumPy arrays are discouraged.

Callables supplied as an annotation should take a single *key* argument and
produce the appropriate annotation. Individual task keys in the annotated collection
are supplied to the callable.

* **Parameters:**
  **\*\*annotations**

#### SEE ALSO
[`get_annotations`](#dask.get_annotations)

### Examples

All tasks within array A should have priority 100 and be retried 3 times
on failure.

```pycon
>>> import dask
>>> import dask.array as da
>>> with dask.annotate(priority=100, retries=3):
...     A = da.ones((10000, 10000))
```

Prioritise tasks within Array A on flattened block ID.

```pycon
>>> nblocks = (10, 10)
>>> with dask.annotate(priority=lambda k: k[1]*nblocks[1] + k[2]):
...     A = da.ones((1000, 1000), chunks=(100, 100))
```

Annotations may be nested.

```pycon
>>> with dask.annotate(priority=1):
...     with dask.annotate(retries=3):
...         A = da.ones((1000, 1000))
...     B = A + 1
```

<!-- !! processed by numpydoc !! -->

### dask.get_annotations() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Get current annotations.

* **Returns:**
  Dict of all current annotations

#### SEE ALSO
[`annotate`](#dask.annotate)

<!-- !! processed by numpydoc !! -->

### dask.compute(\*args, traverse=True, optimize_graph=True, scheduler=None, get=None, \*\*kwargs)

Compute several dask collections at once.

* **Parameters:**
  **args**
  : Any number of objects. If it is a dask object, it’s computed and the
    result is returned. By default, python builtin collections are also
    traversed to look for dask objects (for more information see the
    `traverse` keyword). Non-dask arguments are passed through unchanged.

  **traverse**
  : By default dask traverses builtin python collections looking for dask
    objects passed to `compute`. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the optimizations for each collection are applied
    before computation. Otherwise the graph is run as is. This can be
    useful for debugging.

  **get**
  : Should be left to `None` The get= keyword has been removed.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

### Examples

```pycon
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> dask.compute(a, b)
(np.int64(45), np.float64(4.5))
```

By default, dask objects inside python collections will also be computed:

```pycon
>>> dask.compute({'a': a, 'b': b, 'c': 1})
({'a': np.int64(45), 'b': np.float64(4.5), 'c': 1},)
```

<!-- !! processed by numpydoc !! -->

### dask.is_dask_collection(x) → [bool](https://docs.python.org/3/library/functions.html#bool)

Returns `True` if `x` is a dask collection.

* **Parameters:**
  **x**
  : Object to test.
* **Returns:**
  **result**
  : `True` if x is a Dask collection.

### Notes

The DaskCollection typing.Protocol implementation defines a Dask
collection as a class that returns a Mapping from the
`__dask_graph__` method. This helper function existed before the
implementation of the protocol.

<!-- !! processed by numpydoc !! -->

### dask.optimize(\*args, traverse=True, \*\*kwargs)

Optimize several dask collections at once.

Returns equivalent dask collections that all share the same merged and
optimized underlying graph. This can be useful if converting multiple
collections to delayed objects, or to manually apply the optimizations at
strategic points.

Note that in most cases you shouldn’t need to call this function directly.

Warning:

```default
This function triggers a materialization of the collections and looses
any annotations attached to HLG layers.
```

* **Parameters:**
  **\*args**
  : Any number of objects. If a dask object, its graph is optimized and
    merged with all those of all other dask objects before returning an
    equivalent dask collection. Non-dask arguments are passed through
    unchanged.

  **traverse**
  : By default dask traverses builtin python collections looking for dask
    objects passed to `optimize`. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

  **optimizations**
  : Additional optimization passes to perform.

  **\*\*kwargs**
  : Extra keyword arguments to forward to the optimization passes.

### Examples

```pycon
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> a2, b2 = dask.optimize(a, b)
```

```pycon
>>> a2.compute() == a.compute()
np.True_
>>> b2.compute() == b.compute()
np.True_
```

<!-- !! processed by numpydoc !! -->

### dask.persist(\*args, traverse=True, optimize_graph=True, scheduler=None, \*\*kwargs)

Persist multiple Dask collections into memory

This turns lazy Dask collections into Dask collections with the same
metadata, but now with their results fully computed or actively computing
in the background.

For example a lazy dask.array built up from many lazy calls will now be a
dask.array of the same shape, dtype, chunks, etc., but now with all of
those previously lazy tasks either computed in memory as many small `numpy.array`
(in the single-machine case) or asynchronously running in the
background on a cluster (in the distributed case).

This function operates differently if a `dask.distributed.Client` exists
and is connected to a distributed scheduler.  In this case this function
will return as soon as the task graph has been submitted to the cluster,
but before the computations have completed.  Computations will continue
asynchronously in the background.  When using this function with the single
machine scheduler it blocks until the computations have finished.

When using Dask on a single machine you should ensure that the dataset fits
entirely within memory.

* **Parameters:**
  **\*args: Dask collections**

  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **traverse**
  : By default dask traverses builtin python collections looking for dask
    objects passed to `persist`. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

### Examples

```pycon
>>> df = dd.read_csv('/path/to/*.csv')
>>> df = df[df.name == 'Alice']
>>> df['in-debt'] = df.balance < 0
>>> df = df.persist()  # triggers computation
```

```pycon
>>> df.value().min()  # future computations are now fast
-10
>>> df.value().max()
100
```

```pycon
>>> from dask import persist  # use persist function on multiple collections
>>> a, b = persist(a, b)
```

<!-- !! processed by numpydoc !! -->

### dask.visualize(\*args, filename='mydask', traverse=True, optimize_graph=False, maxval=None, engine: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['cytoscape', 'ipycytoscape', 'graphviz'] | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Visualize several dask graphs simultaneously.

Requires `graphviz` to be installed. All options that are not the dask
graph(s) should be passed as keyword arguments.

* **Parameters:**
  **args**
  : Any number of objects. If it is a dask collection (for example, a
    dask DataFrame, Array, Bag, or Delayed), its associated graph
    will be included in the output of visualize. By default, python builtin
    collections are also traversed to look for dask objects (for more
    information see the `traverse` keyword). Arguments lacking an
    associated graph will be ignored.

  **filename**
  : The name of the file to write to disk. If the provided filename
    doesn’t include an extension, ‘.png’ will be used by default.
    If filename is None, no file will be written, and we communicate
    with dot using only pipes.

  **format**
  : Format in which to write output file.  Default is ‘png’.

  **traverse**
  : By default, dask traverses builtin python collections looking for dask
    objects passed to `visualize`. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

  **optimize_graph**
  : If True, the graph is optimized before rendering.  Otherwise,
    the graph is displayed as is. Default is False.

  **color**
  : Options to color nodes. colormap:
    - None, the default, no colors.
    - ‘order’, colors the nodes’ border based on the order they appear in the graph.
    - ‘ages’, how long the data of a node is held.
    - ‘freed’, the number of dependencies released after running a node.
    - ‘memoryincreases’, how many more outputs are held after the lifetime of a node.
      Large values may indicate nodes that should have run later.
    - ‘memorydecreases’, how many fewer outputs are held after the lifetime of a node.
      Large values may indicate nodes that should have run sooner.
    - ‘memorypressure’, the number of data held when the node is run (circle), or
      the data is released (rectangle).

  **maxval**
  : Maximum value for colormap to normalize form 0 to 1.0. Default is `None`
    will make it the max number of values

  **collapse_outputs**
  : Whether to collapse output boxes, which often have empty labels.
    Default is False.

  **verbose**
  : Whether to label output and input boxes even if the data aren’t chunked.
    Beware: these labels can get very long. Default is False.

  **engine**
  : The visualization engine to use. If not provided, this checks the dask config
    value “visualization.engine”. If that is not set, it tries to import `graphviz`
    and `ipycytoscape`, using the first one to succeed.

  **\*\*kwargs**
  : Additional keyword arguments to forward to the visualization engine.
* **Returns:**
  **result**
  : See dask.dot.dot_graph for more information.

#### SEE ALSO
`dask.dot.dot_graph`

### Notes

For more information on optimization see here:

[https://docs.dask.org/en/latest/optimize.html](https://docs.dask.org/en/latest/optimize.html)

### Examples

```pycon
>>> x.visualize(filename='dask.pdf')
>>> x.visualize(filename='dask.pdf', color='order')
```

<!-- !! processed by numpydoc !! -->

## Datasets

Dask has a few helpers for generating demo datasets

### dask.datasets.make_people(npartitions=10, records_per_partition=1000, seed=None, locale='en')

Make a dataset of random people

This makes a Dask Bag with dictionary records of randomly generated people.
This requires the optional library `mimesis` to generate records.

* **Parameters:**
  **npartitions**
  : Number of partitions

  **records_per_partition**
  : Number of records in each partition

  **seed**
  : Random seed

  **locale**
  : Language locale, like ‘en’, ‘fr’, ‘zh’, or ‘ru’
* **Returns:**
  b: Dask Bag

<!-- !! processed by numpydoc !! -->

### dask.datasets.timeseries(start='2000-01-01', end='2000-01-31', freq='1s', partition_freq='1D', dtypes=None, seed=None, \*\*kwargs)

Create timeseries dataframe with random data

* **Parameters:**
  **start**
  : Start of time series

  **end**
  : End of time series

  **dtypes**
  : Mapping of column names to types.
    Valid types include {float, int, str, ‘category’}

  **freq**
  : String like ‘2s’ or ‘1H’ or ‘12W’ for the time series frequency

  **partition_freq**
  : String like ‘1M’ or ‘2Y’ to divide the dataframe into partitions

  **seed**
  : Randomstate seed

  **kwargs:**
  : Keywords to pass down to individual column creation functions.
    Keywords should be prefixed by the column name and then an underscore.

### Examples

```pycon
>>> import dask
>>> df = dask.datasets.timeseries()
>>> df.head()
          timestamp    id     name         x         y
2000-01-01 00:00:00   967    Jerry -0.031348 -0.040633
2000-01-01 00:00:01  1066  Michael -0.262136  0.307107
2000-01-01 00:00:02   988    Wendy -0.526331  0.128641
2000-01-01 00:00:03  1016   Yvonne  0.620456  0.767270
2000-01-01 00:00:04   998   Ursula  0.684902 -0.463278
>>> df = dask.datasets.timeseries(
...     '2000', '2010',
...     freq='2h', partition_freq='1D', seed=1,  # data frequency
...     dtypes={'value': float, 'name': str, 'id': int},  # data types
...     id_lam=1000  # control number of items in id column
... )
```

<!-- !! processed by numpydoc !! -->

## Datasets with defined specs

The following helpers are still experimental:

### dask.dataframe.io.demo.with_spec(spec: [DatasetSpec](#dask.dataframe.io.demo.DatasetSpec), seed: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None)

Generate a random dataset according to provided spec

* **Parameters:**
  **spec**
  : Specify all the parameters of the dataset

  **seed: int (optional)**
  : Randomstate seed

### Notes

This API is still experimental, and will likely change in the future

### Examples

```pycon
>>> from dask.dataframe.io.demo import ColumnSpec, DatasetSpec, with_spec
>>> ddf = with_spec(
...     DatasetSpec(
...         npartitions=10,
...         nrecords=10_000,
...         column_specs=[
...             ColumnSpec(dtype=int, number=2, prefix="p"),
...             ColumnSpec(dtype=int, number=2, prefix="n", method="normal"),
...             ColumnSpec(dtype=float, number=2, prefix="f"),
...             ColumnSpec(dtype=str, prefix="s", number=2, random=True, length=10),
...             ColumnSpec(dtype="category", prefix="c", choices=["Y", "N"]),
...         ],
...     ), seed=42)
>>> ddf.head(10)
     p1    p2    n1    n2        f1        f2          s1          s2 c1
0  1002   972  -811    20  0.640846 -0.176875  L#h98#}J`?  _8C607/:6e  N
1   985   982 -1663  -777  0.790257  0.792796  u:XI3,omoZ  w~@ /d)'-@  N
2   947   970   799  -269  0.740869 -0.118413  O$dnwCuq\  !WtSe+(;#9  Y
3  1003   983  1133   521 -0.987459  0.278154  j+Qr_2{XG&  &XV7cy$y1T  Y
4  1017  1049   826     5 -0.875667 -0.744359  bJ3E-{:o  {+jC).?vK+  Y
5   984  1017  -492  -399  0.748181  0.293761  ~zUNHNgD"!  yuEkXeVot|  Y
6   992  1027  -856    67 -0.125132 -0.234529  j.7z;o]Gc9  g|Fi5*}Y92  Y
7  1011   974   762 -1223  0.471696  0.937935  yT?j~N/-u]  JhEB[W-}^$  N
8   984   974   856    74  0.109963  0.367864  _j"&@ i&;/  OYXQ)w{hoH  N
9  1030  1001  -792  -262  0.435587 -0.647970  Pmrwl{{|.K  3UTqM$86Sg  N
```

<!-- !! processed by numpydoc !! -->

### The `ColumnSpec` class

### *class* dask.dataframe.io.demo.ColumnSpec(prefix: str | None = None, dtype: str | type | None = None, number: int = 1, nunique: int | None = None, choices: list = <factory>, low: int | None = None, high: int | None = None, length: int | None = None, random: bool = False, method: str | None = None, args: tuple[~typing.Any, ...] = <factory>, kwargs: dict[str, ~typing.Any] = <factory>)

Bases: [`object`](https://docs.python.org/3/library/functions.html#object)

Encapsulates properties of a family of columns with the same dtype.
Different method can be specified for integer dtype (“poisson”, “uniform”,
“binomial”, etc.)

### Notes

This API is still experimental, and will likely change in the future

<!-- !! processed by numpydoc !! -->

#### args *: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[Any](https://docs.python.org/3/library/typing.html#typing.Any), ...]*

Args to pass into the method

<!-- !! processed by numpydoc !! -->

#### choices *: [list](https://docs.python.org/3/library/stdtypes.html#list)*

For a “category” or str column, list of possible values

<!-- !! processed by numpydoc !! -->

#### dtype *: [str](https://docs.python.org/3/library/stdtypes.html#str) | [type](https://docs.python.org/3/library/functions.html#type) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

Column data type. Only supports numpy dtypes

<!-- !! processed by numpydoc !! -->

#### high *: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

For an int column, high end of range

<!-- !! processed by numpydoc !! -->

#### kwargs *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]*

Any other kwargs to pass into the method

<!-- !! processed by numpydoc !! -->

#### length *: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

For a str or “category” column with random=True, how large a string to generate

<!-- !! processed by numpydoc !! -->

#### low *: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

Start value for an int column. Optional if random=True, since `randint` doesn’t accept
high and low.

<!-- !! processed by numpydoc !! -->

#### method *: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

For an int column, method to use when generating the value, such as “poisson”, “uniform”, “binomial”.
Default “poisson”. Delegates to the same method of `RandomState`

<!-- !! processed by numpydoc !! -->

#### number *: [int](https://docs.python.org/3/library/functions.html#int)* *= 1*

How many columns to create with these properties. Default 1.
If more than one columns are specified, they will be numbered: “int1”, “int2”, etc.

<!-- !! processed by numpydoc !! -->

#### nunique *: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

For a “category” column, how many unique categories to generate

<!-- !! processed by numpydoc !! -->

#### prefix *: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

Column prefix. If not specified, will default to str(dtype)

<!-- !! processed by numpydoc !! -->

#### random *: [bool](https://docs.python.org/3/library/functions.html#bool)* *= False*

For an int column, whether to use `randint`. For a string column produces a random string
of specified `length`

<!-- !! processed by numpydoc !! -->

### The `RangeIndexSpec` class

### *class* dask.dataframe.io.demo.RangeIndexSpec(dtype: str | type = <class 'int'>, step: int = 1)

Bases: [`object`](https://docs.python.org/3/library/functions.html#object)

Properties of the dataframe RangeIndex

### Notes

This API is still experimental, and will likely change in the future

<!-- !! processed by numpydoc !! -->

#### dtype

Index dtype

<!-- !! processed by numpydoc !! -->

alias of [`int`](https://docs.python.org/3/library/functions.html#int)

#### step *: [int](https://docs.python.org/3/library/functions.html#int)* *= 1*

Step for a RangeIndex

<!-- !! processed by numpydoc !! -->

### The `DatetimeIndexSpec` class

### *class* dask.dataframe.io.demo.DatetimeIndexSpec(dtype: str | type = <class 'int'>, start: str | None = None, freq: str = '1H', partition_freq: str | None = None)

Bases: [`object`](https://docs.python.org/3/library/functions.html#object)

Properties of the dataframe DatetimeIndex

### Notes

This API is still experimental, and will likely change in the future

<!-- !! processed by numpydoc !! -->

#### dtype

Index dtype

<!-- !! processed by numpydoc !! -->

alias of [`int`](https://docs.python.org/3/library/functions.html#int)

#### freq *: [str](https://docs.python.org/3/library/stdtypes.html#str)* *= '1H'*

Frequency for the index (“1H”, “1D”, etc.)

<!-- !! processed by numpydoc !! -->

#### partition_freq *: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

Partition frequency (“1D”, “1M”, etc.)

<!-- !! processed by numpydoc !! -->

#### start *: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)* *= None*

First value of the index

<!-- !! processed by numpydoc !! -->

### The `DatasetSpec` class

### *class* dask.dataframe.io.demo.DatasetSpec(npartitions: int = 1, nrecords: int = 1000, index_spec: ~dask.dataframe.io.demo.RangeIndexSpec | ~dask.dataframe.io.demo.DatetimeIndexSpec = <factory>, column_specs: list[~dask.dataframe.io.demo.ColumnSpec] = <factory>)

Bases: [`object`](https://docs.python.org/3/library/functions.html#object)

Defines a dataset with random data, such as which columns and data types to generate

### Notes

This API is still experimental, and will likely change in the future

<!-- !! processed by numpydoc !! -->

#### column_specs *: [list](https://docs.python.org/3/library/stdtypes.html#list)[[ColumnSpec](#dask.dataframe.io.demo.ColumnSpec)]*

List of column definitions

<!-- !! processed by numpydoc !! -->

#### index_spec *: [RangeIndexSpec](#dask.dataframe.io.demo.RangeIndexSpec) | [DatetimeIndexSpec](#dask.dataframe.io.demo.DatetimeIndexSpec)*

Properties of the index

<!-- !! processed by numpydoc !! -->

#### npartitions *: [int](https://docs.python.org/3/library/functions.html#int)* *= 1*

How many partitions generate in the dataframe. If the dataframe has a DatetimeIndex, specify
its `partition_freq` instead

<!-- !! processed by numpydoc !! -->

#### nrecords *: [int](https://docs.python.org/3/library/functions.html#int)* *= 1000*

Total number of records to generate

<!-- !! processed by numpydoc !! -->

<a id="api-utilities"></a>

## Utilities

Dask has some public utility methods. These are primarily used for parsing
configuration values.

### dask.utils.apply(func, args, kwargs=None)

Apply a function given its positional and keyword arguments.

Equivalent to `func(*args, **kwargs)`
Most Dask users will never need to use the `apply` function.
It is typically only used by people who need to inject
keyword argument values into a low level Dask task graph.

* **Parameters:**
  **func**
  : The function you want to apply.

  **args**
  : A tuple containing all the positional arguments needed for `func`
    (eg: `(arg_1, arg_2, arg_3)`)

  **kwargs**
  : A dictionary mapping the keyword arguments
    (eg: `{"kwarg_1": value, "kwarg_2": value}`

### Examples

```pycon
>>> from dask.utils import apply
>>> def add(number, second_number=5):
...     return number + second_number
...
>>> apply(add, (10,), {"second_number": 2})  # equivalent to add(*args, **kwargs)
12
```

```pycon
>>> task = apply(add, (10,), {"second_number": 2})
>>> dsk = {'task-name': task}  # adds the task to a low level Dask task graph
```

<!-- !! processed by numpydoc !! -->

### dask.utils.format_bytes(n: [int](https://docs.python.org/3/library/functions.html#int)) → [str](https://docs.python.org/3/library/stdtypes.html#str)

Format bytes as text

```pycon
>>> from dask.utils import format_bytes
>>> format_bytes(1)
'1 B'
>>> format_bytes(1234)
'1.21 kiB'
>>> format_bytes(12345678)
'11.77 MiB'
>>> format_bytes(1234567890)
'1.15 GiB'
>>> format_bytes(1234567890000)
'1.12 TiB'
>>> format_bytes(1234567890000000)
'1.10 PiB'
```

For all values < 2\*\*60, the output is always <= 10 characters.

<!-- !! processed by numpydoc !! -->

### dask.utils.format_time(n: [float](https://docs.python.org/3/library/functions.html#float)) → [str](https://docs.python.org/3/library/stdtypes.html#str)

format integers as time

```pycon
>>> from dask.utils import format_time
>>> format_time(1)
'1.00 s'
>>> format_time(0.001234)
'1.23 ms'
>>> format_time(0.00012345)
'123.45 us'
>>> format_time(123.456)
'123.46 s'
>>> format_time(1234.567)
'20m 34s'
>>> format_time(12345.67)
'3hr 25m'
>>> format_time(123456.78)
'34hr 17m'
>>> format_time(1234567.89)
'14d 6hr'
```

<!-- !! processed by numpydoc !! -->

### dask.utils.parse_bytes(s: [float](https://docs.python.org/3/library/functions.html#float) | [str](https://docs.python.org/3/library/stdtypes.html#str)) → [int](https://docs.python.org/3/library/functions.html#int)

Parse byte string to numbers

```pycon
>>> from dask.utils import parse_bytes
>>> parse_bytes('100')
100
>>> parse_bytes('100 MB')
100000000
>>> parse_bytes('100M')
100000000
>>> parse_bytes('5kB')
5000
>>> parse_bytes('5.4 kB')
5400
>>> parse_bytes('1kiB')
1024
>>> parse_bytes('1e6')
1000000
>>> parse_bytes('1e6 kB')
1000000000
>>> parse_bytes('MB')
1000000
>>> parse_bytes(123)
123
>>> parse_bytes('5 foos')
Traceback (most recent call last):
    ...
ValueError: Could not interpret 'foos' as a byte unit
```

<!-- !! processed by numpydoc !! -->

### dask.utils.parse_timedelta(s: [None](https://docs.python.org/3/library/constants.html#None), default: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)[False] = 'seconds') → [None](https://docs.python.org/3/library/constants.html#None)

### dask.utils.parse_timedelta(s: [str](https://docs.python.org/3/library/stdtypes.html#str) | [float](https://docs.python.org/3/library/functions.html#float) | [timedelta](https://docs.python.org/3/library/datetime.html#datetime.timedelta), default: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)[False] = 'seconds') → [float](https://docs.python.org/3/library/functions.html#float)

Parse timedelta string to number of seconds

* **Parameters:**
  **s**

  **default: str or False, optional**
  : Unit of measure if s  does not specify one. Defaults to seconds.
    Set to False to require s to explicitly specify its own unit.

### Examples

```pycon
>>> from datetime import timedelta
>>> from dask.utils import parse_timedelta
>>> parse_timedelta('3s')
3
>>> parse_timedelta('3.5 seconds')
3.5
>>> parse_timedelta('300ms')
0.3
>>> parse_timedelta(timedelta(seconds=3))  # also supports timedeltas
3
```

<!-- !! processed by numpydoc !! -->


# array-api.html.md

# API

## Top level functions

| [`abs`](generated/dask.array.abs.md#dask.array.abs)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.absolute.                                                                                                                         |
|-------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`absolute`](generated/dask.array.absolute.md#dask.array.absolute)(x, /[, out, where, casting, order, ...])                   | This docstring was copied from numpy.absolute.                                                                                                                         |
| [`add`](generated/dask.array.add.md#dask.array.add)(x1, x2, /[, out, where, casting, order, ...])                             | This docstring was copied from numpy.add.                                                                                                                              |
| [`all`](generated/dask.array.all.md#dask.array.all)(a[, axis, keepdims, split_every, out])                                    | Test whether all array elements along a given axis evaluate to True.                                                                                                   |
| [`allclose`](generated/dask.array.allclose.md#dask.array.allclose)(arr1, arr2[, rtol, atol, equal_nan])                       | Returns True if two arrays are element-wise equal within a tolerance.                                                                                                  |
| [`angle`](generated/dask.array.angle.md#dask.array.angle)(x[, deg])                                                           | Return the angle of the complex argument.                                                                                                                              |
| [`any`](generated/dask.array.any.md#dask.array.any)(a[, axis, keepdims, split_every, out])                                    | Test whether any array element along a given axis evaluates to True.                                                                                                   |
| [`append`](generated/dask.array.append.md#dask.array.append)(arr, values[, axis])                                             | Append values to the end of an array.                                                                                                                                  |
| [`apply_along_axis`](generated/dask.array.apply_along_axis.md#dask.array.apply_along_axis)(func1d, axis, arr, \*args[, ...])  | Apply a function to 1-D slices along the given axis.                                                                                                                   |
| [`apply_over_axes`](generated/dask.array.apply_over_axes.md#dask.array.apply_over_axes)(func, a, axes)                        | Apply a function repeatedly over multiple axes.                                                                                                                        |
| [`arange`](generated/dask.array.arange.md#dask.array.arange)([start, stop, step, chunks, like, dtype])                        | Return evenly spaced values from start to stop with step size step.                                                                                                    |
| [`arccos`](generated/dask.array.arccos.md#dask.array.arccos)(x, /[, out, where, casting, order, ...])                         | This docstring was copied from numpy.arccos.                                                                                                                           |
| [`arccosh`](generated/dask.array.arccosh.md#dask.array.arccosh)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.arccosh.                                                                                                                          |
| [`arcsin`](generated/dask.array.arcsin.md#dask.array.arcsin)(x, /[, out, where, casting, order, ...])                         | This docstring was copied from numpy.arcsin.                                                                                                                           |
| [`arcsinh`](generated/dask.array.arcsinh.md#dask.array.arcsinh)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.arcsinh.                                                                                                                          |
| [`arctan`](generated/dask.array.arctan.md#dask.array.arctan)(x, /[, out, where, casting, order, ...])                         | This docstring was copied from numpy.arctan.                                                                                                                           |
| [`arctan2`](generated/dask.array.arctan2.md#dask.array.arctan2)(x1, x2, /[, out, where, casting, ...])                        | This docstring was copied from numpy.arctan2.                                                                                                                          |
| [`arctanh`](generated/dask.array.arctanh.md#dask.array.arctanh)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.arctanh.                                                                                                                          |
| [`argmax`](generated/dask.array.argmax.md#dask.array.argmax)(a[, axis, keepdims, split_every, out])                           | Returns the indices of the maximum values along an axis.                                                                                                               |
| [`argmin`](generated/dask.array.argmin.md#dask.array.argmin)(a[, axis, keepdims, split_every, out])                           | Returns the indices of the minimum values along an axis.                                                                                                               |
| [`argtopk`](generated/dask.array.argtopk.md#dask.array.argtopk)(a, k[, axis, split_every])                                    | Extract the indices of the k largest elements from a on the given axis, and return them sorted from largest to smallest.                                               |
| [`argwhere`](generated/dask.array.argwhere.md#dask.array.argwhere)(a)                                                         | Find the indices of array elements that are non-zero, grouped by element.                                                                                              |
| [`around`](generated/dask.array.around.md#dask.array.around)(x[, decimals])                                                   | Round an array to the given number of decimals.                                                                                                                        |
| [`array`](generated/dask.array.array.md#dask.array.array)(object[, dtype, copy, order, subok, ...])                           | This docstring was copied from numpy.array.                                                                                                                            |
| [`asanyarray`](generated/dask.array.asanyarray.md#dask.array.asanyarray)(a[, dtype, order, like, inline_array])               | Convert the input to a dask array.                                                                                                                                     |
| [`asarray`](generated/dask.array.asarray.md#dask.array.asarray)(a[, allow_unknown_chunksizes, ...])                           | Convert the input to a dask array.                                                                                                                                     |
| [`atleast_1d`](generated/dask.array.atleast_1d.md#dask.array.atleast_1d)(\*arys)                                              | Convert inputs to arrays with at least one dimension.                                                                                                                  |
| [`atleast_2d`](generated/dask.array.atleast_2d.md#dask.array.atleast_2d)(\*arys)                                              | View inputs as arrays with at least two dimensions.                                                                                                                    |
| [`atleast_3d`](generated/dask.array.atleast_3d.md#dask.array.atleast_3d)(\*arys)                                              | View inputs as arrays with at least three dimensions.                                                                                                                  |
| [`average`](generated/dask.array.average.md#dask.array.average)(a[, axis, weights, returned, keepdims])                       | Compute the weighted average along the specified axis.                                                                                                                 |
| [`bincount`](generated/dask.array.bincount.md#dask.array.bincount)(x, /[, weights, minlength])                                | This docstring was copied from numpy.bincount.                                                                                                                         |
| [`bitwise_and`](generated/dask.array.bitwise_and.md#dask.array.bitwise_and)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.bitwise_and.                                                                                                                      |
| [`bitwise_not`](generated/dask.array.bitwise_not.md#dask.array.bitwise_not)(x, /[, out, where, casting, ...])                 | This docstring was copied from numpy.invert.                                                                                                                           |
| [`bitwise_or`](generated/dask.array.bitwise_or.md#dask.array.bitwise_or)(x1, x2, /[, out, where, casting, ...])               | This docstring was copied from numpy.bitwise_or.                                                                                                                       |
| [`bitwise_xor`](generated/dask.array.bitwise_xor.md#dask.array.bitwise_xor)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.bitwise_xor.                                                                                                                      |
| [`block`](generated/dask.array.block.md#dask.array.block)(arrays[, allow_unknown_chunksizes])                                 | Assemble an nd-array from nested lists of blocks.                                                                                                                      |
| [`blockwise`](generated/dask.array.blockwise.md#dask.array.blockwise)(func, out_ind, \*args[, name, ...])                     | Tensor operation: Generalized inner and outer products                                                                                                                 |
| [`broadcast_arrays`](generated/dask.array.broadcast_arrays.md#dask.array.broadcast_arrays)(\*args[, subok])                   | Broadcast any number of arrays against each other.                                                                                                                     |
| [`broadcast_to`](generated/dask.array.broadcast_to.md#dask.array.broadcast_to)(x, shape[, chunks, meta])                      | Broadcast an array to a new shape.                                                                                                                                     |
| [`cbrt`](generated/dask.array.cbrt.md#dask.array.cbrt)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.cbrt.                                                                                                                             |
| [`coarsen`](generated/dask.array.coarsen.md#dask.array.coarsen)(reduction, x, axes[, trim_excess])                            | Coarsen array by applying reduction to fixed size neighborhoods                                                                                                        |
| [`ceil`](generated/dask.array.ceil.md#dask.array.ceil)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.ceil.                                                                                                                             |
| [`choose`](generated/dask.array.choose.md#dask.array.choose)(a, choices)                                                      | Construct an array from an index array and a list of arrays to choose from.                                                                                            |
| [`clip`](generated/dask.array.clip.md#dask.array.clip)(\*args, \*\*kwargs)                                                    | Clip (limit) the values in an array.                                                                                                                                   |
| [`compress`](generated/dask.array.compress.md#dask.array.compress)(condition, a[, axis])                                      | Return selected slices of an array along given axis.                                                                                                                   |
| [`concatenate`](generated/dask.array.concatenate.md#dask.array.concatenate)(seq[, axis, ...])                                 | Concatenate arrays along an existing axis                                                                                                                              |
| [`conj`](generated/dask.array.conj.md#dask.array.conj)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.conjugate.                                                                                                                        |
| [`copysign`](generated/dask.array.copysign.md#dask.array.copysign)(x1, x2, /[, out, where, casting, ...])                     | This docstring was copied from numpy.copysign.                                                                                                                         |
| [`corrcoef`](generated/dask.array.corrcoef.md#dask.array.corrcoef)(x[, y, rowvar])                                            | Return Pearson product-moment correlation coefficients.                                                                                                                |
| [`cos`](generated/dask.array.cos.md#dask.array.cos)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.cos.                                                                                                                              |
| [`cosh`](generated/dask.array.cosh.md#dask.array.cosh)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.cosh.                                                                                                                             |
| [`count_nonzero`](generated/dask.array.count_nonzero.md#dask.array.count_nonzero)(a[, axis])                                  | Counts the number of non-zero values in the array `a`.                                                                                                                 |
| [`cov`](generated/dask.array.cov.md#dask.array.cov)(m[, y, rowvar, bias, ddof, fweights, ...])                                | Estimate a covariance matrix, given data and weights.                                                                                                                  |
| [`cumprod`](generated/dask.array.cumprod.md#dask.array.cumprod)(x[, axis, dtype, out, method])                                | Return the cumulative product of elements along a given axis.                                                                                                          |
| [`cumsum`](generated/dask.array.cumsum.md#dask.array.cumsum)(x[, axis, dtype, out, method])                                   | Return the cumulative sum of the elements along a given axis.                                                                                                          |
| [`deg2rad`](generated/dask.array.deg2rad.md#dask.array.deg2rad)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.deg2rad.                                                                                                                          |
| [`degrees`](generated/dask.array.degrees.md#dask.array.degrees)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.degrees.                                                                                                                          |
| [`delete`](generated/dask.array.delete.md#dask.array.delete)(arr, obj, axis)                                                  | Return a new array with sub-arrays along an axis deleted.                                                                                                              |
| [`diag`](generated/dask.array.diag.md#dask.array.diag)(v[, k])                                                                | Extract a diagonal or construct a diagonal array.                                                                                                                      |
| [`diagonal`](generated/dask.array.diagonal.md#dask.array.diagonal)(a[, offset, axis1, axis2])                                 | Return specified diagonals.                                                                                                                                            |
| [`diff`](generated/dask.array.diff.md#dask.array.diff)(a[, n, axis, prepend, append])                                         | Calculate the n-th discrete difference along the given axis.                                                                                                           |
| [`divide`](generated/dask.array.divide.md#dask.array.divide)(x1, x2, /[, out, where, casting, ...])                           | This docstring was copied from numpy.divide.                                                                                                                           |
| [`divmod`](generated/dask.array.divmod.md#dask.array.divmod)(x1, x2[, out1, out2], / [[, out, ...])                           | This docstring was copied from numpy.divmod.                                                                                                                           |
| [`digitize`](generated/dask.array.digitize.md#dask.array.digitize)(a, bins[, right])                                          | Return the indices of the bins to which each value in input array belongs.                                                                                             |
| [`dot`](generated/dask.array.dot.md#dask.array.dot)(a, b[, out])                                                              | This docstring was copied from numpy.dot.                                                                                                                              |
| [`dstack`](generated/dask.array.dstack.md#dask.array.dstack)(tup[, allow_unknown_chunksizes])                                 | Stack arrays in sequence depth wise (along third axis).                                                                                                                |
| [`ediff1d`](generated/dask.array.ediff1d.md#dask.array.ediff1d)(ary[, to_end, to_begin])                                      | The differences between consecutive elements of an array.                                                                                                              |
| [`einsum`](generated/dask.array.einsum.md#dask.array.einsum)(subscripts, \*operands[, out, dtype, ...])                       | This docstring was copied from numpy.einsum.                                                                                                                           |
| [`empty`](generated/dask.array.empty.md#dask.array.empty)(\*args, \*\*kwargs)                                                 | Blocked variant of empty_like                                                                                                                                          |
| [`empty_like`](generated/dask.array.empty_like.md#dask.array.empty_like)(a[, dtype, order, chunks, name, ...])                | Return a new array with the same shape and type as a given array.                                                                                                      |
| [`equal`](generated/dask.array.equal.md#dask.array.equal)(x1, x2, /[, out, where, casting, ...])                              | This docstring was copied from numpy.equal.                                                                                                                            |
| [`exp`](generated/dask.array.exp.md#dask.array.exp)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.exp.                                                                                                                              |
| [`exp2`](generated/dask.array.exp2.md#dask.array.exp2)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.exp2.                                                                                                                             |
| [`expand_dims`](generated/dask.array.expand_dims.md#dask.array.expand_dims)(a, axis)                                          | Expand the shape of an array.                                                                                                                                          |
| [`expm1`](generated/dask.array.expm1.md#dask.array.expm1)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.expm1.                                                                                                                            |
| [`extract`](generated/dask.array.extract.md#dask.array.extract)(condition, arr)                                               | Return the elements of an array that satisfy some condition.                                                                                                           |
| [`eye`](generated/dask.array.eye.md#dask.array.eye)(N[, chunks, M, k, dtype])                                                 | Return a 2-D Array with ones on the diagonal and zeros elsewhere.                                                                                                      |
| [`fabs`](generated/dask.array.fabs.md#dask.array.fabs)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.fabs.                                                                                                                             |
| [`fix`](generated/dask.array.fix.md#dask.array.fix)(\*args, \*\*kwargs)                                                       | Round to nearest integer towards zero.                                                                                                                                 |
| [`flatnonzero`](generated/dask.array.flatnonzero.md#dask.array.flatnonzero)(a)                                                | Return indices that are non-zero in the flattened version of a.                                                                                                        |
| [`flip`](generated/dask.array.flip.md#dask.array.flip)(m[, axis])                                                             | Reverse element order along axis.                                                                                                                                      |
| [`flipud`](generated/dask.array.flipud.md#dask.array.flipud)(m)                                                               | Reverse the order of elements along axis 0 (up/down).                                                                                                                  |
| [`fliplr`](generated/dask.array.fliplr.md#dask.array.fliplr)(m)                                                               | Reverse the order of elements along axis 1 (left/right).                                                                                                               |
| [`float_power`](generated/dask.array.float_power.md#dask.array.float_power)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.float_power.                                                                                                                      |
| [`floor`](generated/dask.array.floor.md#dask.array.floor)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.floor.                                                                                                                            |
| [`floor_divide`](generated/dask.array.floor_divide.md#dask.array.floor_divide)(x1, x2, /[, out, where, ...])                  | This docstring was copied from numpy.floor_divide.                                                                                                                     |
| [`fmax`](generated/dask.array.fmax.md#dask.array.fmax)(x1, x2, /[, out, where, casting, ...])                                 | This docstring was copied from numpy.fmax.                                                                                                                             |
| [`fmin`](generated/dask.array.fmin.md#dask.array.fmin)(x1, x2, /[, out, where, casting, ...])                                 | This docstring was copied from numpy.fmin.                                                                                                                             |
| [`fmod`](generated/dask.array.fmod.md#dask.array.fmod)(x1, x2, /[, out, where, casting, ...])                                 | This docstring was copied from numpy.fmod.                                                                                                                             |
| [`frexp`](generated/dask.array.frexp.md#dask.array.frexp)(x[, out1, out2], / [[, out, where, ...])                            | This docstring was copied from numpy.frexp.                                                                                                                            |
| [`fromfunction`](generated/dask.array.fromfunction.md#dask.array.fromfunction)(func[, chunks, shape, dtype])                  | Construct an array by executing a function over each coordinate.                                                                                                       |
| [`frompyfunc`](generated/dask.array.frompyfunc.md#dask.array.frompyfunc)(func, /, nin, nout, \*[, identity])                  | This docstring was copied from numpy.frompyfunc.                                                                                                                       |
| [`full`](generated/dask.array.full.md#dask.array.full)(shape, fill_value, \*args, \*\*kwargs)                                 | Blocked variant of full_like                                                                                                                                           |
| [`full_like`](generated/dask.array.full_like.md#dask.array.full_like)(a, fill_value[, order, dtype, ...])                     | Return a full array with the same shape and type as a given array.                                                                                                     |
| [`gradient`](generated/dask.array.gradient.md#dask.array.gradient)(f, \*varargs[, axis])                                      | Return the gradient of an N-dimensional array.                                                                                                                         |
| [`greater`](generated/dask.array.greater.md#dask.array.greater)(x1, x2, /[, out, where, casting, ...])                        | This docstring was copied from numpy.greater.                                                                                                                          |
| [`greater_equal`](generated/dask.array.greater_equal.md#dask.array.greater_equal)(x1, x2, /[, out, where, ...])               | This docstring was copied from numpy.greater_equal.                                                                                                                    |
| [`histogram`](generated/dask.array.histogram.md#dask.array.histogram)(a[, bins, range, normed, weights, ...])                 | Blocked variant of [`numpy.histogram()`](https://numpy.org/doc/stable/reference/generated/numpy.histogram.html#numpy.histogram).                                       |
| [`histogram2d`](generated/dask.array.histogram2d.md#dask.array.histogram2d)(x, y[, bins, range, normed, ...])                 | Blocked variant of [`numpy.histogram2d()`](https://numpy.org/doc/stable/reference/generated/numpy.histogram2d.html#numpy.histogram2d).                                 |
| [`histogramdd`](generated/dask.array.histogramdd.md#dask.array.histogramdd)(sample, bins[, range, normed, ...])               | Blocked variant of [`numpy.histogramdd()`](https://numpy.org/doc/stable/reference/generated/numpy.histogramdd.html#numpy.histogramdd).                                 |
| [`hstack`](generated/dask.array.hstack.md#dask.array.hstack)(tup[, allow_unknown_chunksizes])                                 | Stack arrays in sequence horizontally (column wise).                                                                                                                   |
| [`hypot`](generated/dask.array.hypot.md#dask.array.hypot)(x1, x2, /[, out, where, casting, ...])                              | This docstring was copied from numpy.hypot.                                                                                                                            |
| [`i0`](generated/dask.array.i0.md#dask.array.i0)(\*args, \*\*kwargs)                                                          | Modified Bessel function of the first kind, order 0.                                                                                                                   |
| [`imag`](generated/dask.array.imag.md#dask.array.imag)(\*args, \*\*kwargs)                                                    | Return the imaginary part of the complex argument.                                                                                                                     |
| [`indices`](generated/dask.array.indices.md#dask.array.indices)(dimensions[, dtype, chunks])                                  | Implements NumPy's `indices` for Dask Arrays.                                                                                                                          |
| [`insert`](generated/dask.array.insert.md#dask.array.insert)(arr, obj, values, axis)                                          | Insert values along the given axis before the given indices.                                                                                                           |
| [`invert`](generated/dask.array.invert.md#dask.array.invert)(x, /[, out, where, casting, order, ...])                         | This docstring was copied from numpy.invert.                                                                                                                           |
| [`isclose`](generated/dask.array.isclose.md#dask.array.isclose)(arr1, arr2[, rtol, atol, equal_nan])                          | Returns a boolean array where two arrays are element-wise equal within a tolerance.                                                                                    |
| [`iscomplex`](generated/dask.array.iscomplex.md#dask.array.iscomplex)(\*args, \*\*kwargs)                                     | Returns a bool array, where True if input element is complex.                                                                                                          |
| [`isfinite`](generated/dask.array.isfinite.md#dask.array.isfinite)(x, /[, out, where, casting, order, ...])                   | This docstring was copied from numpy.isfinite.                                                                                                                         |
| [`isin`](generated/dask.array.isin.md#dask.array.isin)(element, test_elements[, ...])                                         | Calculates `element in test_elements`, broadcasting over element only.                                                                                                 |
| [`isinf`](generated/dask.array.isinf.md#dask.array.isinf)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.isinf.                                                                                                                            |
| [`isneginf`](generated/dask.array.isneginf.md#dask.array.isneginf)                                                            | This docstring was copied from numpy.equal.                                                                                                                            |
| [`isnan`](generated/dask.array.isnan.md#dask.array.isnan)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.isnan.                                                                                                                            |
| [`isnull`](generated/dask.array.isnull.md#dask.array.isnull)(values)                                                          | pandas.isnull for dask arrays                                                                                                                                          |
| [`isposinf`](generated/dask.array.isposinf.md#dask.array.isposinf)                                                            | This docstring was copied from numpy.equal.                                                                                                                            |
| [`isreal`](generated/dask.array.isreal.md#dask.array.isreal)(\*args, \*\*kwargs)                                              | Returns a bool array, where True if input element is real.                                                                                                             |
| [`ldexp`](generated/dask.array.ldexp.md#dask.array.ldexp)(x1, x2, /[, out, where, casting, ...])                              | This docstring was copied from numpy.ldexp.                                                                                                                            |
| [`left_shift`](generated/dask.array.left_shift.md#dask.array.left_shift)(x1, x2, /[, out, where, casting, ...])               | This docstring was copied from numpy.left_shift.                                                                                                                       |
| [`less`](generated/dask.array.less.md#dask.array.less)(x1, x2, /[, out, where, casting, ...])                                 | This docstring was copied from numpy.less.                                                                                                                             |
| [`less_equal`](generated/dask.array.less_equal.md#dask.array.less_equal)(x1, x2, /[, out, where, casting, ...])               | This docstring was copied from numpy.less_equal.                                                                                                                       |
| [`linspace`](generated/dask.array.linspace.md#dask.array.linspace)(start, stop[, num, endpoint, ...])                         | Return num evenly spaced values over the closed interval [start, stop].                                                                                                |
| [`log`](generated/dask.array.log.md#dask.array.log)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.log.                                                                                                                              |
| [`log10`](generated/dask.array.log10.md#dask.array.log10)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.log10.                                                                                                                            |
| [`log1p`](generated/dask.array.log1p.md#dask.array.log1p)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.log1p.                                                                                                                            |
| [`log2`](generated/dask.array.log2.md#dask.array.log2)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.log2.                                                                                                                             |
| [`logaddexp`](generated/dask.array.logaddexp.md#dask.array.logaddexp)(x1, x2, /[, out, where, casting, ...])                  | This docstring was copied from numpy.logaddexp.                                                                                                                        |
| [`logaddexp2`](generated/dask.array.logaddexp2.md#dask.array.logaddexp2)(x1, x2, /[, out, where, casting, ...])               | This docstring was copied from numpy.logaddexp2.                                                                                                                       |
| [`logical_and`](generated/dask.array.logical_and.md#dask.array.logical_and)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.logical_and.                                                                                                                      |
| [`logical_not`](generated/dask.array.logical_not.md#dask.array.logical_not)(x, /[, out, where, casting, ...])                 | This docstring was copied from numpy.logical_not.                                                                                                                      |
| [`logical_or`](generated/dask.array.logical_or.md#dask.array.logical_or)(x1, x2, /[, out, where, casting, ...])               | This docstring was copied from numpy.logical_or.                                                                                                                       |
| [`logical_xor`](generated/dask.array.logical_xor.md#dask.array.logical_xor)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.logical_xor.                                                                                                                      |
| [`map_overlap`](generated/dask.array.map_overlap.md#dask.array.map_overlap)(func, \*args[, depth, boundary, ...])             | Map a function over blocks of arrays with some overlap                                                                                                                 |
| [`map_blocks`](generated/dask.array.map_blocks.md#dask.array.map_blocks)(func, \*args[, name, token, ...])                    | Map a function across all blocks of a dask array.                                                                                                                      |
| [`matmul`](generated/dask.array.matmul.md#dask.array.matmul)(x1, x2, /[, out, casting, order, ...])                           | This docstring was copied from numpy.matmul.                                                                                                                           |
| [`max`](generated/dask.array.max.md#dask.array.max)(a[, axis, keepdims, split_every, out])                                    | Return the maximum of an array or maximum along an axis.                                                                                                               |
| [`maximum`](generated/dask.array.maximum.md#dask.array.maximum)(x1, x2, /[, out, where, casting, ...])                        | This docstring was copied from numpy.maximum.                                                                                                                          |
| [`mean`](generated/dask.array.mean.md#dask.array.mean)(a[, axis, dtype, keepdims, ...])                                       | Compute the arithmetic mean along the specified axis.                                                                                                                  |
| [`median`](generated/dask.array.median.md#dask.array.median)(a[, axis, keepdims, out])                                        | Compute the median along the specified axis.                                                                                                                           |
| [`meshgrid`](generated/dask.array.meshgrid.md#dask.array.meshgrid)(\*xi[, sparse, indexing])                                  | Return a tuple of coordinate matrices from coordinate vectors.                                                                                                         |
| [`min`](generated/dask.array.min.md#dask.array.min)(a[, axis, keepdims, split_every, out])                                    | Return the minimum of an array or minimum along an axis.                                                                                                               |
| [`minimum`](generated/dask.array.minimum.md#dask.array.minimum)(x1, x2, /[, out, where, casting, ...])                        | This docstring was copied from numpy.minimum.                                                                                                                          |
| [`mod`](generated/dask.array.mod.md#dask.array.mod)(x1, x2, /[, out, where, casting, order, ...])                             | This docstring was copied from numpy.remainder.                                                                                                                        |
| [`modf`](generated/dask.array.modf.md#dask.array.modf)(x[, out1, out2], / [[, out, where, ...])                               | This docstring was copied from numpy.modf.                                                                                                                             |
| [`moment`](generated/dask.array.moment.md#dask.array.moment)(a, order[, axis, dtype, keepdims, ...])                          | Calculate the nth centralized moment.                                                                                                                                  |
| [`moveaxis`](generated/dask.array.moveaxis.md#dask.array.moveaxis)(a, source, destination)                                    | Move axes of an array to new positions.                                                                                                                                |
| [`multiply`](generated/dask.array.multiply.md#dask.array.multiply)(x1, x2, /[, out, where, casting, ...])                     | This docstring was copied from numpy.multiply.                                                                                                                         |
| [`nanargmax`](generated/dask.array.nanargmax.md#dask.array.nanargmax)(a[, axis, keepdims, split_every, out])                  | Return the indices of the maximum values in the specified axis ignoring NaNs.                                                                                          |
| [`nanargmin`](generated/dask.array.nanargmin.md#dask.array.nanargmin)(a[, axis, keepdims, split_every, out])                  | Return the indices of the minimum values in the specified axis ignoring NaNs.                                                                                          |
| [`nancumprod`](generated/dask.array.nancumprod.md#dask.array.nancumprod)(x, axis[, dtype, out, method])                       | Return the cumulative product of array elements over a given axis treating Not a Numbers (NaNs) as one.                                                                |
| [`nancumsum`](generated/dask.array.nancumsum.md#dask.array.nancumsum)(x, axis[, dtype, out, method])                          | Return the cumulative sum of array elements over a given axis treating Not a Numbers (NaNs) as zero.                                                                   |
| [`nanmax`](generated/dask.array.nanmax.md#dask.array.nanmax)(a[, axis, keepdims, split_every, out])                           | Return the maximum of an array or maximum along an axis, ignoring any NaNs.                                                                                            |
| [`nanmean`](generated/dask.array.nanmean.md#dask.array.nanmean)(a[, axis, dtype, keepdims, ...])                              | Compute the arithmetic mean along the specified axis, ignoring NaNs.                                                                                                   |
| [`nanmedian`](generated/dask.array.nanmedian.md#dask.array.nanmedian)(a[, axis, keepdims, out])                               | Compute the median along the specified axis, while ignoring NaNs.                                                                                                      |
| [`nanmin`](generated/dask.array.nanmin.md#dask.array.nanmin)(a[, axis, keepdims, split_every, out])                           | Return minimum of an array or minimum along an axis, ignoring any NaNs.                                                                                                |
| [`nanprod`](generated/dask.array.nanprod.md#dask.array.nanprod)(a[, axis, dtype, keepdims, ...])                              | Return the product of array elements over a given axis treating Not a Numbers (NaNs) as ones.                                                                          |
| [`nanquantile`](generated/dask.array.nanquantile.md#dask.array.nanquantile)(a, q[, axis, out, ...])                           | Compute the qth quantile of the data along the specified axis, while ignoring nan values.                                                                              |
| [`nanpercentile`](generated/dask.array.nanpercentile.md#dask.array.nanpercentile)(a, q, \*\*kwargs)                           | Compute the qth percentile of the data along the specified axis, while ignoring nan values.                                                                            |
| [`nanstd`](generated/dask.array.nanstd.md#dask.array.nanstd)(a[, axis, dtype, keepdims, ddof, ...])                           | Compute the standard deviation along the specified axis, while ignoring NaNs.                                                                                          |
| [`nansum`](generated/dask.array.nansum.md#dask.array.nansum)(a[, axis, dtype, keepdims, ...])                                 | Return the sum of array elements over a given axis treating Not a Numbers (NaNs) as zero.                                                                              |
| [`nanvar`](generated/dask.array.nanvar.md#dask.array.nanvar)(a[, axis, dtype, keepdims, ddof, ...])                           | Compute the variance along the specified axis, while ignoring NaNs.                                                                                                    |
| [`nan_to_num`](generated/dask.array.nan_to_num.md#dask.array.nan_to_num)(\*args, \*\*kwargs)                                  | Replace NaN with zero and infinity with large finite numbers (default behaviour) or with the numbers defined by the user using the nan, posinf and/or neginf keywords. |
| [`negative`](generated/dask.array.negative.md#dask.array.negative)(x, /[, out, where, casting, order, ...])                   | This docstring was copied from numpy.negative.                                                                                                                         |
| [`nextafter`](generated/dask.array.nextafter.md#dask.array.nextafter)(x1, x2, /[, out, where, casting, ...])                  | This docstring was copied from numpy.nextafter.                                                                                                                        |
| [`nonzero`](generated/dask.array.nonzero.md#dask.array.nonzero)(a)                                                            | Return the indices of the elements that are non-zero.                                                                                                                  |
| [`not_equal`](generated/dask.array.not_equal.md#dask.array.not_equal)(x1, x2, /[, out, where, casting, ...])                  | This docstring was copied from numpy.not_equal.                                                                                                                        |
| [`notnull`](generated/dask.array.notnull.md#dask.array.notnull)(values)                                                       | pandas.notnull for dask arrays                                                                                                                                         |
| [`ones`](generated/dask.array.ones.md#dask.array.ones)(\*args, \*\*kwargs)                                                    | Blocked variant of ones_like                                                                                                                                           |
| [`ones_like`](generated/dask.array.ones_like.md#dask.array.ones_like)(a[, dtype, order, chunks, name, shape])                 | Return an array of ones with the same shape and type as a given array.                                                                                                 |
| [`outer`](generated/dask.array.outer.md#dask.array.outer)(a, b)                                                               | Compute the outer product of two vectors.                                                                                                                              |
| [`pad`](generated/dask.array.pad.md#dask.array.pad)(array, pad_width[, mode])                                                 | Pad an array.                                                                                                                                                          |
| [`percentile`](generated/dask.array.percentile.md#dask.array.percentile)(a, q[, method, internal_method])                     | Approximate percentile of 1-D array                                                                                                                                    |
| [`push`](generated/dask.array.push.md#dask.array.push)(array, n, axis)                                                        | Dask-version of bottleneck.push                                                                                                                                        |
| [`PerformanceWarning`](generated/dask.array.core.PerformanceWarning.md#dask.array.core.PerformanceWarning)                    | A warning given when bad chunking may cause poor performance                                                                                                           |
| [`piecewise`](generated/dask.array.piecewise.md#dask.array.piecewise)(x, condlist, funclist, \*args, \*\*kw)                  | Evaluate a piecewise-defined function.                                                                                                                                 |
| [`positive`](generated/dask.array.positive.md#dask.array.positive)(x, /[, out, where, casting, order, ...])                   | This docstring was copied from numpy.positive.                                                                                                                         |
| [`power`](generated/dask.array.power.md#dask.array.power)(x1, x2, /[, out, where, casting, ...])                              | This docstring was copied from numpy.power.                                                                                                                            |
| [`prod`](generated/dask.array.prod.md#dask.array.prod)(a[, axis, dtype, keepdims, ...])                                       | Return the product of array elements over a given axis.                                                                                                                |
| [`ptp`](generated/dask.array.ptp.md#dask.array.ptp)(a[, axis])                                                                | Range of values (maximum - minimum) along an axis.                                                                                                                     |
| [`quantile`](generated/dask.array.quantile.md#dask.array.quantile)(a, q[, axis, out, overwrite_input, ...])                   | Compute the q-th quantile of the data along the specified axis.                                                                                                        |
| [`rad2deg`](generated/dask.array.rad2deg.md#dask.array.rad2deg)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.rad2deg.                                                                                                                          |
| [`radians`](generated/dask.array.radians.md#dask.array.radians)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.radians.                                                                                                                          |
| [`ravel`](generated/dask.array.ravel.md#dask.array.ravel)(array_like)                                                         | Return a contiguous flattened array.                                                                                                                                   |
| [`ravel_multi_index`](generated/dask.array.ravel_multi_index.md#dask.array.ravel_multi_index)(multi_index, dims[, mode, ...]) | Converts a tuple of index arrays into an array of flat indices, applying boundary modes to the multi-index.                                                            |
| [`real`](generated/dask.array.real.md#dask.array.real)(\*args, \*\*kwargs)                                                    | Return the real part of the complex argument.                                                                                                                          |
| [`reciprocal`](generated/dask.array.reciprocal.md#dask.array.reciprocal)(x, /[, out, where, casting, ...])                    | This docstring was copied from numpy.reciprocal.                                                                                                                       |
| [`rechunk`](generated/dask.array.rechunk.md#dask.array.rechunk)(x[, chunks, threshold, ...])                                  | Convert blocks in dask array x for new chunks.                                                                                                                         |
| [`reduction`](generated/dask.array.reduction.md#dask.array.reduction)(x, chunk, aggregate[, axis, ...])                       | General version of reductions                                                                                                                                          |
| [`register_chunk_type`](generated/dask.array.register_chunk_type.md#dask.array.register_chunk_type)(type)                     | Register the given type as a valid chunk and downcast array type                                                                                                       |
| [`remainder`](generated/dask.array.remainder.md#dask.array.remainder)(x1, x2, /[, out, where, casting, ...])                  | This docstring was copied from numpy.remainder.                                                                                                                        |
| [`repeat`](generated/dask.array.repeat.md#dask.array.repeat)(a, repeats[, axis])                                              | Repeat each element of an array after themselves                                                                                                                       |
| [`reshape`](generated/dask.array.reshape.md#dask.array.reshape)(x, shape[, merge_chunks, limit])                              | Reshape array to new shape                                                                                                                                             |
| [`reshape_blockwise`](generated/dask.array.reshape_blockwise.md#dask.array.reshape_blockwise)(x, shape[, chunks])             | Blockwise-reshape into a new shape.                                                                                                                                    |
| [`result_type`](generated/dask.array.result_type.md#dask.array.result_type)(\*arrays_and_dtypes)                              | This docstring was copied from numpy.result_type.                                                                                                                      |
| [`right_shift`](generated/dask.array.right_shift.md#dask.array.right_shift)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.right_shift.                                                                                                                      |
| [`rint`](generated/dask.array.rint.md#dask.array.rint)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.rint.                                                                                                                             |
| [`roll`](generated/dask.array.roll.md#dask.array.roll)(array, shift[, axis])                                                  | Roll array elements along a given axis.                                                                                                                                |
| [`rollaxis`](generated/dask.array.rollaxis.md#dask.array.rollaxis)(a, axis[, start])                                          |                                                                                                                                                                        |
| [`rot90`](generated/dask.array.rot90.md#dask.array.rot90)(m[, k, axes])                                                       | Rotate an array by 90 degrees in the plane specified by axes.                                                                                                          |
| [`round`](generated/dask.array.round.md#dask.array.round)(a[, decimals])                                                      | Evenly round to the given number of decimals.                                                                                                                          |
| [`searchsorted`](generated/dask.array.searchsorted.md#dask.array.searchsorted)(a, v[, side, sorter])                          | Find indices where elements should be inserted to maintain order.                                                                                                      |
| [`select`](generated/dask.array.select.md#dask.array.select)(condlist, choicelist[, default])                                 | Return an array drawn from elements in choicelist, depending on conditions.                                                                                            |
| [`shape`](generated/dask.array.shape.md#dask.array.shape)(array)                                                              | Return the shape of an array.                                                                                                                                          |
| [`shuffle`](generated/dask.array.shuffle.md#dask.array.shuffle)(x, indexer, axis[, chunks])                                   | Reorders one dimensions of a Dask Array based on an indexer.                                                                                                           |
| [`sign`](generated/dask.array.sign.md#dask.array.sign)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.sign.                                                                                                                             |
| [`signbit`](generated/dask.array.signbit.md#dask.array.signbit)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.signbit.                                                                                                                          |
| [`sin`](generated/dask.array.sin.md#dask.array.sin)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.sin.                                                                                                                              |
| [`sinc`](generated/dask.array.sinc.md#dask.array.sinc)(\*args, \*\*kwargs)                                                    | Return the normalized sinc function.                                                                                                                                   |
| [`sinh`](generated/dask.array.sinh.md#dask.array.sinh)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.sinh.                                                                                                                             |
| [`spacing`](generated/dask.array.spacing.md#dask.array.spacing)(x, /[, out, where, casting, order, ...])                      | This docstring was copied from numpy.spacing.                                                                                                                          |
| [`sqrt`](generated/dask.array.sqrt.md#dask.array.sqrt)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.sqrt.                                                                                                                             |
| [`square`](generated/dask.array.square.md#dask.array.square)(x, /[, out, where, casting, order, ...])                         | This docstring was copied from numpy.square.                                                                                                                           |
| [`squeeze`](generated/dask.array.squeeze.md#dask.array.squeeze)(a[, axis])                                                    | Remove axes of length one from a.                                                                                                                                      |
| [`stack`](generated/dask.array.stack.md#dask.array.stack)(seq[, axis, allow_unknown_chunksizes])                              | Stack arrays along a new axis                                                                                                                                          |
| [`std`](generated/dask.array.std.md#dask.array.std)(a[, axis, dtype, keepdims, ddof, ...])                                    | Compute the standard deviation along the specified axis.                                                                                                               |
| [`subtract`](generated/dask.array.subtract.md#dask.array.subtract)(x1, x2, /[, out, where, casting, ...])                     | This docstring was copied from numpy.subtract.                                                                                                                         |
| [`sum`](generated/dask.array.sum.md#dask.array.sum)(a[, axis, dtype, keepdims, split_every, out])                             | Sum of array elements over a given axis.                                                                                                                               |
| [`swapaxes`](generated/dask.array.swapaxes.md#dask.array.swapaxes)(a, axis1, axis2)                                           | Interchange two axes of an array.                                                                                                                                      |
| [`take`](generated/dask.array.take.md#dask.array.take)(a, indices[, axis])                                                    | Take elements from an array along an axis.                                                                                                                             |
| [`tan`](generated/dask.array.tan.md#dask.array.tan)(x, /[, out, where, casting, order, ...])                                  | This docstring was copied from numpy.tan.                                                                                                                              |
| [`tanh`](generated/dask.array.tanh.md#dask.array.tanh)(x, /[, out, where, casting, order, ...])                               | This docstring was copied from numpy.tanh.                                                                                                                             |
| [`tensordot`](generated/dask.array.tensordot.md#dask.array.tensordot)(lhs, rhs[, axes])                                       | Compute tensor dot product along specified axes.                                                                                                                       |
| [`tile`](generated/dask.array.tile.md#dask.array.tile)(A, reps)                                                               | Construct an array by repeating A the number of times given by reps.                                                                                                   |
| [`topk`](generated/dask.array.topk.md#dask.array.topk)(a, k[, axis, split_every])                                             | Extract the k largest elements from a on the given axis, and return them sorted from largest to smallest.                                                              |
| [`trace`](generated/dask.array.trace.md#dask.array.trace)(a[, offset, axis1, axis2, dtype])                                   | Return the sum along diagonals of the array.                                                                                                                           |
| [`transpose`](generated/dask.array.transpose.md#dask.array.transpose)(a[, axes])                                              | Returns an array with axes transposed.                                                                                                                                 |
| [`tri`](generated/dask.array.tri.md#dask.array.tri)(N[, M, k, dtype, chunks, like])                                           | An array with ones at and below the given diagonal and zeros elsewhere.                                                                                                |
| [`tril`](generated/dask.array.tril.md#dask.array.tril)(m[, k])                                                                | Lower triangle of an array.                                                                                                                                            |
| [`tril_indices`](generated/dask.array.tril_indices.md#dask.array.tril_indices)(n[, k, m, chunks])                             | Return the indices for the lower-triangle of an (n, m) array.                                                                                                          |
| [`tril_indices_from`](generated/dask.array.tril_indices_from.md#dask.array.tril_indices_from)(arr[, k])                       | Return the indices for the lower-triangle of arr.                                                                                                                      |
| [`triu`](generated/dask.array.triu.md#dask.array.triu)(m[, k])                                                                | Upper triangle of an array.                                                                                                                                            |
| [`triu_indices`](generated/dask.array.triu_indices.md#dask.array.triu_indices)(n[, k, m, chunks])                             | Return the indices for the upper-triangle of an (n, m) array.                                                                                                          |
| [`triu_indices_from`](generated/dask.array.triu_indices_from.md#dask.array.triu_indices_from)(arr[, k])                       | Return the indices for the upper-triangle of arr.                                                                                                                      |
| [`true_divide`](generated/dask.array.true_divide.md#dask.array.true_divide)(x1, x2, /[, out, where, ...])                     | This docstring was copied from numpy.divide.                                                                                                                           |
| [`trunc`](generated/dask.array.trunc.md#dask.array.trunc)(x, /[, out, where, casting, order, ...])                            | This docstring was copied from numpy.trunc.                                                                                                                            |
| [`union1d`](generated/dask.array.union1d.md#dask.array.union1d)(ar1, ar2)                                                     | Find the union of two arrays.                                                                                                                                          |
| [`unique`](generated/dask.array.unique.md#dask.array.unique)(ar[, return_index, return_inverse, ...])                         | Find the unique elements of an array.                                                                                                                                  |
| [`unravel_index`](generated/dask.array.unravel_index.md#dask.array.unravel_index)(indices, shape[, order])                    | This docstring was copied from numpy.unravel_index.                                                                                                                    |
| [`var`](generated/dask.array.var.md#dask.array.var)(a[, axis, dtype, keepdims, ddof, ...])                                    | Compute the variance along the specified axis.                                                                                                                         |
| [`vdot`](generated/dask.array.vdot.md#dask.array.vdot)(a, b, /)                                                               | This docstring was copied from numpy.vdot.                                                                                                                             |
| [`vstack`](generated/dask.array.vstack.md#dask.array.vstack)(tup[, allow_unknown_chunksizes])                                 | Stack arrays in sequence vertically (row wise).                                                                                                                        |
| [`where`](generated/dask.array.where.md#dask.array.where)(condition, [x, y], /)                                               | This docstring was copied from numpy.where.                                                                                                                            |
| [`zeros`](generated/dask.array.zeros.md#dask.array.zeros)(\*args, \*\*kwargs)                                                 | Blocked variant of zeros_like                                                                                                                                          |
| [`zeros_like`](generated/dask.array.zeros_like.md#dask.array.zeros_like)(a[, dtype, order, chunks, name, ...])                | Return an array of zeros with the same shape and type as a given array.                                                                                                |

## Array

| [`Array`](generated/dask.array.Array.md#dask.array.Array)(dask, name, chunks[, dtype, meta, shape])                                     | Parallel Dask Array                                                                                            |
|-----------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------|
| [`Array.all`](generated/dask.array.Array.all.md#dask.array.Array.all)([axis, keepdims, split_every, out])                               | Returns True if all elements evaluate to True.                                                                 |
| [`Array.any`](generated/dask.array.Array.any.md#dask.array.Array.any)([axis, keepdims, split_every, out])                               | Returns True if any of the elements evaluate to True.                                                          |
| [`Array.argmax`](generated/dask.array.Array.argmax.md#dask.array.Array.argmax)([axis, keepdims, split_every, out])                      | Return indices of the maximum values along the given axis.                                                     |
| [`Array.argmin`](generated/dask.array.Array.argmin.md#dask.array.Array.argmin)([axis, keepdims, split_every, out])                      | Return indices of the minimum values along the given axis.                                                     |
| [`Array.argtopk`](generated/dask.array.Array.argtopk.md#dask.array.Array.argtopk)(k[, axis, split_every])                               | The indices of the top k elements of an array.                                                                 |
| [`Array.astype`](generated/dask.array.Array.astype.md#dask.array.Array.astype)(dtype, \*\*kwargs)                                       | Copy of the array, cast to a specified type.                                                                   |
| [`Array.blocks`](generated/dask.array.Array.blocks.md#dask.array.Array.blocks)                                                          | An array-like interface to the blocks of an array.                                                             |
| [`Array.choose`](generated/dask.array.Array.choose.md#dask.array.Array.choose)(choices)                                                 | Use an index array to construct a new array from a set of choices.                                             |
| [`Array.chunks`](generated/dask.array.Array.chunks.md#dask.array.Array.chunks)                                                          | Chunks property.                                                                                               |
| [`Array.chunksize`](generated/dask.array.Array.chunksize.md#dask.array.Array.chunksize)                                                 |                                                                                                                |
| [`Array.clip`](generated/dask.array.Array.clip.md#dask.array.Array.clip)([min, max])                                                    | Return an array whose values are limited to `[min, max]`.                                                      |
| [`Array.compute`](generated/dask.array.Array.compute.md#dask.array.Array.compute)(\*\*kwargs)                                           | Compute this dask collection                                                                                   |
| [`Array.compute_chunk_sizes`](generated/dask.array.Array.compute_chunk_sizes.md#dask.array.Array.compute_chunk_sizes)()                 | Compute the chunk sizes for a Dask array.                                                                      |
| [`Array.conj`](generated/dask.array.Array.conj.md#dask.array.Array.conj)()                                                              | Complex-conjugate all elements.                                                                                |
| [`Array.copy`](generated/dask.array.Array.copy.md#dask.array.Array.copy)()                                                              | Copy array.                                                                                                    |
| [`Array.cumprod`](generated/dask.array.Array.cumprod.md#dask.array.Array.cumprod)(axis[, dtype, out, method])                           | Return the cumulative product of the elements along the given axis.                                            |
| [`Array.cumsum`](generated/dask.array.Array.cumsum.md#dask.array.Array.cumsum)(axis[, dtype, out, method])                              | Return the cumulative sum of the elements along the given axis.                                                |
| [`Array.dask`](generated/dask.array.Array.dask.md#dask.array.Array.dask)                                                                |                                                                                                                |
| [`Array.dot`](generated/dask.array.Array.dot.md#dask.array.Array.dot)(other)                                                            | Dot product of self and other.                                                                                 |
| [`Array.dtype`](generated/dask.array.Array.dtype.md#dask.array.Array.dtype)                                                             |                                                                                                                |
| [`Array.flatten`](generated/dask.array.Array.flatten.md#dask.array.Array.flatten)()                                                     | Return a flattened array.                                                                                      |
| [`Array.imag`](generated/dask.array.Array.imag.md#dask.array.Array.imag)                                                                |                                                                                                                |
| [`Array.itemsize`](generated/dask.array.Array.itemsize.md#dask.array.Array.itemsize)                                                    | Length of one array element in bytes                                                                           |
| [`Array.map_blocks`](generated/dask.array.Array.map_blocks.md#dask.array.Array.map_blocks)(\*args[, name, token, ...])                  | Map a function across all blocks of a dask array.                                                              |
| [`Array.map_overlap`](generated/dask.array.Array.map_overlap.md#dask.array.Array.map_overlap)(func, depth[, boundary, trim])            | Map a function over blocks of the array with some overlap                                                      |
| [`Array.max`](generated/dask.array.Array.max.md#dask.array.Array.max)([axis, keepdims, split_every, out])                               | Return the maximum along a given axis.                                                                         |
| [`Array.mean`](generated/dask.array.Array.mean.md#dask.array.Array.mean)([axis, dtype, keepdims, ...])                                  | Returns the average of the array elements along given axis.                                                    |
| [`Array.min`](generated/dask.array.Array.min.md#dask.array.Array.min)([axis, keepdims, split_every, out])                               | Return the minimum along a given axis.                                                                         |
| [`Array.moment`](generated/dask.array.Array.moment.md#dask.array.Array.moment)(order[, axis, dtype, keepdims, ...])                     | Calculate the nth centralized moment.                                                                          |
| [`Array.name`](generated/dask.array.Array.name.md#dask.array.Array.name)                                                                |                                                                                                                |
| [`Array.nbytes`](generated/dask.array.Array.nbytes.md#dask.array.Array.nbytes)                                                          | Number of bytes in array                                                                                       |
| [`Array.ndim`](generated/dask.array.Array.ndim.md#dask.array.Array.ndim)                                                                |                                                                                                                |
| [`Array.nonzero`](generated/dask.array.Array.nonzero.md#dask.array.Array.nonzero)()                                                     | Return the indices of the elements that are non-zero.                                                          |
| [`Array.npartitions`](generated/dask.array.Array.npartitions.md#dask.array.Array.npartitions)                                           |                                                                                                                |
| [`Array.numblocks`](generated/dask.array.Array.numblocks.md#dask.array.Array.numblocks)                                                 |                                                                                                                |
| [`Array.partitions`](generated/dask.array.Array.partitions.md#dask.array.Array.partitions)                                              | Slice an array by partitions.                                                                                  |
| [`Array.persist`](generated/dask.array.Array.persist.md#dask.array.Array.persist)(\*\*kwargs)                                           | Persist this dask collection into memory                                                                       |
| [`Array.prod`](generated/dask.array.Array.prod.md#dask.array.Array.prod)([axis, dtype, keepdims, ...])                                  | Return the product of the array elements over the given axis                                                   |
| [`Array.ravel`](generated/dask.array.Array.ravel.md#dask.array.Array.ravel)()                                                           | Return a flattened array.                                                                                      |
| [`Array.real`](generated/dask.array.Array.real.md#dask.array.Array.real)                                                                |                                                                                                                |
| [`Array.rechunk`](generated/dask.array.Array.rechunk.md#dask.array.Array.rechunk)([chunks, threshold, ...])                             | Convert blocks in dask array x for new chunks.                                                                 |
| [`Array.repeat`](generated/dask.array.Array.repeat.md#dask.array.Array.repeat)(repeats[, axis])                                         | Repeat elements of an array.                                                                                   |
| [`Array.reshape`](generated/dask.array.Array.reshape.md#dask.array.Array.reshape)(\*shape[, merge_chunks, limit])                       | Reshape array to new shape                                                                                     |
| [`Array.round`](generated/dask.array.Array.round.md#dask.array.Array.round)([decimals])                                                 | Return array with each element rounded to the given number of decimals.                                        |
| [`Array.shape`](generated/dask.array.Array.shape.md#dask.array.Array.shape)                                                             |                                                                                                                |
| [`Array.shuffle`](generated/dask.array.Array.shuffle.md#dask.array.Array.shuffle)(indexer, axis[, chunks])                              | Reorders one dimensions of a Dask Array based on an indexer.                                                   |
| [`Array.size`](generated/dask.array.Array.size.md#dask.array.Array.size)                                                                | Number of elements in array                                                                                    |
| [`Array.squeeze`](generated/dask.array.Array.squeeze.md#dask.array.Array.squeeze)([axis])                                               | Remove axes of length one from array.                                                                          |
| [`Array.std`](generated/dask.array.Array.std.md#dask.array.Array.std)([axis, dtype, keepdims, ddof, ...])                               | Returns the standard deviation of the array elements along given axis.                                         |
| [`Array.store`](generated/dask.array.Array.store.md#dask.array.Array.store)(targets[, lock, regions, ...])                              | Store dask arrays in array-like objects, overwrite data in target                                              |
| [`Array.sum`](generated/dask.array.Array.sum.md#dask.array.Array.sum)([axis, dtype, keepdims, ...])                                     | Return the sum of the array elements over the given axis.                                                      |
| [`Array.swapaxes`](generated/dask.array.Array.swapaxes.md#dask.array.Array.swapaxes)(axis1, axis2)                                      | Return a view of the array with `axis1` and `axis2` interchanged.                                              |
| [`Array.to_backend`](generated/dask.array.Array.to_backend.md#dask.array.Array.to_backend)([backend])                                   | Move to a new Array backend                                                                                    |
| [`Array.to_dask_dataframe`](generated/dask.array.Array.to_dask_dataframe.md#dask.array.Array.to_dask_dataframe)([columns, index, meta]) | Convert dask Array to dask Dataframe                                                                           |
| [`Array.to_delayed`](generated/dask.array.Array.to_delayed.md#dask.array.Array.to_delayed)([optimize_graph])                            | Convert into an array of [`dask.delayed.Delayed`](delayed-api.md#dask.delayed.Delayed) objects, one per chunk. |
| [`Array.to_hdf5`](generated/dask.array.Array.to_hdf5.md#dask.array.Array.to_hdf5)(filename, datapath, \*\*kwargs)                       | Store array in HDF5 file                                                                                       |
| [`Array.to_svg`](generated/dask.array.Array.to_svg.md#dask.array.Array.to_svg)([size])                                                  | Convert chunks from Dask Array into an SVG Image                                                               |
| [`Array.to_tiledb`](generated/dask.array.Array.to_tiledb.md#dask.array.Array.to_tiledb)(uri, \*args, \*\*kwargs)                        | Save array to the TileDB storage manager                                                                       |
| [`Array.to_zarr`](generated/dask.array.Array.to_zarr.md#dask.array.Array.to_zarr)(\*args, \*\*kwargs)                                   | Save array to the zarr storage format                                                                          |
| [`Array.topk`](generated/dask.array.Array.topk.md#dask.array.Array.topk)(k[, axis, split_every])                                        | The top k elements of an array.                                                                                |
| [`Array.trace`](generated/dask.array.Array.trace.md#dask.array.Array.trace)([offset, axis1, axis2, dtype])                              | Return the sum along diagonals of the array.                                                                   |
| [`Array.transpose`](generated/dask.array.Array.transpose.md#dask.array.Array.transpose)(\*axes)                                         | Reverse or permute the axes of an array.                                                                       |
| [`Array.var`](generated/dask.array.Array.var.md#dask.array.Array.var)([axis, dtype, keepdims, ddof, ...])                               | Returns the variance of the array elements, along given axis.                                                  |
| [`Array.view`](generated/dask.array.Array.view.md#dask.array.Array.view)([dtype, order])                                                | Get a view of the array as a new data type                                                                     |
| [`Array.vindex`](generated/dask.array.Array.vindex.md#dask.array.Array.vindex)                                                          | Vectorized indexing with broadcasting.                                                                         |
| [`Array.visualize`](generated/dask.array.Array.visualize.md#dask.array.Array.visualize)([filename, format, ...])                        | Render the computation of this object's task graph using graphviz.                                             |

## Fast Fourier Transforms

| [`fft.fft_wrap`](generated/dask.array.fft.fft_wrap.md#dask.array.fft.fft_wrap)(fft_func[, kind, dtype, ...])   | Wrap 1D, 2D, and ND real and complex FFT functions                                     |
|----------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| [`fft.fft`](generated/dask.array.fft.fft.md#dask.array.fft.fft)(a[, n, axis, norm])                            | Wrapping of numpy.fft.fft                                                              |
| [`fft.fft2`](generated/dask.array.fft.fft2.md#dask.array.fft.fft2)(a[, s, axes, norm])                         | Wrapping of numpy.fft.fft2                                                             |
| [`fft.fftn`](generated/dask.array.fft.fftn.md#dask.array.fft.fftn)(a[, s, axes, norm])                         | Wrapping of numpy.fft.fftn                                                             |
| [`fft.ifft`](generated/dask.array.fft.ifft.md#dask.array.fft.ifft)(a[, n, axis, norm])                         | Wrapping of numpy.fft.ifft                                                             |
| [`fft.ifft2`](generated/dask.array.fft.ifft2.md#dask.array.fft.ifft2)(a[, s, axes, norm])                      | Wrapping of numpy.fft.ifft2                                                            |
| [`fft.ifftn`](generated/dask.array.fft.ifftn.md#dask.array.fft.ifftn)(a[, s, axes, norm])                      | Wrapping of numpy.fft.ifftn                                                            |
| [`fft.rfft`](generated/dask.array.fft.rfft.md#dask.array.fft.rfft)(a[, n, axis, norm])                         | Wrapping of numpy.fft.rfft                                                             |
| [`fft.rfft2`](generated/dask.array.fft.rfft2.md#dask.array.fft.rfft2)(a[, s, axes, norm])                      | Wrapping of numpy.fft.rfft2                                                            |
| [`fft.rfftn`](generated/dask.array.fft.rfftn.md#dask.array.fft.rfftn)(a[, s, axes, norm])                      | Wrapping of numpy.fft.rfftn                                                            |
| [`fft.irfft`](generated/dask.array.fft.irfft.md#dask.array.fft.irfft)(a[, n, axis, norm])                      | Wrapping of numpy.fft.irfft                                                            |
| [`fft.irfft2`](generated/dask.array.fft.irfft2.md#dask.array.fft.irfft2)(a[, s, axes, norm])                   | Wrapping of numpy.fft.irfft2                                                           |
| [`fft.irfftn`](generated/dask.array.fft.irfftn.md#dask.array.fft.irfftn)(a[, s, axes, norm])                   | Wrapping of numpy.fft.irfftn                                                           |
| [`fft.hfft`](generated/dask.array.fft.hfft.md#dask.array.fft.hfft)(a[, n, axis, norm])                         | Wrapping of numpy.fft.hfft                                                             |
| [`fft.ihfft`](generated/dask.array.fft.ihfft.md#dask.array.fft.ihfft)(a[, n, axis, norm])                      | Wrapping of numpy.fft.ihfft                                                            |
| [`fft.fftfreq`](generated/dask.array.fft.fftfreq.md#dask.array.fft.fftfreq)(n[, d, chunks])                    | Return the Discrete Fourier Transform sample frequencies.                              |
| [`fft.rfftfreq`](generated/dask.array.fft.rfftfreq.md#dask.array.fft.rfftfreq)(n[, d, chunks])                 | Return the Discrete Fourier Transform sample frequencies (for usage with rfft, irfft). |
| [`fft.fftshift`](generated/dask.array.fft.fftshift.md#dask.array.fft.fftshift)(x[, axes])                      | Shift the zero-frequency component to the center of the spectrum.                      |
| [`fft.ifftshift`](generated/dask.array.fft.ifftshift.md#dask.array.fft.ifftshift)(x[, axes])                   | The inverse of fftshift.                                                               |

## Linear Algebra

| [`linalg.cholesky`](generated/dask.array.linalg.cholesky.md#dask.array.linalg.cholesky)(a[, lower])                              | Returns the Cholesky decomposition, $A = L L^*$ or $A = U^* U$ of a Hermitian positive-definite matrix A.   |
|----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
| [`linalg.inv`](generated/dask.array.linalg.inv.md#dask.array.linalg.inv)(a)                                                      | Compute the inverse of a matrix with LU decomposition and forward / backward substitutions.                 |
| [`linalg.lstsq`](generated/dask.array.linalg.lstsq.md#dask.array.linalg.lstsq)(a, b)                                             | Return the least-squares solution to a linear matrix equation using QR decomposition.                       |
| [`linalg.lu`](generated/dask.array.linalg.lu.md#dask.array.linalg.lu)(a)                                                         | Compute the lu decomposition of a matrix.                                                                   |
| [`linalg.norm`](generated/dask.array.linalg.norm.md#dask.array.linalg.norm)(x[, ord, axis, keepdims])                            | Matrix or vector norm.                                                                                      |
| [`linalg.qr`](generated/dask.array.linalg.qr.md#dask.array.linalg.qr)(a)                                                         | Compute the qr factorization of a matrix.                                                                   |
| [`linalg.solve`](generated/dask.array.linalg.solve.md#dask.array.linalg.solve)(a, b[, sym_pos, assume_a])                        | Solve the equation `a x = b` for `x`.                                                                       |
| [`linalg.solve_triangular`](generated/dask.array.linalg.solve_triangular.md#dask.array.linalg.solve_triangular)(a, b[, lower])   | Solve the equation a x = b for x, assuming a is a triangular matrix.                                        |
| [`linalg.svd`](generated/dask.array.linalg.svd.md#dask.array.linalg.svd)(a[, coerce_signs, full_matrices])                       | Compute the singular value decomposition of a matrix.                                                       |
| [`linalg.svd_compressed`](generated/dask.array.linalg.svd_compressed.md#dask.array.linalg.svd_compressed)(a, k[, iterator, ...]) | Randomly compressed rank-k thin Singular Value Decomposition.                                               |
| [`linalg.sfqr`](generated/dask.array.linalg.sfqr.md#dask.array.linalg.sfqr)(data[, name])                                        | Direct Short-and-Fat QR                                                                                     |
| [`linalg.tsqr`](generated/dask.array.linalg.tsqr.md#dask.array.linalg.tsqr)(data[, compute_svd, ...])                            | Direct Tall-and-Skinny QR algorithm                                                                         |

## Masked Arrays

| [`ma.average`](generated/dask.array.ma.average.md#dask.array.ma.average)(a[, axis, weights, returned, ...])                       | Return the weighted average of array over the given axis.                     |
|-----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
| [`ma.empty_like`](generated/dask.array.ma.empty_like.md#dask.array.ma.empty_like)(a, \*\*kwargs)                                  | Return a new array with the same shape and type as a given array.             |
| [`ma.filled`](generated/dask.array.ma.filled.md#dask.array.ma.filled)(a[, fill_value])                                            | Return input as an ~numpy.ndarray, with masked values replaced by fill_value. |
| [`ma.fix_invalid`](generated/dask.array.ma.fix_invalid.md#dask.array.ma.fix_invalid)(a[, fill_value])                             | Return input with invalid data masked and replaced by a fill value.           |
| [`ma.getdata`](generated/dask.array.ma.getdata.md#dask.array.ma.getdata)(a)                                                       | Return the data of a masked array as an ndarray.                              |
| [`ma.getmaskarray`](generated/dask.array.ma.getmaskarray.md#dask.array.ma.getmaskarray)(a)                                        | Return the mask of a masked array, or full boolean array of False.            |
| [`ma.masked_array`](generated/dask.array.ma.masked_array.md#dask.array.ma.masked_array)(data[, mask, fill_value])                 | An array class with possibly masked values.                                   |
| [`ma.masked_equal`](generated/dask.array.ma.masked_equal.md#dask.array.ma.masked_equal)(a, value)                                 | Mask an array where equal to a given value.                                   |
| [`ma.masked_greater`](generated/dask.array.ma.masked_greater.md#dask.array.ma.masked_greater)(x, value[, copy])                   | Mask an array where greater than a given value.                               |
| [`ma.masked_greater_equal`](generated/dask.array.ma.masked_greater_equal.md#dask.array.ma.masked_greater_equal)(x, value[, copy]) | Mask an array where greater than or equal to a given value.                   |
| [`ma.masked_inside`](generated/dask.array.ma.masked_inside.md#dask.array.ma.masked_inside)(x, v1, v2)                             | Mask an array inside a given interval.                                        |
| [`ma.masked_invalid`](generated/dask.array.ma.masked_invalid.md#dask.array.ma.masked_invalid)(a)                                  | Mask an array where invalid values occur (NaNs or infs).                      |
| [`ma.masked_less`](generated/dask.array.ma.masked_less.md#dask.array.ma.masked_less)(x, value[, copy])                            | Mask an array where less than a given value.                                  |
| [`ma.masked_less_equal`](generated/dask.array.ma.masked_less_equal.md#dask.array.ma.masked_less_equal)(x, value[, copy])          | Mask an array where less than or equal to a given value.                      |
| [`ma.masked_not_equal`](generated/dask.array.ma.masked_not_equal.md#dask.array.ma.masked_not_equal)(x, value[, copy])             | Mask an array where *not* equal to a given value.                             |
| [`ma.masked_outside`](generated/dask.array.ma.masked_outside.md#dask.array.ma.masked_outside)(x, v1, v2)                          | Mask an array outside a given interval.                                       |
| [`ma.masked_values`](generated/dask.array.ma.masked_values.md#dask.array.ma.masked_values)(x, value[, rtol, atol, shrink])        | Mask using floating point equality.                                           |
| [`ma.masked_where`](generated/dask.array.ma.masked_where.md#dask.array.ma.masked_where)(condition, a)                             | Mask an array where a condition is met.                                       |
| [`ma.nonzero`](generated/dask.array.ma.nonzero.md#dask.array.ma.nonzero)(self)                                                    | This docstring was copied from numpy.ma.core.nonzero.                         |
| [`ma.ones_like`](generated/dask.array.ma.ones_like.md#dask.array.ma.ones_like)(a, \*\*kwargs)                                     | Return an array of ones with the same shape and type as a given array.        |
| [`ma.set_fill_value`](generated/dask.array.ma.set_fill_value.md#dask.array.ma.set_fill_value)(a, fill_value)                      | Set the filling value of a, if a is a masked array.                           |
| [`ma.where`](generated/dask.array.ma.where.md#dask.array.ma.where)(condition[, x, y])                                             | Return a masked array with elements from x or y, depending on condition.      |
| [`ma.zeros_like`](generated/dask.array.ma.zeros_like.md#dask.array.ma.zeros_like)(a, \*\*kwargs)                                  | Return an array of zeros with the same shape and type as a given array.       |

## Random

| [`random.beta`](generated/dask.array.random.beta.md#dask.array.random.beta)(\*args, \*\*kwargs)                                                 | Draw samples from a Beta distribution.                                                                                |
|-------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| [`random.binomial`](generated/dask.array.random.binomial.md#dask.array.random.binomial)(\*args, \*\*kwargs)                                     | Draw samples from a binomial distribution.                                                                            |
| [`random.chisquare`](generated/dask.array.random.chisquare.md#dask.array.random.chisquare)(\*args, \*\*kwargs)                                  | Draw samples from a chi-square distribution.                                                                          |
| [`random.choice`](generated/dask.array.random.choice.md#dask.array.random.choice)(\*args, \*\*kwargs)                                           | Generates a random sample from a given 1-D array                                                                      |
| [`random.default_rng`](generated/dask.array.random.default_rng.md#dask.array.random.default_rng)([seed])                                        | Construct a new Generator with the default BitGenerator (PCG64).                                                      |
| [`random.exponential`](generated/dask.array.random.exponential.md#dask.array.random.exponential)(\*args, \*\*kwargs)                            | Draw samples from an exponential distribution.                                                                        |
| [`random.f`](generated/dask.array.random.f.md#dask.array.random.f)(\*args, \*\*kwargs)                                                          | Draw samples from an F distribution.                                                                                  |
| [`random.gamma`](generated/dask.array.random.gamma.md#dask.array.random.gamma)(\*args, \*\*kwargs)                                              | Draw samples from a Gamma distribution.                                                                               |
| [`random.geometric`](generated/dask.array.random.geometric.md#dask.array.random.geometric)(\*args, \*\*kwargs)                                  | Draw samples from the geometric distribution.                                                                         |
| [`random.gumbel`](generated/dask.array.random.gumbel.md#dask.array.random.gumbel)(\*args, \*\*kwargs)                                           | Draw samples from a Gumbel distribution.                                                                              |
| [`random.hypergeometric`](generated/dask.array.random.hypergeometric.md#dask.array.random.hypergeometric)(\*args, \*\*kwargs)                   | Draw samples from a Hypergeometric distribution.                                                                      |
| [`random.laplace`](generated/dask.array.random.laplace.md#dask.array.random.laplace)(\*args, \*\*kwargs)                                        | Draw samples from the Laplace or double exponential distribution with specified location (or mean) and scale (decay). |
| [`random.logistic`](generated/dask.array.random.logistic.md#dask.array.random.logistic)(\*args, \*\*kwargs)                                     | Draw samples from a logistic distribution.                                                                            |
| [`random.lognormal`](generated/dask.array.random.lognormal.md#dask.array.random.lognormal)(\*args, \*\*kwargs)                                  | Draw samples from a log-normal distribution.                                                                          |
| [`random.logseries`](generated/dask.array.random.logseries.md#dask.array.random.logseries)(\*args, \*\*kwargs)                                  | Draw samples from a logarithmic series distribution.                                                                  |
| [`random.multinomial`](generated/dask.array.random.multinomial.md#dask.array.random.multinomial)(\*args, \*\*kwargs)                            | Draw samples from a multinomial distribution.                                                                         |
| [`random.negative_binomial`](generated/dask.array.random.negative_binomial.md#dask.array.random.negative_binomial)(\*args, \*\*kwargs)          | Draw samples from a negative binomial distribution.                                                                   |
| [`random.noncentral_chisquare`](generated/dask.array.random.noncentral_chisquare.md#dask.array.random.noncentral_chisquare)(\*args, \*\*kwargs) | Draw samples from a noncentral chi-square distribution.                                                               |
| [`random.noncentral_f`](generated/dask.array.random.noncentral_f.md#dask.array.random.noncentral_f)(\*args, \*\*kwargs)                         | Draw samples from the noncentral F distribution.                                                                      |
| [`random.normal`](generated/dask.array.random.normal.md#dask.array.random.normal)(\*args, \*\*kwargs)                                           | Draw random samples from a normal (Gaussian) distribution.                                                            |
| [`random.pareto`](generated/dask.array.random.pareto.md#dask.array.random.pareto)(\*args, \*\*kwargs)                                           | Draw samples from a Pareto II or Lomax distribution with specified shape.                                             |
| [`random.permutation`](generated/dask.array.random.permutation.md#dask.array.random.permutation)(\*args, \*\*kwargs)                            | Randomly permute a sequence, or return a permuted range.                                                              |
| [`random.poisson`](generated/dask.array.random.poisson.md#dask.array.random.poisson)(\*args, \*\*kwargs)                                        | Draw samples from a Poisson distribution.                                                                             |
| [`random.power`](generated/dask.array.random.power.md#dask.array.random.power)(\*args, \*\*kwargs)                                              | Draws samples in [0, 1] from a power distribution with positive exponent a - 1.                                       |
| [`random.randint`](generated/dask.array.random.randint.md#dask.array.random.randint)(\*args, \*\*kwargs)                                        | Return random integers from low (inclusive) to high (exclusive).                                                      |
| [`random.random`](generated/dask.array.random.random.md#dask.array.random.random)(\*args, \*\*kwargs)                                           | Return random floats in the half-open interval [0.0, 1.0).                                                            |
| [`random.random_integers`](generated/dask.array.random.random_integers.md#dask.array.random.random_integers)(\*args, \*\*kwargs)                | Random integers of type numpy.int_ between low and high, inclusive.                                                   |
| [`random.random_sample`](generated/dask.array.random.random_sample.md#dask.array.random.random_sample)(\*args, \*\*kwargs)                      | Return random floats in the half-open interval [0.0, 1.0).                                                            |
| [`random.rayleigh`](generated/dask.array.random.rayleigh.md#dask.array.random.rayleigh)(\*args, \*\*kwargs)                                     | Draw samples from a Rayleigh distribution.                                                                            |
| [`random.standard_cauchy`](generated/dask.array.random.standard_cauchy.md#dask.array.random.standard_cauchy)(\*args, \*\*kwargs)                | Draw samples from a standard Cauchy distribution with mode = 0.                                                       |
| [`random.standard_exponential`](generated/dask.array.random.standard_exponential.md#dask.array.random.standard_exponential)(\*args, \*\*kwargs) | Draw samples from the standard exponential distribution.                                                              |
| [`random.standard_gamma`](generated/dask.array.random.standard_gamma.md#dask.array.random.standard_gamma)(\*args, \*\*kwargs)                   | Draw samples from a standard Gamma distribution.                                                                      |
| [`random.standard_normal`](generated/dask.array.random.standard_normal.md#dask.array.random.standard_normal)(\*args, \*\*kwargs)                | Draw samples from a standard Normal distribution (mean=0, stdev=1).                                                   |
| [`random.standard_t`](generated/dask.array.random.standard_t.md#dask.array.random.standard_t)(\*args, \*\*kwargs)                               | Draw samples from a standard Student's t distribution with df degrees of freedom.                                     |
| [`random.triangular`](generated/dask.array.random.triangular.md#dask.array.random.triangular)(\*args, \*\*kwargs)                               | Draw samples from the triangular distribution over the interval `[left, right]`.                                      |
| [`random.uniform`](generated/dask.array.random.uniform.md#dask.array.random.uniform)(\*args, \*\*kwargs)                                        | Draw samples from a uniform distribution.                                                                             |
| [`random.vonmises`](generated/dask.array.random.vonmises.md#dask.array.random.vonmises)(\*args, \*\*kwargs)                                     | Draw samples from a von Mises distribution.                                                                           |
| [`random.wald`](generated/dask.array.random.wald.md#dask.array.random.wald)(\*args, \*\*kwargs)                                                 | Draw samples from a Wald, or inverse Gaussian, distribution.                                                          |
| [`random.weibull`](generated/dask.array.random.weibull.md#dask.array.random.weibull)(\*args, \*\*kwargs)                                        | Draw samples from a Weibull distribution.                                                                             |
| [`random.zipf`](generated/dask.array.random.zipf.md#dask.array.random.zipf)(\*args, \*\*kwargs)                                                 | Draw samples from a Zipf distribution.                                                                                |

## Stats

| [`stats.ttest_ind`](generated/dask.array.stats.ttest_ind.md#dask.array.stats.ttest_ind)(a, b[, axis, equal_var])                        | Calculate the T-test for the means of *two independent* samples of scores.   |
|-----------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| [`stats.ttest_1samp`](generated/dask.array.stats.ttest_1samp.md#dask.array.stats.ttest_1samp)(a, popmean[, axis, nan_policy])           | Calculate the T-test for the mean of ONE group of scores.                    |
| [`stats.ttest_rel`](generated/dask.array.stats.ttest_rel.md#dask.array.stats.ttest_rel)(a, b[, axis, nan_policy])                       | Calculate the t-test on TWO RELATED samples of scores, a and b.              |
| [`stats.chisquare`](generated/dask.array.stats.chisquare.md#dask.array.stats.chisquare)(f_obs[, f_exp, ddof, axis])                     | Calculate a one-way chi-square test.                                         |
| [`stats.power_divergence`](generated/dask.array.stats.power_divergence.md#dask.array.stats.power_divergence)(f_obs[, f_exp, ddof, ...]) | Cressie-Read power divergence statistic and goodness of fit test.            |
| [`stats.skew`](generated/dask.array.stats.skew.md#dask.array.stats.skew)(a[, axis, bias, nan_policy])                                   | Compute the sample skewness of a data set.                                   |
| [`stats.skewtest`](generated/dask.array.stats.skewtest.md#dask.array.stats.skewtest)(a[, axis, nan_policy])                             | Test whether the skew is different from the normal distribution.             |
| [`stats.kurtosis`](generated/dask.array.stats.kurtosis.md#dask.array.stats.kurtosis)(a[, axis, fisher, bias, ...])                      | Compute the kurtosis (Fisher or Pearson) of a dataset.                       |
| [`stats.kurtosistest`](generated/dask.array.stats.kurtosistest.md#dask.array.stats.kurtosistest)(a[, axis, nan_policy])                 | Test whether a dataset has normal kurtosis.                                  |
| [`stats.normaltest`](generated/dask.array.stats.normaltest.md#dask.array.stats.normaltest)(a[, axis, nan_policy])                       | Test whether a sample differs from a normal distribution.                    |
| [`stats.f_oneway`](generated/dask.array.stats.f_oneway.md#dask.array.stats.f_oneway)(\*args)                                            | Perform one-way ANOVA.                                                       |
| [`stats.moment`](generated/dask.array.stats.moment.md#dask.array.stats.moment)(a[, moment, axis, nan_policy])                           | Calculate the nth moment about the mean for a sample.                        |

## Image Support

| [`image.imread`](generated/dask.array.image.imread.md#dask.array.image.imread)(filename[, imread, preprocess])   | Read a stack of images into a dask array   |
|------------------------------------------------------------------------------------------------------------------|--------------------------------------------|

## Slightly Overlapping Computations

| [`overlap.overlap`](generated/dask.array.overlap.overlap.md#dask.array.overlap.overlap)(x, depth, boundary, \*[, ...])                                            | Share boundaries between neighboring blocks                              |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
| [`overlap.map_overlap`](generated/dask.array.overlap.map_overlap.md#dask.array.overlap.map_overlap)(func, \*args[, depth, ...])                                   | Map a function over blocks of arrays with some overlap                   |
| [`lib.stride_tricks.sliding_window_view`](generated/dask.array.lib.stride_tricks.sliding_window_view.md#dask.array.lib.stride_tricks.sliding_window_view)(x, ...) | Create a sliding window view into the array with the given window shape. |
| [`overlap.trim_internal`](generated/dask.array.overlap.trim_internal.md#dask.array.overlap.trim_internal)(x, axes[, boundary])                                    | Trim sides from each block                                               |
| [`overlap.trim_overlap`](generated/dask.array.overlap.trim_overlap.md#dask.array.overlap.trim_overlap)(x, depth[, boundary])                                      | Trim sides from each block.                                              |

## Create and Store Arrays

| [`from_array`](generated/dask.array.from_array.md#dask.array.from_array)(x[, chunks, name, lock, asarray, ...])   | Create dask array from something that looks like an array.        |
|-------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`from_delayed`](generated/dask.array.from_delayed.md#dask.array.from_delayed)(value, shape[, dtype, meta, name]) | Create a dask array from a dask delayed value                     |
| [`from_npy_stack`](generated/dask.array.from_npy_stack.md#dask.array.from_npy_stack)(dirname[, mmap_mode])        | Load dask array from stack of npy files                           |
| [`from_zarr`](generated/dask.array.from_zarr.md#dask.array.from_zarr)(url[, component, storage_options, ...])     | Load array from the zarr storage format                           |
| [`from_tiledb`](generated/dask.array.from_tiledb.md#dask.array.from_tiledb)(uri[, attribute, chunks, ...])        | Load array from the TileDB storage format                         |
| [`store`](generated/dask.array.store.md#dask.array.store)(sources, targets[, lock, regions, ...])                 | Store dask arrays in array-like objects, overwrite data in target |
| [`to_hdf5`](generated/dask.array.to_hdf5.md#dask.array.to_hdf5)(filename, \*args[, chunks])                       | Store arrays in HDF5 file                                         |
| [`to_zarr`](generated/dask.array.to_zarr.md#dask.array.to_zarr)(arr, url[, component, ...])                       | Save array to the zarr storage format                             |
| [`to_npy_stack`](generated/dask.array.to_npy_stack.md#dask.array.to_npy_stack)(dirname, x[, axis])                | Write dask array to a stack of .npy files                         |
| [`to_tiledb`](generated/dask.array.to_tiledb.md#dask.array.to_tiledb)(darray, uri[, compute, ...])                | Save array to the TileDB storage format                           |

## Generalized Ufuncs

| [`apply_gufunc`](generated/dask.array.gufunc.apply_gufunc.md#dask.array.gufunc.apply_gufunc)(func, signature, \*args[, axes, ...])   | Apply a generalized ufunc or similar python function to arrays.   |
|--------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`as_gufunc`](generated/dask.array.gufunc.as_gufunc.md#dask.array.gufunc.as_gufunc)([signature])                                     | Decorator for `dask.array.gufunc`.                                |
| [`gufunc`](generated/dask.array.gufunc.gufunc.md#dask.array.gufunc.gufunc)(pyfunc, \*[, signature, vectorize, ...])                  | Binds pyfunc into `dask.array.apply_gufunc` when called.          |

## Downstream Libraries API

| [`normalize_chunks_cached`](generated/dask.array.api.normalize_chunks_cached.md#dask.array.api.normalize_chunks_cached)(chunks[, shape, ...])   | Cached version of normalize_chunks.   |
|-------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
| [`normalize_chunks`](generated/dask.array.api.normalize_chunks.md#dask.array.api.normalize_chunks)(chunks[, shape, limit, ...])                 | Normalize chunks to tuple of tuples   |

## Internal functions

| [`blockwise`](generated/dask.array.core.blockwise.md#dask.array.core.blockwise)(func, out_ind, \*args[, name, ...])               | Tensor operation: Generalized inner and outer products   |
|-----------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| [`normalize_chunks`](generated/dask.array.core.normalize_chunks.md#dask.array.core.normalize_chunks)(chunks[, shape, limit, ...]) | Normalize chunks to tuple of tuples                      |
| [`unify_chunks`](generated/dask.array.core.unify_chunks.md#dask.array.core.unify_chunks)(\*args, \*\*kwargs)                      | Unify chunks across a sequence of arrays                 |

## Dask Metadata

| [`meta_from_array`](generated/dask.array.utils.meta_from_array.md#dask.array.utils.meta_from_array)(x[, ndim, dtype])   | Normalize an array to appropriate meta object   |
|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|


# array-assignment.html.md

<a id="array-assignment"></a>

# Assignment

Dask Array supports most of the NumPy assignment indexing syntax. In
particular, it supports combinations of the following:

* Indexing by integers: `x[1] = y`
* Indexing by slices: `x[2::-1] = y`
* Indexing by a list of integers: `x[[0, -1, 1]] = y`
* Indexing by a 1-d `numpy` array of integers: `x[np.arange(3)] = y`
* Indexing by a 1-d [`Array`](generated/dask.array.Array.md#dask.array.Array) of integers: `x[da.arange(3)] = y`, `x[da.from_array([0, -1, 1])] = y`, `x[da.where(np.array([1, 2, 3]) < 3)[0]] = y`
* Indexing by a list of booleans: `x[[False, True, True]] = y`
* Indexing by a 1-d `numpy` array of booleans: `x[np.arange(3) > 0] = y`

It also supports:

* Indexing by one broadcastable [`Array`](generated/dask.array.Array.md#dask.array.Array) of
  booleans: `x[x > 0] = y`.

However, it does not currently support the following:

* Indexing with lists in multiple axes: `x[[1, 2, 3], [3, 1, 2]] = y`

<a id="array-assignment-broadcasting"></a>

## Broadcasting

The normal NumPy broadcasting rules apply:

```python
>>> x = da.zeros((2, 6))
>>> x[0] = 1
>>> x[..., 1] = 2.0
>>> x[:, 2] = [3, 4]
>>> x[:, 5:2:-2] = [[6, 5]]
>>> x.compute()
array([[1., 2., 3., 5., 1., 6.],
       [0., 2., 4., 5., 0., 6.]])
>>> x[1] = -x[0]
>>> x.compute()
array([[ 1.,  2.,  3.,  5.,  1.,  6.],
       [-1., -2., -3., -5., -1., -6.]])
```

<a id="array-assignment-masking"></a>

## Masking

Elements may be masked by assigning to the NumPy masked value, or to an
array with masked values:

```python
>>> x = da.ones((2, 6))
>>> x[0, [1, -2]] = np.ma.masked
>>> x[1] = np.ma.array([0, 1, 2, 3, 4, 5], mask=[0, 1, 1, 0, 0, 0])
>>> print(x.compute())
[[1.0 -- 1.0 1.0 -- 1.0]
 [0.0 -- -- 3.0 4.0 5.0]]
>>> x[:, 0] = x[:, 1]
>>> print(x.compute())
[[1.0 -- 1.0 1.0 -- 1.0]
 [0.0 -- -- 3.0 4.0 5.0]]
>>> x[:, 0] = x[:, 1]
>>> print(x.compute())
[[-- -- 1.0 1.0 -- 1.0]
 [-- -- -- 3.0 4.0 5.0]]
```

If, and only if, a single broadcastable [`Array`](generated/dask.array.Array.md#dask.array.Array) of
booleans is provided then masked array assignment does not yet work as
expected. In this case the data underlying the mask are assigned:

```python
>>> x = da.arange(12).reshape(2, 6)
>>> x[x > 7] = np.ma.array(-99, mask=True)
>>> print(x.compute())
[[  0   1   2   3   4   5]
 [  6   7 -99 -99 -99 -99]]
```

Note that masked assignments do work when a boolean
[`Array`](generated/dask.array.Array.md#dask.array.Array) index used in a tuple, or implicit tuple,
of indices:

```python
>>> x = da.arange(12).reshape(2, 6)
>>> x[1, x[0] > 3] = np.ma.masked
>>> print(x.compute())
[[0 1 2 3 4 5]
 [6 7 8 9 -- --]]
>>> x = da.arange(12).reshape(2, 6)
>>> print(x.compute())
[[ 0  1  2  3  4  5]
 [ 6  7  8  9 10 11]]
>>> x[(x[:, 2] < 4,)] = np.ma.masked
>>> print(x.compute())
[[-- -- -- -- -- --]
 [6 7 8 9 10 11]]
```


# array-best-practices.html.md

# Best Practices

It is easy to get started with Dask arrays, but using them *well* does require
some experience.  This page contains suggestions for best practices, and
includes solutions to common problems.

## Use NumPy

If your data fits comfortably in RAM and you are not performance bound, then
using NumPy might be the right choice.  Dask adds another layer of complexity
which may get in the way.

If you are just looking for speedups rather than scalability then you may want
to consider a project like [Numba](https://numba.pydata.org)

## Select a good chunk size

A common performance problem among Dask Array users is that they have chosen a
chunk size that is either too small (leading to lots of overhead) or poorly
aligned with their data (leading to inefficient reading).

While optimal sizes and shapes are highly problem specific, it is rare to see
chunk sizes below 100 MB in size.  If you are dealing with float64 data then
this is around `(4000, 4000)` in size for a 2D array or `(100, 400, 400)`
for a 3D array.

You want to choose a chunk size that is large in order to reduce the number of
chunks that Dask has to think about (which affects overhead) but also small
enough so that many of them can fit in memory at once.  Dask will often have as
many chunks in memory as twice the number of active threads.

## Orient your chunks

When reading data you should align your chunks with your storage format.
Most array storage formats store data in chunks themselves.  If your Dask array
chunks aren’t multiples of these chunk shapes then you will have to read the
same data repeatedly, which can be expensive.  Note though that often
storage formats choose chunk sizes that are much smaller than is ideal for
Dask, closer to 1MB than 100MB.  In these cases you should choose a Dask chunk
size that aligns with the storage chunk size and that every Dask chunk
dimension is a multiple of the storage chunk dimension.

So for example if we have an HDF file that has chunks of size `(128, 64)`, we
might choose a chunk shape of `(1280, 6400)`.

```python
>>> import h5py
>>> storage = h5py.File('myfile.hdf5')['x']
>>> storage.chunks
(128, 64)

>>> import dask.array as da
>>> x = da.from_array(storage, chunks=(1280, 6400))
```

Note that if you provide `chunks='auto'` then Dask Array will look for a
`.chunks` attribute and use that to provide a good chunking.

## Avoid Oversubscribing Threads

By default Dask will run as many concurrent tasks as you have logical cores.
It assumes that each task will consume about one core.  However, many
array-computing libraries are themselves multi-threaded, which can cause
contention and low performance.  In particular the BLAS/LAPACK libraries that
back most of NumPy’s linear algebra routines are often multi-threaded, and need
to be told to use only one thread explicitly.  You can do this with the
following environment variables (using bash `export` command below, but this
may vary depending on your operating system).

```bash
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
```

You need to run this before you start your Python process for it to take effect.

## Consider Xarray

The [Xarray](http://xarray.pydata.org/en/stable/) package wraps around Dask
Array, and so offers the same scalability, but also adds convenience when
dealing with complex datasets.  In particular Xarray can help with the
following:

1. Manage multiple arrays together as a consistent dataset
2. Read from a stack of HDF or NetCDF files at once
3. Switch between Dask Array and NumPy with a consistent API

Xarray is used in wide range of fields, including physics, astronomy, geoscience,
microscopy, bioinformatics, engineering, finance, and deep learning.
Xarray also has a thriving user community that is good at providing support.

## Build your own Operations

Often we want to perform computations for which there is no exact function
in Dask Array.  In these cases we may be able to use some of the more generic
functions to build our own.  These include:

| [`blockwise`](generated/dask.array.blockwise.md#dask.array.blockwise)(func, out_ind, \*args[, name, ...])         | Tensor operation: Generalized inner and outer products   |
|-------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| [`map_blocks`](generated/dask.array.map_blocks.md#dask.array.map_blocks)(func, \*args[, name, token, ...])        | Map a function across all blocks of a dask array.        |
| [`map_overlap`](generated/dask.array.map_overlap.md#dask.array.map_overlap)(func, \*args[, depth, boundary, ...]) | Map a function over blocks of arrays with some overlap   |
| [`reduction`](generated/dask.array.reduction.md#dask.array.reduction)(x, chunk, aggregate[, axis, ...])           | General version of reductions                            |

These functions may help you to apply a function that you write for NumPy
functions onto larger Dask arrays.


# array-chunks.html.md

<a id="array-chunks"></a>

# Chunks

Dask arrays are composed of many NumPy (or NumPy-like) arrays. How these arrays
are arranged can significantly affect performance.  For example, for a square
array you might arrange your chunks along rows, along columns, or in a more
square-like fashion. Different arrangements of NumPy arrays will be faster or
slower for different algorithms.

Thinking about and controlling chunking is important to optimize advanced
algorithms.

## Specifying Chunk shapes

We always specify a `chunks` argument to tell dask.array how to break up the
underlying array into chunks.  We can specify `chunks` in a variety of ways:

1. A uniform dimension size like `1000`, meaning chunks of size `1000` in each dimension
2. A uniform chunk shape like `(1000, 2000, 3000)`, meaning chunks of size `1000` in the
   first axis, `2000` in the second axis, and `3000` in the third
3. Fully explicit sizes of all blocks along all dimensions,
   like `((1000, 1000, 500), (400, 400), (5, 5, 5, 5, 5))`
4. A dictionary specifying chunk size per dimension like `{0: 1000, 1: 2000,
   2: 3000}`.  This is just another way of writing the forms 2 and 3 above

Your chunks input will be normalized and stored in the third and most explicit
form.  Note that `chunks` stands for “chunk shape” rather than “number of
chunks”, so specifying `chunks=1` means that you will have many chunks,
each with exactly one element.

For performance, a good choice of `chunks` follows the following rules:

1. A chunk should be small enough to fit comfortably in memory.  We’ll
   have many chunks in memory at once
2. A chunk must be large enough so that computations on that chunk take
   significantly longer than the 1ms overhead per task that Dask scheduling
   incurs.  A task should take longer than 100ms
3. Chunk sizes between 10MB-1GB are common, depending on the availability of
   RAM and the duration of computations
4. Chunks should align with the computation that you want to do.

   For example, if you plan to frequently slice along a particular dimension,
   then it’s more efficient if your chunks are aligned so that you have to
   touch fewer chunks.  If you want to add two arrays, then its convenient if
   those arrays have matching chunks patterns
5. Chunks should align with your storage, if applicable.

   Array data formats are often chunked as well.  When loading or saving data,
   it is useful to have Dask array chunks that are aligned with the chunking
   of your storage, often an even multiple times larger in each direction

Learn more in [Choosing good chunk sizes in Dask](https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes) by Genevieve Buckley.

## Unknown Chunks

Some arrays have unknown chunk sizes.  This arises whenever the size of an
array depends on lazy computations that we haven’t yet performed like the
following:

```python
>>> rng = np.random.default_rng()
>>> x = da.from_array(rng.standard_normal(100), chunks=20)
>>> x += 0.1
>>> y = x[x > 0]  # don't know how many values are greater than 0 ahead of time
```

Operations like the above result in arrays with unknown shapes and unknown
chunk sizes.  Unknown values within shape or chunks are designated using
`np.nan` rather than an integer.  These arrays support many (but not all)
operations.  In particular, operations like slicing are not possible and will
result in an error.

```python
>>> y.shape
(np.nan,)
>>> y[4]
...
ValueError: Array chunk sizes unknown

A possible solution: https://docs.dask.org/en/latest/array-chunks.html#unknown-chunks.
Summary: to compute chunks sizes, use

    x.compute_chunk_sizes()  # for Dask Array
    ddf.to_dask_array(lengths=True)  # for Dask DataFrame ddf
```

Using [`compute_chunk_sizes()`](generated/dask.array.Array.compute_chunk_sizes.md#dask.array.Array.compute_chunk_sizes)  allows this example run:

```python
>>> y.compute_chunk_sizes()
dask.array<..., chunksize=(19,), ...>
>>> y.shape
(44,)
>>> y[4].compute()
0.78621774046566
```

Note that [`compute_chunk_sizes()`](generated/dask.array.Array.compute_chunk_sizes.md#dask.array.Array.compute_chunk_sizes) immediately performs computation and
modifies the array in-place.

Unknown chunksizes also occur when using a Dask DataFrame to create a Dask array:

```python
>>> ddf = dask.dataframe.from_pandas(...)
>>> ddf.to_dask_array()
dask.array<..., shape=(nan, 2), ..., chunksize=(nan, 2)>
```

Using [`to_dask_array()`](generated/dask.dataframe.DataFrame.to_dask_array.md#dask.dataframe.DataFrame.to_dask_array) resolves this issue:

```python
>>> ddf.to_dask_array(lengths=True)
dask.array<..., shape=(100, 2), ..., chunksize=(20, 2)>
```

More details on [`to_dask_array()`](generated/dask.dataframe.DataFrame.to_dask_array.md#dask.dataframe.DataFrame.to_dask_array) are in mentioned in how to create a Dask
array from a Dask DataFrame in the [documentation on Dask array creation](array-creation.md).

## Chunks Examples

In this example we show how different inputs for `chunks=` cut up the following array:

```default
1 2 3 4 5 6
7 8 9 0 1 2
3 4 5 6 7 8
9 0 1 2 3 4
5 6 7 8 9 0
1 2 3 4 5 6
```

Here, we show how different `chunks=` arguments split the array into different blocks

**chunks=3**: Symmetric blocks of size 3:

```default
1 2 3  4 5 6
7 8 9  0 1 2
3 4 5  6 7 8

9 0 1  2 3 4
5 6 7  8 9 0
1 2 3  4 5 6
```

**chunks=2**: Symmetric blocks of size 2:

```default
1 2  3 4  5 6
7 8  9 0  1 2

3 4  5 6  7 8
9 0  1 2  3 4

5 6  7 8  9 0
1 2  3 4  5 6
```

**chunks=(3, 2)**: Asymmetric but repeated blocks of size `(3, 2)`:

```default
1 2  3 4  5 6
7 8  9 0  1 2
3 4  5 6  7 8

9 0  1 2  3 4
5 6  7 8  9 0
1 2  3 4  5 6
```

**chunks=(1, 6)**: Asymmetric but repeated blocks of size `(1, 6)`:

```default
1 2 3 4 5 6

7 8 9 0 1 2

3 4 5 6 7 8

9 0 1 2 3 4

5 6 7 8 9 0

1 2 3 4 5 6
```

**chunks=((2, 4), (3, 3))**: Asymmetric and non-repeated blocks:

```default
1 2 3  4 5 6
7 8 9  0 1 2

3 4 5  6 7 8
9 0 1  2 3 4
5 6 7  8 9 0
1 2 3  4 5 6
```

**chunks=((2, 2, 1, 1), (3, 2, 1))**: Asymmetric and non-repeated blocks:

```default
1 2 3  4 5  6
7 8 9  0 1  2

3 4 5  6 7  8
9 0 1  2 3  4

5 6 7  8 9  0

1 2 3  4 5  6
```

**Discussion**

The latter examples are rarely provided by users on original data but arise from complex slicing and broadcasting operations.  Generally people use the simplest form until they need more complex forms.  The choice of chunks should align with the computations you want to do.

For example, if you plan to take out thin slices along the first dimension, then you might want to make that dimension skinnier than the others.  If you plan to do linear algebra, then you might want more symmetric blocks.

## Loading Chunked Data

Modern NDArray storage formats like HDF5, NetCDF, TIFF, and Zarr, allow arrays
to be stored in chunks or tiles so that blocks of data can be pulled out
efficiently without having to seek through a linear data stream.  It is best to
align the chunks of your Dask array with the chunks of your underlying data
store.

However, data stores often chunk more finely than is ideal for Dask array, so
it is common to choose a chunking that is a multiple of your storage chunk
size, otherwise you might incur high overhead.

For example, if you are loading a data store that is chunked in blocks of
`(100, 100)`, then you might choose a chunking more like `(1000, 2000)` that
is larger, but still evenly divisible by `(100, 100)`.  Data storage
technologies will be able to tell you how their data is chunked.

## Rechunking

| [`rechunk`](generated/dask.array.rechunk.md#dask.array.rechunk)(x[, chunks, threshold, ...])   | Convert blocks in dask array x for new chunks.   |
|------------------------------------------------------------------------------------------------|--------------------------------------------------|

Sometimes you need to change the chunking layout of your data.  For example,
perhaps it comes to you chunked row-wise, but you need to do an operation that
is much faster if done across columns.  You can change the chunking with the
`rechunk` method.

```python
x = x.rechunk((50, 1000))
```

Rechunking across axes can be expensive and incur a lot of communication, but
Dask array has fairly efficient algorithms to accomplish this.

Note: The rechunk method expects the output array to have the same shape as the input array and does not support reshaping. It is important to ensure that the desired output shape matches the input shape before using rechunk.

You can pass rechunk any valid chunking form:

```python
x = x.rechunk(1000)
x = x.rechunk((50, 1000))
x = x.rechunk({0: 50, 1: 1000})
```

<a id="array-chunks-reshaping"></a>

## Reshaping

The efficiency of [`dask.array.reshape()`](generated/dask.array.reshape.md#dask.array.reshape) can depend strongly on the chunking
of the input array. In reshaping operations, there’s the concept of “fast-moving”
or “high” axes. For a 2d array the second axis (`axis=1`) is the fastest-moving,
followed by the first. This means that if we draw a line indicating how values
are filled, we move across the “columns” first (along `axis=1`), and then down
to the next row. Consider `np.ones((3, 4)).reshape(12)`:

![Visual representation of a 2-dimensional (3 rows by 4 colurmns) NumPy array being reshaped to 1 dimension (12 columns by 1 row). Arrows indicate the order in which values from the original array are copied to the new array, moving across the columns in axis 1 first before moving down to the next row in axis 0.](images/reshape.png)

Now consider the impact of Dask’s chunking on this operation. If the slow-moving
axis (just `axis=0` in this case) has chunks larger than size 1, we run into
a problem.

![image](images/reshape_problem.png)

The first block has a shape `(2, 2)`. Following the rules of `reshape` we
take the two values from the first row of block 1. But then we cross a chunk
boundary (from 1 to 2) while we still have two “unused” values in the first
block. There’s no way to line up the input blocks with the output shape. We
need to somehow rechunk the input to be compatible with the output shape. We
have two options

1. Merge chunks using the logic in [`dask.array.rechunk()`](generated/dask.array.rechunk.md#dask.array.rechunk). This avoids
   making two many tasks / blocks, at the cost of some communication and
   larger intermediates. This is the default behavior.
2. Use `da.reshape(x, shape, merge_chunks=False)` to avoid merging chunks
   by *splitting the input*. In particular, we can rechunk all the
   slow-moving axes to have a chunksize of 1. This avoids
   communication and moving around large amounts of data, at the cost of
   a larger task graph (potentially much larger, since the number of chunks
   on the slow-moving axes will equal the length of those axes.).

Visually, here’s the second option:

![image](images/reshape_rechunked.png)

Which if these is better depends on your problem. If communication is very
expensive and your data is relatively small along the slow-moving axes, then
`merge_chunks=False` may be better. Let’s compare the task graphs of these
two on a problem reshaping a 3-d array to a 2-d, where the input array doesn’t
have `chunksize=1` on the slow-moving axes.

```python
>>> a = da.from_array(np.arange(24).reshape(2, 3, 4), chunks=((2,), (2, 1), (2, 2)))
>>> a
dask.array<array, shape=(2, 3, 4), dtype=int64, chunksize=(2, 2, 2), chunktype=numpy.ndarray>
>>> a.reshape(6, 4).visualize()
```

![image](images/merge_chunks.png)
```python
>>> a.reshape(6, 4, merge_chunks=False).visualize()
```

![image](images/merge_chunks_false.png)

By default, some intermediate chunks chunks are merged, leading to a more complicated task
graph. With `merge_chunks=False` we split the input chunks (leading to more overall tasks,
depending on the size of the array) but avoid later communication.

## Automatic Chunking

Chunks also includes three special values:

1. `-1`: no chunking along this dimension
2. `None`: no change to the chunking along this dimension (useful for rechunk)
3. `"auto"`: allow the chunking in this dimension to accommodate ideal chunk sizes

So, for example, one could rechunk a 3D array to have no chunking along the zeroth
dimension, but still have sensible chunk sizes as follows:

```python
x = x.rechunk({0: -1, 1: 'auto', 2: 'auto'})
```

Or one can allow *all* dimensions to be auto-scaled to get to a good chunk
size:

```python
x = x.rechunk('auto')
```

Automatic chunking expands or contracts all dimensions marked with `"auto"`
to try to reach chunk sizes with a number of bytes equal to the config value
`array.chunk-size`, which is set to 128MiB by default, but which you can
change in your [configuration](configuration.md).

```python
>>> dask.config.get('array.chunk-size')
'128MiB'
```

Automatic rechunking tries to respect the median chunk shape of the
auto-rescaled dimensions, but will modify this to accommodate the shape of the
full array (can’t have larger chunks than the array itself) and to find
chunk shapes that nicely divide the shape.

These values can also be used when creating arrays with operations like
`dask.array.ones` or `dask.array.from_array`

```python
>>> dask.array.ones((10000, 10000), chunks=(-1, 'auto'))
dask.array<wrapped, shape=(10000, 10000), dtype=float64, chunksize=(10000, 1250), chunktype=numpy.ndarray>
```


# array-creation.html.md

# Create Dask Arrays

You can load or store Dask arrays from a variety of common sources like HDF5,
NetCDF, [Zarr](https://zarr.readthedocs.io/en/stable/), or any format that supports NumPy-style slicing.

| [`from_array`](generated/dask.array.from_array.md#dask.array.from_array)(x[, chunks, name, lock, asarray, ...])   | Create dask array from something that looks like an array.   |
|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| [`from_delayed`](generated/dask.array.from_delayed.md#dask.array.from_delayed)(value, shape[, dtype, meta, name]) | Create a dask array from a dask delayed value                |
| [`from_npy_stack`](generated/dask.array.from_npy_stack.md#dask.array.from_npy_stack)(dirname[, mmap_mode])        | Load dask array from stack of npy files                      |
| [`from_zarr`](generated/dask.array.from_zarr.md#dask.array.from_zarr)(url[, component, storage_options, ...])     | Load array from the zarr storage format                      |
| [`stack`](generated/dask.array.stack.md#dask.array.stack)(seq[, axis, allow_unknown_chunksizes])                  | Stack arrays along a new axis                                |
| [`concatenate`](generated/dask.array.concatenate.md#dask.array.concatenate)(seq[, axis, ...])                     | Concatenate arrays along an existing axis                    |

## NumPy Slicing

| [`from_array`](generated/dask.array.from_array.md#dask.array.from_array)(x[, chunks, name, lock, asarray, ...])   | Create dask array from something that looks like an array.   |
|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|

Many storage formats have Python projects that expose storage using NumPy
slicing syntax.  These include HDF5, NetCDF, BColz, Zarr, GRIB, etc.  For
example, we can load a Dask array from an HDF5 file using [h5py](https://www.h5py.org/):

```Python
>>> import h5py
>>> f = h5py.File('myfile.hdf5') # HDF5 file
>>> d = f['/data/path']          # Pointer on on-disk array
>>> d.shape                      # d can be very large
(1000000, 1000000)

>>> x = d[:5, :5]                # We slice to get numpy arrays
```

Given an object like `d` above that has `dtype` and `shape` properties
and that supports NumPy style slicing, we can construct a lazy Dask array:

```Python
>>> import dask.array as da
>>> x = da.from_array(d, chunks=(1000, 1000))
```

This process is entirely lazy.  Neither creating the h5py object nor wrapping
it with `da.from_array` have loaded any data.

## Random Data

For experimentation or benchmarking it is common to create arrays of random
data.  The `dask.array.random` module implements most of the functions in the
`numpy.random` module.  We list some common functions below but for a full
list see the [Array API](array-api.md):

| [`random.binomial`](generated/dask.array.random.binomial.md#dask.array.random.binomial)(\*args, \*\*kwargs)   | Draw samples from a binomial distribution.                 |
|---------------------------------------------------------------------------------------------------------------|------------------------------------------------------------|
| [`random.normal`](generated/dask.array.random.normal.md#dask.array.random.normal)(\*args, \*\*kwargs)         | Draw random samples from a normal (Gaussian) distribution. |
| [`random.poisson`](generated/dask.array.random.poisson.md#dask.array.random.poisson)(\*args, \*\*kwargs)      | Draw samples from a Poisson distribution.                  |
| [`random.random`](generated/dask.array.random.random.md#dask.array.random.random)(\*args, \*\*kwargs)         | Return random floats in the half-open interval [0.0, 1.0). |
```python
>>> import dask.array as da
>>> rng = da.random.default_rng()
>>> x = rng.random((10000, 10000), chunks=(1000, 1000))
```

## Concatenation and Stacking

| [`stack`](generated/dask.array.stack.md#dask.array.stack)(seq[, axis, allow_unknown_chunksizes])   | Stack arrays along a new axis             |
|----------------------------------------------------------------------------------------------------|-------------------------------------------|
| [`concatenate`](generated/dask.array.concatenate.md#dask.array.concatenate)(seq[, axis, ...])      | Concatenate arrays along an existing axis |

Often we store data in several different locations and want to stitch them together:

```Python
dask_arrays = []
for fn in filenames:
    f = h5py.File(fn)
    d = f['/data']
    array = da.from_array(d, chunks=(1000, 1000))
    dask_arrays.append(array)

x = da.concatenate(dask_arrays, axis=0)  # concatenate arrays along first axis
```

For more information, see [concatenation and stacking](array-stack.md) docs.

## Using `dask.delayed`

| [`from_delayed`](generated/dask.array.from_delayed.md#dask.array.from_delayed)(value, shape[, dtype, meta, name])   | Create a dask array from a dask delayed value   |
|---------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|
| [`stack`](generated/dask.array.stack.md#dask.array.stack)(seq[, axis, allow_unknown_chunksizes])                    | Stack arrays along a new axis                   |
| [`concatenate`](generated/dask.array.concatenate.md#dask.array.concatenate)(seq[, axis, ...])                       | Concatenate arrays along an existing axis       |

Sometimes NumPy-style data resides in formats that do not support NumPy-style
slicing.  We can still construct Dask arrays around this data if we have a
Python function that can generate pieces of the full array if we use
[dask.delayed](delayed.md).  Dask delayed lets us delay a single function
call that would create a NumPy array.  We can then wrap this delayed object
with `da.from_delayed`, providing a dtype and shape to produce a
single-chunked Dask array.  Furthermore, we can use `stack` or `concatenate` from
before to construct a larger lazy array.

As an example, consider loading a stack of images using `skimage.io.imread`:

```python
import skimage.io
import dask.array as da
import dask

imread = dask.delayed(skimage.io.imread, pure=True)  # Lazy version of imread

filenames = sorted(glob.glob('*.jpg'))

lazy_images = [imread(path) for path in filenames]   # Lazily evaluate imread on each path
sample = lazy_images[0].compute()  # load the first image (assume rest are same shape/dtype)

arrays = [da.from_delayed(lazy_image,           # Construct a small Dask array
                          dtype=sample.dtype,   # for every lazy value
                          shape=sample.shape)
          for lazy_image in lazy_images]

stack = da.stack(arrays, axis=0)                # Stack all small Dask arrays into one
```

See [documentation on using dask.delayed with collections](delayed-collections.md).

Often it is substantially faster to use `da.map_blocks` rather than `da.stack`

```python
import glob
import skimage.io
import numpy as np
import dask.array as da

filenames = sorted(glob.glob('*.jpg'))

def read_one_image(block_id, filenames=filenames, axis=0):
    # a function that reads in one chunk of data
    path = filenames[block_id[axis]]
    image = skimage.io.imread(path)
    return np.expand_dims(image, axis=axis)

# load the first image (assume rest are same shape/dtype)
sample = skimage.io.imread(filenames[0])

stack = da.map_blocks(
    read_one_image,
    dtype=sample.dtype,
    chunks=((1,) * len(filenames),  *sample.shape)
)
```

## From Dask DataFrame

There are several ways to create a Dask array from a Dask DataFrame. Dask DataFrames have a `to_dask_array` method:

```python
>>> df = dask.dataframes.from_pandas(...)
>>> df.to_dask_array()
dask.array<values, shape=(nan, 3), dtype=float64, chunksize=(nan, 3), chunktype=numpy.ndarray>
```

This mirrors the [to_numpy](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html)
function in Pandas. The `values` attribute is also supported:

```python
>>> df.values
dask.array<values, shape=(nan, 3), dtype=float64, chunksize=(nan, 3), chunktype=numpy.ndarray>
```

However, these arrays do not have known chunk sizes because dask.dataframe does
not track the number of rows in each partition. This means that some operations
like slicing will not operate correctly.

The chunk sizes can be computed:

```python
>>> df.to_dask_array(lengths=True)
dask.array<array, shape=(100, 3), dtype=float64, chunksize=(50, 3), chunktype=numpy.ndarray>
```

Specifying `lengths=True` triggers immediate computation of the chunk sizes.
This enables downstream computations that rely on having known chunk sizes
(e.g., slicing).

The Dask DataFrame `to_records` method also returns a Dask Array, but does not compute the shape
information:

```python
>>> df.to_records()
dask.array<to_records, shape=(nan,), dtype=(numpy.record, [('index', '<i8'), ('x', '<f8'), ('y', '<f8'), ('z', '<f8')]), chunksize=(nan,), chunktype=numpy.ndarray>
```

If you have a function that converts a Pandas DataFrame into a NumPy array,
then calling `map_partitions` with that function on a Dask DataFrame will
produce a Dask array:

```python
>>> df.map_partitions(np.asarray)
dask.array<asarray, shape=(nan, 3), dtype=float64, chunksize=(nan, 3), chunktype=numpy.ndarray>
```

## Interactions with NumPy arrays

Dask array operations will automatically convert NumPy arrays into single-chunk
dask arrays:

```python
>>> x = da.sum(np.ones(5))
>>> x.compute()
5
```

When NumPy and Dask arrays interact, the result will be a Dask array.  Automatic
rechunking rules will generally slice the NumPy array into the appropriate Dask
chunk shape:

```python
>>> x = da.ones(10, chunks=(5,))
>>> y = np.ones(10)
>>> z = x + y
>>> z
dask.array<add, shape=(10,), dtype=float64, chunksize=(5,), chunktype=numpy.ndarray>
```

These interactions work not just for NumPy arrays but for any object that has
shape and dtype attributes and implements NumPy slicing syntax.

## Memory mapping

Memory mapping can be a highly effective method to access raw binary data since
it has nearly zero overhead if the data is already in the file system cache. For
the threaded scheduler, creating a Dask array from a raw binary file can be as simple as
`a = da.from_array(np.memmap(filename, shape=shape, dtype=dtype, mode='r'))`.

For multiprocessing or distributed schedulers, the memory map for each array
chunk should be created on the correct worker process and not on the main
process to avoid data transfer through the cluster. This can be achieved by
wrapping the function that creates the memory map using `dask.delayed`.

```python
import numpy as np
import dask
import dask.array as da


def mmap_load_chunk(filename, shape, dtype, offset, sl):
    '''
    Memory map the given file with overall shape and dtype and return a slice
    specified by :code:`sl`.

    Parameters
    ----------

    filename : str
    shape : tuple
        Total shape of the data in the file
    dtype:
        NumPy dtype of the data in the file
    offset : int
        Skip :code:`offset` bytes from the beginning of the file.
    sl:
        Object that can be used for indexing or slicing a NumPy array to
        extract a chunk

    Returns
    -------

    numpy.memmap or numpy.ndarray
        View into memory map created by indexing with :code:`sl`,
        or NumPy ndarray in case no view can be created using :code:`sl`.
    '''
    data = np.memmap(filename, mode='r', shape=shape, dtype=dtype, offset=offset)
    return data[sl]


def mmap_dask_array(filename, shape, dtype, offset=0, blocksize=5):
    '''
    Create a Dask array from raw binary data in :code:`filename`
    by memory mapping.

    This method is particularly effective if the file is already
    in the file system cache and if arbitrary smaller subsets are
    to be extracted from the Dask array without optimizing its
    chunking scheme.

    It may perform poorly on Windows if the file is not in the file
    system cache. On Linux it performs well under most circumstances.

    Parameters
    ----------

    filename : str
    shape : tuple
        Total shape of the data in the file
    dtype:
        NumPy dtype of the data in the file
    offset : int, optional
        Skip :code:`offset` bytes from the beginning of the file.
    blocksize : int, optional
        Chunk size for the outermost axis. The other axes remain unchunked.

    Returns
    -------

    dask.array.Array
        Dask array matching :code:`shape` and :code:`dtype`, backed by
        memory-mapped chunks.
    '''
    load = dask.delayed(mmap_load_chunk)
    chunks = []
    for index in range(0, shape[0], blocksize):
        # Truncate the last chunk if necessary
        chunk_size = min(blocksize, shape[0] - index)
        chunk = dask.array.from_delayed(
            load(
                filename,
                shape=shape,
                dtype=dtype,
                offset=offset,
                sl=slice(index, index + chunk_size)
            ),
            shape=(chunk_size, ) + shape[1:],
            dtype=dtype
        )
        chunks.append(chunk)
    return da.concatenate(chunks, axis=0)

x = mmap_dask_array(
    filename='testfile-50-50-100-100-float32.raw',
    shape=(50, 50, 100, 100),
    dtype=np.float32
)
```

## Chunks

See [documentation on Array Chunks](array-chunks.md) for more information.

# Store Dask Arrays

| [`store`](generated/dask.array.store.md#dask.array.store)(sources, targets[, lock, regions, ...])   | Store dask arrays in array-like objects, overwrite data in target   |
|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`to_hdf5`](generated/dask.array.to_hdf5.md#dask.array.to_hdf5)(filename, \*args[, chunks])         | Store arrays in HDF5 file                                           |
| [`to_npy_stack`](generated/dask.array.to_npy_stack.md#dask.array.to_npy_stack)(dirname, x[, axis])  | Write dask array to a stack of .npy files                           |
| [`to_zarr`](generated/dask.array.to_zarr.md#dask.array.to_zarr)(arr, url[, component, ...])         | Save array to the zarr storage format                               |
| `compute`(\*args[, traverse, optimize_graph, ...])                                                  | Compute several dask collections at once.                           |

## In Memory

| `compute`(\*args[, traverse, optimize_graph, ...])   | Compute several dask collections at once.   |
|------------------------------------------------------|---------------------------------------------|

If you have a small amount of data, you can call `np.array` or `.compute()`
on your Dask array to turn in to a normal NumPy array:

```Python
>>> x = da.arange(6, chunks=3)
>>> y = x**2
>>> np.array(y)
array([0, 1, 4, 9, 16, 25])

>>> y.compute()
array([0, 1, 4, 9, 16, 25])
```

## NumPy style slicing

| [`store`](generated/dask.array.store.md#dask.array.store)(sources, targets[, lock, regions, ...])   | Store dask arrays in array-like objects, overwrite data in target   |
|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|

You can store Dask arrays in any object that supports NumPy-style slice
assignment like `h5py.Dataset`:

```Python
>>> import h5py
>>> f = h5py.File('myfile.hdf5')
>>> d = f.require_dataset('/data', shape=x.shape, dtype=x.dtype)
>>> da.store(x, d)
```

Also, you can store several arrays in one computation by passing lists of sources and
destinations:

```Python
>>> da.store([array1, array2], [output1, output2])  # doctest: +SKIP
```

## HDF5

| [`to_hdf5`](generated/dask.array.to_hdf5.md#dask.array.to_hdf5)(filename, \*args[, chunks])   | Store arrays in HDF5 file   |
|-----------------------------------------------------------------------------------------------|-----------------------------|

HDF5 is sufficiently common that there is a special function `to_hdf5` to
store data into HDF5 files using `h5py`:

```Python
>>> da.to_hdf5('myfile.hdf5', '/y', y)  # doctest: +SKIP
```

You can store several arrays in one computation with the function
`da.to_hdf5` by passing in a dictionary:

```Python
>>> da.to_hdf5('myfile.hdf5', {'/x': x, '/y': y})  # doctest: +SKIP
```

## Zarr

The [Zarr](https://zarr.readthedocs.io/en/stable/) format is a chunk-wise binary array
storage file format with a good selection of encoding and compression options.
Due to each chunk being stored in a separate file, it is ideal for parallel
access in both reading and writing (for the latter, if the Dask array
chunks are aligned with the target). Furthermore, storage in
[remote data services](how-to/connect-to-remote-data.md) such as S3 and GCS is
supported.

For example, to save data to a local zarr dataset you would do:

```Python
>>> arr.to_zarr('output.zarr')
```

or to save to a particular bucket on S3:

```Python
>>> arr.to_zarr('s3://mybucket/output.zarr', storage_option={'key': 'mykey',
                'secret': 'mysecret'})
```

or your own custom zarr Array:

```Python
>>> z = zarr.create((10,), dtype=float, store=zarr.ZipStore("output.zarr"))
>>> arr.to_zarr(z)
```

To retrieve those data, you would do `da.from_zarr` with exactly the same arguments. The
chunking of the resultant Dask array is defined by how the files were saved, unless
otherwise specified.

## Compression options

Zarr supports a variety of compression algorithms and compression levels.
When writing Dask arrays to Zarr, compression can be configured by passing
a Zarr-compatible compressor via `storage_options`.

For example, using `numcodecs` to configure Blosc compression:

```Python
>>> from numcodecs import Blosc
>>> import dask.array as da
>>> arr = da.ones((10, 10), chunks=(5, 5))
>>> compressor = Blosc(cname="zstd", clevel=1)
>>> arr.to_zarr(
...     "output.zarr",
...     storage_options={"compressor": compressor},
... )
```

The available compressors and compression settings depend on the Zarr
backend being used. See the [Zarr documentation on compressors]([https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#compressors](https://zarr.readthedocs.io/en/stable/user-guide/arrays.html#compressors)) for details:

## TileDB

[TileDB](https://docs.tiledb.io)  is a binary array format and storage manager with
tunable chunking, layout, and compression options. The TileDB storage manager library
includes support for scalable storage backends such as S3 API compatible object stores
and HDFS, with automatic scaling, and supports multi-threaded and multi-process
reads (consistent) and writes (eventually-consistent).

To save data to a local TileDB array:

```Python
>>> arr.to_tiledb('output.tdb')
```

or to save to a bucket on S3:

```python
>>> arr.to_tiledb('s3://mybucket/output.tdb',
                  storage_options={'vfs.s3.aws_access_key_id': 'mykey',
                                   'vfs.s3.aws_secret_access_key': 'mysecret'})
```

Files may be retrieved by running da.from_tiledb with the same URI, and any
necessary arguments.

## Intermediate storage

| [`store`](generated/dask.array.store.md#dask.array.store)(sources, targets[, lock, regions, ...])   | Store dask arrays in array-like objects, overwrite data in target   |
|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|

In some cases, one may wish to store an intermediate result in long term
storage. This differs from `persist`, which is mainly used to manage
intermediate results within Dask that don’t necessarily have longevity.
Also it differs from storing final results as these mark the end of the Dask
graph. Thus intermediate results are easier to reuse without reloading data.
Intermediate storage is mainly useful in cases where the data is needed
outside of Dask (e.g. on disk, in a database, in the cloud, etc.). It can
be useful as a checkpoint for long running or error-prone computations.

The intermediate storage use case differs from the typical storage use case as
a Dask Array is returned to the user that represents the result of that
storage operation. This is typically done by setting the `store` function’s
`return_stored` flag to `True`.

```python
x.store()  # stores data, returns nothing
x = x.store(return_stored=True)  # stores data, returns new dask array backed by that data
```

The user can then decide whether the
storage operation happens immediately (by setting the `compute` flag to
`True`) or later (by setting the `compute` flag to `False`). In all
other ways, this behaves the same as a normal call to `store`. Some examples
are shown below.

```Python
>>> import dask.array as da
>>> import zarr as zr
>>> c = (2, 2)
>>> d = da.ones((10, 11), chunks=c)
>>> z1 = zr.open_array('lazy.zarr', shape=d.shape, dtype=d.dtype, chunks=c)
>>> z2 = zr.open_array('eager.zarr', shape=d.shape, dtype=d.dtype, chunks=c)
>>> d1 = d.store(z1, compute=False, return_stored=True)
>>> d2 = d.store(z2, compute=True, return_stored=True)
```

This can be combined with any other storage strategies either noted above, in
the docs or for any specialized storage types.

# Plugins

We can run arbitrary user-defined functions on Dask arrays whenever they are
constructed.  This allows us to build a variety of custom behaviors that improve
debugging, user warning, etc.  You can register a list of functions to run on
all Dask arrays to the global `array_plugins=` value:

```python
>>> def f(x):
...     print(x.nbytes)

>>> with dask.config.set(array_plugins=[f]):
...     x = da.ones((10, 1), chunks=(5, 1))
...     y = x.dot(x.T)
80
80
800
800
```

If the plugin function returns None, then the input Dask array will be returned
without change.  If the plugin function returns something else, then that value
will be the result of the constructor.

## Examples

### Automatically compute

We may wish to turn some Dask array code into normal NumPy code.  This is
useful, for example, to track down errors immediately that would otherwise be
hidden by Dask’s lazy semantics:

```python
>>> with dask.config.set(array_plugins=[lambda x: x.compute()]):
...     x = da.arange(5, chunks=2)

>>> x  # this was automatically converted into a numpy array
array([0, 1, 2, 3, 4])
```

### Warn on large chunks

We may wish to warn users if they are creating chunks that are too large:

```python
def warn_on_large_chunks(x):
    shapes = list(itertools.product(*x.chunks))
    nbytes = [x.dtype.itemsize * np.prod(shape) for shape in shapes]
    if any(nb > 1e9 for nb in nbytes):
        warnings.warn("Array contains very large chunks")

with dask.config.set(array_plugins=[warn_on_large_chunks]):
    ...
```

### Combine

You can also combine these plugins into a list.  They will run one after the
other, chaining results through them:

```python
with dask.config.set(array_plugins=[warn_on_large_chunks, lambda x: x.compute()]):
    ...
```


# array-design.html.md

# Internal Design

## Overview

![12 rectangular blocks arranged as a 4-row, 3-column layout. Each block includes 'x' and its location in the table starting with ('x',0,0) in the top-left, and a size of 5x8.](images/array.svg)

Dask arrays define a large array with a grid of blocks of smaller arrays.
These arrays may be actual arrays or functions that produce arrays. We
define a Dask array with the following components:

* A Dask graph with a special set of keys designating blocks
  such as `('x', 0, 0), ('x', 0, 1), ...` (See [Dask graph
  documentation](graphs.md) for more details)
* A sequence of chunk sizes along each dimension called `chunks`,
  for example `((5, 5, 5, 5), (8, 8, 8))`
* A name to identify which keys in the Dask graph refer to this array, like
  `'x'`
* A NumPy dtype

### Example

```python
>>> import dask.array as da
>>> x = da.arange(0, 15, chunks=(5,))

>>> x.name
'arange-539766a'

>>> x.__dask_graph__()
<dask.highlevelgraph.HighLevelGraph at 0x7f9f6f686d68>

>>> dict(x.__dask_graph__())  # somewhat simplified
{('arange-539766a', 0): (np.arange, 0, 5),
 ('arange-539766a', 1): (np.arange, 5, 10),
 ('arange-539766a', 2): (np.arange, 10, 15)}

>>> x.chunks
((5, 5, 5),)

>>> x.dtype
dtype('int64')
```

## Keys of the Dask graph

By special convention, we refer to each block of the array with a tuple of the
form `(name, i, j, k)`, with `i, j, k` being the indices of the block
ranging from `0` to the number of blocks in that dimension.  The Dask graph
must hold key-value pairs referring to these keys.  Moreover, it likely also
holds other key-value pairs required to eventually compute the desired values
(usually organised in a [HighLevelGraph](high-level-graphs.md), but shown
in a flattened form here for illustration):

```python
{
 ('x', 0, 0): (add, 1, ('y', 0, 0)),
 ('x', 0, 1): (add, 1, ('y', 0, 1)),
 ...
 ('y', 0, 0): (getitem, dataset, (slice(0, 1000), slice(0, 1000))),
 ('y', 0, 1): (getitem, dataset, (slice(0, 1000), slice(1000, 2000)))
 ...
}
```

The name of an `Array` object can be found in the `name` attribute.  One
can get a nested list of keys with the `.__dask_keys__()` method.  Additionally,
one can flatten down this list with `dask.array.core.flatten()`. This is sometimes
useful when building new dictionaries.

## Chunks

We also store the size of each block along each axis.  This is composed of
a tuple of tuples such that the length of the outer tuple is equal to the
number of dimensions of the array, and the lengths of the inner tuples are
equal to the number of blocks along each dimension.  In the example illustrated
above this value is as follows:

```default
chunks = ((5, 5, 5, 5), (8, 8, 8))
```

Note that these numbers do not necessarily need to be regular.  We often create
regularly sized grids but blocks change shape after complex slicing.  Beware
that some operations do expect certain symmetries in the block-shapes.  For
example, matrix multiplication requires that blocks on each side have
anti-symmetric shapes.

Some ways in which `chunks` reflects properties of our array:

1. `len(x.chunks) == x.ndim`: the length of chunks is the number of dimensions
2. `tuple(map(sum, x.chunks)) == x.shape`: the sum of each internal chunk is the
   length of that dimension
3. The length of each internal chunk is the number of keys in that dimension.
   For instance, for `chunks == ((a, b), (d, e, f))` and name == `'x'`
   our array has tasks with the following keys:
   ```default
   ('x', 0, 0), ('x', 0, 1), ('x', 0, 2)
   ('x', 1, 0), ('x', 1, 1), ('x', 1, 2)
   ```

## Metadata

Many Array operations rely on knowing the dtype (int, float,..) and type (numpy, cupy,…).  To
keep track of this information, all Dask Array objects have a `_meta`
attribute which contains an empty Numpy object with the same dtypes.
For example:

```python
>>> np_array = np.arange(15).reshape(3, 5)
>>> da_array = da.from_array(np_array, npartitions=2)
>>> da_array._meta
Empty Array
Shape: (0, 0)
dtype: int64
array([], shape=(0, 0), dtype=int64)

>>> ddf._meta.dtype
dtype: int64
```

Internally, Dask Array does its best to propagate this information
through all operations, so most of the time a user shouldn’t have to worry
about this.

## Create an Array Object

In order to create an `da.Array` object we need a graph with these special
keys:

```default
layer = {('x', 0, 0): ...}
dsk = HighLevelGraph.from_collections('x', layer, dependencies=())
```

a name specifying which keys this array refers to:

```default
name = 'x'
```

and a chunks tuple:

```default
chunks = ((5, 5, 5, 5), (8, 8, 8))
```

Then, using these elements, one can construct an array:

```default
x = da.Array(dsk, name, chunks)
```

In short, `dask.array` operations update Dask graphs, update dtypes, and track chunk
shapes.

## Example - `eye` function

As an example, let’s build the `np.eye` function for `dask.array` to make the
identity matrix:

```python
def eye(n, blocksize):
    chunks = ((blocksize,) * (n // blocksize),
              (blocksize,) * (n // blocksize))

    name = 'eye' + next(tokens)  # unique identifier

    layer = {(name, i, j): (np.eye, blocksize)
                           if i == j else
                           (np.zeros, (blocksize, blocksize))
             for i in range(n // blocksize)
             for j in range(n // blocksize)}
    dsk = dask.highlevelgraph.HighLevelGraph.from_collections(name, layer, dependencies=())

    dtype = np.eye(0).dtype  # take dtype default from numpy

    return dask.array.Array(dsk, name, chunks, dtype)
```


# array-gufunc.html.md

# Generalized UFuncs

[NumPy](https://www.numpy.org) provides the concept of [generalized ufuncs](https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html). Generalized ufuncs are functions
that distinguish the various dimensions of passed arrays in the two classes loop dimensions
and core dimensions. To accomplish this, a [signature](https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html#details-of-signature) is specified for NumPy generalized ufuncs.

[Dask](https://dask.org/) integrates interoperability with NumPy’s generalized ufuncs
by adhering to respective [ufunc protocol](https://docs.scipy.org/doc/numpy/reference/arrays.classes.html#numpy.class.__array_ufunc__), and provides a wrapper to make a Python function a generalized ufunc.

## Usage

### NumPy Generalized UFuncs

#### NOTE
[NumPy](https://www.numpy.org) generalized ufuncs are currently (v1.14.3 and below) stored in
`np.linalg._umath_linalg` and might change in the future.

```python
import dask.array as da
import numpy as np

x = da.random.default_rng().normal(size=(3, 10, 10), chunks=(2, 10, 10))

w, v = np.linalg._umath_linalg.eig(x, output_dtypes=(float, float))
```

### Create Generalized UFuncs

It can be difficult to create your own GUFuncs without going into the CPython API.
However, the [Numba](https://numba.pydata.org) project does provide a
nice implementation with their `numba.guvectorize` decorator.  See [Numba’s
documentation](https://numba.pydata.org/numba-doc/dev/user/vectorize.html#the-guvectorize-decorator)
for more information.

### Wrap your own Python function

`gufunc` can be used to make a Python function behave like a generalized ufunc:

```python
x = da.random.default_rng().normal(size=(10, 5), chunks=(2, 5))

def foo(x):
    return np.mean(x, axis=-1)

gufoo = da.gufunc(foo, signature="(i)->()", output_dtypes=float, vectorize=True)

y = gufoo(x)
```

Instead of `gufunc`, also the `as_gufunc` decorator can be used for convenience:

```python
x = da.random.normal(size=(10, 5), chunks=(2, 5))

@da.as_gufunc(signature="(i)->()", output_dtypes=float, vectorize=True)
def gufoo(x):
    return np.mean(x, axis=-1)

y = gufoo(x)
```

## Disclaimer

This experimental generalized ufunc integration is not complete:

* `gufunc` does not create a true generalized ufunc to be used with other input arrays besides Dask.
  I.e., at the moment, `gufunc` casts all input arguments to `dask.array.Array`
* Inferring `output_dtypes` automatically is not implemented yet

## API

| [`apply_gufunc`](generated/dask.array.gufunc.apply_gufunc.md#dask.array.gufunc.apply_gufunc)(func, signature, \*args[, axes, ...])   | Apply a generalized ufunc or similar python function to arrays.   |
|--------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`as_gufunc`](generated/dask.array.gufunc.as_gufunc.md#dask.array.gufunc.as_gufunc)([signature])                                     | Decorator for `dask.array.gufunc`.                                |
| [`gufunc`](generated/dask.array.gufunc.gufunc.md#dask.array.gufunc.gufunc)(pyfunc, \*[, signature, vectorize, ...])                  | Binds pyfunc into `dask.array.apply_gufunc` when called.          |


# array-numpy-compatibility.html.md

# Compatibility with numpy functions

The following table describes the compatibilities between numpy and dask.array
functions.
Please be aware that some inconsistencies with the two versions may exist.

This table has been compiled manually and may not reflect the current Dask state.
Update contributions are welcome.

* A blank entry indicates that the function is not implemented in Dask.
* Direct implementations are direct calls to numpy functions.
* Element-wise implementations are derived from numpy but applied element-wise: the
  argument should be a dask array.
* Dask equivalent are Dask implementations, which may lack or add parameters with respect
  to the numpy function.

A more in-depth comparison in the framework of the [Array API](https://data-apis.org/array-api/latest/)
is available via the [Array API Comparison repository](https://github.com/data-apis/array-api-comparison).

| NumPy                                                                                                                                    | Dask                                                                                                                                                                                          | Implementation                 |
|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|
| [`numpy.absolute`](https://numpy.org/doc/stable/reference/generated/numpy.absolute.html#numpy.absolute)                                  | [`dask.array.absolute`](generated/dask.array.absolute.md#dask.array.absolute) or [`dask.array.abs`](generated/dask.array.abs.md#dask.array.abs)                                               | direct (ufunc)                 |
| [`numpy.add`](https://numpy.org/doc/stable/reference/generated/numpy.add.html#numpy.add)                                                 | [`dask.array.add`](generated/dask.array.add.md#dask.array.add)                                                                                                                                | direct (ufunc)                 |
| [`numpy.all`](https://numpy.org/doc/stable/reference/generated/numpy.all.html#numpy.all)                                                 | [`dask.array.all`](generated/dask.array.all.md#dask.array.all) <sup>[21](#id64)</sup>                                                                                                         | dask equivalent                |
| [`numpy.allclose`](https://numpy.org/doc/stable/reference/generated/numpy.allclose.html#numpy.allclose)                                  | [`dask.array.allclose`](generated/dask.array.allclose.md#dask.array.allclose)                                                                                                                 | dask equivalent                |
| [`numpy.amax`](https://numpy.org/doc/stable/reference/generated/numpy.amax.html#numpy.amax)                                              | [`dask.array.max`](generated/dask.array.max.md#dask.array.max) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                                  | dask equivalent                |
| [`numpy.amin`](https://numpy.org/doc/stable/reference/generated/numpy.amin.html#numpy.amin)                                              | [`dask.array.min`](generated/dask.array.min.md#dask.array.min) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                                  | dask equivalent                |
| [`numpy.angle`](https://numpy.org/doc/stable/reference/generated/numpy.angle.html#numpy.angle)                                           | [`dask.array.angle`](generated/dask.array.angle.md#dask.array.angle) <sup>[23](#id66)</sup>                                                                                                   | dask equivalent                |
| [`numpy.any`](https://numpy.org/doc/stable/reference/generated/numpy.any.html#numpy.any)                                                 | [`dask.array.any`](generated/dask.array.any.md#dask.array.any) <sup>[21](#id64)</sup>                                                                                                         | dask equivalent                |
| [`numpy.append`](https://numpy.org/doc/stable/reference/generated/numpy.append.html#numpy.append)                                        | [`dask.array.append`](generated/dask.array.append.md#dask.array.append)                                                                                                                       | dask equivalent                |
| [`numpy.apply_along_axis`](https://numpy.org/doc/stable/reference/generated/numpy.apply_along_axis.html#numpy.apply_along_axis)          | [`dask.array.apply_along_axis`](generated/dask.array.apply_along_axis.md#dask.array.apply_along_axis)                                                                                         | dask equivalent                |
| [`numpy.apply_over_axes`](https://numpy.org/doc/stable/reference/generated/numpy.apply_over_axes.html#numpy.apply_over_axes)             | [`dask.array.apply_over_axes`](generated/dask.array.apply_over_axes.md#dask.array.apply_over_axes)                                                                                            | dask equivalent                |
| [`numpy.arange`](https://numpy.org/doc/stable/reference/generated/numpy.arange.html#numpy.arange)                                        | [`dask.array.arange`](generated/dask.array.arange.md#dask.array.arange)                                                                                                                       | dask equivalent                |
| [`numpy.arccos`](https://numpy.org/doc/stable/reference/generated/numpy.arccos.html#numpy.arccos)                                        | [`dask.array.arccos`](generated/dask.array.arccos.md#dask.array.arccos)                                                                                                                       | direct (ufunc)                 |
| [`numpy.arccosh`](https://numpy.org/doc/stable/reference/generated/numpy.arccosh.html#numpy.arccosh)                                     | [`dask.array.arccosh`](generated/dask.array.arccosh.md#dask.array.arccosh)                                                                                                                    | direct (ufunc)                 |
| [`numpy.arcsin`](https://numpy.org/doc/stable/reference/generated/numpy.arcsin.html#numpy.arcsin)                                        | [`dask.array.arcsin`](generated/dask.array.arcsin.md#dask.array.arcsin)                                                                                                                       | direct (ufunc)                 |
| [`numpy.arcsinh`](https://numpy.org/doc/stable/reference/generated/numpy.arcsinh.html#numpy.arcsinh)                                     | [`dask.array.arcsinh`](generated/dask.array.arcsinh.md#dask.array.arcsinh)                                                                                                                    | direct (ufunc)                 |
| [`numpy.arctan`](https://numpy.org/doc/stable/reference/generated/numpy.arctan.html#numpy.arctan)                                        | [`dask.array.arctan`](generated/dask.array.arctan.md#dask.array.arctan)                                                                                                                       | direct (ufunc)                 |
| [`numpy.arctan2`](https://numpy.org/doc/stable/reference/generated/numpy.arctan2.html#numpy.arctan2)                                     | [`dask.array.arctan2`](generated/dask.array.arctan2.md#dask.array.arctan2)                                                                                                                    | direct (ufunc)                 |
| [`numpy.arctanh`](https://numpy.org/doc/stable/reference/generated/numpy.arctanh.html#numpy.arctanh)                                     | [`dask.array.arctanh`](generated/dask.array.arctanh.md#dask.array.arctanh)                                                                                                                    | direct (ufunc)                 |
| [`numpy.argmax`](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html#numpy.argmax)                                        | [`dask.array.argmax`](generated/dask.array.argmax.md#dask.array.argmax)                                                                                                                       | dask equivalent                |
| [`numpy.argmin`](https://numpy.org/doc/stable/reference/generated/numpy.argmin.html#numpy.argmin)                                        | [`dask.array.argmin`](generated/dask.array.argmin.md#dask.array.argmin)                                                                                                                       | dask equivalent                |
| [`numpy.argpartition`](https://numpy.org/doc/stable/reference/generated/numpy.argpartition.html#numpy.argpartition)                      | -                                                                                                                                                                                             |                                |
| [`numpy.argsort`](https://numpy.org/doc/stable/reference/generated/numpy.argsort.html#numpy.argsort)                                     | [`dask.array.argtopk`](generated/dask.array.argtopk.md#dask.array.argtopk) <sup>[25](#id68)</sup>                                                                                             |                                |
| [`numpy.argwhere`](https://numpy.org/doc/stable/reference/generated/numpy.argwhere.html#numpy.argwhere)                                  | [`dask.array.argwhere`](generated/dask.array.argwhere.md#dask.array.argwhere)                                                                                                                 | dask equivalent                |
| [`numpy.around`](https://numpy.org/doc/stable/reference/generated/numpy.around.html#numpy.around)                                        | [`dask.array.around`](generated/dask.array.around.md#dask.array.around) <sup>[23](#id66)</sup> <sup>[26](#id69)</sup> or [`dask.array.round`](generated/dask.array.round.md#dask.array.round) | dask equivalent                |
| [`numpy.array`](https://numpy.org/doc/stable/reference/generated/numpy.array.html#numpy.array)                                           | [`dask.array.array`](generated/dask.array.array.md#dask.array.array)                                                                                                                          | dask equivalent                |
| [`numpy.array2string`](https://numpy.org/doc/stable/reference/generated/numpy.array2string.html#numpy.array2string)                      | -                                                                                                                                                                                             |                                |
| [`numpy.array_equal`](https://numpy.org/doc/stable/reference/generated/numpy.array_equal.html#numpy.array_equal)                         | -                                                                                                                                                                                             |                                |
| [`numpy.array_equiv`](https://numpy.org/doc/stable/reference/generated/numpy.array_equiv.html#numpy.array_equiv)                         | -                                                                                                                                                                                             |                                |
| [`numpy.array_repr`](https://numpy.org/doc/stable/reference/generated/numpy.array_repr.html#numpy.array_repr)                            | -                                                                                                                                                                                             |                                |
| [`numpy.array_split`](https://numpy.org/doc/stable/reference/generated/numpy.array_split.html#numpy.array_split)                         | -                                                                                                                                                                                             |                                |
| [`numpy.array_str`](https://numpy.org/doc/stable/reference/generated/numpy.array_str.html#numpy.array_str)                               | -                                                                                                                                                                                             |                                |
| [`numpy.asanyarray`](https://numpy.org/doc/stable/reference/generated/numpy.asanyarray.html#numpy.asanyarray)                            | [`dask.array.asanyarray`](generated/dask.array.asanyarray.md#dask.array.asanyarray)                                                                                                           | dask equivalent                |
| [`numpy.asarray`](https://numpy.org/doc/stable/reference/generated/numpy.asarray.html#numpy.asarray)                                     | [`dask.array.asarray`](generated/dask.array.asarray.md#dask.array.asarray)                                                                                                                    | dask equivalent                |
| [`numpy.asarray_chkfinite`](https://numpy.org/doc/stable/reference/generated/numpy.asarray_chkfinite.html#numpy.asarray_chkfinite)       | -                                                                                                                                                                                             |                                |
| [`numpy.ascontiguousarray`](https://numpy.org/doc/stable/reference/generated/numpy.ascontiguousarray.html#numpy.ascontiguousarray)       | -                                                                                                                                                                                             |                                |
| `numpy.asfarray`                                                                                                                         | -                                                                                                                                                                                             |                                |
| [`numpy.asfortranarray`](https://numpy.org/doc/stable/reference/generated/numpy.asfortranarray.html#numpy.asfortranarray)                | -                                                                                                                                                                                             |                                |
| [`numpy.asmatrix`](https://numpy.org/doc/stable/reference/generated/numpy.asmatrix.html#numpy.asmatrix)                                  | - <sup>[27](#id70)</sup>                                                                                                                                                                      |                                |
| [`numpy.atleast_1d`](https://numpy.org/doc/stable/reference/generated/numpy.atleast_1d.html#numpy.atleast_1d)                            | [`dask.array.atleast_1d`](generated/dask.array.atleast_1d.md#dask.array.atleast_1d)                                                                                                           | dask equivalent                |
| [`numpy.atleast_2d`](https://numpy.org/doc/stable/reference/generated/numpy.atleast_2d.html#numpy.atleast_2d)                            | [`dask.array.atleast_2d`](generated/dask.array.atleast_2d.md#dask.array.atleast_2d)                                                                                                           | dask equivalent                |
| [`numpy.atleast_3d`](https://numpy.org/doc/stable/reference/generated/numpy.atleast_3d.html#numpy.atleast_3d)                            | [`dask.array.atleast_3d`](generated/dask.array.atleast_3d.md#dask.array.atleast_3d)                                                                                                           | dask equivalent                |
| [`numpy.average`](https://numpy.org/doc/stable/reference/generated/numpy.average.html#numpy.average)                                     | [`dask.array.average`](generated/dask.array.average.md#dask.array.average)                                                                                                                    | dask equivalent                |
| [`numpy.bartlett`](https://numpy.org/doc/stable/reference/generated/numpy.bartlett.html#numpy.bartlett)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.bincount`](https://numpy.org/doc/stable/reference/generated/numpy.bincount.html#numpy.bincount)                                  | [`dask.array.bincount`](generated/dask.array.bincount.md#dask.array.bincount)                                                                                                                 | dask equivalent                |
| [`numpy.bitwise_and`](https://numpy.org/doc/stable/reference/generated/numpy.bitwise_and.html#numpy.bitwise_and)                         | [`dask.array.bitwise_and`](generated/dask.array.bitwise_and.md#dask.array.bitwise_and)                                                                                                        | direct (ufunc)                 |
| [`numpy.bitwise_or`](https://numpy.org/doc/stable/reference/generated/numpy.bitwise_or.html#numpy.bitwise_or)                            | [`dask.array.bitwise_or`](generated/dask.array.bitwise_or.md#dask.array.bitwise_or)                                                                                                           | direct (ufunc)                 |
| [`numpy.bitwise_xor`](https://numpy.org/doc/stable/reference/generated/numpy.bitwise_xor.html#numpy.bitwise_xor)                         | [`dask.array.bitwise_xor`](generated/dask.array.bitwise_xor.md#dask.array.bitwise_xor)                                                                                                        | direct (ufunc)                 |
| [`numpy.blackman`](https://numpy.org/doc/stable/reference/generated/numpy.blackman.html#numpy.blackman)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.block`](https://numpy.org/doc/stable/reference/generated/numpy.block.html#numpy.block)                                           | [`dask.array.block`](generated/dask.array.block.md#dask.array.block)                                                                                                                          | dask equivalent                |
| [`numpy.bmat`](https://numpy.org/doc/stable/reference/generated/numpy.bmat.html#numpy.bmat)                                              | - <sup>[27](#id70)</sup>                                                                                                                                                                      |                                |
| [`numpy.broadcast`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast.html#numpy.broadcast)                               | -                                                                                                                                                                                             |                                |
| [`numpy.broadcast_arrays`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast_arrays.html#numpy.broadcast_arrays)          | [`dask.array.broadcast_arrays`](generated/dask.array.broadcast_arrays.md#dask.array.broadcast_arrays)                                                                                         | dask equivalent                |
| [`numpy.broadcast_shapes`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast_shapes.html#numpy.broadcast_shapes)          | -                                                                                                                                                                                             |                                |
| [`numpy.broadcast_to`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast_to.html#numpy.broadcast_to)                      | [`dask.array.broadcast_to`](generated/dask.array.broadcast_to.md#dask.array.broadcast_to)                                                                                                     | dask equivalent                |
| `numpy.byte_bounds`                                                                                                                      | -                                                                                                                                                                                             |                                |
| [`numpy.c_`](https://numpy.org/doc/stable/reference/generated/numpy.c_.html#numpy.c_)                                                    | -                                                                                                                                                                                             |                                |
| [`numpy.can_cast`](https://numpy.org/doc/stable/reference/generated/numpy.can_cast.html#numpy.can_cast)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.cbrt`](https://numpy.org/doc/stable/reference/generated/numpy.cbrt.html#numpy.cbrt)                                              | [`dask.array.cbrt`](generated/dask.array.cbrt.md#dask.array.cbrt)                                                                                                                             | direct (ufunc)                 |
| [`numpy.ceil`](https://numpy.org/doc/stable/reference/generated/numpy.ceil.html#numpy.ceil)                                              | [`dask.array.ceil`](generated/dask.array.ceil.md#dask.array.ceil)                                                                                                                             | direct (ufunc)                 |
| [`numpy.choose`](https://numpy.org/doc/stable/reference/generated/numpy.choose.html#numpy.choose)                                        | [`dask.array.choose`](generated/dask.array.choose.md#dask.array.choose) <sup>[28](#id71)</sup>                                                                                                | dask equivalent                |
| [`numpy.clip`](https://numpy.org/doc/stable/reference/generated/numpy.clip.html#numpy.clip)                                              | [`dask.array.clip`](generated/dask.array.clip.md#dask.array.clip) <sup>[23](#id66)</sup> <sup>[26](#id69)</sup>                                                                               | direct (non-ufunc elementwise) |
| [`numpy.column_stack`](https://numpy.org/doc/stable/reference/generated/numpy.column_stack.html#numpy.column_stack)                      | -                                                                                                                                                                                             |                                |
| [`numpy.common_type`](https://numpy.org/doc/stable/reference/generated/numpy.common_type.html#numpy.common_type)                         | -                                                                                                                                                                                             |                                |
| [`numpy.compress`](https://numpy.org/doc/stable/reference/generated/numpy.compress.html#numpy.compress)                                  | [`dask.array.compress`](generated/dask.array.compress.md#dask.array.compress) <sup>[26](#id69)</sup>                                                                                          | dask equivalent                |
| [`numpy.concatenate`](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html#numpy.concatenate)                         | [`dask.array.concatenate`](generated/dask.array.concatenate.md#dask.array.concatenate)                                                                                                        | dask equivalent                |
| [`numpy.conj`](https://numpy.org/doc/stable/reference/generated/numpy.conj.html#numpy.conj)                                              | [`dask.array.conj`](generated/dask.array.conj.md#dask.array.conj)                                                                                                                             | direct (ufunc)                 |
| [`numpy.conjugate`](https://numpy.org/doc/stable/reference/generated/numpy.conjugate.html#numpy.conjugate)                               | [`dask.array.conj`](generated/dask.array.conj.md#dask.array.conj)                                                                                                                             | direct (ufunc)                 |
| [`numpy.convolve`](https://numpy.org/doc/stable/reference/generated/numpy.convolve.html#numpy.convolve)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.copy`](https://numpy.org/doc/stable/reference/generated/numpy.copy.html#numpy.copy)                                              | -                                                                                                                                                                                             |                                |
| [`numpy.copysign`](https://numpy.org/doc/stable/reference/generated/numpy.copysign.html#numpy.copysign)                                  | [`dask.array.copysign`](generated/dask.array.copysign.md#dask.array.copysign)                                                                                                                 | direct (ufunc)                 |
| [`numpy.copyto`](https://numpy.org/doc/stable/reference/generated/numpy.copyto.html#numpy.copyto)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.corrcoef`](https://numpy.org/doc/stable/reference/generated/numpy.corrcoef.html#numpy.corrcoef)                                  | [`dask.array.corrcoef`](generated/dask.array.corrcoef.md#dask.array.corrcoef)                                                                                                                 | dask equivalent                |
| [`numpy.correlate`](https://numpy.org/doc/stable/reference/generated/numpy.correlate.html#numpy.correlate)                               | -                                                                                                                                                                                             |                                |
| [`numpy.cos`](https://numpy.org/doc/stable/reference/generated/numpy.cos.html#numpy.cos)                                                 | [`dask.array.cos`](generated/dask.array.cos.md#dask.array.cos)                                                                                                                                | direct (ufunc)                 |
| [`numpy.cosh`](https://numpy.org/doc/stable/reference/generated/numpy.cosh.html#numpy.cosh)                                              | [`dask.array.cosh`](generated/dask.array.cosh.md#dask.array.cosh)                                                                                                                             | direct (ufunc)                 |
| [`numpy.count_nonzero`](https://numpy.org/doc/stable/reference/generated/numpy.count_nonzero.html#numpy.count_nonzero)                   | [`dask.array.count_nonzero`](generated/dask.array.count_nonzero.md#dask.array.count_nonzero) <sup>[29](#id72)</sup>                                                                           | dask equivalent                |
| [`numpy.cov`](https://numpy.org/doc/stable/reference/generated/numpy.cov.html#numpy.cov)                                                 | [`dask.array.cov`](generated/dask.array.cov.md#dask.array.cov) <sup>[30](#id73)</sup>                                                                                                         | dask equivalent                |
| [`numpy.cross`](https://numpy.org/doc/stable/reference/generated/numpy.cross.html#numpy.cross)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.cumprod`](https://numpy.org/doc/stable/reference/generated/numpy.cumprod.html#numpy.cumprod)                                     | [`dask.array.cumprod`](generated/dask.array.cumprod.md#dask.array.cumprod) <sup>[23](#id66)</sup> <sup>[38](#id81)</sup>                                                                      | dask equivalent                |
| [`numpy.cumsum`](https://numpy.org/doc/stable/reference/generated/numpy.cumsum.html#numpy.cumsum)                                        | [`dask.array.cumsum`](generated/dask.array.cumsum.md#dask.array.cumsum) <sup>[23](#id66)</sup> <sup>[38](#id81)</sup>                                                                         | dask equivalent                |
| [`numpy.datetime_as_string`](https://numpy.org/doc/stable/reference/generated/numpy.datetime_as_string.html#numpy.datetime_as_string)    | -                                                                                                                                                                                             |                                |
| [`numpy.deg2rad`](https://numpy.org/doc/stable/reference/generated/numpy.deg2rad.html#numpy.deg2rad)                                     | [`dask.array.deg2rad`](generated/dask.array.deg2rad.md#dask.array.deg2rad)                                                                                                                    | direct (ufunc)                 |
| [`numpy.degrees`](https://numpy.org/doc/stable/reference/generated/numpy.degrees.html#numpy.degrees)                                     | [`dask.array.degrees`](generated/dask.array.degrees.md#dask.array.degrees)                                                                                                                    | direct (ufunc)                 |
| [`numpy.delete`](https://numpy.org/doc/stable/reference/generated/numpy.delete.html#numpy.delete)                                        | [`dask.array.delete`](generated/dask.array.delete.md#dask.array.delete)                                                                                                                       | dask equivalent                |
| [`numpy.diag`](https://numpy.org/doc/stable/reference/generated/numpy.diag.html#numpy.diag)                                              | [`dask.array.diag`](generated/dask.array.diag.md#dask.array.diag)                                                                                                                             | dask equivalent                |
| [`numpy.diag_indices`](https://numpy.org/doc/stable/reference/generated/numpy.diag_indices.html#numpy.diag_indices)                      | -                                                                                                                                                                                             |                                |
| [`numpy.diag_indices_from`](https://numpy.org/doc/stable/reference/generated/numpy.diag_indices_from.html#numpy.diag_indices_from)       | -                                                                                                                                                                                             |                                |
| [`numpy.diagflat`](https://numpy.org/doc/stable/reference/generated/numpy.diagflat.html#numpy.diagflat)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.diagonal`](https://numpy.org/doc/stable/reference/generated/numpy.diagonal.html#numpy.diagonal)                                  | [`dask.array.diagonal`](generated/dask.array.diagonal.md#dask.array.diagonal)                                                                                                                 | dask equivalent                |
| [`numpy.diff`](https://numpy.org/doc/stable/reference/generated/numpy.diff.html#numpy.diff)                                              | [`dask.array.diff`](generated/dask.array.diff.md#dask.array.diff)                                                                                                                             | dask equivalent                |
| [`numpy.digitize`](https://numpy.org/doc/stable/reference/generated/numpy.digitize.html#numpy.digitize)                                  | [`dask.array.digitize`](generated/dask.array.digitize.md#dask.array.digitize) <sup>[23](#id66)</sup>                                                                                          | dask equivalent                |
| [`numpy.divide`](https://numpy.org/doc/stable/reference/generated/numpy.divide.html#numpy.divide)                                        | [`dask.array.divide`](generated/dask.array.divide.md#dask.array.divide)                                                                                                                       | direct (ufunc)                 |
| [`numpy.divmod`](https://numpy.org/doc/stable/reference/generated/numpy.divmod.html#numpy.divmod)                                        | [`dask.array.divmod`](generated/dask.array.divmod.md#dask.array.divmod)                                                                                                                       | dask equivalent                |
| [`numpy.dot`](https://numpy.org/doc/stable/reference/generated/numpy.dot.html#numpy.dot)                                                 | [`dask.array.dot`](generated/dask.array.dot.md#dask.array.dot) <sup>[26](#id69)</sup>                                                                                                         | dask equivalent                |
| [`numpy.dsplit`](https://numpy.org/doc/stable/reference/generated/numpy.dsplit.html#numpy.dsplit)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.dstack`](https://numpy.org/doc/stable/reference/generated/numpy.dstack.html#numpy.dstack)                                        | [`dask.array.dstack`](generated/dask.array.dstack.md#dask.array.dstack)                                                                                                                       | dask equivalent                |
| [`numpy.ediff1d`](https://numpy.org/doc/stable/reference/generated/numpy.ediff1d.html#numpy.ediff1d)                                     | [`dask.array.ediff1d`](generated/dask.array.ediff1d.md#dask.array.ediff1d)                                                                                                                    | dask equivalent                |
| [`numpy.einsum`](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html#numpy.einsum)                                        | [`dask.array.einsum`](generated/dask.array.einsum.md#dask.array.einsum) <sup>[26](#id69)</sup>                                                                                                | dask equivalent                |
| [`numpy.einsum_path`](https://numpy.org/doc/stable/reference/generated/numpy.einsum_path.html#numpy.einsum_path)                         | -                                                                                                                                                                                             |                                |
| [`numpy.empty`](https://numpy.org/doc/stable/reference/generated/numpy.empty.html#numpy.empty)                                           | [`dask.array.empty`](generated/dask.array.empty.md#dask.array.empty)                                                                                                                          | dask equivalent                |
| [`numpy.empty_like`](https://numpy.org/doc/stable/reference/generated/numpy.empty_like.html#numpy.empty_like)                            | [`dask.array.empty_like`](generated/dask.array.empty_like.md#dask.array.empty_like)                                                                                                           | dask equivalent                |
| [`numpy.equal`](https://numpy.org/doc/stable/reference/generated/numpy.equal.html#numpy.equal)                                           | [`dask.array.equal`](generated/dask.array.equal.md#dask.array.equal)                                                                                                                          | direct (ufunc)                 |
| [`numpy.exp`](https://numpy.org/doc/stable/reference/generated/numpy.exp.html#numpy.exp)                                                 | [`dask.array.exp`](generated/dask.array.exp.md#dask.array.exp)                                                                                                                                | direct (ufunc)                 |
| [`numpy.exp2`](https://numpy.org/doc/stable/reference/generated/numpy.exp2.html#numpy.exp2)                                              | [`dask.array.exp2`](generated/dask.array.exp2.md#dask.array.exp2)                                                                                                                             | direct (ufunc)                 |
| [`numpy.expand_dims`](https://numpy.org/doc/stable/reference/generated/numpy.expand_dims.html#numpy.expand_dims)                         | [`dask.array.expand_dims`](generated/dask.array.expand_dims.md#dask.array.expand_dims)                                                                                                        | dask equivalent                |
| [`numpy.expm1`](https://numpy.org/doc/stable/reference/generated/numpy.expm1.html#numpy.expm1)                                           | [`dask.array.expm1`](generated/dask.array.expm1.md#dask.array.expm1)                                                                                                                          | direct (ufunc)                 |
| [`numpy.extract`](https://numpy.org/doc/stable/reference/generated/numpy.extract.html#numpy.extract)                                     | [`dask.array.extract`](generated/dask.array.extract.md#dask.array.extract)                                                                                                                    | dask equivalent                |
| [`numpy.eye`](https://numpy.org/doc/stable/reference/generated/numpy.eye.html#numpy.eye)                                                 | [`dask.array.eye`](generated/dask.array.eye.md#dask.array.eye)                                                                                                                                | dask equivalent                |
| [`numpy.fabs`](https://numpy.org/doc/stable/reference/generated/numpy.fabs.html#numpy.fabs)                                              | [`dask.array.fabs`](generated/dask.array.fabs.md#dask.array.fabs)                                                                                                                             | direct (ufunc)                 |
| [`numpy.fill_diagonal`](https://numpy.org/doc/stable/reference/generated/numpy.fill_diagonal.html#numpy.fill_diagonal)                   | -                                                                                                                                                                                             |                                |
| [`numpy.fix`](https://numpy.org/doc/stable/reference/generated/numpy.fix.html#numpy.fix)                                                 | [`dask.array.fix`](generated/dask.array.fix.md#dask.array.fix)                                                                                                                                | direct (non-ufunc elementwise) |
| [`numpy.flatnonzero`](https://numpy.org/doc/stable/reference/generated/numpy.flatnonzero.html#numpy.flatnonzero)                         | [`dask.array.flatnonzero`](generated/dask.array.flatnonzero.md#dask.array.flatnonzero)                                                                                                        | dask equivalent                |
| [`numpy.flip`](https://numpy.org/doc/stable/reference/generated/numpy.flip.html#numpy.flip)                                              | [`dask.array.flip`](generated/dask.array.flip.md#dask.array.flip)                                                                                                                             | dask equivalent                |
| [`numpy.fliplr`](https://numpy.org/doc/stable/reference/generated/numpy.fliplr.html#numpy.fliplr)                                        | [`dask.array.fliplr`](generated/dask.array.fliplr.md#dask.array.fliplr)                                                                                                                       | dask equivalent                |
| [`numpy.flipud`](https://numpy.org/doc/stable/reference/generated/numpy.flipud.html#numpy.flipud)                                        | [`dask.array.flipud`](generated/dask.array.flipud.md#dask.array.flipud)                                                                                                                       | dask equivalent                |
| [`numpy.float_power`](https://numpy.org/doc/stable/reference/generated/numpy.float_power.html#numpy.float_power)                         | [`dask.array.float_power`](generated/dask.array.float_power.md#dask.array.float_power)                                                                                                        | direct (ufunc)                 |
| [`numpy.floor`](https://numpy.org/doc/stable/reference/generated/numpy.floor.html#numpy.floor)                                           | [`dask.array.floor`](generated/dask.array.floor.md#dask.array.floor)                                                                                                                          | direct (ufunc)                 |
| [`numpy.floor_divide`](https://numpy.org/doc/stable/reference/generated/numpy.floor_divide.html#numpy.floor_divide)                      | [`dask.array.floor_divide`](generated/dask.array.floor_divide.md#dask.array.floor_divide)                                                                                                     | direct (ufunc)                 |
| [`numpy.fmax`](https://numpy.org/doc/stable/reference/generated/numpy.fmax.html#numpy.fmax)                                              | [`dask.array.fmax`](generated/dask.array.fmax.md#dask.array.fmax)                                                                                                                             | direct (ufunc)                 |
| [`numpy.fmin`](https://numpy.org/doc/stable/reference/generated/numpy.fmin.html#numpy.fmin)                                              | [`dask.array.fmin`](generated/dask.array.fmin.md#dask.array.fmin)                                                                                                                             | direct (ufunc)                 |
| [`numpy.fmod`](https://numpy.org/doc/stable/reference/generated/numpy.fmod.html#numpy.fmod)                                              | [`dask.array.fmod`](generated/dask.array.fmod.md#dask.array.fmod)                                                                                                                             | direct (ufunc)                 |
| [`numpy.frexp`](https://numpy.org/doc/stable/reference/generated/numpy.frexp.html#numpy.frexp)                                           | [`dask.array.frexp`](generated/dask.array.frexp.md#dask.array.frexp)                                                                                                                          | dask equivalent                |
| [`numpy.from_dlpack`](https://numpy.org/doc/stable/reference/generated/numpy.from_dlpack.html#numpy.from_dlpack)                         | -                                                                                                                                                                                             |                                |
| [`numpy.frombuffer`](https://numpy.org/doc/stable/reference/generated/numpy.frombuffer.html#numpy.frombuffer)                            | -                                                                                                                                                                                             |                                |
| [`numpy.fromfile`](https://numpy.org/doc/stable/reference/generated/numpy.fromfile.html#numpy.fromfile)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.fromfunction`](https://numpy.org/doc/stable/reference/generated/numpy.fromfunction.html#numpy.fromfunction)                      | [`dask.array.fromfunction`](generated/dask.array.fromfunction.md#dask.array.fromfunction) <sup>[31](#id74)</sup>                                                                              | dask equivalent                |
| [`numpy.fromiter`](https://numpy.org/doc/stable/reference/generated/numpy.fromiter.html#numpy.fromiter)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.frompyfunc`](https://numpy.org/doc/stable/reference/generated/numpy.frompyfunc.html#numpy.frompyfunc)                            | [`dask.array.frompyfunc`](generated/dask.array.frompyfunc.md#dask.array.frompyfunc) <sup>[32](#id75)</sup>                                                                                    | dask equivalent                |
| [`numpy.fromregex`](https://numpy.org/doc/stable/reference/generated/numpy.fromregex.html#numpy.fromregex)                               | -                                                                                                                                                                                             |                                |
| [`numpy.fromstring`](https://numpy.org/doc/stable/reference/generated/numpy.fromstring.html#numpy.fromstring)                            | -                                                                                                                                                                                             |                                |
| [`numpy.full`](https://numpy.org/doc/stable/reference/generated/numpy.full.html#numpy.full)                                              | [`dask.array.full`](generated/dask.array.full.md#dask.array.full)                                                                                                                             | dask equivalent                |
| [`numpy.full_like`](https://numpy.org/doc/stable/reference/generated/numpy.full_like.html#numpy.full_like)                               | [`dask.array.full_like`](generated/dask.array.full_like.md#dask.array.full_like)                                                                                                              | dask equivalent                |
| [`numpy.gcd`](https://numpy.org/doc/stable/reference/generated/numpy.gcd.html#numpy.gcd)                                                 | -                                                                                                                                                                                             |                                |
| [`numpy.genfromtxt`](https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt)                            | -                                                                                                                                                                                             |                                |
| [`numpy.geomspace`](https://numpy.org/doc/stable/reference/generated/numpy.geomspace.html#numpy.geomspace)                               | -                                                                                                                                                                                             |                                |
| [`numpy.gradient`](https://numpy.org/doc/stable/reference/generated/numpy.gradient.html#numpy.gradient)                                  | [`dask.array.gradient`](generated/dask.array.gradient.md#dask.array.gradient) <sup>[33](#id76)</sup>                                                                                          | dask equivalent                |
| [`numpy.greater`](https://numpy.org/doc/stable/reference/generated/numpy.greater.html#numpy.greater)                                     | [`dask.array.greater`](generated/dask.array.greater.md#dask.array.greater)                                                                                                                    | direct (ufunc)                 |
| [`numpy.greater_equal`](https://numpy.org/doc/stable/reference/generated/numpy.greater_equal.html#numpy.greater_equal)                   | [`dask.array.greater_equal`](generated/dask.array.greater_equal.md#dask.array.greater_equal)                                                                                                  | direct (ufunc)                 |
| [`numpy.hamming`](https://numpy.org/doc/stable/reference/generated/numpy.hamming.html#numpy.hamming)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.hanning`](https://numpy.org/doc/stable/reference/generated/numpy.hanning.html#numpy.hanning)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.heaviside`](https://numpy.org/doc/stable/reference/generated/numpy.heaviside.html#numpy.heaviside)                               | -                                                                                                                                                                                             |                                |
| [`numpy.histogram`](https://numpy.org/doc/stable/reference/generated/numpy.histogram.html#numpy.histogram)                               | [`dask.array.histogram`](generated/dask.array.histogram.md#dask.array.histogram)                                                                                                              | dask equivalent                |
| [`numpy.histogram2d`](https://numpy.org/doc/stable/reference/generated/numpy.histogram2d.html#numpy.histogram2d)                         | [`dask.array.histogram2d`](generated/dask.array.histogram2d.md#dask.array.histogram2d)                                                                                                        | dask equivalent                |
| [`numpy.histogram_bin_edges`](https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html#numpy.histogram_bin_edges) | -                                                                                                                                                                                             |                                |
| [`numpy.histogramdd`](https://numpy.org/doc/stable/reference/generated/numpy.histogramdd.html#numpy.histogramdd)                         | [`dask.array.histogramdd`](generated/dask.array.histogramdd.md#dask.array.histogramdd) <sup>[34](#id77)</sup>                                                                                 | dask equivalent                |
| [`numpy.hsplit`](https://numpy.org/doc/stable/reference/generated/numpy.hsplit.html#numpy.hsplit)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.hstack`](https://numpy.org/doc/stable/reference/generated/numpy.hstack.html#numpy.hstack)                                        | [`dask.array.hstack`](generated/dask.array.hstack.md#dask.array.hstack)                                                                                                                       | dask equivalent                |
| [`numpy.hypot`](https://numpy.org/doc/stable/reference/generated/numpy.hypot.html#numpy.hypot)                                           | [`dask.array.hypot`](generated/dask.array.hypot.md#dask.array.hypot)                                                                                                                          | direct (ufunc)                 |
| [`numpy.i0`](https://numpy.org/doc/stable/reference/generated/numpy.i0.html#numpy.i0)                                                    | [`dask.array.i0`](generated/dask.array.i0.md#dask.array.i0)                                                                                                                                   | direct (non-ufunc elementwise) |
| [`numpy.identity`](https://numpy.org/doc/stable/reference/generated/numpy.identity.html#numpy.identity)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.imag`](https://numpy.org/doc/stable/reference/generated/numpy.imag.html#numpy.imag)                                              | [`dask.array.imag`](generated/dask.array.imag.md#dask.array.imag)                                                                                                                             | direct (non-ufunc elementwise) |
| `numpy.in1d`                                                                                                                             | -                                                                                                                                                                                             |                                |
| [`numpy.indices`](https://numpy.org/doc/stable/reference/generated/numpy.indices.html#numpy.indices)                                     | [`dask.array.indices`](generated/dask.array.indices.md#dask.array.indices)                                                                                                                    | dask equivalent                |
| [`numpy.inner`](https://numpy.org/doc/stable/reference/generated/numpy.inner.html#numpy.inner)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.insert`](https://numpy.org/doc/stable/reference/generated/numpy.insert.html#numpy.insert)                                        | [`dask.array.insert`](generated/dask.array.insert.md#dask.array.insert) <sup>[35](#id78)</sup>                                                                                                | dask equivalent                |
| [`numpy.interp`](https://numpy.org/doc/stable/reference/generated/numpy.interp.html#numpy.interp)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.intersect1d`](https://numpy.org/doc/stable/reference/generated/numpy.intersect1d.html#numpy.intersect1d)                         | -                                                                                                                                                                                             |                                |
| [`numpy.invert`](https://numpy.org/doc/stable/reference/generated/numpy.invert.html#numpy.invert)                                        | [`dask.array.invert`](generated/dask.array.invert.md#dask.array.invert) or [`dask.array.bitwise_not`](generated/dask.array.bitwise_not.md#dask.array.bitwise_not)                             | direct (ufunc)                 |
| [`numpy.is_busday`](https://numpy.org/doc/stable/reference/generated/numpy.is_busday.html#numpy.is_busday)                               | -                                                                                                                                                                                             |                                |
| [`numpy.isclose`](https://numpy.org/doc/stable/reference/generated/numpy.isclose.html#numpy.isclose)                                     | [`dask.array.isclose`](generated/dask.array.isclose.md#dask.array.isclose)                                                                                                                    | dask equivalent                |
| [`numpy.iscomplex`](https://numpy.org/doc/stable/reference/generated/numpy.iscomplex.html#numpy.iscomplex)                               | [`dask.array.iscomplex`](generated/dask.array.iscomplex.md#dask.array.iscomplex)                                                                                                              | direct (non-ufunc elementwise) |
| [`numpy.iscomplexobj`](https://numpy.org/doc/stable/reference/generated/numpy.iscomplexobj.html#numpy.iscomplexobj)                      | -                                                                                                                                                                                             |                                |
| [`numpy.isfinite`](https://numpy.org/doc/stable/reference/generated/numpy.isfinite.html#numpy.isfinite)                                  | [`dask.array.isfinite`](generated/dask.array.isfinite.md#dask.array.isfinite)                                                                                                                 | direct (ufunc)                 |
| [`numpy.isfortran`](https://numpy.org/doc/stable/reference/generated/numpy.isfortran.html#numpy.isfortran)                               | -                                                                                                                                                                                             |                                |
| [`numpy.isin`](https://numpy.org/doc/stable/reference/generated/numpy.isin.html#numpy.isin)                                              | [`dask.array.isin`](generated/dask.array.isin.md#dask.array.isin)                                                                                                                             | dask equivalent                |
| [`numpy.isinf`](https://numpy.org/doc/stable/reference/generated/numpy.isinf.html#numpy.isinf)                                           | [`dask.array.isinf`](generated/dask.array.isinf.md#dask.array.isinf)                                                                                                                          | direct (ufunc)                 |
| [`numpy.isnan`](https://numpy.org/doc/stable/reference/generated/numpy.isnan.html#numpy.isnan)                                           | [`dask.array.isnan`](generated/dask.array.isnan.md#dask.array.isnan)                                                                                                                          | direct (ufunc)                 |
| [`numpy.isnat`](https://numpy.org/doc/stable/reference/generated/numpy.isnat.html#numpy.isnat)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.isneginf`](https://numpy.org/doc/stable/reference/generated/numpy.isneginf.html#numpy.isneginf)                                  | [`dask.array.isneginf`](generated/dask.array.isneginf.md#dask.array.isneginf)                                                                                                                 | direct (ufunc)                 |
| [`numpy.isposinf`](https://numpy.org/doc/stable/reference/generated/numpy.isposinf.html#numpy.isposinf)                                  | [`dask.array.isposinf`](generated/dask.array.isposinf.md#dask.array.isposinf)                                                                                                                 | direct (ufunc)                 |
| [`numpy.isreal`](https://numpy.org/doc/stable/reference/generated/numpy.isreal.html#numpy.isreal)                                        | [`dask.array.isreal`](generated/dask.array.isreal.md#dask.array.isreal)                                                                                                                       | direct (non-ufunc elementwise) |
| [`numpy.ix_`](https://numpy.org/doc/stable/reference/generated/numpy.ix_.html#numpy.ix_)                                                 | -                                                                                                                                                                                             |                                |
| [`numpy.kaiser`](https://numpy.org/doc/stable/reference/generated/numpy.kaiser.html#numpy.kaiser)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.kron`](https://numpy.org/doc/stable/reference/generated/numpy.kron.html#numpy.kron)                                              | -                                                                                                                                                                                             |                                |
| [`numpy.lcm`](https://numpy.org/doc/stable/reference/generated/numpy.lcm.html#numpy.lcm)                                                 | -                                                                                                                                                                                             |                                |
| [`numpy.ldexp`](https://numpy.org/doc/stable/reference/generated/numpy.ldexp.html#numpy.ldexp)                                           | [`dask.array.ldexp`](generated/dask.array.ldexp.md#dask.array.ldexp)                                                                                                                          | direct (ufunc)                 |
| [`numpy.left_shift`](https://numpy.org/doc/stable/reference/generated/numpy.left_shift.html#numpy.left_shift)                            | [`dask.array.left_shift`](generated/dask.array.left_shift.md#dask.array.left_shift)                                                                                                           | direct (ufunc)                 |
| [`numpy.less`](https://numpy.org/doc/stable/reference/generated/numpy.less.html#numpy.less)                                              | [`dask.array.less`](generated/dask.array.less.md#dask.array.less)                                                                                                                             | direct (ufunc)                 |
| [`numpy.less_equal`](https://numpy.org/doc/stable/reference/generated/numpy.less_equal.html#numpy.less_equal)                            | [`dask.array.less_equal`](generated/dask.array.less_equal.md#dask.array.less_equal)                                                                                                           | direct (ufunc)                 |
| [`numpy.lexsort`](https://numpy.org/doc/stable/reference/generated/numpy.lexsort.html#numpy.lexsort)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.linspace`](https://numpy.org/doc/stable/reference/generated/numpy.linspace.html#numpy.linspace)                                  | [`dask.array.linspace`](generated/dask.array.linspace.md#dask.array.linspace)                                                                                                                 | dask equivalent                |
| [`numpy.load`](https://numpy.org/doc/stable/reference/generated/numpy.load.html#numpy.load)                                              | -                                                                                                                                                                                             |                                |
| [`numpy.loadtxt`](https://numpy.org/doc/stable/reference/generated/numpy.loadtxt.html#numpy.loadtxt)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.log`](https://numpy.org/doc/stable/reference/generated/numpy.log.html#numpy.log)                                                 | [`dask.array.log`](generated/dask.array.log.md#dask.array.log)                                                                                                                                | direct (ufunc)                 |
| [`numpy.log10`](https://numpy.org/doc/stable/reference/generated/numpy.log10.html#numpy.log10)                                           | [`dask.array.log10`](generated/dask.array.log10.md#dask.array.log10)                                                                                                                          | direct (ufunc)                 |
| [`numpy.log1p`](https://numpy.org/doc/stable/reference/generated/numpy.log1p.html#numpy.log1p)                                           | [`dask.array.log1p`](generated/dask.array.log1p.md#dask.array.log1p)                                                                                                                          | direct (ufunc)                 |
| [`numpy.log2`](https://numpy.org/doc/stable/reference/generated/numpy.log2.html#numpy.log2)                                              | [`dask.array.log2`](generated/dask.array.log2.md#dask.array.log2)                                                                                                                             | direct (ufunc)                 |
| [`numpy.logaddexp`](https://numpy.org/doc/stable/reference/generated/numpy.logaddexp.html#numpy.logaddexp)                               | [`dask.array.logaddexp`](generated/dask.array.logaddexp.md#dask.array.logaddexp)                                                                                                              | direct (ufunc)                 |
| [`numpy.logaddexp2`](https://numpy.org/doc/stable/reference/generated/numpy.logaddexp2.html#numpy.logaddexp2)                            | [`dask.array.logaddexp2`](generated/dask.array.logaddexp2.md#dask.array.logaddexp2)                                                                                                           | direct (ufunc)                 |
| [`numpy.logical_and`](https://numpy.org/doc/stable/reference/generated/numpy.logical_and.html#numpy.logical_and)                         | [`dask.array.logical_and`](generated/dask.array.logical_and.md#dask.array.logical_and)                                                                                                        | direct (ufunc)                 |
| [`numpy.logical_not`](https://numpy.org/doc/stable/reference/generated/numpy.logical_not.html#numpy.logical_not)                         | [`dask.array.logical_not`](generated/dask.array.logical_not.md#dask.array.logical_not)                                                                                                        | direct (ufunc)                 |
| [`numpy.logical_or`](https://numpy.org/doc/stable/reference/generated/numpy.logical_or.html#numpy.logical_or)                            | [`dask.array.logical_or`](generated/dask.array.logical_or.md#dask.array.logical_or)                                                                                                           | direct (ufunc)                 |
| [`numpy.logical_xor`](https://numpy.org/doc/stable/reference/generated/numpy.logical_xor.html#numpy.logical_xor)                         | [`dask.array.logical_xor`](generated/dask.array.logical_xor.md#dask.array.logical_xor)                                                                                                        | direct (ufunc)                 |
| [`numpy.logspace`](https://numpy.org/doc/stable/reference/generated/numpy.logspace.html#numpy.logspace)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.mask_indices`](https://numpy.org/doc/stable/reference/generated/numpy.mask_indices.html#numpy.mask_indices)                      | -                                                                                                                                                                                             |                                |
| `numpy.mat`                                                                                                                              | - <sup>[27](#id70)</sup>                                                                                                                                                                      |                                |
| [`numpy.matmul`](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html#numpy.matmul)                                        | [`dask.array.matmul`](generated/dask.array.matmul.md#dask.array.matmul)                                                                                                                       | dask equivalent                |
| [`numpy.matrix`](https://numpy.org/doc/stable/reference/generated/numpy.matrix.html#numpy.matrix)                                        | - <sup>[27](#id70)</sup>                                                                                                                                                                      |                                |
| [`numpy.maximum`](https://numpy.org/doc/stable/reference/generated/numpy.maximum.html#numpy.maximum)                                     | [`dask.array.maximum`](generated/dask.array.maximum.md#dask.array.maximum)                                                                                                                    | direct (ufunc)                 |
| [`numpy.may_share_memory`](https://numpy.org/doc/stable/reference/generated/numpy.may_share_memory.html#numpy.may_share_memory)          | -                                                                                                                                                                                             |                                |
| [`numpy.mean`](https://numpy.org/doc/stable/reference/generated/numpy.mean.html#numpy.mean)                                              | [`dask.array.mean`](generated/dask.array.mean.md#dask.array.mean) <sup>[21](#id64)</sup>                                                                                                      | dask equivalent                |
| [`numpy.median`](https://numpy.org/doc/stable/reference/generated/numpy.median.html#numpy.median)                                        | [`dask.array.median`](generated/dask.array.median.md#dask.array.median) <sup>[36](#id79)</sup>                                                                                                | dask equivalent                |
| [`numpy.memmap`](https://numpy.org/doc/stable/reference/generated/numpy.memmap.html#numpy.memmap)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.meshgrid`](https://numpy.org/doc/stable/reference/generated/numpy.meshgrid.html#numpy.meshgrid)                                  | [`dask.array.meshgrid`](generated/dask.array.meshgrid.md#dask.array.meshgrid) <sup>[37](#id80)</sup>                                                                                          | dask equivalent                |
| [`numpy.mgrid`](https://numpy.org/doc/stable/reference/generated/numpy.mgrid.html#numpy.mgrid)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.minimum`](https://numpy.org/doc/stable/reference/generated/numpy.minimum.html#numpy.minimum)                                     | [`dask.array.minimum`](generated/dask.array.minimum.md#dask.array.minimum)                                                                                                                    | direct (ufunc)                 |
| [`numpy.mintypecode`](https://numpy.org/doc/stable/reference/generated/numpy.mintypecode.html#numpy.mintypecode)                         | -                                                                                                                                                                                             |                                |
| [`numpy.mod`](https://numpy.org/doc/stable/reference/generated/numpy.mod.html#numpy.mod)                                                 | [`dask.array.mod`](generated/dask.array.mod.md#dask.array.mod)                                                                                                                                | direct (ufunc)                 |
| [`numpy.modf`](https://numpy.org/doc/stable/reference/generated/numpy.modf.html#numpy.modf)                                              | [`dask.array.modf`](generated/dask.array.modf.md#dask.array.modf)                                                                                                                             | dask equivalent                |
| [`numpy.moveaxis`](https://numpy.org/doc/stable/reference/generated/numpy.moveaxis.html#numpy.moveaxis)                                  | [`dask.array.moveaxis`](generated/dask.array.moveaxis.md#dask.array.moveaxis)                                                                                                                 | dask equivalent                |
| [`numpy.multiply`](https://numpy.org/doc/stable/reference/generated/numpy.multiply.html#numpy.multiply)                                  | [`dask.array.multiply`](generated/dask.array.multiply.md#dask.array.multiply)                                                                                                                 | direct (ufunc)                 |
| [`numpy.nan_to_num`](https://numpy.org/doc/stable/reference/generated/numpy.nan_to_num.html#numpy.nan_to_num)                            | [`dask.array.nan_to_num`](generated/dask.array.nan_to_num.md#dask.array.nan_to_num)                                                                                                           | direct (non-ufunc elementwise) |
| [`numpy.nanargmax`](https://numpy.org/doc/stable/reference/generated/numpy.nanargmax.html#numpy.nanargmax)                               | [`dask.array.nanargmax`](generated/dask.array.nanargmax.md#dask.array.nanargmax)                                                                                                              | dask equivalent                |
| [`numpy.nanargmin`](https://numpy.org/doc/stable/reference/generated/numpy.nanargmin.html#numpy.nanargmin)                               | [`dask.array.nanargmin`](generated/dask.array.nanargmin.md#dask.array.nanargmin)                                                                                                              | dask equivalent                |
| [`numpy.nancumprod`](https://numpy.org/doc/stable/reference/generated/numpy.nancumprod.html#numpy.nancumprod)                            | [`dask.array.nancumprod`](generated/dask.array.nancumprod.md#dask.array.nancumprod) <sup>[23](#id66)</sup> <sup>[38](#id81)</sup>                                                             | dask equivalent                |
| [`numpy.nancumsum`](https://numpy.org/doc/stable/reference/generated/numpy.nancumsum.html#numpy.nancumsum)                               | [`dask.array.nancumsum`](generated/dask.array.nancumsum.md#dask.array.nancumsum) <sup>[23](#id66)</sup> <sup>[38](#id81)</sup>                                                                | dask equivalent                |
| [`numpy.nanmax`](https://numpy.org/doc/stable/reference/generated/numpy.nanmax.html#numpy.nanmax)                                        | [`dask.array.nanmax`](generated/dask.array.nanmax.md#dask.array.nanmax) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                         | dask equivalent                |
| [`numpy.nanmean`](https://numpy.org/doc/stable/reference/generated/numpy.nanmean.html#numpy.nanmean)                                     | [`dask.array.nanmean`](generated/dask.array.nanmean.md#dask.array.nanmean) <sup>[21](#id64)</sup>                                                                                             | dask equivalent                |
| [`numpy.nanmedian`](https://numpy.org/doc/stable/reference/generated/numpy.nanmedian.html#numpy.nanmedian)                               | [`dask.array.nanmedian`](generated/dask.array.nanmedian.md#dask.array.nanmedian) <sup>[36](#id79)</sup>                                                                                       | dask equivalent                |
| [`numpy.nanmin`](https://numpy.org/doc/stable/reference/generated/numpy.nanmin.html#numpy.nanmin)                                        | [`dask.array.nanmin`](generated/dask.array.nanmin.md#dask.array.nanmin) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                         | dask equivalent                |
| [`numpy.nanpercentile`](https://numpy.org/doc/stable/reference/generated/numpy.nanpercentile.html#numpy.nanpercentile)                   | [`dask.array.nanpercentile`](generated/dask.array.nanpercentile.md#dask.array.nanpercentile)                                                                                                  |                                |
| [`numpy.nanprod`](https://numpy.org/doc/stable/reference/generated/numpy.nanprod.html#numpy.nanprod)                                     | [`dask.array.nanprod`](generated/dask.array.nanprod.md#dask.array.nanprod) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                      | dask equivalent                |
| [`numpy.nanquantile`](https://numpy.org/doc/stable/reference/generated/numpy.nanquantile.html#numpy.nanquantile)                         | [`dask.array.nanquantile`](generated/dask.array.nanquantile.md#dask.array.nanquantile)                                                                                                        |                                |
| [`numpy.nanstd`](https://numpy.org/doc/stable/reference/generated/numpy.nanstd.html#numpy.nanstd)                                        | [`dask.array.nanstd`](generated/dask.array.nanstd.md#dask.array.nanstd) <sup>[21](#id64)</sup>                                                                                                | dask equivalent                |
| [`numpy.nansum`](https://numpy.org/doc/stable/reference/generated/numpy.nansum.html#numpy.nansum)                                        | [`dask.array.nansum`](generated/dask.array.nansum.md#dask.array.nansum) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                         | dask equivalent                |
| [`numpy.nanvar`](https://numpy.org/doc/stable/reference/generated/numpy.nanvar.html#numpy.nanvar)                                        | [`dask.array.nanvar`](generated/dask.array.nanvar.md#dask.array.nanvar) <sup>[21](#id64)</sup>                                                                                                | dask equivalent                |
| [`numpy.ndenumerate`](https://numpy.org/doc/stable/reference/generated/numpy.ndenumerate.html#numpy.ndenumerate)                         | -                                                                                                                                                                                             |                                |
| [`numpy.ndindex`](https://numpy.org/doc/stable/reference/generated/numpy.ndindex.html#numpy.ndindex)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.nditer`](https://numpy.org/doc/stable/reference/generated/numpy.nditer.html#numpy.nditer)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.negative`](https://numpy.org/doc/stable/reference/generated/numpy.negative.html#numpy.negative)                                  | [`dask.array.negative`](generated/dask.array.negative.md#dask.array.negative)                                                                                                                 | direct (ufunc)                 |
| [`numpy.nested_iters`](https://numpy.org/doc/stable/reference/generated/numpy.nested_iters.html#numpy.nested_iters)                      | -                                                                                                                                                                                             |                                |
| [`numpy.nextafter`](https://numpy.org/doc/stable/reference/generated/numpy.nextafter.html#numpy.nextafter)                               | [`dask.array.nextafter`](generated/dask.array.nextafter.md#dask.array.nextafter)                                                                                                              | direct (ufunc)                 |
| [`numpy.nonzero`](https://numpy.org/doc/stable/reference/generated/numpy.nonzero.html#numpy.nonzero)                                     | [`dask.array.nonzero`](generated/dask.array.nonzero.md#dask.array.nonzero)                                                                                                                    | dask equivalent                |
| [`numpy.not_equal`](https://numpy.org/doc/stable/reference/generated/numpy.not_equal.html#numpy.not_equal)                               | [`dask.array.not_equal`](generated/dask.array.not_equal.md#dask.array.not_equal)                                                                                                              | direct (ufunc)                 |
| [`numpy.ogrid`](https://numpy.org/doc/stable/reference/generated/numpy.ogrid.html#numpy.ogrid)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.ones`](https://numpy.org/doc/stable/reference/generated/numpy.ones.html#numpy.ones)                                              | [`dask.array.ones`](generated/dask.array.ones.md#dask.array.ones)                                                                                                                             | dask equivalent                |
| [`numpy.ones_like`](https://numpy.org/doc/stable/reference/generated/numpy.ones_like.html#numpy.ones_like)                               | [`dask.array.ones_like`](generated/dask.array.ones_like.md#dask.array.ones_like)                                                                                                              | dask equivalent                |
| [`numpy.outer`](https://numpy.org/doc/stable/reference/generated/numpy.outer.html#numpy.outer)                                           | [`dask.array.outer`](generated/dask.array.outer.md#dask.array.outer)                                                                                                                          | dask equivalent                |
| [`numpy.packbits`](https://numpy.org/doc/stable/reference/generated/numpy.packbits.html#numpy.packbits)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.pad`](https://numpy.org/doc/stable/reference/generated/numpy.pad.html#numpy.pad)                                                 | [`dask.array.pad`](generated/dask.array.pad.md#dask.array.pad)                                                                                                                                | dask equivalent                |
| [`numpy.partition`](https://numpy.org/doc/stable/reference/generated/numpy.partition.html#numpy.partition)                               | -                                                                                                                                                                                             |                                |
| [`numpy.percentile`](https://numpy.org/doc/stable/reference/generated/numpy.percentile.html#numpy.percentile)                            | [`dask.array.percentile`](generated/dask.array.percentile.md#dask.array.percentile)                                                                                                           | dask equivalent                |
| [`numpy.piecewise`](https://numpy.org/doc/stable/reference/generated/numpy.piecewise.html#numpy.piecewise)                               | [`dask.array.piecewise`](generated/dask.array.piecewise.md#dask.array.piecewise)                                                                                                              | dask equivalent                |
| [`numpy.place`](https://numpy.org/doc/stable/reference/generated/numpy.place.html#numpy.place)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.poly`](https://numpy.org/doc/stable/reference/generated/numpy.poly.html#numpy.poly)                                              | -                                                                                                                                                                                             |                                |
| [`numpy.poly1d`](https://numpy.org/doc/stable/reference/generated/numpy.poly1d.html#numpy.poly1d)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.polyadd`](https://numpy.org/doc/stable/reference/generated/numpy.polyadd.html#numpy.polyadd)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polyder`](https://numpy.org/doc/stable/reference/generated/numpy.polyder.html#numpy.polyder)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polydiv`](https://numpy.org/doc/stable/reference/generated/numpy.polydiv.html#numpy.polydiv)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polyfit`](https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html#numpy.polyfit)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polyint`](https://numpy.org/doc/stable/reference/generated/numpy.polyint.html#numpy.polyint)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polymul`](https://numpy.org/doc/stable/reference/generated/numpy.polymul.html#numpy.polymul)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polysub`](https://numpy.org/doc/stable/reference/generated/numpy.polysub.html#numpy.polysub)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.polyval`](https://numpy.org/doc/stable/reference/generated/numpy.polyval.html#numpy.polyval)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.positive`](https://numpy.org/doc/stable/reference/generated/numpy.positive.html#numpy.positive)                                  | [`dask.array.positive`](generated/dask.array.positive.md#dask.array.positive)                                                                                                                 | direct (ufunc)                 |
| [`numpy.power`](https://numpy.org/doc/stable/reference/generated/numpy.power.html#numpy.power)                                           | [`dask.array.power`](generated/dask.array.power.md#dask.array.power)                                                                                                                          | direct (ufunc)                 |
| [`numpy.prod`](https://numpy.org/doc/stable/reference/generated/numpy.prod.html#numpy.prod)                                              | [`dask.array.prod`](generated/dask.array.prod.md#dask.array.prod)                                                                                                                             | dask equivalent                |
| [`numpy.ptp`](https://numpy.org/doc/stable/reference/generated/numpy.ptp.html#numpy.ptp)                                                 | [`dask.array.ptp`](generated/dask.array.ptp.md#dask.array.ptp)                                                                                                                                | dask equivalent                |
| [`numpy.put`](https://numpy.org/doc/stable/reference/generated/numpy.put.html#numpy.put)                                                 | -                                                                                                                                                                                             |                                |
| [`numpy.put_along_axis`](https://numpy.org/doc/stable/reference/generated/numpy.put_along_axis.html#numpy.put_along_axis)                | -                                                                                                                                                                                             |                                |
| [`numpy.putmask`](https://numpy.org/doc/stable/reference/generated/numpy.putmask.html#numpy.putmask)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.quantile`](https://numpy.org/doc/stable/reference/generated/numpy.quantile.html#numpy.quantile)                                  | [`dask.array.quantile`](generated/dask.array.quantile.md#dask.array.quantile)                                                                                                                 |                                |
| [`numpy.r_`](https://numpy.org/doc/stable/reference/generated/numpy.r_.html#numpy.r_)                                                    | -                                                                                                                                                                                             |                                |
| [`numpy.rad2deg`](https://numpy.org/doc/stable/reference/generated/numpy.rad2deg.html#numpy.rad2deg)                                     | [`dask.array.rad2deg`](generated/dask.array.rad2deg.md#dask.array.rad2deg)                                                                                                                    | direct (ufunc)                 |
| [`numpy.radians`](https://numpy.org/doc/stable/reference/generated/numpy.radians.html#numpy.radians)                                     | [`dask.array.radians`](generated/dask.array.radians.md#dask.array.radians)                                                                                                                    | direct (ufunc)                 |
| [`numpy.ravel`](https://numpy.org/doc/stable/reference/generated/numpy.ravel.html#numpy.ravel)                                           | [`dask.array.ravel`](generated/dask.array.ravel.md#dask.array.ravel) <sup>[23](#id66)</sup> <sup>[24](#id67)</sup>                                                                            | dask equivalent                |
| [`numpy.ravel_multi_index`](https://numpy.org/doc/stable/reference/generated/numpy.ravel_multi_index.html#numpy.ravel_multi_index)       | [`dask.array.ravel_multi_index`](generated/dask.array.ravel_multi_index.md#dask.array.ravel_multi_index)                                                                                      | dask equivalent                |
| [`numpy.real`](https://numpy.org/doc/stable/reference/generated/numpy.real.html#numpy.real)                                              | [`dask.array.real`](generated/dask.array.real.md#dask.array.real)                                                                                                                             | direct (non-ufunc elementwise) |
| [`numpy.real_if_close`](https://numpy.org/doc/stable/reference/generated/numpy.real_if_close.html#numpy.real_if_close)                   | -                                                                                                                                                                                             |                                |
| [`numpy.reciprocal`](https://numpy.org/doc/stable/reference/generated/numpy.reciprocal.html#numpy.reciprocal)                            | [`dask.array.reciprocal`](generated/dask.array.reciprocal.md#dask.array.reciprocal)                                                                                                           | direct (ufunc)                 |
| [`numpy.remainder`](https://numpy.org/doc/stable/reference/generated/numpy.remainder.html#numpy.remainder)                               | [`dask.array.remainder`](generated/dask.array.remainder.md#dask.array.remainder)                                                                                                              | direct (ufunc)                 |
| [`numpy.repeat`](https://numpy.org/doc/stable/reference/generated/numpy.repeat.html#numpy.repeat)                                        | [`dask.array.repeat`](generated/dask.array.repeat.md#dask.array.repeat)                                                                                                                       | dask equivalent                |
| [`numpy.require`](https://numpy.org/doc/stable/reference/generated/numpy.require.html#numpy.require)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.reshape`](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html#numpy.reshape)                                     | [`dask.array.reshape`](generated/dask.array.reshape.md#dask.array.reshape)                                                                                                                    | dask equivalent                |
| [`numpy.resize`](https://numpy.org/doc/stable/reference/generated/numpy.resize.html#numpy.resize)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.result_type`](https://numpy.org/doc/stable/reference/generated/numpy.result_type.html#numpy.result_type)                         | [`dask.array.result_type`](generated/dask.array.result_type.md#dask.array.result_type)                                                                                                        | dask equivalent                |
| [`numpy.right_shift`](https://numpy.org/doc/stable/reference/generated/numpy.right_shift.html#numpy.right_shift)                         | [`dask.array.right_shift`](generated/dask.array.right_shift.md#dask.array.right_shift)                                                                                                        | direct (ufunc)                 |
| [`numpy.rint`](https://numpy.org/doc/stable/reference/generated/numpy.rint.html#numpy.rint)                                              | [`dask.array.rint`](generated/dask.array.rint.md#dask.array.rint)                                                                                                                             | direct (ufunc)                 |
| [`numpy.roll`](https://numpy.org/doc/stable/reference/generated/numpy.roll.html#numpy.roll)                                              | [`dask.array.roll`](generated/dask.array.roll.md#dask.array.roll)                                                                                                                             | dask equivalent                |
| [`numpy.rollaxis`](https://numpy.org/doc/stable/reference/generated/numpy.rollaxis.html#numpy.rollaxis)                                  | [`dask.array.rollaxis`](generated/dask.array.rollaxis.md#dask.array.rollaxis)                                                                                                                 | dask equivalent                |
| [`numpy.roots`](https://numpy.org/doc/stable/reference/generated/numpy.roots.html#numpy.roots)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.rot90`](https://numpy.org/doc/stable/reference/generated/numpy.rot90.html#numpy.rot90)                                           | [`dask.array.rot90`](generated/dask.array.rot90.md#dask.array.rot90)                                                                                                                          | dask equivalent                |
| `numpy.row_stack`                                                                                                                        | -                                                                                                                                                                                             |                                |
| [`numpy.save`](https://numpy.org/doc/stable/reference/generated/numpy.save.html#numpy.save)                                              | -                                                                                                                                                                                             |                                |
| [`numpy.savetxt`](https://numpy.org/doc/stable/reference/generated/numpy.savetxt.html#numpy.savetxt)                                     | -                                                                                                                                                                                             |                                |
| [`numpy.savez`](https://numpy.org/doc/stable/reference/generated/numpy.savez.html#numpy.savez)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.savez_compressed`](https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html#numpy.savez_compressed)          | -                                                                                                                                                                                             |                                |
| [`numpy.searchsorted`](https://numpy.org/doc/stable/reference/generated/numpy.searchsorted.html#numpy.searchsorted)                      | [`dask.array.searchsorted`](generated/dask.array.searchsorted.md#dask.array.searchsorted)                                                                                                     | dask equivalent                |
| [`numpy.select`](https://numpy.org/doc/stable/reference/generated/numpy.select.html#numpy.select)                                        | [`dask.array.select`](generated/dask.array.select.md#dask.array.select)                                                                                                                       | dask equivalent                |
| [`numpy.setdiff1d`](https://numpy.org/doc/stable/reference/generated/numpy.setdiff1d.html#numpy.setdiff1d)                               | -                                                                                                                                                                                             |                                |
| [`numpy.setxor1d`](https://numpy.org/doc/stable/reference/generated/numpy.setxor1d.html#numpy.setxor1d)                                  | -                                                                                                                                                                                             |                                |
| [`numpy.shape`](https://numpy.org/doc/stable/reference/generated/numpy.shape.html#numpy.shape)                                           | [`dask.array.shape`](generated/dask.array.shape.md#dask.array.shape) <sup>[23](#id66)</sup>                                                                                                   | dask equivalent                |
| [`numpy.shares_memory`](https://numpy.org/doc/stable/reference/generated/numpy.shares_memory.html#numpy.shares_memory)                   | -                                                                                                                                                                                             |                                |
| [`numpy.sign`](https://numpy.org/doc/stable/reference/generated/numpy.sign.html#numpy.sign)                                              | [`dask.array.sign`](generated/dask.array.sign.md#dask.array.sign)                                                                                                                             | direct (ufunc)                 |
| [`numpy.signbit`](https://numpy.org/doc/stable/reference/generated/numpy.signbit.html#numpy.signbit)                                     | [`dask.array.signbit`](generated/dask.array.signbit.md#dask.array.signbit)                                                                                                                    | direct (ufunc)                 |
| [`numpy.sin`](https://numpy.org/doc/stable/reference/generated/numpy.sin.html#numpy.sin)                                                 | [`dask.array.sin`](generated/dask.array.sin.md#dask.array.sin)                                                                                                                                | direct (ufunc)                 |
| [`numpy.sinc`](https://numpy.org/doc/stable/reference/generated/numpy.sinc.html#numpy.sinc)                                              | [`dask.array.sinc`](generated/dask.array.sinc.md#dask.array.sinc)                                                                                                                             | direct (non-ufunc elementwise) |
| [`numpy.sinh`](https://numpy.org/doc/stable/reference/generated/numpy.sinh.html#numpy.sinh)                                              | [`dask.array.sinh`](generated/dask.array.sinh.md#dask.array.sinh)                                                                                                                             | direct (ufunc)                 |
| [`numpy.sort`](https://numpy.org/doc/stable/reference/generated/numpy.sort.html#numpy.sort)                                              | [`dask.array.topk`](generated/dask.array.topk.md#dask.array.topk) <sup>[25](#id68)</sup>                                                                                                      |                                |
| [`numpy.sort_complex`](https://numpy.org/doc/stable/reference/generated/numpy.sort_complex.html#numpy.sort_complex)                      | -                                                                                                                                                                                             |                                |
| `numpy.source`                                                                                                                           | -                                                                                                                                                                                             |                                |
| [`numpy.spacing`](https://numpy.org/doc/stable/reference/generated/numpy.spacing.html#numpy.spacing)                                     | [`dask.array.spacing`](generated/dask.array.spacing.md#dask.array.spacing)                                                                                                                    | direct (ufunc)                 |
| [`numpy.split`](https://numpy.org/doc/stable/reference/generated/numpy.split.html#numpy.split)                                           | -                                                                                                                                                                                             |                                |
| [`numpy.sqrt`](https://numpy.org/doc/stable/reference/generated/numpy.sqrt.html#numpy.sqrt)                                              | [`dask.array.sqrt`](generated/dask.array.sqrt.md#dask.array.sqrt)                                                                                                                             | direct (ufunc)                 |
| [`numpy.square`](https://numpy.org/doc/stable/reference/generated/numpy.square.html#numpy.square)                                        | [`dask.array.square`](generated/dask.array.square.md#dask.array.square)                                                                                                                       | direct (ufunc)                 |
| [`numpy.squeeze`](https://numpy.org/doc/stable/reference/generated/numpy.squeeze.html#numpy.squeeze)                                     | [`dask.array.squeeze`](generated/dask.array.squeeze.md#dask.array.squeeze)                                                                                                                    | dask equivalent                |
| [`numpy.stack`](https://numpy.org/doc/stable/reference/generated/numpy.stack.html#numpy.stack)                                           | [`dask.array.stack`](generated/dask.array.stack.md#dask.array.stack)                                                                                                                          | dask equivalent                |
| [`numpy.std`](https://numpy.org/doc/stable/reference/generated/numpy.std.html#numpy.std)                                                 | [`dask.array.std`](generated/dask.array.std.md#dask.array.std) <sup>[21](#id64)</sup>                                                                                                         | dask equivalent                |
| [`numpy.subtract`](https://numpy.org/doc/stable/reference/generated/numpy.subtract.html#numpy.subtract)                                  | [`dask.array.subtract`](generated/dask.array.subtract.md#dask.array.subtract)                                                                                                                 | direct (ufunc)                 |
| [`numpy.sum`](https://numpy.org/doc/stable/reference/generated/numpy.sum.html#numpy.sum)                                                 | [`dask.array.sum`](generated/dask.array.sum.md#dask.array.sum) <sup>[21](#id64)</sup> <sup>[22](#id65)</sup>                                                                                  | dask equivalent                |
| [`numpy.swapaxes`](https://numpy.org/doc/stable/reference/generated/numpy.swapaxes.html#numpy.swapaxes)                                  | [`dask.array.swapaxes`](generated/dask.array.swapaxes.md#dask.array.swapaxes)                                                                                                                 | dask equivalent                |
| [`numpy.take`](https://numpy.org/doc/stable/reference/generated/numpy.take.html#numpy.take)                                              | [`dask.array.take`](generated/dask.array.take.md#dask.array.take) <sup>[28](#id71)</sup>                                                                                                      | dask equivalent                |
| [`numpy.take_along_axis`](https://numpy.org/doc/stable/reference/generated/numpy.take_along_axis.html#numpy.take_along_axis)             | -                                                                                                                                                                                             |                                |
| [`numpy.tan`](https://numpy.org/doc/stable/reference/generated/numpy.tan.html#numpy.tan)                                                 | [`dask.array.tan`](generated/dask.array.tan.md#dask.array.tan)                                                                                                                                | direct (ufunc)                 |
| [`numpy.tanh`](https://numpy.org/doc/stable/reference/generated/numpy.tanh.html#numpy.tanh)                                              | [`dask.array.tanh`](generated/dask.array.tanh.md#dask.array.tanh)                                                                                                                             | direct (ufunc)                 |
| [`numpy.tensordot`](https://numpy.org/doc/stable/reference/generated/numpy.tensordot.html#numpy.tensordot)                               | [`dask.array.tensordot`](generated/dask.array.tensordot.md#dask.array.tensordot)                                                                                                              | dask equivalent                |
| [`numpy.tile`](https://numpy.org/doc/stable/reference/generated/numpy.tile.html#numpy.tile)                                              | [`dask.array.tile`](generated/dask.array.tile.md#dask.array.tile)                                                                                                                             | dask equivalent                |
| [`numpy.trace`](https://numpy.org/doc/stable/reference/generated/numpy.trace.html#numpy.trace)                                           | [`dask.array.trace`](generated/dask.array.trace.md#dask.array.trace) <sup>[26](#id69)</sup>                                                                                                   | dask equivalent                |
| [`numpy.transpose`](https://numpy.org/doc/stable/reference/generated/numpy.transpose.html#numpy.transpose)                               | [`dask.array.transpose`](generated/dask.array.transpose.md#dask.array.transpose)                                                                                                              | dask equivalent                |
| `numpy.trapz`                                                                                                                            | -                                                                                                                                                                                             |                                |
| [`numpy.tri`](https://numpy.org/doc/stable/reference/generated/numpy.tri.html#numpy.tri)                                                 | [`dask.array.tri`](generated/dask.array.tri.md#dask.array.tri)                                                                                                                                | dask equivalent                |
| [`numpy.tril`](https://numpy.org/doc/stable/reference/generated/numpy.tril.html#numpy.tril)                                              | [`dask.array.tril`](generated/dask.array.tril.md#dask.array.tril)                                                                                                                             | dask equivalent                |
| [`numpy.tril_indices`](https://numpy.org/doc/stable/reference/generated/numpy.tril_indices.html#numpy.tril_indices)                      | [`dask.array.tril_indices`](generated/dask.array.tril_indices.md#dask.array.tril_indices)                                                                                                     | dask equivalent                |
| [`numpy.tril_indices_from`](https://numpy.org/doc/stable/reference/generated/numpy.tril_indices_from.html#numpy.tril_indices_from)       | [`dask.array.tril_indices_from`](generated/dask.array.tril_indices_from.md#dask.array.tril_indices_from)                                                                                      | dask equivalent                |
| [`numpy.trim_zeros`](https://numpy.org/doc/stable/reference/generated/numpy.trim_zeros.html#numpy.trim_zeros)                            | -                                                                                                                                                                                             |                                |
| [`numpy.triu`](https://numpy.org/doc/stable/reference/generated/numpy.triu.html#numpy.triu)                                              | [`dask.array.triu`](generated/dask.array.triu.md#dask.array.triu)                                                                                                                             | dask equivalent                |
| [`numpy.triu_indices`](https://numpy.org/doc/stable/reference/generated/numpy.triu_indices.html#numpy.triu_indices)                      | [`dask.array.triu_indices`](generated/dask.array.triu_indices.md#dask.array.triu_indices)                                                                                                     | dask equivalent                |
| [`numpy.triu_indices_from`](https://numpy.org/doc/stable/reference/generated/numpy.triu_indices_from.html#numpy.triu_indices_from)       | [`dask.array.triu_indices_from`](generated/dask.array.triu_indices_from.md#dask.array.triu_indices_from)                                                                                      | dask equivalent                |
| [`numpy.true_divide`](https://numpy.org/doc/stable/reference/generated/numpy.true_divide.html#numpy.true_divide)                         | [`dask.array.true_divide`](generated/dask.array.true_divide.md#dask.array.true_divide)                                                                                                        | direct (ufunc)                 |
| [`numpy.trunc`](https://numpy.org/doc/stable/reference/generated/numpy.trunc.html#numpy.trunc)                                           | [`dask.array.trunc`](generated/dask.array.trunc.md#dask.array.trunc)                                                                                                                          | direct (ufunc)                 |
| [`numpy.union1d`](https://numpy.org/doc/stable/reference/generated/numpy.union1d.html#numpy.union1d)                                     | [`dask.array.union1d`](generated/dask.array.union1d.md#dask.array.union1d)                                                                                                                    | dask equivalent                |
| [`numpy.unique`](https://numpy.org/doc/stable/reference/generated/numpy.unique.html#numpy.unique)                                        | [`dask.array.unique`](generated/dask.array.unique.md#dask.array.unique) <sup>[39](#id82)</sup>                                                                                                | dask equivalent                |
| [`numpy.unpackbits`](https://numpy.org/doc/stable/reference/generated/numpy.unpackbits.html#numpy.unpackbits)                            | -                                                                                                                                                                                             |                                |
| [`numpy.unravel_index`](https://numpy.org/doc/stable/reference/generated/numpy.unravel_index.html#numpy.unravel_index)                   | [`dask.array.unravel_index`](generated/dask.array.unravel_index.md#dask.array.unravel_index)                                                                                                  | dask equivalent                |
| [`numpy.unwrap`](https://numpy.org/doc/stable/reference/generated/numpy.unwrap.html#numpy.unwrap)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.vander`](https://numpy.org/doc/stable/reference/generated/numpy.vander.html#numpy.vander)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.var`](https://numpy.org/doc/stable/reference/generated/numpy.var.html#numpy.var)                                                 | [`dask.array.var`](generated/dask.array.var.md#dask.array.var) <sup>[21](#id64)</sup>                                                                                                         | dask equivalent                |
| [`numpy.vdot`](https://numpy.org/doc/stable/reference/generated/numpy.vdot.html#numpy.vdot)                                              | [`dask.array.vdot`](generated/dask.array.vdot.md#dask.array.vdot)                                                                                                                             | dask equivalent                |
| [`numpy.vsplit`](https://numpy.org/doc/stable/reference/generated/numpy.vsplit.html#numpy.vsplit)                                        | -                                                                                                                                                                                             |                                |
| [`numpy.vstack`](https://numpy.org/doc/stable/reference/generated/numpy.vstack.html#numpy.vstack)                                        | [`dask.array.vstack`](generated/dask.array.vstack.md#dask.array.vstack) <sup>[40](#id83)</sup>                                                                                                | dask equivalent                |
| [`numpy.where`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where)                                           | [`dask.array.where`](generated/dask.array.where.md#dask.array.where)                                                                                                                          | dask equivalent                |
| [`numpy.zeros`](https://numpy.org/doc/stable/reference/generated/numpy.zeros.html#numpy.zeros)                                           | [`dask.array.zeros`](generated/dask.array.zeros.md#dask.array.zeros)                                                                                                                          | dask equivalent                |
| [`numpy.zeros_like`](https://numpy.org/doc/stable/reference/generated/numpy.zeros_like.html#numpy.zeros_like)                            | [`dask.array.zeros_like`](generated/dask.array.zeros_like.md#dask.array.zeros_like)                                                                                                           | dask equivalent                |

### Footnotes

* <a id='id64'>**[21]**</a> `where` parameter not supported.
* <a id='id65'>**[22]**</a> `initial` parameter not supported.
* <a id='id66'>**[23]**</a> Input must be a dask array.
* <a id='id67'>**[24]**</a> `order` parameter not supported.
* <a id='id68'>**[25]**</a> Sort operations are notoriously difficult to do in parallel. Parallel-friendly alternatives sort the k largest elements.
* <a id='id69'>**[26]**</a> `out` parameter not supported.
* <a id='id70'>**[27]**</a> Use of numpy.matrix is discouraged in NumPy and thus there is no need to add it.
* <a id='id71'>**[28]**</a> `mode` parameter not supported.
* <a id='id72'>**[29]**</a> `keepdims` parameter not supported.
* <a id='id73'>**[30]**</a> `fweights`, `aweights`, `dtype` parameters not supported.
* <a id='id74'>**[31]**</a> `like` parameter not supported. Callable functions not supported.
* <a id='id75'>**[32]**</a> Not implemented with more than one output.
* <a id='id76'>**[33]**</a> `edge_order` parameter not supported.
* <a id='id77'>**[34]**</a> Chunking of the input data (sample) is only allowed along the 0th (row) axis.
* <a id='id78'>**[35]**</a> Only implemented for monotonic `obj` arguments.
* <a id='id79'>**[36]**</a> `overwrite_input` parameter not supported.
* <a id='id80'>**[37]**</a> `copy` parameter not supported.
* <a id='id81'>**[38]**</a> Dask implementation introduces an additional parameter `method`.
* <a id='id82'>**[39]**</a> `axis` parameter not supported.
* <a id='id83'>**[40]**</a> `casting` parameter not supported.


# array-overlap.html.md

# Overlapping Computations

Some array operations require communication of borders between neighboring
blocks.  Example operations include the following:

* Convolve a filter across an image
* Sliding sum/mean/max, …
* Search for image motifs like a Gaussian blob that might span the border of a
  block
* Evaluate a partial derivative
* Play the game of [Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life)

Dask Array supports these operations by creating a new array where each
block is slightly expanded by the borders of its neighbors.  This costs an
excess copy and the communication of many small chunks, but allows localized
functions to evaluate in an embarrassingly parallel manner.

The main API for these computations is the `map_overlap` method defined
below:

| [`map_overlap`](generated/dask.array.map_overlap.md#dask.array.map_overlap)(func, \*args[, depth, boundary, ...])   | Map a function over blocks of arrays with some overlap   |
|---------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|

### dask.array.map_overlap(func, \*args, depth=None, boundary=None, trim=True, align_arrays=True, allow_rechunk=True, \*\*kwargs)

Map a function over blocks of arrays with some overlap

We share neighboring zones between blocks of the array, map a
function, and then trim away the neighboring strips. If depth is
larger than any chunk along a particular axis, then the array is
rechunked.

Note that this function will attempt to automatically determine the output
array type before computing it, please refer to the `meta` keyword argument
in `map_blocks` if you expect that the function will not succeed when
operating on 0-d arrays.

* **Parameters:**
  **func: function**
  : The function to apply to each extended block.
    If multiple arrays are provided, then the function should expect to
    receive chunks of each array in the same order.

  **args**

  **depth: int, tuple, dict or list, keyword only**
  : The number of elements that each block should share with its neighbors
    If a tuple or dict then this can be different per axis.
    If a list then each element of that list must be an int, tuple or dict
    defining depth for the corresponding array in args.
    Asymmetric depths may be specified using a dict value of (-/+) tuples.
    Note that asymmetric depths are currently only supported when
    `boundary` is ‘none’.
    The default value is 0.

  **boundary: str, tuple, dict or list, keyword only**
  : How to handle the boundaries.
    Values include ‘reflect’, ‘periodic’, ‘nearest’, ‘none’,
    or any constant value like 0 or np.nan.
    If a list then each element must be a str, tuple or dict defining the
    boundary for the corresponding array in args.

  **trim: bool, keyword only**
  : Whether or not to trim `depth` elements from each block after
    calling the map function.
    Set this to False if your mapping function already does this for you

  **align_arrays: bool, keyword only**
  : Whether or not to align chunks along equally sized dimensions when
    multiple arrays are provided.  This allows for larger chunks in some
    arrays to be broken into smaller ones that match chunk sizes in other
    arrays such that they are compatible for block function mapping. If
    this is false, then an error will be thrown if arrays do not already
    have the same number of blocks in each dimension.

  **allow_rechunk: bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.

  **\*\*kwargs:**
  : Other keyword arguments valid in `map_blocks`

### Examples

```pycon
>>> import numpy as np
>>> import dask.array as da
```

```pycon
>>> x = np.array([1, 1, 2, 3, 3, 3, 2, 1, 1])
>>> x = da.from_array(x, chunks=5)
>>> def derivative(x):
...     return x - np.roll(x, 1)
```

```pycon
>>> y = x.map_overlap(derivative, depth=1, boundary=0)
>>> y.compute()
array([ 1,  0,  1,  1,  0,  0, -1, -1,  0])
```

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> d.map_overlap(lambda x: x + x.size, depth=1, boundary='reflect').compute()
array([[16, 17, 18, 19],
       [20, 21, 22, 23],
       [24, 25, 26, 27],
       [28, 29, 30, 31]])
```

```pycon
>>> func = lambda x: x + x.size
>>> depth = {0: 1, 1: 1}
>>> boundary = {0: 'reflect', 1: 'none'}
>>> d.map_overlap(func, depth, boundary).compute()
array([[12,  13,  14,  15],
       [16,  17,  18,  19],
       [20,  21,  22,  23],
       [24,  25,  26,  27]])
```

The `da.map_overlap` function can also accept multiple arrays.

```pycon
>>> func = lambda x, y: x + y
>>> x = da.arange(8).reshape(2, 4).rechunk((1, 2))
>>> y = da.arange(4).rechunk(2)
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect').compute()
array([[ 0,  2,  4,  6],
       [ 4,  6,  8,  10]])
```

When multiple arrays are given, they do not need to have the
same number of dimensions but they must broadcast together.
Arrays are aligned block by block (just as in `da.map_blocks`)
so the blocks must have a common chunk size.  This common chunking
is determined automatically as long as `align_arrays` is True.

```pycon
>>> x = da.arange(8, chunks=4)
>>> y = da.arange(8, chunks=2)
>>> r = da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=True)
>>> len(r.to_delayed())
4
```

```pycon
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=False).compute()
Traceback (most recent call last):
    ...
ValueError: Shapes do not align {'.0': {2, 4}}
```

Note also that this function is equivalent to `map_blocks`
by default.  A non-zero `depth` must be defined for any
overlap to appear in the arrays provided to `func`.

```pycon
>>> func = lambda x: x.sum()
>>> x = da.ones(10, dtype='int')
>>> block_args = dict(chunks=(), drop_axis=0)
>>> da.map_blocks(func, x, **block_args).compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, boundary='reflect').compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, depth=1, boundary='reflect').compute()
np.int64(12)
```

For functions that may not handle 0-d arrays, it’s also possible to specify
`meta` with an empty array matching the type of the expected result. In
the example below, `func` will result in an `IndexError` when computing
`meta`:

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=np.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

Similarly, it’s possible to specify a non-NumPy array to `meta`:

```pycon
>>> import cupy
>>> x = cupy.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=cupy.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=cupy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

<!-- !! processed by numpydoc !! -->

## Explanation

Consider two neighboring blocks in a Dask array:

![Two neighboring blocks which do not overlap.](images/unoverlapping-neighbors.svg)

We extend each block by trading thin nearby slices between arrays:

![Two neighboring block with thin strips along their shared border representing data shared between them.](images/overlapping-neighbors.svg)

We do this in all directions, including also diagonal interactions with the
overlap function:

![A two-dimensional grid of blocks where each one has thin strips around their borders representing data shared from their neighbors. They include small corner bits for data shared from diagonal neighbors as well.](images/overlapping-blocks.svg)
```python
>>> import dask.array as da
>>> import numpy as np

>>> x = np.arange(64).reshape((8, 8))
>>> d = da.from_array(x, chunks=(4, 4))
>>> d.chunks
((4, 4), (4, 4))

>>> g = da.overlap.overlap(d, depth={0: 2, 1: 1},
...                       boundary={0: 100, 1: 'reflect'})
>>> g.chunks
((8, 8), (6, 6))

>>> np.array(g)
array([[100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [  0,   0,   1,   2,   3,   4,   3,   4,   5,   6,   7,   7],
       [  8,   8,   9,  10,  11,  12,  11,  12,  13,  14,  15,  15],
       [ 16,  16,  17,  18,  19,  20,  19,  20,  21,  22,  23,  23],
       [ 24,  24,  25,  26,  27,  28,  27,  28,  29,  30,  31,  31],
       [ 32,  32,  33,  34,  35,  36,  35,  36,  37,  38,  39,  39],
       [ 40,  40,  41,  42,  43,  44,  43,  44,  45,  46,  47,  47],
       [ 16,  16,  17,  18,  19,  20,  19,  20,  21,  22,  23,  23],
       [ 24,  24,  25,  26,  27,  28,  27,  28,  29,  30,  31,  31],
       [ 32,  32,  33,  34,  35,  36,  35,  36,  37,  38,  39,  39],
       [ 40,  40,  41,  42,  43,  44,  43,  44,  45,  46,  47,  47],
       [ 48,  48,  49,  50,  51,  52,  51,  52,  53,  54,  55,  55],
       [ 56,  56,  57,  58,  59,  60,  59,  60,  61,  62,  63,  63],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100]])
```

## Boundaries

With respect to overlapping, you can specify how to handle the boundaries.  Current policies
include the following:

* `periodic` - wrap borders around to the other side
* `reflect` - reflect each border outwards
* `any-constant` - pad the border with this value

An example boundary kind argument might look like the following:

```python
{0: 'periodic',
 1: 'reflect',
 2: np.nan}
```

Alternatively, you can use [`dask.array.pad()`](generated/dask.array.pad.md#dask.array.pad) for other types of
paddings.

## Map a function across blocks

Overlapping goes hand-in-hand with mapping a function across blocks.  This
function can now use the additional information copied over from the neighbors
that is not stored locally in each block:

```python
>>> from scipy.ndimage import gaussian_filter
>>> def func(block):
...    return gaussian_filter(block, sigma=1)

>>> filt = g.map_blocks(func)
```

While in this case we used a SciPy function, any arbitrary function could have been
used instead. This is a good interaction point with [Numba](https://numba.pydata.org/).

If your function does not preserve the shape of the block, then you will need to
provide a `chunks` keyword argument. If your block size is regular, then this
argument can take a block shape of, for example, `(1000, 1000)`. In case of
irregular block sizes, it must be a tuple with the full chunks shape like
`((1000, 700, 1000), (200, 300))`.

```python
>>> g.map_blocks(myfunc, chunks=(5, 5))
```

If your function needs to know the location of the block on which it operates,
you can give your function a keyword argument `block_id`:

```python
def func(block, block_id=None):
    ...
```

This extra keyword argument will be given a tuple that provides the block
location like `(0, 0)` for the upper-left block or `(0, 1)` for the block
just to the right of that block.

## Trim Excess

After mapping a blocked function, you may want to trim off the borders from each
block by the same amount by which they were expanded.  The function
`trim_internal` is useful here and takes the same `depth` argument
given to `overlap`:

```python
>>> x.chunks
((10, 10, 10, 10), (10, 10, 10, 10))

>>> y = da.overlap.trim_internal(x, {0: 2, 1: 1})
>>> y.chunks
((6, 6, 6, 6), (8, 8, 8, 8))
```

## Full Workflow

And so, a pretty typical overlapping workflow includes `overlap`, `map_blocks`
and `trim_internal`:

```python
>>> x = ...
>>> g = da.overlap.overlap(x, depth={0: 2, 1: 2},
...                        boundary={0: 'periodic', 1: 'periodic'})
>>> g2 = g.map_blocks(myfunc)
>>> result = da.overlap.trim_internal(g2, {0: 2, 1: 2})
```


# array-random.html.md

# Random Number Generation

Dask’s random number routines produce pseudo random numbers using combinations
of a `BitGenerator` to create sequences and a `Generator` to use those
sequences to sample from different statistical distributions.

Since Dask version 2023.2.1, the `Generator` can be initialized with a number
of different `BitGenerator` classes. It exposes many different probability
distributions. The legacy `RandomState` random number routines are still
available, but are considered frozen and will not be getting any updates.

## Differences with NumPy

Dask follows the NumPy interface for random number generation with some
differences:

- Methods under `dask.array.random` take a `chunks` keyword.
- Dask tries to be backend agnostic. In other words, you can mostly use CuPy
  and NumPy interchangeably as a backend for random number generation. Any
  library providing a similar interface should also work with some effort.

## Notes

- **BitGenerators:** Objects that generate random sequences. These are
  provided by a backend library such as NumPy or CuPy and are typically
  unsigned integer words filled with sequences of either 32 or 64 random
  bits.
- **Generators:** Objects that transform sequences of random bits from a
  `BitGenerator` into sequences of numbers that follow a specific probability
  distribution (such as uniform, Normal or Binomial) within a specified
  interval.
- Dask does not guarantee that the same number generator is used across versions.
  This means that numbers generated by `dask.array.random` by a new version may
  not be the same as the previous one, even when the same seed and distribution
  are used. As better algorithms evolve, the bit stream may change.
- Dask does not guarantee parity in the generated numbers with any third party
  library. In particular, numbers generated by Dask and NumPy or CuPy will differ
  even when given the same seed and `BitGenerator`. Dask tends to spawn `SeedSequence`
  children to produce independent random number streams in parallel.
- Many of the RandomState methods are exported as functions in `dask.array.random`.
  This usage is discouraged, as it is implemented via a global RandomState instance
  which is not advised on two counts:
  1. It uses global state, which means results will change as the code changes.
  2. It uses a RandomState rather than the more modern Generator.

  For backward compatible legacy reasons, we cannot change this. Use
  `dask.array.random.default_rng()` to get a Generator and use its methods instead.
- `Generator.integers` is now the canonical way to generate integer random numbers
  from a discrete uniform distribution. The endpoint keyword can be used to
  specify open or closed intervals. This replaces both randint and random_integers.
- `Generator.random` is now the canonical way to generate floating-point random
  numbers, which replaces random_sample. The `dask.array.random.random`
  method still uses `RandomState` for backwards compatibility and should be
  avoided for new code. Please use `Generator.random` instead.

## Quick Start

Call `default_rng` to get a new instance of a Generator, then call its methods to
obtain samples from different distributions. By default, `Generator` uses bits
provided by PCG64 which has better statistical properties than the legacy MT19937
used in `RandomState`.

```python
# Do this (new version)
import dask.array as da
rng = da.random.default_rng()
vals = rng.standard_normal(10)
more_vals = rng.standard_normal(10)

# instead of this (legacy version)
import dask.array as da
vals = da.random.standard_normal(10)
more_vals = da.random.standard_normal(10)
```

For further info, please see [NumPy docs](https://numpy.org/devdocs/reference/random/index.html)


# array-slicing.html.md

# Slicing

Dask Array supports most of the NumPy slicing syntax.  In particular, it
supports the following:

* Slicing by integers and slices: `x[0, :5]`
* Slicing by lists/arrays of integers: `x[[1, 2, 4]]`
* Slicing by lists/arrays of booleans: `x[[False, True, True, False, True]]`
* Slicing one [`Array`](generated/dask.array.Array.md#dask.array.Array) with an [`Array`](generated/dask.array.Array.md#dask.array.Array) of bools: `x[x > 0]`
* Slicing one [`Array`](generated/dask.array.Array.md#dask.array.Array) with a zero or one-dimensional [`Array`](generated/dask.array.Array.md#dask.array.Array)
  of ints: `a[b.argtopk(5)]`

However, it does not currently support the following:

* Slicing with lists in multiple axes: `x[[1, 2, 3], [3, 2, 1]]`

  This is straightforward to add though.  If you have a use case then raise an
  issue. Also, users interested in this should take a look at
  [`vindex`](generated/dask.array.Array.vindex.md#dask.array.Array.vindex).
* Slicing one [`Array`](generated/dask.array.Array.md#dask.array.Array) with a multi-dimensional [`Array`](generated/dask.array.Array.md#dask.array.Array) of ints

<a id="array-slicing-efficiency"></a>

## Efficiency

The normal Dask schedulers are smart enough to compute only those blocks that
are necessary to achieve the desired slicing.  Hence, large operations may be cheap
if only a small output is desired.

In the example below, we create a Dask array with a trillion elements with million
element sized blocks.  We then operate on the entire array and finally slice out
only a portion of the output:

```python
>>> # Trillion element array of ones, in 1000 by 1000 blocks
>>> x = da.ones((1000000, 1000000), chunks=(1000, 1000))

>>> da.exp(x)[:1500, :1500]
...
```

This only needs to compute the top-left four blocks to achieve the result.  We
are slightly wasteful on those blocks where we need only partial results.  Moreover,
we are also a bit wasteful in that we still need to manipulate the Dask graph
with a million or so tasks in it.  This can cause an interactive overhead of a
second or two.

Slicing with concrete indexers (a list of integers, say) has a couple of possible
failure modes that are worth mentioning. First, when you’re indexing a chunked
axis, Dask will typically “match” the chunking on the output.

```python
# Array of ones, chunked along axis 0
>>> a = da.ones((4, 10000, 10000), chunks=(1, -1, -1))
```

If we slice that with a *sorted* sequence of integers, Dask will return one chunk
per input chunk (notice the output chunksize is 1, since the indices `0`
and `1` are in separate chunks in the input).

```pycon
>>> a[[0, 1], :, :]
dask.array<getitem, shape=(2, 10000, 10000), dtype=float64, chunksize=(1, 10000, 10000), chunktype=numpy.ndarray>
```

But what about repeated indices? Dask continues to return one chunk per input chunk,
but if you have many repetitions from the same input chunk, your output chunk could
be much larger.

```python
>>> a[[0] * 15, :, :]
PerformanceWarning: Slicing is producing a large chunk. To accept the large
chunk and silence this warning, set the option
    >>> with dask.config.set({'array.slicing.split_large_chunks': False}):
    ...     array[indexer]

To avoid creating the large chunks, set the option
    >>> with dask.config.set({'array.slicing.split_large_chunks': True}):
    ...     array[indexer]
dask.array<getitem, shape=(15, 10000, 10000), dtype=float64, chunksize=(15, 10000, 10000), chunktype=numpy.ndarray>
```

Previously we had a chunksize of `1` along the first dimension since we selected
just one element from each input chunk. But now we’ve selected 15 elements
from the first chunk, producing a large output chunk.

Dask warns when indexing like this produces a chunk that’s 5x larger
than the `array.chunk-size` config option. You have two options to deal with
that warning:

1. Set `dask.config.set({"array.slicing.split_large_chunks": False})` to
   allow the large chunk and silence the warning.
2. Set `dask.config.set({"array.slicing.split_large_chunks": True})` to
   avoid creating the large chunk in the first place.

The right choice will depend on your downstream operations. See [Chunks](array-chunks.md#array-chunks)
for more on choosing chunk sizes.


# array-sparse.html.md

# Sparse Arrays

By swapping out in-memory NumPy arrays with in-memory sparse arrays, we can
reuse the blocked algorithms of Dask’s Array to achieve parallel and distributed
sparse arrays.

The blocked algorithms in Dask Array normally parallelize around in-memory
NumPy arrays.  However, if another in-memory array library supports the NumPy
interface, then it too can take advantage of Dask Array’s parallel algorithms.
In particular the [sparse](https://github.com/pydata/sparse/) array library
satisfies a subset of the NumPy API and works well with (and is tested against)
Dask Array.

## Example

Say we have a Dask array with mostly zeros:

```python
rng = da.random.default_rng()
x = rng.random((100000, 100000), chunks=(1000, 1000))
x[x < 0.95] = 0
```

We can convert each of these chunks of NumPy arrays into a **sparse.COO** array:

```python
import sparse
s = x.map_blocks(sparse.COO)
```

Now, our array is not composed of many NumPy arrays, but rather of many
sparse arrays.  Semantically, this does not change anything.  Operations that
work will continue to work identically (assuming that the behavior of `numpy`
and `sparse` are identical), but performance characteristics and storage costs
may change significantly:

```python
>>> s.sum(axis=0)[:100].compute()
<COO: shape=(100,), dtype=float64, nnz=100>

>>> _.todense()
array([ 4803.06859272,  4913.94964525,  4877.13266438,  4860.7470773 ,
        4938.94446802,  4849.51326473,  4858.83977856,  4847.81468485,
        ... ])
```

## Requirements

Any in-memory library that copies the NumPy ndarray interface should work here.
The [sparse](https://github.com/pydata/sparse/) library is a minimal
example.  In particular, an in-memory library should implement at least the
following operations:

1. Simple slicing with slices, lists, and elements (for slicing, rechunking,
   reshaping, etc)
2. A `concatenate` function matching the interface of `np.concatenate`.
   This must be registered in `dask.array.core.concatenate_lookup`
3. All ufuncs must support the full ufunc interface, including `dtype=` and
   `out=` parameters (even if they don’t function properly)
4. All reductions must support the full `axis=` and `keepdims=` keywords
   and behave like NumPy in this respect
5. The array class should follow the `__array_priority__` protocol and be
   prepared to respond to other arrays of lower priority
6. If `dot` support is desired, a `tensordot` function matching the
   interface of `np.tensordot` should be registered in
   `dask.array.core.tensordot_lookup`

The implementation of other operations like reshape, transpose, etc.,
should follow standard NumPy conventions regarding shape and dtype.  Not
implementing these is fine; the parallel `dask.array` will err at runtime if
these operations are attempted.

## Mixed Arrays

Dask’s Array supports mixing different kinds of in-memory arrays.  This relies
on the in-memory arrays knowing how to interact with each other when necessary.
When two arrays interact, the functions from the array with the highest
`__array_priority__` will take precedence (for example, for concatenate,
tensordot, etc.).


# array-stack.html.md

# Stack, Concatenate, and Block

Often we have many arrays stored on disk that we want to stack together and
think of as one large array.  This is common with geospatial data in which we
might have many HDF5/NetCDF files on disk, one for every day, but we want to do
operations that span multiple days.

To solve this problem, we use the functions `da.stack`, `da.concatenate`,
and `da.block`.

## Stack

We stack many existing Dask arrays into a new array, creating a new dimension
as we go.

```python
>>> import dask.array as da

>>> arr0 = da.from_array(np.zeros((3, 4)), chunks=(1, 2))
>>> arr1 = da.from_array(np.ones((3, 4)), chunks=(1, 2))

>>> data = [arr0, arr1]

>>> x = da.stack(data, axis=0)
>>> x.shape
(2, 3, 4)

>>> da.stack(data, axis=1).shape
(3, 2, 4)

>>> da.stack(data, axis=-1).shape
(3, 4, 2)
```

This creates a new dimension with length equal to the number of slices

## Concatenate

We concatenate existing arrays into a new array, extending them along an
existing dimension

```python
>>> import dask.array as da
>>> import numpy as np

>>> arr0 = da.from_array(np.zeros((3, 4)), chunks=(1, 2))
>>> arr1 = da.from_array(np.ones((3, 4)), chunks=(1, 2))

>>> data = [arr0, arr1]

>>> x = da.concatenate(data, axis=0)
>>> x.shape
(6, 4)

>>> da.concatenate(data, axis=1).shape
(3, 8)
```

## Block

We can handle a larger variety of cases with `da.block` as it allows
concatenation to be applied over multiple dimensions at once.  This is useful if
your chunks tile a space, for example if small squares tile a larger 2-D plane.

```python
>>> import dask.array as da
>>> import numpy as np

>>> arr0 = da.from_array(np.zeros((3, 4)), chunks=(1, 2))
>>> arr1 = da.from_array(np.ones((3, 4)), chunks=(1, 2))

>>> data = [
...     [arr0, arr1],
...     [arr1, arr0]
... ]

>>> x = da.block(data)
>>> x.shape
(6, 8)
```


# array-stats.html.md

# Stats

Dask Array implements a subset of the [scipy.stats](https://docs.scipy.org/doc/scipy-0.19.0/reference/stats.html) package.

## Statistical Functions

You can calculate various measures of an array including skewness, kurtosis, and arbitrary moments.

```python
>>> from dask.array import stats
>>> rng = da.random.default_rng()
>>> x = rng.beta(1, 1, size=(1000,), chunks=10)
>>> k, s, m = [stats.kurtosis(x), stats.skew(x), stats.moment(x, 5)]
>>> dask.compute(k, s, m)
(1.7612340817172787, -0.064073498030693302, -0.00054523780628304799)
```

## Statistical Tests

You can perform basic statistical tests on Dask arrays.
Each of these tests return a `dask.delayed` wrapping one of the scipy `namedtuple`
results.

```python
>>> rng = da.random.default_rng()
>>> a = rng.uniform(size=(50,), chunks=(25,))
>>> b = a + rng.uniform(low=-0.15, high=0.15, size=(50,), chunks=(25,))
>>> result = stats.ttest_rel(a, b)
>>> result.compute()
Ttest_relResult(statistic=-1.5102104380013242, pvalue=0.13741197274874514)
```


# array.html.md

# Array

Dask Array implements a subset of the NumPy ndarray interface using blocked
algorithms, cutting up the large array into many small arrays. This lets us
compute on arrays larger than memory using all of our cores.  We coordinate
these blocked algorithms using Dask graphs.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/9h_61hXCDuI"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

## Examples

Dask Array is used across a wide variety of applications — anywhere where working
with large array datasets.

This
[Analyzing the National Water Model with Xarray, Dask, and Coiled](https://docs.coiled.io/user_guide/xarray.html?utm_source=dask-docs&utm_medium=array)
example process 6 TB of geospatial data on a cluster using Xarray and Dask Array.
The cluster in this example is deployed with
[Coiled](https://coiled.io/?utm_source=dask-docs&utm_medium=array),
but there are many options for managing and deploying Dask.
See our [Deploy Dask Clusters](deploying.md) documentation for more information on deployment options.

You can also visit [https://examples.dask.org/array.html](https://examples.dask.org/array.html) for a collection of additional examples.

## Design

![Dask arrays coordinate many numpy arrays](images/dask-array.svg)

Dask arrays coordinate many NumPy arrays (or “duck arrays” that are
sufficiently NumPy-like in API such as CuPy or Sparse arrays) arranged into a
grid. These arrays may live on disk or on other machines.

New duck array chunk types (types below Dask on
[NEP-13’s type-casting hierarchy](https://numpy.org/neps/nep-0013-ufunc-overrides.html#type-casting-hierarchy)) can be registered via
[`register_chunk_type()`](generated/dask.array.register_chunk_type.md#dask.array.register_chunk_type). Any other duck array types that are
not registered will be deferred to in binary operations and NumPy
ufuncs/functions (that is, Dask will return `NotImplemented`). Note, however,
that *any* ndarray-like type can be inserted into a Dask Array using
`from_array()`.

## Common Uses

Dask Array is used in fields like atmospheric and oceanographic science, large
scale imaging, genomics, numerical algorithms for optimization or statistics,
and more.

## Scope

Dask arrays support most of the NumPy interface like the following:

- Arithmetic and scalar mathematics: `+, *, exp, log, ...`
- Reductions along axes: `sum(), mean(), std(), sum(axis=0), ...`
- Tensor contractions / dot products / matrix multiply: `tensordot`
- Axis reordering / transpose: `transpose`
- Slicing: `x[:100, 500:100:-2]`
- Fancy indexing along single axes with lists or NumPy arrays: `x[:, [10, 1, 5]]`
- Array protocols like `__array__` and `__array_ufunc__`
- Some linear algebra: `svd, qr, solve, solve_triangular, lstsq`
- …

However, Dask Array does not implement the entire NumPy interface.  Users expecting this
will be disappointed.  Notably, Dask Array lacks the following features:

- Much of `np.linalg` has not been implemented.
  This has been done by a number of excellent BLAS/LAPACK implementations,
  and is the focus of numerous ongoing academic research projects
- Arrays with unknown shapes do not support all operations
- Operations like `sort` which are notoriously
  difficult to do in parallel, and are of somewhat diminished value on very
  large data (you rarely actually need a full sort).
  Often we include parallel-friendly alternatives like `topk`
- Dask Array doesn’t implement operations like `tolist` that would be very
  inefficient for larger datasets. Likewise, it is very inefficient to iterate
  over a Dask array with for loops
- Dask development is driven by immediate need, hence many lesser used
  functions have not been implemented. Community contributions are encouraged

See [the dask.array API](array-api.md) for a more extensive list of
functionality.

## Execution

By default, Dask Array uses the threaded scheduler in order to avoid data
transfer costs, and because NumPy releases the GIL well.  It is also quite
effective on a cluster using the [dask.distributed](https://distributed.dask.org/en/latest/) scheduler.


# bag-api.html.md

# API

## Create Bags

| [`from_sequence`](generated/dask.bag.from_sequence.md#dask.bag.from_sequence)(seq[, partition_size, npartitions])   | Create a dask Bag from Python sequence.    |
|---------------------------------------------------------------------------------------------------------------------|--------------------------------------------|
| [`from_delayed`](generated/dask.bag.from_delayed.md#dask.bag.from_delayed)(values)                                  | Create bag from many dask Delayed objects. |
| [`from_url`](generated/dask.bag.from_url.md#dask.bag.from_url)(urls)                                                | Create a dask Bag from a url.              |
| [`range`](generated/dask.bag.range.md#dask.bag.range)(n, npartitions)                                               | Numbers from zero to n                     |
| [`read_text`](generated/dask.bag.read_text.md#dask.bag.read_text)(urlpath[, blocksize, compression, ...])           | Read lines from text files                 |
| [`read_avro`](generated/dask.bag.read_avro.md#dask.bag.read_avro)(urlpath[, blocksize, ...])                        | Read set of avro files                     |

### From DataFrame

| [`DataFrame.to_bag`](generated/dask.dataframe.DataFrame.to_bag.md#dask.dataframe.DataFrame.to_bag)([index, format])   | Create a Dask Bag from a Series   |
|-----------------------------------------------------------------------------------------------------------------------|-----------------------------------|
| [`Series.to_bag`](generated/dask.dataframe.Series.to_bag.md#dask.dataframe.Series.to_bag)([index, format])            | Create a Dask Bag from a Series   |

## Top-level functions

| [`concat`](generated/dask.bag.concat.md#dask.bag.concat)(bags)                                             | Concatenate many bags together, unioning all elements.                    |
|------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|
| [`map`](generated/dask.bag.map.md#dask.bag.map)(func, \*args, \*\*kwargs)                                  | Apply a function elementwise across one or more bags.                     |
| [`map_partitions`](generated/dask.bag.map_partitions.md#dask.bag.map_partitions)(func, \*args, \*\*kwargs) | Apply a function to every partition across one or more bags.              |
| [`to_textfiles`](generated/dask.bag.to_textfiles.md#dask.bag.to_textfiles)(b, path[, name_function, ...])  | Write dask Bag to disk, one filename per partition, one line per element. |
| [`zip`](generated/dask.bag.zip.md#dask.bag.zip)(\*bags)                                                    | Partition-wise bag zip                                                    |

## Random Sampling

| [`random.choices`](generated/dask.bag.random.choices.md#dask.bag.random.choices)(population[, k, split_every])   | Return a k sized list of elements chosen with replacement.   |
|------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| [`random.sample`](generated/dask.bag.random.sample.md#dask.bag.random.sample)(population, k[, split_every])      | Chooses k unique random elements from a bag.                 |

## Turn Bags into other things

| [`Bag.to_textfiles`](generated/dask.bag.Bag.to_textfiles.md#dask.bag.Bag.to_textfiles)(path[, name_function, ...])      | Write dask Bag to disk, one filename per partition, one line per element.   |
|-------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
| [`Bag.to_dataframe`](generated/dask.bag.Bag.to_dataframe.md#dask.bag.Bag.to_dataframe)([meta, columns, optimize_graph]) | Create Dask Dataframe from a Dask Bag.                                      |
| [`Bag.to_delayed`](generated/dask.bag.Bag.to_delayed.md#dask.bag.Bag.to_delayed)([optimize_graph])                      | Convert into a list of `dask.delayed` objects, one per partition.           |
| [`Bag.to_avro`](generated/dask.bag.Bag.to_avro.md#dask.bag.Bag.to_avro)(filename, schema[, ...])                        | Write bag to set of avro files                                              |

## Bag Methods

| [`Bag`](generated/dask.bag.Bag.md#dask.bag.Bag)(dsk, name, npartitions)                                                 | Parallel collection of Python objects                                     |
|-------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|
| [`Bag.accumulate`](generated/dask.bag.Bag.accumulate.md#dask.bag.Bag.accumulate)(binop[, initial])                      | Repeatedly apply binary function to a sequence, accumulating results.     |
| [`Bag.all`](generated/dask.bag.Bag.all.md#dask.bag.Bag.all)([split_every])                                              | Are all elements truthy?                                                  |
| [`Bag.any`](generated/dask.bag.Bag.any.md#dask.bag.Bag.any)([split_every])                                              | Are any of the elements truthy?                                           |
| [`Bag.compute`](generated/dask.bag.Bag.compute.md#dask.bag.Bag.compute)(\*\*kwargs)                                     | Compute this dask collection                                              |
| [`Bag.count`](generated/dask.bag.Bag.count.md#dask.bag.Bag.count)([split_every])                                        | Count the number of elements.                                             |
| [`Bag.distinct`](generated/dask.bag.Bag.distinct.md#dask.bag.Bag.distinct)([key])                                       | Distinct elements of collection                                           |
| [`Bag.filter`](generated/dask.bag.Bag.filter.md#dask.bag.Bag.filter)(predicate)                                         | Filter elements in collection by a predicate function.                    |
| [`Bag.flatten`](generated/dask.bag.Bag.flatten.md#dask.bag.Bag.flatten)()                                               | Concatenate nested lists into one long list.                              |
| [`Bag.fold`](generated/dask.bag.Bag.fold.md#dask.bag.Bag.fold)(binop[, combine, initial, ...])                          | Parallelizable reduction                                                  |
| [`Bag.foldby`](generated/dask.bag.Bag.foldby.md#dask.bag.Bag.foldby)(key, binop[, initial, combine, ...])               | Combined reduction and groupby.                                           |
| [`Bag.frequencies`](generated/dask.bag.Bag.frequencies.md#dask.bag.Bag.frequencies)([split_every, sort])                | Count number of occurrences of each distinct element.                     |
| [`Bag.groupby`](generated/dask.bag.Bag.groupby.md#dask.bag.Bag.groupby)(grouper[, method, npartitions, ...])            | Group collection by key function                                          |
| [`Bag.join`](generated/dask.bag.Bag.join.md#dask.bag.Bag.join)(other, on_self[, on_other])                              | Joins collection with another collection.                                 |
| [`Bag.map`](generated/dask.bag.Bag.map.md#dask.bag.Bag.map)(func, \*args, \*\*kwargs)                                   | Apply a function elementwise across one or more bags.                     |
| [`Bag.map_partitions`](generated/dask.bag.Bag.map_partitions.md#dask.bag.Bag.map_partitions)(func, \*args, \*\*kwargs)  | Apply a function to every partition across one or more bags.              |
| [`Bag.max`](generated/dask.bag.Bag.max.md#dask.bag.Bag.max)([split_every])                                              | Maximum element                                                           |
| [`Bag.mean`](generated/dask.bag.Bag.mean.md#dask.bag.Bag.mean)()                                                        | Arithmetic mean                                                           |
| [`Bag.min`](generated/dask.bag.Bag.min.md#dask.bag.Bag.min)([split_every])                                              | Minimum element                                                           |
| [`Bag.persist`](generated/dask.bag.Bag.persist.md#dask.bag.Bag.persist)(\*\*kwargs)                                     | Persist this dask collection into memory                                  |
| [`Bag.pluck`](generated/dask.bag.Bag.pluck.md#dask.bag.Bag.pluck)(key[, default])                                       | Select item from all tuples/dicts in collection.                          |
| [`Bag.product`](generated/dask.bag.Bag.product.md#dask.bag.Bag.product)(other)                                          | Cartesian product between two bags.                                       |
| [`Bag.reduction`](generated/dask.bag.Bag.reduction.md#dask.bag.Bag.reduction)(perpartition, aggregate[, ...])           | Reduce collection with reduction operators.                               |
| [`Bag.random_sample`](generated/dask.bag.Bag.random_sample.md#dask.bag.Bag.random_sample)(prob[, random_state])         | Return elements from bag with probability of `prob`.                      |
| [`Bag.remove`](generated/dask.bag.Bag.remove.md#dask.bag.Bag.remove)(predicate)                                         | Remove elements in collection that match predicate.                       |
| [`Bag.repartition`](generated/dask.bag.Bag.repartition.md#dask.bag.Bag.repartition)([npartitions, partition_size])      | Repartition Bag across new divisions.                                     |
| [`Bag.starmap`](generated/dask.bag.Bag.starmap.md#dask.bag.Bag.starmap)(func, \*\*kwargs)                               | Apply a function using argument tuples from the given bag.                |
| [`Bag.std`](generated/dask.bag.Bag.std.md#dask.bag.Bag.std)([ddof])                                                     | Standard deviation                                                        |
| [`Bag.sum`](generated/dask.bag.Bag.sum.md#dask.bag.Bag.sum)([split_every])                                              | Sum all elements                                                          |
| [`Bag.take`](generated/dask.bag.Bag.take.md#dask.bag.Bag.take)(k[, npartitions, compute, warn])                         | Take the first k elements.                                                |
| [`Bag.to_avro`](generated/dask.bag.Bag.to_avro.md#dask.bag.Bag.to_avro)(filename, schema[, ...])                        | Write bag to set of avro files                                            |
| [`Bag.to_dataframe`](generated/dask.bag.Bag.to_dataframe.md#dask.bag.Bag.to_dataframe)([meta, columns, optimize_graph]) | Create Dask Dataframe from a Dask Bag.                                    |
| [`Bag.to_delayed`](generated/dask.bag.Bag.to_delayed.md#dask.bag.Bag.to_delayed)([optimize_graph])                      | Convert into a list of `dask.delayed` objects, one per partition.         |
| [`Bag.to_textfiles`](generated/dask.bag.Bag.to_textfiles.md#dask.bag.Bag.to_textfiles)(path[, name_function, ...])      | Write dask Bag to disk, one filename per partition, one line per element. |
| [`Bag.topk`](generated/dask.bag.Bag.topk.md#dask.bag.Bag.topk)(k[, key, split_every])                                   | K largest elements in collection                                          |
| [`Bag.var`](generated/dask.bag.Bag.var.md#dask.bag.Bag.var)([ddof])                                                     | Variance                                                                  |
| [`Bag.visualize`](generated/dask.bag.Bag.visualize.md#dask.bag.Bag.visualize)([filename, format, optimize_graph])       | Render the computation of this object's task graph using graphviz.        |

## Item Methods

| [`Item`](generated/dask.bag.Item.md#dask.bag.Item)(dsk, key[, layer])                                     |                                                                    |
|-----------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
| [`Item.apply`](generated/dask.bag.Item.apply.md#dask.bag.Item.apply)(func)                                |                                                                    |
| [`Item.compute`](generated/dask.bag.Item.compute.md#dask.bag.Item.compute)(\*\*kwargs)                    | Compute this dask collection                                       |
| [`Item.from_delayed`](generated/dask.bag.Item.from_delayed.md#dask.bag.Item.from_delayed)(value)          | Create bag item from a dask.delayed value.                         |
| [`Item.persist`](generated/dask.bag.Item.persist.md#dask.bag.Item.persist)(\*\*kwargs)                    | Persist this dask collection into memory                           |
| [`Item.to_delayed`](generated/dask.bag.Item.to_delayed.md#dask.bag.Item.to_delayed)([optimize_graph])     | Convert into a `dask.delayed` object.                              |
| [`Item.visualize`](generated/dask.bag.Item.visualize.md#dask.bag.Item.visualize)([filename, format, ...]) | Render the computation of this object's task graph using graphviz. |


# bag-creation.html.md

# Create Dask Bags

There are several ways to create Dask bags around your data:

## `db.from_sequence`

You can create a bag from an existing Python iterable:

```python
>>> import dask.bag as db
>>> b = db.from_sequence([1, 2, 3, 4, 5, 6])
```

You can control the number of partitions into which this data is binned:

```python
>>> b = db.from_sequence([1, 2, 3, 4, 5, 6], npartitions=2)
```

This controls the granularity of the parallelism that you expose.  By default,
Dask will try to partition your data into about 100 partitions.

IMPORTANT: do not load your data into Python and then load that data into a
Dask bag.  Instead, use Dask Bag to load your data.  This
parallelizes the loading step and reduces inter-worker communication:

```python
>>> b = db.from_sequence(['1.dat', '2.dat', ...]).map(load_from_filename)
```

## `db.read_text`

Dask Bag can load data directly from text files.  You can pass either a
single file name, a list of file names, or a globstring.  The resulting
bag will have one item per line and one file per partition:

```python
>>> b = db.read_text('myfile.txt')
>>> b = db.read_text(['myfile.1.txt', 'myfile.2.txt', ...])
>>> b = db.read_text('myfile.*.txt')
```

This handles standard compression libraries like `gzip`, `bz2`, `xz`, or
any easily installed compression library that has a file-like object.
Compression will be inferred by the file name extension, or by using the
`compression='gzip'` keyword:

```python
>>> b = db.read_text('myfile.*.txt.gz')
```

The resulting items in the bag are strings. If you have encoded data like
line-delimited JSON, then you may want to map a decoding or load function across
the bag:

```python
>>> import json
>>> b = db.read_text('myfile.*.json').map(json.loads)
```

Or do string munging tasks.  For convenience, there is a string namespace
attached directly to bags with `.str.methodname`:

```python
>>> b = db.read_text('myfile.*.csv').str.strip().str.split(',')
```

## `db.read_avro`

Dask Bag can read binary files in the [Avro](https://avro.apache.org/docs/1.8.2/) format if [fastavro](https://fastavro.readthedocs.io) is installed.
A bag can be made from one or more files, with optional chunking within files.
The resulting bag will have one item per Avro record, which will be a dictionary
of the form given by the Avro schema.  There will be at least one partition per
input file:

```python
>>> b = db.read_avro('datafile.avro')
>>> b = db.read_avro('data.*.avro')
```

By default, Dask will split data files into chunks of approximately `blocksize`
bytes in size.  The actual blocks you would get depend on the internal blocking
of the file.

For files that are compressed after creation (this is not the same as the internal
“codec” used by Avro), no chunking should be used, and there will be exactly one
partition per file:

```python
> b = bd.read_avro('compressed.*.avro.gz', blocksize=None, compression='gzip')
```

## `db.from_delayed`

You can construct a Dask bag from [dask.delayed](delayed.md) values using the
`db.from_delayed` function.  For more information, see
[documentation on using dask.delayed with collections](delayed-collections.md).

# Store Dask Bags

## In Memory

You can convert a Dask bag to a list or Python iterable by calling `compute()`
or by converting the object into a list:

```python
>>> result = b.compute()
or
>>> result = list(b)
```

## To Text Files

You can convert a Dask bag into a sequence of files on disk by calling the
`.to_textfiles()` method:

### dask.bag.core.to_textfiles(b, path, name_function=None, compression='infer', encoding='utf-8', compute=True, storage_options=None, last_endline=False, \*\*kwargs)

Write dask Bag to disk, one filename per partition, one line per element.

**Paths**: This will create one file for each partition in your bag. You
can specify the filenames in a variety of ways.

Use a globstring

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz')
```

The \* will be replaced by the increasing sequence 1, 2, …

```default
/path/to/data/0.json.gz
/path/to/data/1.json.gz
```

Use a globstring and a `name_function=` keyword argument.  The
name_function function should expect an integer and produce a string.
Strings produced by name_function must preserve the order of their
respective partition indices.

```pycon
>>> from datetime import date, timedelta
>>> def name(i):
...     return str(date(2015, 1, 1) + i * timedelta(days=1))
```

```pycon
>>> name(0)
'2015-01-01'
>>> name(15)
'2015-01-16'
```

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz', name_function=name)
```

```default
/path/to/data/2015-01-01.json.gz
/path/to/data/2015-01-02.json.gz
...
```

You can also provide an explicit list of paths.

```pycon
>>> paths = ['/path/to/data/alice.json.gz', '/path/to/data/bob.json.gz', ...]
>>> b.to_textfiles(paths)
```

**Compression**: Filenames with extensions corresponding to known
compression algorithms (gz, bz2) will be compressed accordingly.

**Bag Contents**: The bag calling `to_textfiles` must be a bag of
text strings. For example, a bag of dictionaries could be written to
JSON text files by mapping `json.dumps` on to the bag first, and
then calling `to_textfiles` :

```pycon
>>> b_dict.map(json.dumps).to_textfiles("/path/to/data/*.json")
```

**Last endline**: By default the last line does not end with a newline
character. Pass `last_endline=True` to invert the default.

<!-- !! processed by numpydoc !! -->

## To Avro

Dask bags can be written directly to Avro binary format using [fastavro](https://fastavro.readthedocs.io). One file
will be written per bag partition. This requires the user to provide a fully-specified
schema dictionary (see the docstring of the `.to_avro()` method).

### dask.bag.avro.to_avro(b, filename, schema, name_function=None, storage_options=None, codec='null', sync_interval=16000, metadata=None, compute=True, \*\*kwargs)

Write bag to set of avro files

The schema is a complex dictionary describing the data, see
[https://avro.apache.org/docs/1.8.2/gettingstartedpython.html#Defining+a+schema](https://avro.apache.org/docs/1.8.2/gettingstartedpython.html#Defining+a+schema)
and [https://fastavro.readthedocs.io/en/latest/writer.html](https://fastavro.readthedocs.io/en/latest/writer.html) .
Its structure is as follows:

```default
{'name': 'Test',
 'namespace': 'Test',
 'doc': 'Descriptive text',
 'type': 'record',
 'fields': [
    {'name': 'a', 'type': 'int'},
 ]}
```

where the “name” field is required, but “namespace” and “doc” are optional
descriptors; “type” must always be “record”. The list of fields should
have an entry for every key of the input records, and the types are
like the primitive, complex or logical types of the Avro spec
( [https://avro.apache.org/docs/1.8.2/spec.html](https://avro.apache.org/docs/1.8.2/spec.html) ).

Results in one avro file per input partition.

* **Parameters:**
  **b: dask.bag.Bag**

  **filename: list of str or str**
  : Filenames to write to. If a list, number must match the number of
    partitions. If a string, must include a glob character “\*”, which will
    be expanded using name_function

  **schema: dict**
  : Avro schema dictionary, see above

  **name_function: None or callable**
  : Expands integers into strings, see
    `dask.bytes.utils.build_name_function`

  **storage_options: None or dict**
  : Extra key/value options to pass to the backend file-system

  **codec: ‘null’, ‘deflate’, or ‘snappy’**
  : Compression algorithm

  **sync_interval: int**
  : Number of records to include in each block within a file

  **metadata: None or dict**
  : Included in the file header

  **compute: bool**
  : If True, files are written immediately, and function blocks. If False,
    returns delayed objects, which can be computed by the user where
    convenient.

  **kwargs: passed to compute(), if compute=True**

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([{'name': 'Alice', 'value': 100},
...                       {'name': 'Bob', 'value': 200}])
>>> schema = {'name': 'People', 'doc': "Set of people's scores",
...           'type': 'record',
...           'fields': [
...               {'name': 'name', 'type': 'string'},
...               {'name': 'value', 'type': 'int'}]}
>>> b.to_avro('my-data.*.avro', schema)
['my-data.0.avro', 'my-data.1.avro']
```

<!-- !! processed by numpydoc !! -->

## To DataFrames

You can convert a Dask bag into a [Dask DataFrame](dataframe.md) and use
those storage solutions.

#### Bag.to_dataframe(meta=None, columns=None, optimize_graph=True)

Create Dask Dataframe from a Dask Bag.

Bag should contain tuples, dict records, or scalars.

Index will not be particularly meaningful.  Use `reindex` afterwards
if necessary.

* **Parameters:**
  **meta**
  : An empty `pd.DataFrame` that matches the dtypes and column names
    of the output. This metadata is necessary for many algorithms in
    dask dataframe to work.  For ease of use, some alternative inputs
    are also available. Instead of a `DataFrame`, a `dict` of
    `{name: dtype}` or iterable of `(name, dtype)` can be provided.
    If not provided or a list, a single element from the first
    partition will be computed, triggering a potentially expensive call
    to `compute`. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

  **columns**
  : Column names to use. If the passed data do not have names
    associated with them, this argument provides names for the columns.
    Otherwise this argument indicates the order of the columns in the
    result (any names not found in the data will become all-NA
    columns).  Note that if `meta` is provided, column names will be
    taken from there and this parameter is invalid.

  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    [`dask.dataframe.DataFrame`](generated/dask.dataframe.DataFrame.md#dask.dataframe.DataFrame).

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([{'name': 'Alice',   'balance': 100},
...                       {'name': 'Bob',     'balance': 200},
...                       {'name': 'Charlie', 'balance': 300}],
...                      npartitions=2)
>>> df = b.to_dataframe()
```

```pycon
>>> df.compute()
      name  balance
0    Alice      100
1      Bob      200
0  Charlie      300
```

<!-- !! processed by numpydoc !! -->

## To Delayed Values

You can convert a Dask bag into a list of [Dask delayed values](delayed.md)
and custom storage solutions from there.

#### Bag.to_delayed(optimize_graph=True)

Convert into a list of `dask.delayed` objects, one per partition.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

#### SEE ALSO
[`dask.bag.from_delayed`](generated/dask.bag.from_delayed.md#dask.bag.from_delayed)

<!-- !! processed by numpydoc !! -->


# bag.html.md

# Bag

Dask Bag implements operations like `map`, `filter`, `fold`, and
`groupby` on collections of generic Python objects.  It does this in parallel with a
small memory footprint using Python iterators.  It is similar to a parallel
version of [PyToolz](https://toolz.readthedocs.io/en/latest/) or a Pythonic version of the [PySpark RDD](https://spark.apache.org/docs/latest/api/python).

## Examples

Visit [https://examples.dask.org/bag.html](https://examples.dask.org/bag.html) to see and run examples using Dask Bag.

## Design

Dask bags coordinate many Python lists or Iterators, each of which forms a
partition of a larger collection.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/-qIiJ1XtSv0"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

## Common Uses

Dask bags are often used to parallelize simple computations on unstructured or
semi-structured data like text data, log files, JSON records, or user defined
Python objects.

## Execution

Execution on bags provides two benefits:

1. Parallel: data is split up, allowing multiple cores or machines to execute
   in parallel
2. Iterating: data processes lazily, allowing smooth execution of
   larger-than-memory data, even on a single machine within a single partition

### Default scheduler

By default, `dask.bag` uses `dask.multiprocessing` for computation.  As a
benefit, Dask bypasses the [GIL](https://docs.python.org/3/glossary.html#term-gil) and uses multiple cores on pure Python objects.
As a drawback, Dask Bag doesn’t perform well on computations that include a
great deal of inter-worker communication.  For common operations this is rarely
an issue as most Dask Bag workflows are embarrassingly parallel or result in
reductions with little data moving between workers.

### Shuffle

Some operations, like `groupby`, require substantial inter-worker
communication. On a single machine, Dask uses [partd](https://github.com/mrocklin/partd) to perform efficient,
parallel, spill-to-disk shuffles. When working in a cluster, Dask uses a task
based shuffle.

These shuffle operations are expensive and better handled by projects like
`dask.dataframe`. It is best to use `dask.bag` to clean and process data,
then transform it into an array or DataFrame before embarking on the more
complex operations that require shuffle steps.

## Known Limitations

Bags provide very general computation (any Python function).  This generality
comes at cost.  Bags have the following known limitations:

1. By default, they rely on the multiprocessing scheduler, which has its own
   set of known limitations (see [Shared Memory](shared.md))
2. Bags are immutable and so you can not change individual elements
3. Bag operations tend to be slower than array/DataFrame computations in the
   same way that standard Python containers tend to be slower than NumPy
   arrays and Pandas DataFrames
4. Bag’s `groupby` is slow.  You should try to use Bag’s `foldby` if possible.
   Using `foldby` requires more thought though

## Name

*Bag* is the mathematical name for an unordered collection allowing repeats. It
is a friendly synonym to [multiset](https://en.wikipedia.org/wiki/Bag_(mathematics)). A bag, or a multiset, is a generalization of
the concept of a set that, unlike a set, allows multiple instances of the
multiset’s elements:

* `list`: *ordered* collection *with repeats*, `[1, 2, 3, 2]`
* `set`: *unordered* collection *without repeats*,  `{1, 2, 3}`
* `bag`: *unordered* collection *with repeats*, `{1, 2, 2, 3}`

So, a bag is like a list, but it doesn’t guarantee an ordering among elements.
There can be repeated elements but you can’t ask for the ith element.


# best-practices.html.md

# Dask Best Practices

It is easy to get started with Dask’s APIs, but using them *well* requires some
experience. This page contains suggestions for Dask best practices and
includes solutions to common Dask problems.

This document specifically focuses on best practices that are shared among all
of the Dask APIs.  Readers may first want to investigate one of the
API-specific Best Practices documents first.

- [Arrays](array-best-practices.md)
- [DataFrames](dataframe-best-practices.md)
- [Delayed](delayed-best-practices.md)

<iframe width="560" height="315" src="https://www.youtube.com/embed/Gvuf6gSGA9M" title="YouTube video player" frameborder="0" style="margin: 0 auto 20px auto; display: block;" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

## Start Small

Parallelism brings extra complexity and overhead.
Sometimes it’s necessary for larger problems, but often it’s not.
Before adding a parallel computing system like Dask to your workload you may
want to first try some alternatives:

- **Use better algorithms or data structures**:  NumPy, pandas, Scikit-learn
  may have faster functions for what you’re trying to do.  It may be worth
  consulting with an expert or reading through their docs again to find a
  better pre-built algorithm.
- **Better file formats**:  Efficient binary formats that support random
  access can often help you manage larger-than-memory datasets efficiently and
  simply.  See the [Store Data Efficiently]() section below.
- **Compiled code**:  Compiling your Python code with Numba or Cython might
  make parallelism unnecessary.  Or you might use the multi-core parallelism
  available within those libraries.
- **Sampling**:  Even if you have a lot of data, there might not be much
  advantage from using all of it.  By sampling intelligently you might be able
  to derive the same insight from a much more manageable subset.
- **Profile**:  If you’re trying to speed up slow code it’s important that
  you first understand why it is slow.  Modest time investments in profiling
  your code can help you to identify what is slowing you down.  This
  information can help you make better decisions about if parallelism is likely
  to help, or if other approaches are likely to be more effective.

## Use The Dashboard

Dask’s dashboard helps you to understand the state of your workers.
This information can help to guide you to efficient solutions.
In parallel and distributed computing there are new costs to be aware of and so
your old intuition may no longer be true.  Working with the dashboard can help
you relearn about what is fast and slow and how to deal with it.

See [Documentation on Dask’s dashboard](dashboard.md) for more
information.

## Avoid Very Large Partitions

Your chunks of data should be small enough so that many of them fit in a
worker’s available memory at once.  You often control this when you select
partition size in Dask DataFrame (see [DataFrame Partitions](dataframe-design.md#dataframe-design-partitions))
or chunk size in Dask Array (see [Array Chunks](array-chunks.md)).

Dask will likely manipulate as many chunks in parallel on one machine as you
have cores on that machine.  So if you have 1 GB chunks and ten
cores, then Dask is likely to use *at least* 10 GB of memory.  Additionally,
it’s common for Dask to have 2-3 times as many chunks available to work on so
that it always has something to work on.

If you have a machine with 100 GB and 10 cores, then you might want to choose
chunks in the 1GB range.  You have space for ten chunks per core which gives
Dask a healthy margin, without having tasks that are too small.

Note that you also want to avoid chunk sizes that are too small.  See the next
section for details. For a more detailed guide to choosing chunk sizes for
Dask Arrays, see this blog post on [Choosing good chunk sizes](https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes).

## Avoid Very Large Graphs

Dask workloads are composed of *tasks*.
A task is a Python function, like `np.sum` applied onto a Python object,
like a pandas DataFrame or NumPy array.  If you are working with Dask
collections with many partitions, then every operation you do, like `x + 1`
likely generates many tasks, at least as many as partitions in your collection.

Every task comes with some overhead.  This is somewhere between 200us and 1ms.
If you have a computation with thousands of tasks this is fine, there will be
about a second of overhead, and that may not trouble you.

However when you have very large graphs with millions of tasks then this may
become troublesome, both because overhead is now in the 10 minutes to hours
range, and also because the overhead of dealing with such a large graph can
start to overwhelm the scheduler.

You can build smaller graphs by:

- **Increasing your chunk size:**  If you have a 1,000 GB of data and are using
  10 MB chunks, then you have 100,000 partitions.  Every operation on such
  a collection will generate at least 100,000 tasks.

  However if you increase your chunksize to 1 GB or even a few GB then you
  reduce the overhead by orders of magnitude.  This requires that your
  workers have much more than 1 GB of memory, but that’s typical for larger
  workloads.
- **Fusing operations together:** Dask will do a bit of this on its own, but you
  can help it.  If you have a very complex operation with dozens of
  sub-operations, maybe you can pack that into a single Python function
  and use a function like `da.map_blocks` or `dd.map_partitions`.

  In general, the more administrative work you can move into your functions
  the better.  That way the Dask scheduler doesn’t need to think about all
  of the fine-grained operations.
- **Breaking up your computation:** For very large workloads you may also want to
  try sending smaller chunks to Dask at a time.  For example if you’re
  processing a petabyte of data but find that Dask is only happy with 100
  TB, maybe you can break up your computation into ten pieces and submit
  them one after the other.

## Learn Techniques For Customization

The high level Dask collections (array, DataFrame, bag) include common
operations that follow standard Python APIs from NumPy and pandas.
However, many Python workloads are complex and may require operations that are
not included in these high level APIs.

Fortunately, there are many options to support custom workloads:

- All collections have a `map_partitions` or `map_blocks` function, that
  applies a user provided function across every pandas DataFrame or NumPy array
  in the collection.  Because Dask collections are made up of normal Python
  objects, it’s often quite easy to map custom functions across partitions of a
  dataset without much modification.
  ```python
  df.map_partitions(my_custom_func)
  ```
- More complex `map_*` functions.  Sometimes your custom behavior isn’t
  embarrassingly parallel, but requires more advanced communication.  For
  example maybe you need to communicate a little bit of information from one
  partition to the next, or maybe you want to build a custom aggregation.

  Dask collections include methods for these as well.
- For even more complex workloads you can convert your collections into
  individual blocks, and arrange those blocks as you like using Dask Delayed.
  There is usually a `to_delayed` method on every collection.

| [`map_partitions`](generated/dask.dataframe.map_partitions.md#dask.dataframe.map_partitions)(func, \*args[, meta, ...])   | Apply Python function on each DataFrame partition.                         |
|---------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|
| [`map_overlap`](generated/dask.dataframe.map_overlap.md#dask.dataframe.map_overlap)(func, df, before, after, \*args)      | Apply a function to each partition, sharing rows with adjacent partitions. |
| [`groupby.Aggregation`](generated/dask.dataframe.Aggregation.md#dask.dataframe.Aggregation)(name, chunk, agg[, finalize]) | User defined groupby-aggregation.                                          |

| [`blockwise`](generated/dask.array.blockwise.md#dask.array.blockwise)(func, out_ind, \*args[, name, ...])         | Tensor operation: Generalized inner and outer products   |
|-------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|
| [`map_blocks`](generated/dask.array.map_blocks.md#dask.array.map_blocks)(func, \*args[, name, token, ...])        | Map a function across all blocks of a dask array.        |
| [`map_overlap`](generated/dask.array.map_overlap.md#dask.array.map_overlap)(func, \*args[, depth, boundary, ...]) | Map a function over blocks of arrays with some overlap   |
| [`reduction`](generated/dask.array.reduction.md#dask.array.reduction)(x, chunk, aggregate[, axis, ...])           | General version of reductions                            |

## Store Data Efficiently

As your ability to compute increases you will likely find that data access and
I/O take up a larger portion of your total time.  Additionally, parallel
computing will often add new constraints to how your store your data,
particularly around providing random access to blocks of your data that are in
line with how you plan to compute on it.

For example:

- For compression you’ll probably find that you drop gzip and bz2, and embrace
  newer systems like lz4, snappy, and Z-Standard that provide better
  performance and random access.
- For storage formats you may find that you want self-describing formats that
  are optimized for random access, metadata storage, and binary encoding like
  [Parquet](https://parquet.apache.org/), [ORC](https://orc.apache.org/),
  [Zarr](https://zarr.readthedocs.io/en/stable/),
  [HDF5](https://portal.hdfgroup.org/display/HDF5/HDF5), and
  [GeoTIFF](https://en.wikipedia.org/wiki/GeoTIFF).
- When working on the cloud you may find that some older formats like HDF5 may
  not work as well.
- You may want to partition or chunk your data in ways that align well to
  common queries.  In Dask DataFrame this might mean choosing a column to
  sort by for fast selection and joins.  For Dask Array this might mean
  choosing chunk sizes that are aligned with your access patterns and
  algorithms.

## Processes, Threads and VM sizes

If you’re doing mostly numeric work with Numpy, pandas, Scikit-learn, Numba,
and other libraries that release the [GIL](https://docs.python.org/3/glossary.html#term-global-interpreter-lock), then use mostly threads.  If you’re
doing work on text data or Python collections like lists and dicts then use
mostly processes.

If you’re on larger machines with a high thread count (much greater than 4),
then you should probably split things up into at least a few processes
regardless. Python can be highly productive with about 4 threads per process
with numeric work, but not 50 threads.

This is advise that generalizes to cloud computing and picking appropriate VM
instance sizes. There is a lot of nuance to picking the \_perfect_ instance but
a good starting point is a 1:4 CPU to RAM ratio with one Worker instance per VM. You can adjust from there given your workload.

For more information on threads, processes, and how to configure them in Dask, see
[the scheduler documentation](scheduling.md).

## Load Data with Dask

A common anti-pattern we see is people creating large Python objects like a DataFrame
or an Array on the client (i.e. their local machine) outside of Dask and then embedding
them into the computation. This means that Dask has to send these objects over the network
multiple times instead of just passing pointers to the data.

This incurs a lot of overhead and slows down a computation quite significantly, especially
so if the network connection between the client and the scheduler is slow. It can
also overload the scheduler so that it errors with out of memory errors. Instead, you
should use Dask methods to load the data and use Dask to control the results.

Here are some common patterns to avoid and nicer alternatives:

### DataFrames

We are using Dask to read a parquet dataset before appending a set of pandas
DataFrames to it. We are loading the csv files into memory before sending the
data to Dask.

```python
ddf = dd.read_parquet(...)

pandas_dfs = []
for fn in filenames:
    pandas_dfs(pandas.read_csv(fn))     # Read locally with pandas
ddf = dd.concat([ddf] + pandas_dfs)     # Give to Dask
```

Instead, we can use Dask to read the csv files directly, keeping all data
on the cluster.

```python
ddf = dd.read_parquet(...)
ddf2 = dd.read_csv(filenames)
ddf = dd.concat([ddf, ddf2])
```

### Arrays

We are using NumPy to create an in-memory array before handing it over to
Dask, forcing Dask to embed the array into the task graph instead of handling
pointers to the data.

```python
f = h5py.File(...)

x = np.asarray(f["x"])  # Get data as a NumPy array locally
x = da.from_array(x)   # Hand NumPy array to Dask
```

Instead, we can use Dask to read the file directly, keeping all data
on the cluster.

```python
f = h5py.File(...)
x = da.from_array(f["x"])  # Let Dask do the reading
```

### Delayed

We are using pandas to read a large CSV file before building a Graph
with delayed to parallelize a computation on the data.

```python
@dask.delayed
def process(a, b):
    ...

df = pandas.read_csv("some-large-file.csv")  # Create large object locally
results = []
for item in L:
    result = process(item, df)  # include df in every delayed call
    results.append(result)
```

Instead, we can use delayed to read the data as well. This avoid embedding the
large file into the graph, Dask can just pass a reference to the delayed
object around.

```python
@dask.delayed
def process(a, b):
   ...

df = dask.delayed(pandas.read_csv)("some-large-file.csv")  # Let Dask build object
results = []
for item in L:
   result = process(item, df)  # include pointer to df in every delayed call
   results.append(result)
```

Embedding large objects like pandas DataFrames or Arrays into the computation is a
frequent pain point for Dask users. It adds a significant delay until the scheduler
has received and is able to start the computation and stresses the scheduler during
the computation.

Using Dask to load these objects instead avoids these issues and improves the performance
of a computation significantly.

## Avoid calling compute repeatedly

Calling `compute` will block the execution on the client until the Dask computation
completes. A pattern we regularly see is users calling `compute` in a loop or sequentially
on slightly different queries.

This prohibits Dask from parallelizing different computations on the cluster and from
sharing intermediate results between different queries.

```python
foo = ...
results = []
for i in range(...):
     results.append(foo.select(...).compute())
```

This holds execution every time that the iteration arrives at the compute call computing
one query at a time.

```python
foo = ...
results = []
for i in range(...):
     results.append(foo.select(...))  # no compute here
results = dask.compute(*results)
```

This allows Dask to compute the shared parts of the computation (like the
`foo` object above) only once, rather than once per `compute` call and allows
Dask to parallelize across the different selects as well instead of running
them sequentially.


# caching.html.md

# Opportunistic Caching

Dask usually removes intermediate values as quickly as possible in order to
make space for more data to flow through your computation.  However, in some
cases, we may want to hold onto intermediate values, because they might be
useful for future computations in an interactive session.

We need to balance the following concerns:

1. Intermediate results might be useful in future unknown computations
2. Intermediate results also fill up memory, reducing space for the rest of our
   current computation

Negotiating between these two concerns helps us to leverage the memory that we
have available to speed up future, unanticipated computations.  Which intermediate results
should we keep?

This document explains an experimental, opportunistic caching mechanism that automatically
picks out and stores useful tasks.

## Motivating Example

Consider computing the maximum value of a column in a CSV file:

```python
>>> import dask.dataframe as dd
>>> df = dd.read_csv('myfile.csv')
>>> df.columns
['first-name', 'last-name', 'amount', 'id', 'timestamp']

>>> df.amount.max().compute()
1000
```

Even though our full dataset may be too large to fit in memory, the single
`df.amount` column may be small enough to hold in memory just in case it
might be useful in the future.  This is often the case during data exploration,
because we investigate the same subset of our data repeatedly before moving on.

For example, we may now want to find the minimum of the amount column:

```python
>>> df.amount.min().compute()
-1000
```

Under normal operations, this would need to read through the entire CSV file over
again.  This is somewhat wasteful and stymies interactive data exploration.

## Two Simple Solutions

If we know ahead of time that we want both the maximum and minimum, we can
compute them simultaneously.  Dask will share intermediates intelligently,
reading through the dataset only once:

```python
>>> dd.compute(df.amount.max(), df.amount.min())
(1000, -1000)
```

If we know that this column fits in memory, then we can also explicitly
compute the column and then continue forward with straight Pandas:

```python
>>> amount = df.amount.compute()
>>> amount.max()
1000
>>> amount.min()
-1000
```

If either of these solutions work for you, great.  Otherwise, continue on for a third approach.

## Automatic Opportunistic Caching

Another approach is to watch *all* intermediate computations, and *guess* which
ones might be valuable to keep for the future.  Dask has an *opportunistic
caching mechanism* that stores intermediate tasks that show the following
characteristics:

1. Expensive to compute
2. Cheap to store
3. Frequently used

We can activate a fixed sized cache as a [callback](diagnostics-local.html#custom-callbacks):

```python
>>> from dask.cache import Cache
>>> cache = Cache(2e9)  # Leverage two gigabytes of memory
>>> cache.register()    # Turn cache on globally
```

Now the cache will watch every small part of the computation and judge the
value of that part based on the three characteristics listed above (expensive
to compute, cheap to store, and frequently used).

Dask will hold on to 2GB of the
best intermediate results it can find, evicting older results as better results
come in.  If the `df.amount` column fits in 2GB, then probably all of it will
be stored while we keep working on it.

If we start work on something else,
then the `df.amount` column will likely be evicted to make space for other
more timely results:

```python
>>> df.amount.max().compute()  # slow the first time
1000
>>> df.amount.min().compute()  # fast because df.amount is in the cache
-1000
>>> df.id.nunique().compute()  # starts to push out df.amount from cache
```

## Cache tasks, not expressions

This caching happens at the low-level scheduling layer, not the high-level
Dask DataFrame or Dask Array layer.  We don’t explicitly cache the column
`df.amount`.  Instead, we cache the hundreds of small pieces of that column
that form the dask graph.  It could be that we end up caching only a fraction
of the column.

This means that the opportunistic caching mechanism described above works for *all* Dask
computations, as long as those computations employ a consistent naming scheme
(as all of Dask DataFrame, Dask Array, and Dask Delayed do).

You can see which tasks are held by the cache by inspecting the following
attributes of the cache object:

```python
>>> cache.cache.data
<stored values>
>>> cache.cache.heap.heap
<scores of items in cache>
>>> cache.cache.nbytes
<number of bytes per item in cache>
```

The cache object is powered by [cachey](https://github.com/blaze/cachey), a tiny library for opportunistic
caching.

## Disclaimer

Opportunistic caching is not available when using the distributed scheduler.

Restricting your cache to a fixed size like 2GB requires Dask to accurately count
the size of each of our objects in memory.  This can be tricky, particularly
for Pythonic objects like lists and tuples, and for DataFrames that contain
object dtypes.

It is entirely possible that the caching mechanism will
*undercount* the size of objects, causing it to use up more memory than
anticipated, which can lead to blowing up RAM and crashing your session.


# changelog.html.md

# Changelog

#### NOTE
This is not exhaustive. For an exhaustive list of changes, see the git log.

<a id="v2026-7-0"></a>

## 2026.7.0

### Highlights

- Support composite expressions with the `__dask_exprs__` protocol and avoid graph materialization when checking expr-wrapper Dask collections ([dask#12457](https://github.com/dask/dask/pull/12457), [dask#12476](https://github.com/dask/dask/pull/12476)) [Matthew Rocklin](https://github.com/mrocklin)
- Update compatibility for NumPy 2.5 type stubs, pandas 3 nightlies, pytest 9.1, and free-threaded msgpack builds ([dask#12483](https://github.com/dask/dask/pull/12483), [dask#12469](https://github.com/dask/dask/pull/12469), [dask#12465](https://github.com/dask/dask/pull/12465), [dask#12467](https://github.com/dask/dask/pull/12467), [distributed#9312](https://github.com/dask/distributed/pull/9312), [distributed#9304](https://github.com/dask/distributed/pull/9304), [distributed#9303](https://github.com/dask/distributed/pull/9303), [distributed#9302](https://github.com/dask/distributed/pull/9302)) [Guido Imperiale](https://github.com/crusaderky)
- Return all workers from `scheduler_info()` by default ([distributed#9308](https://github.com/dask/distributed/pull/9308)) [Matthew Rocklin](https://github.com/mrocklin)
- Keep `Client.scatter` from unpacking custom containers ([distributed#9298](https://github.com/dask/distributed/pull/9298)) [Adrin Jalali](https://github.com/adrinjalali)
- Fix Distributed worker startup and aborted TCP communication races ([distributed#9313](https://github.com/dask/distributed/pull/9313), [distributed#9315](https://github.com/dask/distributed/pull/9315)) [Guido Imperiale](https://github.com/crusaderky)

### Additional changes

- Upgrade pixi ([dask#12460](https://github.com/dask/dask/pull/12460)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix release publisher warnings ([dask#12456](https://github.com/dask/dask/pull/12456), [distributed#9299](https://github.com/dask/distributed/pull/9299)) [Matthew Rocklin](https://github.com/mrocklin)
- XFAIL `test_concat_categorical` (again) ([dask#12463](https://github.com/dask/dask/pull/12463)) [Guido Imperiale](https://github.com/crusaderky)
- CI: Suppress warning about slow disk ([dask#12464](https://github.com/dask/dask/pull/12464)) [Guido Imperiale](https://github.com/crusaderky)
- Support for pytest 9.1 ([dask#12465](https://github.com/dask/dask/pull/12465), [distributed#9303](https://github.com/dask/distributed/pull/9303)) [Guido Imperiale](https://github.com/crusaderky)
- Free-threading: use upstream msgpack ([dask#12467](https://github.com/dask/dask/pull/12467), [distributed#9302](https://github.com/dask/distributed/pull/9302)) [Guido Imperiale](https://github.com/crusaderky)
- CI: Suppress numpy `generic` timedelta deprecation from pandas ([dask#12468](https://github.com/dask/dask/pull/12468)) [Guido Imperiale](https://github.com/crusaderky)
- Pin pandas 3 nightlies ([dask#12469](https://github.com/dask/dask/pull/12469), [distributed#9304](https://github.com/dask/distributed/pull/9304)) [Guido Imperiale](https://github.com/crusaderky)
- Bump actions/checkout from 6 to 7 ([dask#12473](https://github.com/dask/dask/pull/12473), [distributed#9306](https://github.com/dask/distributed/pull/9306))
- Fix randomly failing s3fs checkouts in nightly CI ([dask#12474](https://github.com/dask/dask/pull/12474)) [Guido Imperiale](https://github.com/crusaderky)
- Avoid graph materialization in `is_dask_collection` for expr wrappers ([dask#12476](https://github.com/dask/dask/pull/12476)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump prefix-dev/setup-pixi from 0.9.6 to 0.10.0 ([dask#12478](https://github.com/dask/dask/pull/12478), [distributed#9310](https://github.com/dask/distributed/pull/9310))
- Bump actions/cache from 5 to 6 ([dask#12479](https://github.com/dask/dask/pull/12479), [distributed#9309](https://github.com/dask/distributed/pull/9309))
- Avoid NumPy 2.5 type stubs ([dask#12483](https://github.com/dask/dask/pull/12483), [distributed#9312](https://github.com/dask/distributed/pull/9312)) [Guido Imperiale](https://github.com/crusaderky)
- Bump actions/download-artifact from 7 to 8 ([dask#12461](https://github.com/dask/dask/pull/12461), [distributed#9301](https://github.com/dask/distributed/pull/9301))
- Support composite expressions with `__dask_exprs__` protocol ([dask#12457](https://github.com/dask/dask/pull/12457)) [Matthew Rocklin](https://github.com/mrocklin)
- Return all workers from `scheduler_info()` by default ([distributed#9308](https://github.com/dask/distributed/pull/9308)) [Matthew Rocklin](https://github.com/mrocklin)
- `Client.scatter` should not unpack custom containers ([distributed#9298](https://github.com/dask/distributed/pull/9298)) [Adrin Jalali](https://github.com/adrinjalali)
- Suppress mindeps warning when running a single test in CI [crusaderky](https://github.com/crusaderky)
- AI agent skill to debug flaky tests ([distributed#9319](https://github.com/dask/distributed/pull/9319)) [Guido Imperiale](https://github.com/crusaderky)
- Fix Nanny race conditions when worker fails to start ([distributed#9313](https://github.com/dask/distributed/pull/9313)) [Guido Imperiale](https://github.com/crusaderky)
- Fix server hanging on aborted TCP comms ([distributed#9315](https://github.com/dask/distributed/pull/9315)) [Guido Imperiale](https://github.com/crusaderky)
- Fix conda build CI ([distributed#9314](https://github.com/dask/distributed/pull/9314)) [Guido Imperiale](https://github.com/crusaderky)
- Update deprecated CUDA system requirements in Pixi ([distributed#9311](https://github.com/dask/distributed/pull/9311)) [Guido Imperiale](https://github.com/crusaderky)

<a id="v2026-6-0"></a>

## 2026.6.0

### Highlights

- Improve pandas 3.1 compatibility, including `add_prefix`/`add_suffix` and `DataFrame.drop` changes ([dask#12414](https://github.com/dask/dask/pull/12414), [dask#12447](https://github.com/dask/dask/pull/12447)) [Guido Imperiale](https://github.com/crusaderky)
- Fix groupby compatibility with pandas 2.2 and 2.3 and test against intermediate NumPy and pandas versions ([dask#12372](https://github.com/dask/dask/pull/12372)) [Guido Imperiale](https://github.com/crusaderky)
- Store empty arrays with Zarr 3.2.0 ([dask#12366](https://github.com/dask/dask/pull/12366)) [Guido Imperiale](https://github.com/crusaderky)
- Fix quantile and nanquantile behavior ([dask#12380](https://github.com/dask/dask/pull/12380)) [Guido Imperiale](https://github.com/crusaderky)
- Add support for Cholesky decomposition with complex dtypes ([dask#12416](https://github.com/dask/dask/pull/12416)) [Niclas Rieger](https://github.com/nicrie)
- Fix dataframe correctness issues in `DataFrame.merge` and `Series.map` ([dask#12430](https://github.com/dask/dask/pull/12430), [dask#12432](https://github.com/dask/dask/pull/12432)) [nn](https://github.com/hebian1994), [Mike Evdokimov](https://github.com/xronocode)
- Overhaul the `publish_dataset` extension ([distributed#9217](https://github.com/dask/distributed/pull/9217)) [Guido Imperiale](https://github.com/crusaderky)
- Overhaul task stream handling ([distributed#9230](https://github.com/dask/distributed/pull/9230), [distributed#9282](https://github.com/dask/distributed/pull/9282), [distributed#9293](https://github.com/dask/distributed/pull/9293)) [Guido Imperiale](https://github.com/crusaderky), [MohammadYusif](https://github.com/MohammadYusif)
- Fix a race condition in `SubprocessCluster` startup ([distributed#9292](https://github.com/dask/distributed/pull/9292)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated Distributed APIs across scheduler, worker, CLI, plugin, and progress handling ([distributed#9222](https://github.com/dask/distributed/pull/9222), [distributed#9225](https://github.com/dask/distributed/pull/9225), [distributed#9238](https://github.com/dask/distributed/pull/9238), [distributed#9240](https://github.com/dask/distributed/pull/9240), [distributed#9250](https://github.com/dask/distributed/pull/9250)) [Guido Imperiale](https://github.com/crusaderky)
- Migrate CI and local workflows to Pixi ([dask#12389](https://github.com/dask/dask/pull/12389), [distributed#9276](https://github.com/dask/distributed/pull/9276)) [Guido Imperiale](https://github.com/crusaderky)
- Add AI contribution guidance with AGENTS.md and CLAUDE.md ([dask#12415](https://github.com/dask/dask/pull/12415), [distributed#9279](https://github.com/dask/distributed/pull/9279)) [Guido Imperiale](https://github.com/crusaderky)
- Add PyPI release workflows with GitHub Actions Trusted Publishing ([dask#12452](https://github.com/dask/dask/pull/12452), [distributed#9297](https://github.com/dask/distributed/pull/9297)) [Matthew Rocklin](https://github.com/mrocklin)

### Additional changes

- Pandas 3.1: The inplace keyword in DataFrame.drop is deprecated ([dask#12447](https://github.com/dask/dask/pull/12447)) [Guido Imperiale](https://github.com/crusaderky)
- `test_concat_categorical` is still flaky on Pandas 2.0 ([dask#12454](https://github.com/dask/dask/pull/12454)) [Guido Imperiale](https://github.com/crusaderky)
- Add tests for the array `assert_eq` helper ([dask#12441](https://github.com/dask/dask/pull/12441)) [きょうすけ](https://github.com/kyo5uke)
- Revert security autoescape change in HTML reprs ([dask#12451](https://github.com/dask/dask/pull/12451)) [Guido Imperiale](https://github.com/crusaderky)
- Fix Dataframe.merge losing data when there are 128 or 129 partitions ([dask#12430](https://github.com/dask/dask/pull/12430)) [nn](https://github.com/hebian1994)
- Security: enable Jinja2 autoescape to prevent XSS in HTML reprs ([dask#12423](https://github.com/dask/dask/pull/12423)) [dfgvaetyj3456356-hash](https://github.com/dfgvaetyj3456356-hash)
- Test vs. NumPy 1.26 and PyArrow 18 ([dask#12445](https://github.com/dask/dask/pull/12445)) [Guido Imperiale](https://github.com/crusaderky)
- Do not build the dask-core pixi package twice ([dask#12443](https://github.com/dask/dask/pull/12443)) [Guido Imperiale](https://github.com/crusaderky)
- Fix broken condition in `test_cpu_affinity_taskset` ([dask#12399](https://github.com/dask/dask/pull/12399)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Fix flaky `test_interrupt` ([dask#12437](https://github.com/dask/dask/pull/12437)) [Guido Imperiale](https://github.com/crusaderky)
- Free-threading: enable msgpack C extension ([dask#12439](https://github.com/dask/dask/pull/12439)) [Guido Imperiale](https://github.com/crusaderky)
- Fix `Series.map(Series)` producing wrong results for non-co-aligned inputs ([dask#12432](https://github.com/dask/dask/pull/12432)) [Mike Evdokimov](https://github.com/xronocode)
- Repair Upstream CI ([dask#12436](https://github.com/dask/dask/pull/12436)) [Guido Imperiale](https://github.com/crusaderky)
- Strictly expect test failures ([dask#12435](https://github.com/dask/dask/pull/12435)) [Guido Imperiale](https://github.com/crusaderky)
- Update pixi lockfile ([dask#12433](https://github.com/dask/dask/pull/12433)) [Guido Imperiale](https://github.com/crusaderky)
- Add more thorough tests for `Series.map` and fix pandas nightly CI ([dask#12425](https://github.com/dask/dask/pull/12425)) [Guido Imperiale](https://github.com/crusaderky)
- Add AGENTS.md / CLAUDE.md and clarify guidelines for AI pull requests ([dask#12415](https://github.com/dask/dask/pull/12415)) [Guido Imperiale](https://github.com/crusaderky)
- Tweak local coverage HTML report ([dask#12422](https://github.com/dask/dask/pull/12422)) [Guido Imperiale](https://github.com/crusaderky)
- Make tests less dependent on urllib3 and requests ([dask#12419](https://github.com/dask/dask/pull/12419)) [Guido Imperiale](https://github.com/crusaderky)
- Fix trivial Sphinx warning [crusaderky](https://github.com/crusaderky)
- Do not run scheduled tests on forks ([dask#12418](https://github.com/dask/dask/pull/12418)) [Guido Imperiale](https://github.com/crusaderky)
- Only upload conda packages from main branch [crusaderky](https://github.com/crusaderky)
- Repair conda upload ([dask#12417](https://github.com/dask/dask/pull/12417)) [Guido Imperiale](https://github.com/crusaderky)
- Fix `add_prefix`/`add_suffix` in pandas nightly and support explicit axis ([dask#12414](https://github.com/dask/dask/pull/12414)) [Guido Imperiale](https://github.com/crusaderky)
- Remove legacy pandas dtype testing strings vs. decimals ([dask#12413](https://github.com/dask/dask/pull/12413)) [Guido Imperiale](https://github.com/crusaderky)
- Better type annotations for `import_optional_dependency` ([dask#12412](https://github.com/dask/dask/pull/12412)) [Guido Imperiale](https://github.com/crusaderky)
- Resurrect PySpark tests ([dask#12410](https://github.com/dask/dask/pull/12410)) [Guido Imperiale](https://github.com/crusaderky)
- Test on Linux ARM ([dask#12408](https://github.com/dask/dask/pull/12408)) [Guido Imperiale](https://github.com/crusaderky)
- Fix failures in `test_describe` vs. pandas nightly ([dask#12409](https://github.com/dask/dask/pull/12409)) [Guido Imperiale](https://github.com/crusaderky)
- Migrate CI to Pixi ([dask#12389](https://github.com/dask/dask/pull/12389)) [Guido Imperiale](https://github.com/crusaderky)
- Fix typos ([dask#12405](https://github.com/dask/dask/pull/12405)) [Guido Imperiale](https://github.com/crusaderky)
- XFAIL regression in click 8.4.0 ([dask#12407](https://github.com/dask/dask/pull/12407)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_shared_tasks` in free-threading ([dask#12398](https://github.com/dask/dask/pull/12398)) [Guido Imperiale](https://github.com/crusaderky)
- Enforce Python 3.10 compatibility in black ([dask#12397](https://github.com/dask/dask/pull/12397)) [Guido Imperiale](https://github.com/crusaderky)
- Update Optuna+Dask example to use optuna-integration ([dask#12385](https://github.com/dask/dask/pull/12385)) [Mike Evdokimov](https://github.com/xronocode)
- Fix flaky `test_store_locks` ([dask#12393](https://github.com/dask/dask/pull/12393)) [Guido Imperiale](https://github.com/crusaderky)
- XFAIL flaky `test_len` ([dask#12396](https://github.com/dask/dask/pull/12396)) [Guido Imperiale](https://github.com/crusaderky)
- Run some tests serially in pytest-xdist ([dask#12395](https://github.com/dask/dask/pull/12395)) [Guido Imperiale](https://github.com/crusaderky)
- `test_orc` segfaults with PyArrow 16 ([dask#12394](https://github.com/dask/dask/pull/12394)) [Guido Imperiale](https://github.com/crusaderky)
- Unskip `test_map_block_series` ([dask#12392](https://github.com/dask/dask/pull/12392)) [Guido Imperiale](https://github.com/crusaderky)
- Speed up `test_from_delayed_future` ([dask#12391](https://github.com/dask/dask/pull/12391)) [Guido Imperiale](https://github.com/crusaderky)
- Fix duplicated words in dask.array routines and slicing docstrings ([dask#12390](https://github.com/dask/dask/pull/12390)) [lphuc2250gma](https://github.com/lphuc2250gma)
- Simplify `test_dot` and suppress spurious test outputs ([dask#12388](https://github.com/dask/dask/pull/12388)) [Guido Imperiale](https://github.com/crusaderky)
- Test vs. intermediate versions of numpy and pandas ([dask#12372](https://github.com/dask/dask/pull/12372)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up obsolete version check in `test_http` ([dask#12387](https://github.com/dask/dask/pull/12387)) [Guido Imperiale](https://github.com/crusaderky)
- Fix LaTeX formula for split_every in docstrings ([dask#12379](https://github.com/dask/dask/pull/12379)) [stephenworsley](https://github.com/stephenworsley)
- Remove numexpr ([dask#12384](https://github.com/dask/dask/pull/12384)) [Guido Imperiale](https://github.com/crusaderky)
- Unit-less timedelta64 is deprecated in NumPy nightly ([dask#12383](https://github.com/dask/dask/pull/12383)) [Guido Imperiale](https://github.com/crusaderky)
- Require `pyyaml >=5.4.1` ([dask#12382](https://github.com/dask/dask/pull/12382)) [Guido Imperiale](https://github.com/crusaderky)
- Re-raise ModuleNotFoundError instead of ImportError ([dask#12381](https://github.com/dask/dask/pull/12381)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_ffill_bfill` ([dask#12378](https://github.com/dask/dask/pull/12378)) [Guido Imperiale](https://github.com/crusaderky)
- Raise NotImplementedError for `da.quantile` with weights on NumPy < 2.0 ([dask#12370](https://github.com/dask/dask/pull/12370)) [Dr Alex Mitre](https://github.com/mitre88)
- Fix typo in array-chunks.rst ([dask#12375](https://github.com/dask/dask/pull/12375)) [Henry](https://github.com/HGWright)
- Minor docs tweaks ([dask#12377](https://github.com/dask/dask/pull/12377)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up minimum pyarrow version checks ([dask#12376](https://github.com/dask/dask/pull/12376)) [Guido Imperiale](https://github.com/crusaderky)
- Use Python 3.14 in additional and upstream envs and dev docs ([dask#12373](https://github.com/dask/dask/pull/12373)) [Guido Imperiale](https://github.com/crusaderky)
- More test fixes for pandas-nightly ([dask#12369](https://github.com/dask/dask/pull/12369)) [Guido Imperiale](https://github.com/crusaderky)
- Switch back to conda ([dask#12368](https://github.com/dask/dask/pull/12368)) [Guido Imperiale](https://github.com/crusaderky)
- Disable CI for Windows 3.14t ([dask#12367](https://github.com/dask/dask/pull/12367)) [Guido Imperiale](https://github.com/crusaderky)
- Use f-strings ([dask#12362](https://github.com/dask/dask/pull/12362)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Store empty arrays with zarr 3.2.0 ([dask#12366](https://github.com/dask/dask/pull/12366)) [Guido Imperiale](https://github.com/crusaderky)
- Fix some pandas nightly deprecation warnings ([dask#12341](https://github.com/dask/dask/pull/12341)) [Trevin Chow](https://github.com/tmchow)
- 3.14t CI: tweak notes ([dask#12363](https://github.com/dask/dask/pull/12363)) [Guido Imperiale](https://github.com/crusaderky)
- Use f-strings ([dask#12354](https://github.com/dask/dask/pull/12354)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Docs: order array arguments in docstring ([dask#12342](https://github.com/dask/dask/pull/12342)) [Peter A. Jonsson](https://github.com/pjonsson)
- numba is now available on conda-forge for 3.14t ([dask#12358](https://github.com/dask/dask/pull/12358)) [Guido Imperiale](https://github.com/crusaderky)
- `__package__` to `__spec__.parent` ([dask#12333](https://github.com/dask/dask/pull/12333)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update pre-commit black hook ([dask#12344](https://github.com/dask/dask/pull/12344)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update pre-commit ruff hook ([dask#12345](https://github.com/dask/dask/pull/12345)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix typos ([dask#12339](https://github.com/dask/dask/pull/12339)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Add numba, sparse, and h5py to 3.14t CI ([dask#12338](https://github.com/dask/dask/pull/12338)) [Guido Imperiale](https://github.com/crusaderky)
- Improve Contribution Policy ([dask#12320](https://github.com/dask/dask/pull/12320)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix test failures caused by port 8787 already in use ([distributed#9296](https://github.com/dask/distributed/pull/9296)) [Guido Imperiale](https://github.com/crusaderky)
- Migrate from black to `ruff format` ([distributed#9295](https://github.com/dask/distributed/pull/9295)) [Guido Imperiale](https://github.com/crusaderky)
- Fix race condition in SubprocessCluster startup ([distributed#9292](https://github.com/dask/distributed/pull/9292)) [Guido Imperiale](https://github.com/crusaderky)
- Refactor `get_task_stream` ([distributed#9293](https://github.com/dask/distributed/pull/9293)) [Guido Imperiale](https://github.com/crusaderky)
- Do not build the distributed pixi package twice ([distributed#9289](https://github.com/dask/distributed/pull/9289)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_steal_twice` and `test_steal_when_more_tasks` ([distributed#9288](https://github.com/dask/distributed/pull/9288)) [Guido Imperiale](https://github.com/crusaderky)
- Clarify scheduler unpickling in protocol docs ([distributed#9286](https://github.com/dask/distributed/pull/9286)) [Peter Chen J.](https://github.com/peter941221)
- Reinstate no-queue tests ([distributed#9287](https://github.com/dask/distributed/pull/9287)) [Guido Imperiale](https://github.com/crusaderky)
- Clarify unsatisfied resource restrictions in resource docs ([distributed#9269](https://github.com/dask/distributed/pull/9269)) [Peter Chen J.](https://github.com/peter941221)
- Fix import-time coverage ([distributed#9284](https://github.com/dask/distributed/pull/9284)) [Guido Imperiale](https://github.com/crusaderky)
- Improve CLI error message for unknown options to dask worker ([distributed#9281](https://github.com/dask/distributed/pull/9281)) [Zeus Almightee](https://github.com/ernestprovo23)
- Run most CUDA tests in pixi ([distributed#9285](https://github.com/dask/distributed/pull/9285)) [Guido Imperiale](https://github.com/crusaderky)
- Migrate CI to pixi ([distributed#9276](https://github.com/dask/distributed/pull/9276)) [Guido Imperiale](https://github.com/crusaderky)
- Add AGENTS.md / CLAUDE.md and guidelines for AI pull requests ([distributed#9279](https://github.com/dask/distributed/pull/9279)) [Guido Imperiale](https://github.com/crusaderky)
- Fix tests vs. NumPy nightly builds ([distributed#9280](https://github.com/dask/distributed/pull/9280)) [Guido Imperiale](https://github.com/crusaderky)
- uvloop tests ([distributed#9278](https://github.com/dask/distributed/pull/9278)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_call_stack_future` ([distributed#9277](https://github.com/dask/distributed/pull/9277)) [Guido Imperiale](https://github.com/crusaderky)
- Drop dependency on urllib3 ([distributed#9273](https://github.com/dask/distributed/pull/9273)) [James Lamb](https://github.com/jameslamb)
- Remove `avoid_ci` mark ([distributed#9275](https://github.com/dask/distributed/pull/9275)) [Guido Imperiale](https://github.com/crusaderky)
- Print host_info in CI without needing NumPy ([distributed#9274](https://github.com/dask/distributed/pull/9274)) [Guido Imperiale](https://github.com/crusaderky)
- Fix typos ([distributed#9268](https://github.com/dask/distributed/pull/9268)) [Guido Imperiale](https://github.com/crusaderky)
- Support pixi in dask/dask CI ([distributed#9264](https://github.com/dask/distributed/pull/9264)) [Guido Imperiale](https://github.com/crusaderky)
- Truncate large coro repr in retry log output ([distributed#9197](https://github.com/dask/distributed/pull/9197)) [Ernest Provo](https://github.com/ernestprovo23)
- Add explicit check for name not None ([distributed#9265](https://github.com/dask/distributed/pull/9265)) [Maneesh Sutar](https://github.com/maneesh29s)
- XFAIL `test_scheduler_bokeh.py::test_simple` in Windows ([distributed#9266](https://github.com/dask/distributed/pull/9266)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated loop properties ([distributed#9231](https://github.com/dask/distributed/pull/9231)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in Scheduler, Worker, Nanny ([distributed#9238](https://github.com/dask/distributed/pull/9238)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated client context in different tasks/threads ([distributed#9233](https://github.com/dask/distributed/pull/9233)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated Prometheus metrics ([distributed#9249](https://github.com/dask/distributed/pull/9249)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `distributed.deploy` ([distributed#9244](https://github.com/dask/distributed/pull/9244)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated `stream` RPC handler argument ([distributed#9242](https://github.com/dask/distributed/pull/9242)) [Guido Imperiale](https://github.com/crusaderky)
- Deprecations in CLI ([distributed#9240](https://github.com/dask/distributed/pull/9240)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `security` ([distributed#9239](https://github.com/dask/distributed/pull/9239)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated rpc synchronous context manager ([distributed#9235](https://github.com/dask/distributed/pull/9235)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated async `listener.stop()` ([distributed#9234](https://github.com/dask/distributed/pull/9234)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `register_plugin` ([distributed#9225](https://github.com/dask/distributed/pull/9225)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `remove_worker` ([distributed#9222](https://github.com/dask/distributed/pull/9222)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up minimum pyarrow version checks ([distributed#9260](https://github.com/dask/distributed/pull/9260)) [Guido Imperiale](https://github.com/crusaderky)
- Partial review of task streams ([distributed#9230](https://github.com/dask/distributed/pull/9230)) [Guido Imperiale](https://github.com/crusaderky)
- Update outdated work stealing docs regarding worker restrictions ([distributed#9214](https://github.com/dask/distributed/pull/9214)) [Kevin Ziroldi](https://github.com/kevinziroldi)
- Fix memray CI; move back from mamba to conda ([distributed#9258](https://github.com/dask/distributed/pull/9258)) [Guido Imperiale](https://github.com/crusaderky)
- Install keras with conda ([distributed#9259](https://github.com/dask/distributed/pull/9259)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up Cython ([distributed#9257](https://github.com/dask/distributed/pull/9257)) [Guido Imperiale](https://github.com/crusaderky)
- Use `@dataclass(slots=True)` with constraints from Python <3.14 ([distributed#9256](https://github.com/dask/distributed/pull/9256)) [Guido Imperiale](https://github.com/crusaderky)
- Type annotations for `client.Future` ([distributed#9255](https://github.com/dask/distributed/pull/9255)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated Lock client parameter ([distributed#9237](https://github.com/dask/distributed/pull/9237)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecated nested_deserialize ([distributed#9243](https://github.com/dask/distributed/pull/9243)) [Guido Imperiale](https://github.com/crusaderky)
- Deprecations in `distributed.compatibility` ([distributed#9232](https://github.com/dask/distributed/pull/9232)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `utils_test` ([distributed#9226](https://github.com/dask/distributed/pull/9226)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in `distributed.utils` ([distributed#9236](https://github.com/dask/distributed/pull/9236)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up deprecations in ProgressBar ([distributed#9250](https://github.com/dask/distributed/pull/9250)) [Guido Imperiale](https://github.com/crusaderky)
- Run more tests when requests is not installed ([distributed#9251](https://github.com/dask/distributed/pull/9251)) [Guido Imperiale](https://github.com/crusaderky)
- Standardize and increase `async_poll_for` timeout ([distributed#9248](https://github.com/dask/distributed/pull/9248)) [Guido Imperiale](https://github.com/crusaderky)
- Relax unreasonably short test timeouts ([distributed#9247](https://github.com/dask/distributed/pull/9247)) [Guido Imperiale](https://github.com/crusaderky)
- Use f-strings ([distributed#9245](https://github.com/dask/distributed/pull/9245)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update link to bokeh sources in comment ([distributed#9241](https://github.com/dask/distributed/pull/9241)) [Guido Imperiale](https://github.com/crusaderky)
- Apply ruff/pyupgrade rule UP031 ([distributed#9204](https://github.com/dask/distributed/pull/9204)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Homogeneous environment names ([distributed#9209](https://github.com/dask/distributed/pull/9209)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Overhaul `publish_dataset` extension ([distributed#9217](https://github.com/dask/distributed/pull/9217)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_mixing_clients_same_scheduler` ([distributed#9229](https://github.com/dask/distributed/pull/9229)) [Guido Imperiale](https://github.com/crusaderky)
- Type hints: `@contextmanager` should return `Generator[T]` ([distributed#9228](https://github.com/dask/distributed/pull/9228)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky `test_actor.py::test_failed_worker` ([distributed#9227](https://github.com/dask/distributed/pull/9227)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up info parameters in transition to memory ([distributed#9221](https://github.com/dask/distributed/pull/9221)) [Guido Imperiale](https://github.com/crusaderky)
- Move state validation from Scheduler to SchedulerState ([distributed#9224](https://github.com/dask/distributed/pull/9224)) [Guido Imperiale](https://github.com/crusaderky)
- Bump to black 26.3.1 ([distributed#9223](https://github.com/dask/distributed/pull/9223)) [Guido Imperiale](https://github.com/crusaderky)
- Tweak scheduler annotations ([distributed#9220](https://github.com/dask/distributed/pull/9220)) [Guido Imperiale](https://github.com/crusaderky)
- Fix typos ([distributed#9203](https://github.com/dask/distributed/pull/9203)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Bug: dashboard would not show no-worker tasks ([distributed#9215](https://github.com/dask/distributed/pull/9215)) [Guido Imperiale](https://github.com/crusaderky)

<a id="v2026-3-0"></a>

## 2026.3.0

### Highlights

- Preliminary Python 3.14t support ([dask#12223](https://github.com/dask/dask/pull/12223)) [Guido Imperiale](https://github.com/crusaderky)
- Bokeh 3.9.0 compatibility ([distributed#9205](https://github.com/dask/distributed/pull/9205)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)

### Additional changes

- docs: document approximate algorithm and Dask-specific params in describe() ([dask#12300](https://github.com/dask/dask/pull/12300)) 

  ```
  `Maxime Grenu`_
  ```
- docs: clarify coarsen reduction function contract ([dask#12314](https://github.com/dask/dask/pull/12314)) 

  ```
  `monkeyjack123`_
  ```
- Fix misleading TypeError for scalar overflow in dask.array elemwise ([dask#12301](https://github.com/dask/dask/pull/12301)) 

  ```
  `Maxime Grenu`_
  ```
- Stricter warnings filter ([dask#12274](https://github.com/dask/dask/pull/12274)) [Guido Imperiale](https://github.com/crusaderky)
- Clean up obsolete PANDAS_GE markers ([dask#12279](https://github.com/dask/dask/pull/12279)) [Guido Imperiale](https://github.com/crusaderky)
- Remove mention of obsolete default value for ‘boundary’ parameter. ([dask#12304](https://github.com/dask/dask/pull/12304)) 

  ```
  `Marianne Corvellec`_
  ```
- Pandas in 3.14t CI ([dask#12284](https://github.com/dask/dask/pull/12284)) [Guido Imperiale](https://github.com/crusaderky)
- Quadratic definition time in xarray.DataArray.to_zarr(compute=False) ([dask#12299](https://github.com/dask/dask/pull/12299)) [Guido Imperiale](https://github.com/crusaderky)
- `test_tokenize_range_index` fails if cityhash is not installed ([dask#12286](https://github.com/dask/dask/pull/12286)) [Guido Imperiale](https://github.com/crusaderky)
- Bump minimum version of scipy ([dask#12271](https://github.com/dask/dask/pull/12271)) [Guido Imperiale](https://github.com/crusaderky)
- Fix flaky categorical concat test ([dask#12276](https://github.com/dask/dask/pull/12276)) 

  ```
  `Harshith J`_
  ```
- Doc: document Zarr compression options for to_zarr ([dask#12269](https://github.com/dask/dask/pull/12269)) 

  ```
  `Harshith J`_
  ```
- Disable the GIL on 3.14t Windows CI ([dask#12280](https://github.com/dask/dask/pull/12280)) [Guido Imperiale](https://github.com/crusaderky)
- Update obsolete pandas URLs ([dask#12278](https://github.com/dask/dask/pull/12278)) [Guido Imperiale](https://github.com/crusaderky)
- Suppress warning: Consolidated metadata is not part of Zarr 3 ([dask#12273](https://github.com/dask/dask/pull/12273)) [Guido Imperiale](https://github.com/crusaderky)
- Pandas4Warning: Copy-on-Write is always enabled with pandas >= 3.0 ([dask#12272](https://github.com/dask/dask/pull/12272)) [Guido Imperiale](https://github.com/crusaderky)
- Disable the GIL in 3.14t CI ([dask#12270](https://github.com/dask/dask/pull/12270)) [Guido Imperiale](https://github.com/crusaderky)
- Propagate contextvars to worker threads; catch warnings in 3.14t ([dask#12224](https://github.com/dask/dask/pull/12224)) [Guido Imperiale](https://github.com/crusaderky)
- Fix bugs in env.yaml / pytest.xml upload ([dask#12266](https://github.com/dask/dask/pull/12266)) [Guido Imperiale](https://github.com/crusaderky)
- Added `full_matrices` parameter to `dask.array.linalg.svd` ([dask#12292](https://github.com/dask/dask/pull/12292)) 

  ```
  `Ayan Bag`_
  ```
- fix: `zarr.create_array` for better backward compatibility ([dask#12291](https://github.com/dask/dask/pull/12291)) [Wouter-Michiel Vierdag](https://github.com/melonora)
- Silence deprecations in global config if local config overrides them ([dask#12315](https://github.com/dask/dask/pull/12315)) [Guido Imperiale](https://github.com/crusaderky)
- Fix Total CPU % on /workers tab to normalize by total nthreads ([distributed#9195](https://github.com/dask/distributed/pull/9195)) [Ernest Provo](https://github.com/ernestprovo23)
- setproctitle: avoid being caught by dask.config; add to test envs ([distributed#9202](https://github.com/dask/distributed/pull/9202)) [Guido Imperiale](https://github.com/crusaderky)
- Add return type annotation for Client._register_plugin ([distributed#9201](https://github.com/dask/distributed/pull/9201)) [Simon-Martin Schröder](https://github.com/moi90)
- docs: fix Scheduler.close docstring ([distributed#9198](https://github.com/dask/distributed/pull/9198)) 

  ```
  `Chase Naples`_
  ```
- Fix Total CPU % on /workers tab to normalize by total nthreads ([distributed#9195](https://github.com/dask/distributed/pull/9195)) [Ernest Provo](https://github.com/ernestprovo23)
- XFAIL test_handle_null_partitions_2 ([distributed#9191](https://github.com/dask/distributed/pull/9191)) [Guido Imperiale](https://github.com/crusaderky)
- Type hints for Future.status ([distributed#9188](https://github.com/dask/distributed/pull/9188)) 

  ```
  `Navid`_
  ```
- Pin sphinx=8 ([distributed#9190](https://github.com/dask/distributed/pull/9190)) [Guido Imperiale](https://github.com/crusaderky)

<a id="v2026-2-0"></a>

## 2026.2.0

### Highlights

### Additional changes

- Minimum version of optional dependency scipy bumped to 1.10.0 (was 1.7.2)

<a id="v2026-1-2"></a>

## 2026.1.2

### Highlights

- dask.dataframe now requires PyArrow 16 or greater (was 14)
- Have `**kwargs` in `to_zarr` follow zarr-python API and add `mode` argument ([dask#12205](https://github.com/dask/dask/pull/12205)) [Wouter-Michiel Vierdag](https://github.com/melonora)

#### NOTE
Passing on io-related arguments in `**kwargs` in `to_zarr` will be deprecated
and `read_kwargs` argument as well as `zarr_array_kwargs` (dict) introduced in 2025.12.0
has been removed.
If you passed on either `mode` or read_only as `**kwargs` or `read_kwargs` in
`to_zarr`, please use the new `mode` argument. The `read_only` argument can still
be passed on, but it will give a warning and have no effect (given that `to_zarr`
is meant to write this should not be an issue). For now no error will be thrown.
`**kwargs` in `to_zarr` has been renamed as `**zarr_array_kwargs` to indicate
that this  directly follows the `zarr-python` API of `Group.create_array`
when `zarr>v3.0.0` and `zarr.create` for `zarr<v3.0.0`. Please see
[`dask.array.to_zarr()`](generated/dask.array.to_zarr.md#dask.array.to_zarr) for more.

### Additional changes

- Minimum version of optional dependency h5py bumped to 3.7.0 (was 3.4.0)
- Minimum version of optional dependency python-snappy bumped to 0.7.1 (was 0.6.0)
- Minimum version of optional dependency tiledb bumped to 0.27.0 (was 0.12.0)

<a id="v2026-1-1"></a>

## 2026.1.1

### Highlights

- Fix XSS vulnerability [CVE-2026-23528](https://github.com/dask/distributed/security/advisories/GHSA-c336-7962-wfj2) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Support duck-typed Futures in task graph processing ([dask#12213](https://github.com/dask/dask/pull/12213)) [Matthew Rocklin](https://github.com/mrocklin)

### Additional changes

- Remove the Python 2 Comment ([dask#12229](https://github.com/dask/dask/pull/12229)) [Vipin Kataria](https://github.com/vipinkataria2209)
- Fix changelog: distributed-pr -> pr-distributed ([dask#12227](https://github.com/dask/dask/pull/12227)) [Matthew Plough](https://github.com/mplough-kobold)
- Support duck-typed Futures in task graph processing ([dask#12213](https://github.com/dask/dask/pull/12213)) [Matthew Rocklin](https://github.com/mrocklin)
- Relax test_serialization ([dask#12226](https://github.com/dask/dask/pull/12226)) [Guido Imperiale](https://github.com/crusaderky)
- [cosmetic] Reorganise dependency groups in CI environment files ([dask#12222](https://github.com/dask/dask/pull/12222)) [Guido Imperiale](https://github.com/crusaderky)
- Review \_array_expr_enabled() ([dask#12217](https://github.com/dask/dask/pull/12217)) [Guido Imperiale](https://github.com/crusaderky)
- Increase coverage; lower codecov threshold to pass ([dask#12214](https://github.com/dask/dask/pull/12214)) [Guido Imperiale](https://github.com/crusaderky)
- Test array expr on mindeps ([dask#12216](https://github.com/dask/dask/pull/12216)) [Guido Imperiale](https://github.com/crusaderky)
- Disable some Mac builds ([dask#12218](https://github.com/dask/dask/pull/12218)) [Guido Imperiale](https://github.com/crusaderky)
- Typing tweaks ([dask#12215](https://github.com/dask/dask/pull/12215)) [Guido Imperiale](https://github.com/crusaderky)
- [CI] unbreak codecov ([dask#12211](https://github.com/dask/dask/pull/12211)) [Guido Imperiale](https://github.com/crusaderky)
- Test array expr on Python 3.14 ([dask#12212](https://github.com/dask/dask/pull/12212)) [Guido Imperiale](https://github.com/crusaderky)
- Fix pickle compatibility for Python 3.14 ([dask#12206](https://github.com/dask/dask/pull/12206)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove deprecated dask._compatibility.entry_points ([dask#12202](https://github.com/dask/dask/pull/12202)) [Guido Imperiale](https://github.com/crusaderky)
- Tweak MacOS CI ([dask#12200](https://github.com/dask/dask/pull/12200)) [Guido Imperiale](https://github.com/crusaderky)
- Remove obsolete CI pins ([dask#12199](https://github.com/dask/dask/pull/12199)) [Guido Imperiale](https://github.com/crusaderky)
- Fix XSS vulnerability [CVE-2026-23528](https://github.com/dask/distributed/security/advisories/GHSA-c336-7962-wfj2) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Clean up obsolete pins in CI ([distributed#9172](https://github.com/dask/distributed/pull/9172)) [Guido Imperiale](https://github.com/crusaderky)
- Fix incompatibility of pyparsing vs. packaging in mindeps CI ([distributed#9170](https://github.com/dask/distributed/pull/9170)) [Guido Imperiale](https://github.com/crusaderky)
- Bump mypy; fix mypy failure ([distributed#9171](https://github.com/dask/distributed/pull/9171)) [Guido Imperiale](https://github.com/crusaderky)

<a id="v2026-1-0"></a>

## 2026.1.0

Broken yanked release, please ignore.

<a id="v2025-12-0"></a>

## 2025.12.0

### Highlights

- More improvements for pandas 3.x [Tom Augspurger](https://github.com/tomaugspurger)
- Support zarr sharding through create_array ([dask#12153](https://github.com/dask/dask/pull/12153)) [Wouter-Michiel Vierdag](https://github.com/melonora)
- Various improvements for project linting and type hinting [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Add new “optimization.tune.active” configuration option to disable partition fusion ([dask#12194](https://github.com/dask/dask/pull/12194)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Additional changes

- Stable sort in Series.value_counts for pandas 3.x ([dask#12191](https://github.com/dask/dask/pull/12191)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add new “optimization.tune.active” configuration option to disable partition fusion ([dask#12194](https://github.com/dask/dask/pull/12194)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Build llms.txt files in Sphinx documentation ([dask#12192](https://github.com/dask/dask/pull/12192)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Support zarr sharding through create_array ([dask#12153](https://github.com/dask/dask/pull/12153)) [Wouter-Michiel Vierdag](https://github.com/melonora)
- Support min/max of datetime ([dask#12183](https://github.com/dask/dask/pull/12183)) [Julia Signell](https://github.com/jsignell)
- pandas 3.x compatibility ([dask#12180](https://github.com/dask/dask/pull/12180)) [Tom Augspurger](https://github.com/tomaugspurger)
- Minimal version of setuptools-scm ([dask#12184](https://github.com/dask/dask/pull/12184)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update test_ufunc_meta for upstream-dev failure ([dask#12170](https://github.com/dask/dask/pull/12170)) [Tom Augspurger](https://github.com/tomaugspurger)
- Upstream compat ([dask#12165](https://github.com/dask/dask/pull/12165)) [Tom Augspurger](https://github.com/tomaugspurger)
- Enforce a few more ruff rules ([dask#12157](https://github.com/dask/dask/pull/12157)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Enforce ruff/refurb rules (FURB) ([dask#12144](https://github.com/dask/dask/pull/12144)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- DEP: bump minimal requirement on toolz (0.10.0 -> 0.12.0) ([dask#12163](https://github.com/dask/dask/pull/12163)) [Clément Robert](https://github.com/neutrinoceros)
- Fix execution stop in da.to_zarr due to (misleading) PerformanceWarning raised as exception ([dask#12161](https://github.com/dask/dask/pull/12161)) [Marvin Albert](https://github.com/m-albert)
- Use f-string interpolation where possible ([dask#12140](https://github.com/dask/dask/pull/12140)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- pre-commit black hook: use implicit defaults ([dask#12156](https://github.com/dask/dask/pull/12156)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Enforce ruff/pygrep-hooks rules (PGH) ([dask#12143](https://github.com/dask/dask/pull/12143)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply Repo-Review rules ([dask#12148](https://github.com/dask/dask/pull/12148)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Document groupby: split_every, split_out ([dask#12135](https://github.com/dask/dask/pull/12135)) [Jayesh Manani](https://github.com/jayeshmanani)
- isort → ruff ([dask#12149](https://github.com/dask/dask/pull/12149)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Enforce ruff/pyupgrade rule UP031 ([dask#12137](https://github.com/dask/dask/pull/12137)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Replace pre-commit hook with ruff rule ([dask#12142](https://github.com/dask/dask/pull/12142)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix reify to handle sparse arrays and other objects without \_\_len_\_ ([dask#12103](https://github.com/dask/dask/pull/12103)) [Gautham Hullikunte](https://github.com/batcity)
- Ruff supersedes absolufy-imports ([dask#12141](https://github.com/dask/dask/pull/12141)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Enforce ruff/pyupgrade rule UP032 ([dask#12136](https://github.com/dask/dask/pull/12136)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Typing fixes ([distributed#9159](https://github.com/dask/distributed/pull/9159)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Explicit setuptools-scm minimum version ([distributed#9160](https://github.com/dask/distributed/pull/9160)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Enforce ruff rules (RUF) ([distributed#9153](https://github.com/dask/distributed/pull/9153)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Clean up MANIFEST.in ([distributed#9149](https://github.com/dask/distributed/pull/9149)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- isort → ruff ([distributed#9152](https://github.com/dask/distributed/pull/9152)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Ruff supersedes absolufy-imports ([distributed#9154](https://github.com/dask/distributed/pull/9154)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Bump minimum supported `toolz` to 0.12.0 ([distributed#9151](https://github.com/dask/distributed/pull/9151)) [James Bourbeau](https://github.com/jrbourbeau)
- flake8, bugbear, pyupgrade → ruff ([distributed#9147](https://github.com/dask/distributed/pull/9147)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix typos found by codespell ([distributed#9145](https://github.com/dask/distributed/pull/9145)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Clean up setuptools-specific configuration ([distributed#9150](https://github.com/dask/distributed/pull/9150)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- PEP 639 compliance ([distributed#9146](https://github.com/dask/distributed/pull/9146)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update black ([distributed#9148](https://github.com/dask/distributed/pull/9148)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix empty progress bar ([distributed#9144](https://github.com/dask/distributed/pull/9144)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Exclude broken tblib versions in CI ([distributed#9141](https://github.com/dask/distributed/pull/9141)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2025-11-0"></a>

## 2025.11.0

### Highlights

- Use shard shape when available in `to_zarr` ([dask#12105](https://github.com/dask/dask/pull/12105)) [Davis Bennett](https://github.com/d-v-b)
- Improve worker and nanny support for ipv6 ([distributed#9133](https://github.com/dask/distributed/pull/9133)) [Jianyu Sun](https://github.com/csfldf)
- Linting and type hinting improvements across the codebase

### Additional changes

- Replace `versioneer` with `setuptools-scm` ([dask#12133](https://github.com/dask/dask/pull/12133)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Apply ruff/Pylint Refactor rules (PLR) ([dask#12010](https://github.com/dask/dask/pull/12010)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Remove files from `MANIFEST.in` ([dask#12041](https://github.com/dask/dask/pull/12041)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Stabilize `test_filter_nonpartition_columns` ([dask#12131](https://github.com/dask/dask/pull/12131)) [DongWon](https://github.com/dongwonmoon)
- Enforce ruff/pyupgrade rules UP007 and UP033 ([dask#12125](https://github.com/dask/dask/pull/12125)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update `np.accumulate` workaround comment ([dask#12129](https://github.com/dask/dask/pull/12129)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- `flake8`, `bugbear`, `pyupgrade` → `ruff` ([dask#12002](https://github.com/dask/dask/pull/12002)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Adjust pyarrow version skip in `test_parquet` ([dask#12124](https://github.com/dask/dask/pull/12124)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix ufunc in `dask.array.cumreduction` ([dask#12119](https://github.com/dask/dask/pull/12119)) [Tony Ding](https://github.com/tonyyuyiding)
- Fix docs footer ([dask#12120](https://github.com/dask/dask/pull/12120)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Use integer multiple of shard shape when rechunking in `to_zarr` ([dask#12106](https://github.com/dask/dask/pull/12106)) [Davis Bennett](https://github.com/d-v-b)
- Ensure that the shard shape is used as the default chunk shape for sharded Zarr arrays ([dask#12104](https://github.com/dask/dask/pull/12104)) [Davis Bennett](https://github.com/d-v-b)
- Skip `test_parquet` for `pyarrow==22.0` ([dask#12116](https://github.com/dask/dask/pull/12116)) [Tom Augspurger](https://github.com/tomaugspurger)
- Clean up setuptools-specific configuration ([dask#12040](https://github.com/dask/dask/pull/12040)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- PEP 639 compliance ([dask#12024](https://github.com/dask/dask/pull/12024)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix deprecated quantile `interpolation` being passed to numpy ([dask#12108](https://github.com/dask/dask/pull/12108)) [David Hoese](https://github.com/djhoese)
- Add `uv.lock` to `.gitignore` ([dask#12110](https://github.com/dask/dask/pull/12110)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Use shard shape when available in `to_zarr` ([dask#12105](https://github.com/dask/dask/pull/12105)) [Davis Bennett](https://github.com/d-v-b)
- Add more optional dependencies to Python 3.13 CI builds ([dask#12100](https://github.com/dask/dask/pull/12100)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `pip` pin for docs ([dask#12102](https://github.com/dask/dask/pull/12102)) [James Bourbeau](https://github.com/jrbourbeau)
- Address collection-based `meta` arguments in `GroupByApply` ([dask#12099](https://github.com/dask/dask/pull/12099)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Replace versioneer with setuptools-scm ([distributed#9137](https://github.com/dask/distributed/pull/9137)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Improve worker and nanny support for ipv6 ([distributed#9133](https://github.com/dask/distributed/pull/9133)) [Jianyu Sun](https://github.com/csfldf)
- Fix CI Multiple aliased keys in file `/Users/runner/.condarc` ([distributed#9136](https://github.com/dask/distributed/pull/9136)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Remove `pip` pin for docs ([distributed#9132](https://github.com/dask/distributed/pull/9132)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove UCX configuration schema ([distributed#9127](https://github.com/dask/distributed/pull/9127)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add generic type support to `Future` and `Client` methods ([distributed#9123](https://github.com/dask/distributed/pull/9123)) [Simon-Martin Schröder](https://github.com/moi90)

<a id="v2025-10-0"></a>

## 2025.10.0

### Highlights

- Several Dask Array bug fixes including [dask#12097](https://github.com/dask/dask/pull/12097), [dask#12089](https://github.com/dask/dask/pull/12089), [dask#12088](https://github.com/dask/dask/pull/12088), and [dask#12090](https://github.com/dask/dask/pull/12090).

### Additional changes

- Use updated docs theme ([dask#12093](https://github.com/dask/dask/pull/12093)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix: `dask.array.cumprod` does not deal with `dtype` ([dask#12097](https://github.com/dask/dask/pull/12097)) [Tony Ding](https://github.com/tonyyuyiding)
- CuPy compatibility for percentile ([dask#12098](https://github.com/dask/dask/pull/12098)) [Tom Augspurger](https://github.com/tomaugspurger)
- Avoid using `methods.concat` on empty lists ([dask#12096](https://github.com/dask/dask/pull/12096)) [Tony Ding](https://github.com/tonyyuyiding)
- Add distribution check for optional dependencies ([dask#12087](https://github.com/dask/dask/pull/12087)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix percentile inconsistencies ([dask#12088](https://github.com/dask/dask/pull/12088)) [Oisin-M](https://github.com/Oisin-M)
- Fix warning in `test_ufunc_where_no_out` ([dask#12094](https://github.com/dask/dask/pull/12094)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix/choose trivial case ([dask#12090](https://github.com/dask/dask/pull/12090)) [Oisin-M](https://github.com/Oisin-M)
- Add input validation on `dask.dataframe.read_sql_query()` ([dask#12091](https://github.com/dask/dask/pull/12091)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Numpy 2.2 updates for `cov` function with tests ([dask#12079](https://github.com/dask/dask/pull/12079)) [Mike McCarty](https://github.com/mmccarty)
- Fix `nanvar` ([dask#12089](https://github.com/dask/dask/pull/12089)) [Oisin-M](https://github.com/Oisin-M)
- Document manually triggering the conda-forge bots ([dask#12083](https://github.com/dask/dask/pull/12083)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix mixed HLG/Expr handling in `_ExprSequence._simplify_down` ([dask#12081](https://github.com/dask/dask/pull/12081)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `dask.tokenize` to API docs ([dask#12080](https://github.com/dask/dask/pull/12080)) [Username46786](https://github.com/Username46786)
- `CreateOverlappingPartitions`: Add before and after to prepend name ([dask#11965](https://github.com/dask/dask/pull/11965)) [Fabien Aulaire](https://github.com/faulaire)
- Fix `scipy.sparce.csc_matrix` scalar declaration in `_array_like_safe` ([dask#12078](https://github.com/dask/dask/pull/12078)) [Ilan Gold](https://github.com/ilan-gold)
- Update docs theme and remove docs env pins ([distributed#9125](https://github.com/dask/distributed/pull/9125)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add worker name as prefix to `ThreadPoolExecutor` name ([distributed#9120](https://github.com/dask/distributed/pull/9120)) [Maneesh Sutar](https://github.com/maneesh29s)
- Skip hanging SSH tests on Windows ([distributed#9115](https://github.com/dask/distributed/pull/9115)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix macOS CI failure during job startup ([distributed#9113](https://github.com/dask/distributed/pull/9113)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Prevent task stream dashboard showing 1970 date ([distributed#9109](https://github.com/dask/distributed/pull/9109)) [Guillaume Eynard-Bontemps](https://github.com/guillaumeeb)

<a id="v2025-9-2"></a>

## 2025.9.2

This is a backport security release only.

See [CVE-2026-23528](https://github.com/dask/distributed/security/advisories/GHSA-c336-7962-wfj2) for more details.

<a id="v2025-9-1"></a>

## 2025.9.1

### Highlights

- Avoid unconditional pyarrow dependency in dataframe.backends ([dask#12075](https://github.com/dask/dask/pull/12075)) [Tom Augspurger](https://github.com/tomaugspurger)
- pandas 3.x compatibility for .groups ([dask#12071](https://github.com/dask/dask/pull/12071)) [Tom Augspurger](https://github.com/tomaugspurger)

### Additional changes

- Avoid unconditional pyarrow dependency in dataframe.backends ([dask#12075](https://github.com/dask/dask/pull/12075)) [Tom Augspurger](https://github.com/tomaugspurger)
- pandas 3.x compatibility for .groups ([dask#12071](https://github.com/dask/dask/pull/12071)) [Tom Augspurger](https://github.com/tomaugspurger)
- Expose details about worker start timeout in the exception message ([distributed#9092](https://github.com/dask/distributed/pull/9092)) [Taylor Braun-Jones](https://github.com/nocnokneo)
- pynvml => nvidia-ml-py in CI ([distributed#9111](https://github.com/dask/distributed/pull/9111)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2025-9-0"></a>

## 2025.9.0

### Highlights

- pandas 3.x compatibility ([dask#12025](https://github.com/dask/dask/pull/12025)) [Tom Augspurger](https://github.com/tomaugspurger)
- Remove protocol=”ucx” support in favor of distributed-ucxx ([distributed#9105](https://github.com/dask/distributed/pull/9105)) [Peter Andreas Entschev](https://github.com/pentschev)

### Additional changes

- Fix 0 scalar setting for scipy.sparse ([dask#12027](https://github.com/dask/dask/pull/12027)) [Ilan Gold](https://github.com/ilan-gold)
- Workaround failing upstream-dev tests ([dask#12061](https://github.com/dask/dask/pull/12061)) [Tom Augspurger](https://github.com/tomaugspurger)
- avoid instantiating a potentially very large arange in take ([dask#11998](https://github.com/dask/dask/pull/11998)) [Justus Magin](https://github.com/keewis)
- MAINT: address NumPy deprecation in np.minimum ([dask#12059](https://github.com/dask/dask/pull/12059)) [Marco Edward Gorelli](https://github.com/MarcoGorelli)
- CI fixes ([dask#12058](https://github.com/dask/dask/pull/12058)) [Tom Augspurger](https://github.com/tomaugspurger)
- MAINT: Address NumPy DeprecationWarning ([dask#12056](https://github.com/dask/dask/pull/12056)) [Marco Edward Gorelli](https://github.com/MarcoGorelli)
- Fix `test_enforce_columns` on Python 3.14 ([dask#12047](https://github.com/dask/dask/pull/12047)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Fix “th” –> “the” typo in DataFrame SQL docs ([dask#12038](https://github.com/dask/dask/pull/12038)) [Peter A. Jonsson](https://github.com/pjonsson)
- Advance rng state in permutation ([dask#12031](https://github.com/dask/dask/pull/12031)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `pyarrow` chunked array conversion ([dask#12034](https://github.com/dask/dask/pull/12034)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `xfail` condition for `pyarrow` `large_string` issue ([dask#12032](https://github.com/dask/dask/pull/12032)) [James Bourbeau](https://github.com/jrbourbeau)
- pandas 3.x compatibility ([dask#12025](https://github.com/dask/dask/pull/12025)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix name not propagated correctly in map_blocks ([dask#11952](https://github.com/dask/dask/pull/11952)) [Ilan Gold](https://github.com/ilan-gold)
- Clean tuples dict keys from workers_info in /api/v1/retire_workers. ([distributed#8996](https://github.com/dask/distributed/pull/8996)) [Florian Courtial](https://github.com/fcourtial)
- Remove protocol=”ucx” support in favor of distributed-ucxx ([distributed#9105](https://github.com/dask/distributed/pull/9105)) [Peter Andreas Entschev](https://github.com/pentschev)

<a id="v2025-7-0"></a>

## 2025.7.0

### Highlights

- Account for `__main__` in `pickle` normalization ([dask#11970](https://github.com/dask/dask/pull/11970)) [James Bourbeau](https://github.com/jrbourbeau)
- Enable column projection in `MapPartitions` ([dask#11875](https://github.com/dask/dask/pull/11875)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add config option for `direct-to-workers` ([distributed#9097](https://github.com/dask/distributed/pull/9097)) [James Bourbeau](https://github.com/jrbourbeau)

### Additional changes

- CI: update actions location ([dask#12019](https://github.com/dask/dask/pull/12019)) [Brigitta Sipőcz](https://github.com/bsipocz)
- Apply `ruff/flake8-comprehensions` rules (C4) ([dask#12004](https://github.com/dask/dask/pull/12004)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/flake8-pie` rules (PIE) ([dask#12006](https://github.com/dask/dask/pull/12006)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/Pylint Error` rules (PLE) ([dask#12013](https://github.com/dask/dask/pull/12013)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/Pylint Convention` rules (PLC) ([dask#12012](https://github.com/dask/dask/pull/12012)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/flake8-pyi` rules (PYI) ([dask#12007](https://github.com/dask/dask/pull/12007)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/flake8-simplify` rules (SIM) ([dask#12008](https://github.com/dask/dask/pull/12008)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/Pylint` Warning rules (PLW) ([dask#12011](https://github.com/dask/dask/pull/12011)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/flake8-implicit-str-concat` rules (ISC) ([dask#12005](https://github.com/dask/dask/pull/12005)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Apply `ruff/pycodestyle` rule E714 ([dask#12000](https://github.com/dask/dask/pull/12000)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix typos found by `codespell` ([dask#12001](https://github.com/dask/dask/pull/12001)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Update PyPI URL for official nightly `pyarrow` repository ([dask#11996](https://github.com/dask/dask/pull/11996)) [Raúl Cumplido](https://github.com/raulcd)
- Fall-back to textual repr in case `jinja2` is not installed ([dask#11987](https://github.com/dask/dask/pull/11987)) [Lukas Bindreiter](https://github.com/lukasbindreiter)
- Prevent `builtins.any` from being shadowed in `dask.array.reductions` ([dask#11988](https://github.com/dask/dask/pull/11988)) [Marvin Albert](https://github.com/m-albert)
- Bump `conda-incubator/setup-miniconda` from 3.1.1 to 3.2.0 ([dask#11982](https://github.com/dask/dask/pull/11982))
- Skip groupby cov test for pandas 3.x ([dask#11977](https://github.com/dask/dask/pull/11977)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix `upstream` CI installation ([dask#11976](https://github.com/dask/dask/pull/11976)) [James Bourbeau](https://github.com/jrbourbeau)
- Make module name logic more resilient in `Dispatch` ([dask#11974](https://github.com/dask/dask/pull/11974)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure `memray` profiler runs on all workers ([distributed#9095](https://github.com/dask/distributed/pull/9095)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `def` to `class` typo in actors docs ([distributed#9091](https://github.com/dask/distributed/pull/9091)) [Peter Fackeldey](https://github.com/pfackeldey)
- Bump `conda-incubator/setup-miniconda` from 3.1.1 to 3.2.0 ([distributed#9090](https://github.com/dask/distributed/pull/9090))
- Update persist in tests for async clients ([distributed#9089](https://github.com/dask/distributed/pull/9089)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix `pyarrow` `FileInfo` import ([distributed#9078](https://github.com/dask/distributed/pull/9078)) [James Bourbeau](https://github.com/jrbourbeau)
- Make module name logic more resilient in `_always_use_pickle_for` ([distributed#9086](https://github.com/dask/distributed/pull/9086)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily pin `pytest` in CI to avoid coverage error  ([distributed#9088](https://github.com/dask/distributed/pull/9088)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `s3fs` from testing CI environment ([distributed#9087](https://github.com/dask/distributed/pull/9087)) [James Bourbeau](https://github.com/jrbourbeau)
- Reuse `Comm` objects in `Scheduler.broadcast` ([distributed#9083](https://github.com/dask/distributed/pull/9083)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix `test_resubmit_nondeterministic_task_different_deps` ([distributed#9085](https://github.com/dask/distributed/pull/9085)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2025-5-1"></a>

## 2025.5.1

### Highlights

Fixed Dask Array slicing regression introduced in the 2025.5.0 release.
See [dask#11947](https://github.com/dask/dask/pull/11947) from [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Speed up slicing graph generation ([dask#11945](https://github.com/dask/dask/pull/11945)) [Florian Jetter](https://github.com/fjetter)
- Revert “Don’t handle tuple in `task_spec.parse_input`” ([dask#11953](https://github.com/dask/dask/pull/11953)) [Florian Jetter](https://github.com/fjetter)
- Optimize slicing graph generation ([dask#11946](https://github.com/dask/dask/pull/11946)) [Florian Jetter](https://github.com/fjetter)
- Fix `xarray` slicing regression ([dask#11947](https://github.com/dask/dask/pull/11947)) [Florian Jetter](https://github.com/fjetter)
- Don’t handle tuple in `task_spec.parse_input` ([dask#11948](https://github.com/dask/dask/pull/11948)) [Florian Jetter](https://github.com/fjetter)

<a id="v2025-5-0"></a>

## 2025.5.0

### Highlights

- Fixed Array `setitem` when both the array and the indexer have unknown shape.
  See [dask#11753](https://github.com/dask/dask/pull/11753) from [Tom Augspurger](https://github.com/tomaugspurger) for more details.
- Fixed several `delayed` graph handling issues introduced in the 2025.4.0 release.
  See [dask#11917](https://github.com/dask/dask/pull/11917), [dask#11907](https://github.com/dask/dask/pull/11907), and [distributed#9071](https://github.com/dask/distributed/pull/9071) from [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Speed up slicing graph generation ([dask#11945](https://github.com/dask/dask/pull/11945)) [Florian Jetter](https://github.com/fjetter)
- Optimize dask order for worst case of `get_target` ([dask#11935](https://github.com/dask/dask/pull/11935)) [Florian Jetter](https://github.com/fjetter)
- Raise on local executor if tasks are missing dependency ([dask#11944](https://github.com/dask/dask/pull/11944)) [Florian Jetter](https://github.com/fjetter)
- Fix `to_dask_array` for single partition ([dask#11931](https://github.com/dask/dask/pull/11931)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure parquet plan is fully cached during optimization ([dask#11933](https://github.com/dask/dask/pull/11933)) [Florian Jetter](https://github.com/fjetter)
- Better documentation for expression system ([dask#11915](https://github.com/dask/dask/pull/11915)) [Florian Jetter](https://github.com/fjetter)
- Simplify (and speed up) culling ([dask#11899](https://github.com/dask/dask/pull/11899)) [Florian Jetter](https://github.com/fjetter)
- Update pre-commit ([dask#11926](https://github.com/dask/dask/pull/11926)) [Florian Jetter](https://github.com/fjetter)
- Don’t run post `setup-miniconda` step in CI ([dask#11925](https://github.com/dask/dask/pull/11925)) [James Bourbeau](https://github.com/jrbourbeau)
- Try to pin pip for readthedocs ([dask#11923](https://github.com/dask/dask/pull/11923)) [Florian Jetter](https://github.com/fjetter)
- Fix windows CI ([dask#11919](https://github.com/dask/dask/pull/11919)) [Florian Jetter](https://github.com/fjetter)
- Use stable `crick` for py310 ([distributed#9072](https://github.com/dask/distributed/pull/9072)) [Florian Jetter](https://github.com/fjetter)
- Remove internal dependencies mapping in `update_graph` ([distributed#9036](https://github.com/dask/distributed/pull/9036)) [Florian Jetter](https://github.com/fjetter)
- Partially forgotten dependencies ([distributed#9068](https://github.com/dask/distributed/pull/9068)) [Florian Jetter](https://github.com/fjetter)
- Replace `filesystem-spec` in CI environment with `fsspec` ([distributed#9069](https://github.com/dask/distributed/pull/9069)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure actors set erred state properly in case of worker failure ([distributed#9067](https://github.com/dask/distributed/pull/9067)) [Florian Jetter](https://github.com/fjetter)
- Refactor timeouts in start cluster ([distributed#9062](https://github.com/dask/distributed/pull/9062)) [Florian Jetter](https://github.com/fjetter)
- Fix workers / threads / memory displayed in client repr ([distributed#9066](https://github.com/dask/distributed/pull/9066)) [James Bourbeau](https://github.com/jrbourbeau)
- Pin pip for readthedocs ([distributed#9063](https://github.com/dask/distributed/pull/9063)) [Florian Jetter](https://github.com/fjetter)
- Skip TLS functional tests ([distributed#9061](https://github.com/dask/distributed/pull/9061)) [Florian Jetter](https://github.com/fjetter)
- Ensure client submit does not serialize unnecessarily ([distributed#9057](https://github.com/dask/distributed/pull/9057)) [Florian Jetter](https://github.com/fjetter)

<a id="v2025-4-1"></a>

## 2025.4.1

### Highlights

This release contains several graph optimization fixes for issues introduced in the `2025.4.0` release.

See [dask#11906](https://github.com/dask/dask/pull/11906), [dask#11898](https://github.com/dask/dask/pull/11898), [dask#11903](https://github.com/dask/dask/pull/11903), and [dask#11904](https://github.com/dask/dask/pull/11904) by [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Implement `ufuncs` and `gufunc` for array-expr ([dask#11818](https://github.com/dask/dask/pull/11818)) [Patrick Hoefler](https://github.com/phofl)
- Implement `map_overlap` for array-expr ([dask#11822](https://github.com/dask/dask/pull/11822)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2025-4-0"></a>

## 2025.4.0

### Highlights

- When computing multiple Dask-Expr backed collections like DataFrames, they are
  now optimized together instead of individually.
- Graph materialization and low level optimization is now being performed on the
  scheduler of a distributed cluster (if available).
- New kwarg `force` for `DataFrame.shuffle` which signals the optimizer to not
  drop the shuffle during optimization.
- Collections that are passed to Dask methods as arguments
  are now properly optimized. If multiple collections are passed as arguments
  they will be optimized together. Collections passed this way are prohibited
  from being being reused, i.e. if the collection is used again in another
  function call it will be computed again. This pattern is used to avoid
  pipeline breakers which typically drive memory usage. Avoiding those should
  reduce memory pressure on the cluster but can cause runtime regressions.
- (Special case of above point) Collections passed to Delayed objects are now
  optimized automatically.

### Breaking changes

- Support for custom low level optimizers removed.
- Top level `dask.optimize` will now always trigger graph materialization.
  Previously this was not always the case. This also causes any low level HLG
  annotations to be dropped.
- DataFrame and Array compute results are now always concatenated on the
  cluster. Previously, the behavior was dependent on the API used to call
  compute (`dask.compute`, `DaskCollection.compute`, or `Client.compute`).
- `dask.base.collections_to_dsk` has been renamed to `collections_to_expr` and
  no longer returns a `HighLevelGraph` or `dict` object but instead
  guarantees an `dask._expr.Expr` object. Further, it no longer performs low
  level optimization immediately but instead delays until the `Expr` instance
  is materialized, i.e. the returned object is no longer a mapping such that
  converting it to `dict` or iterating over it is not possible any more.

### Additional changes

- Ensure `Future` value is in `da.from_delayed` task graph ([dask#11896](https://github.com/dask/dask/pull/11896)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix annotations passed to `delayed` ([dask#11893](https://github.com/dask/dask/pull/11893)) [Florian Jetter](https://github.com/fjetter)
- Migrate `delayed` `unpack_collections` ([dask#11881](https://github.com/dask/dask/pull/11881)) [Florian Jetter](https://github.com/fjetter)
- Remove `Pub` / `Sub` references from docs ([dask#11891](https://github.com/dask/dask/pull/11891)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure only classes without custom init are singletons ([dask#11886](https://github.com/dask/dask/pull/11886)) [Florian Jetter](https://github.com/fjetter)
- Remove custom initializers for `delayed` expressions ([dask#11888](https://github.com/dask/dask/pull/11888)) [Florian Jetter](https://github.com/fjetter)
- Fix persisting multiple DFs at the same time ([dask#11887](https://github.com/dask/dask/pull/11887)) [Florian Jetter](https://github.com/fjetter)
- Avoid always parsing list inputs to `DataFrame.isin` as object type `numpy` arrays ([dask#11869](https://github.com/dask/dask/pull/11869)) [Matthew Roeschke](https://github.com/mroeschke)
- Unskip pandas-dev `cov` / `corr` tests ([dask#11873](https://github.com/dask/dask/pull/11873)) [Tom Augspurger](https://github.com/tomaugspurger)
- HLG `blockwise` fix ([dask#11871](https://github.com/dask/dask/pull/11871)) [Florian Jetter](https://github.com/fjetter)
- Ensure annotations for HLG objects are properly generated ([dask#11866](https://github.com/dask/dask/pull/11866)) [Florian Jetter](https://github.com/fjetter)
- Factor out singleton logic from base `Expr` class ([dask#11868](https://github.com/dask/dask/pull/11868)) [Florian Jetter](https://github.com/fjetter)
- Ensure HLGs are using dependencies properly in optimization ([dask#11859](https://github.com/dask/dask/pull/11859)) [Florian Jetter](https://github.com/fjetter)
- Ensure dictionaries tokenize deterministically ([dask#11867](https://github.com/dask/dask/pull/11867)) [Florian Jetter](https://github.com/fjetter)
- Ensure default dask scheduler only compute what’s needed ([dask#11861](https://github.com/dask/dask/pull/11861)) [Florian Jetter](https://github.com/fjetter)
- Faster tokenization of `pd.RangeIndex` ([dask#11863](https://github.com/dask/dask/pull/11863)) [Florian Jetter](https://github.com/fjetter)
- Update link to Quansight in community doc ([dask#11860](https://github.com/dask/dask/pull/11860)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Relax tolerance in `autocorr` test ([dask#11857](https://github.com/dask/dask/pull/11857)) [Tom Augspurger](https://github.com/tomaugspurger)
- Use `map_blocks` in `array.store` to avoid materialization and dropping of annotations ([dask#11844](https://github.com/dask/dask/pull/11844)) [Florian Jetter](https://github.com/fjetter)
- Ensure `repartition` does not trigger memory size computation during lowering (i.e. on the scheduler) ([dask#11855](https://github.com/dask/dask/pull/11855)) [Florian Jetter](https://github.com/fjetter)
- Support `args` and `kwargs` for rolling aggregations ([dask#11856](https://github.com/dask/dask/pull/11856)) [Florian Jetter](https://github.com/fjetter)
- Remove nightly `h5py` from `upstream` CI job ([dask#11847](https://github.com/dask/dask/pull/11847)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure `HLGExpr` tokenize uniquely ([dask#11849](https://github.com/dask/dask/pull/11849)) [Florian Jetter](https://github.com/fjetter)
- Do not inject median in describe for `pandas` 3 ([dask#11846](https://github.com/dask/dask/pull/11846)) [Florian Jetter](https://github.com/fjetter)
- Fixed `Expr.__setattr__` for subclasses ([dask#11845](https://github.com/dask/dask/pull/11845)) [Tom Augspurger](https://github.com/tomaugspurger)
- Wrap HLGs in an `Expr` to avoid `Client` side materialization ([dask#11736](https://github.com/dask/dask/pull/11736)) [Florian Jetter](https://github.com/fjetter)
- Improve error when submitting work from a closed client ([distributed#9049](https://github.com/dask/distributed/pull/9049)) [James Bourbeau](https://github.com/jrbourbeau)
- Return a default value if address resolution fails ([distributed#9051](https://github.com/dask/distributed/pull/9051)) [Sandro](https://github.com/penguinpee)
- Avoid `deepcopy` when submitting graph ([distributed#8633](https://github.com/dask/distributed/pull/8633)) [Florian Jetter](https://github.com/fjetter)
- Dynamically scale heartbeat and `scheduler_info` intervals ([distributed#9046](https://github.com/dask/distributed/pull/9046)) [Florian Jetter](https://github.com/fjetter)
- Speed up process startup time by avoiding importing packages on version check ([distributed#9048](https://github.com/dask/distributed/pull/9048)) [Florian Jetter](https://github.com/fjetter)
- Reduce size of `scheduler_info` ([distributed#9045](https://github.com/dask/distributed/pull/9045)) [Florian Jetter](https://github.com/fjetter)
- Cache `WorkerState` host property ([distributed#9044](https://github.com/dask/distributed/pull/9044)) [Florian Jetter](https://github.com/fjetter)
- Clear ci env cache ([distributed#9047](https://github.com/dask/distributed/pull/9047)) [Florian Jetter](https://github.com/fjetter)
- Remove deprecated `Pub` / `Sub` ([distributed#9039](https://github.com/dask/distributed/pull/9039)) [Florian Jetter](https://github.com/fjetter)
- Perform explicit culling step only if LLG is submitted ([distributed#9040](https://github.com/dask/distributed/pull/9040)) [Florian Jetter](https://github.com/fjetter)
- Do not fully materialize global annotations by type ([distributed#9035](https://github.com/dask/distributed/pull/9035)) [Florian Jetter](https://github.com/fjetter)
- Allow nested `worker_client` calls ([distributed#9038](https://github.com/dask/distributed/pull/9038)) [George Sakkis](https://github.com/gsakkis)
- Dump ci cache ([distributed#9037](https://github.com/dask/distributed/pull/9037)) [Florian Jetter](https://github.com/fjetter)
- Scheduler type annotations ([distributed#9030](https://github.com/dask/distributed/pull/9030)) [Florian Jetter](https://github.com/fjetter)
- Reduce `dask.order` overhead by removing `stripped_dep` computation ([distributed#9031](https://github.com/dask/distributed/pull/9031)) [Florian Jetter](https://github.com/fjetter)
- Use `Expr` instead of HLG ([distributed#9008](https://github.com/dask/distributed/pull/9008)) [Florian Jetter](https://github.com/fjetter)

<a id="v2025-3-0"></a>

## 2025.3.0

### Highlights

#### Automatically adjust chunksizes in `xarray.apply_ufunc`

`apply_ufunc` requires the core dimension to have `chunksize=-1`. The underlying
rechunking operation will automatically adjust the chunksize of the core dimension
but keep the other dimensions the same. This can cause exploding chunksizes under the hood.

This release adds an intermediate step that resizes the non-core dimensions by the same factor
that the core dimension will increase to keep the maximum chunksize under control. This behavior
is automatically enabled when `allow_rechunk=True` is set.

```default
import xarray as xr
import dask.array as da

arr = xr.DataArray(
    da.random.random((1, 750, 45910), chunks=(1, "auto", -1)),
    dims=["band", "y", "x"],
)

result = arr.interp(
    y=arr.coords["y"],
    method="linear",
)
```

**Previously**

Individual chunks are exploding to 25 GiB, likely causing out of memory errors.

![Individual chunks are exploding to 25 GiB, likely causing out of memory errors.](images/changelog/gufunc_chunksizes_exploding.png)

**Now**

Dask will now automatically split individual chunks into chunks that will have the
same chunksize minus a small tolerance.

![Individual chunks are now roughly the same size](images/changelog/gufunc_chunksizes_constant.png)

### Additional changes

- Fix dataset info cache assignment ([dask#11840](https://github.com/dask/dask/pull/11840)) [Florian Jetter](https://github.com/fjetter)
- Expr `setattr` ([dask#11836](https://github.com/dask/dask/pull/11836)) [Florian Jetter](https://github.com/fjetter)
- Follow up to expression tokenization caching ([dask#11837](https://github.com/dask/dask/pull/11837)) [Florian Jetter](https://github.com/fjetter)
- Consolidate `getattr` for expr classes ([dask#11835](https://github.com/dask/dask/pull/11835)) [Florian Jetter](https://github.com/fjetter)
- Reduce pickle size of `ReadParquet` expression ([dask#11797](https://github.com/dask/dask/pull/11797)) [Florian Jetter](https://github.com/fjetter)
- `arange` loses precision on `~2**63` ([dask#11801](https://github.com/dask/dask/pull/11801)) [Guido Imperiale](https://github.com/crusaderky)
- Remove `numbagg` from upstream build ([dask#11821](https://github.com/dask/dask/pull/11821)) [Patrick Hoefler](https://github.com/phofl)
- Dispatch to `numbagg` for `nanmedian` and `nanquantile` ([dask#11817](https://github.com/dask/dask/pull/11817)) [Patrick Hoefler](https://github.com/phofl)
- Make missing `meta` warning more ergonomic ([dask#11814](https://github.com/dask/dask/pull/11814)) [Patrick Hoefler](https://github.com/phofl)
- Remove `name` doc from `from_pandas` ([dask#11812](https://github.com/dask/dask/pull/11812)) [Patrick Hoefler](https://github.com/phofl)
- Implement an Array Scalar ([dask#11810](https://github.com/dask/dask/pull/11810)) [Patrick Hoefler](https://github.com/phofl)
- Added `to_orc` to DataFrame API ([dask#11807](https://github.com/dask/dask/pull/11807)) [Tom Augspurger](https://github.com/tomaugspurger)
- Implement reverse indexing for DataFrames ([dask#11803](https://github.com/dask/dask/pull/11803)) [Patrick Hoefler](https://github.com/phofl)
- Add lazy `to_pandas_dispatch` registration for `cudf` ([dask#11799](https://github.com/dask/dask/pull/11799)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix missing imports in array-expr ([dask#11796](https://github.com/dask/dask/pull/11796)) [Florian Jetter](https://github.com/fjetter)
- Cache tokens on expressions and restore after pickle roundtrip ([dask#11791](https://github.com/dask/dask/pull/11791)) [Florian Jetter](https://github.com/fjetter)
- Use random dashboard ports for `LocalCluster` in distributed tests ([dask#11795](https://github.com/dask/dask/pull/11795)) [Florian Jetter](https://github.com/fjetter)
- Implement slicing for array-expr ([dask#11783](https://github.com/dask/dask/pull/11783)) [Patrick Hoefler](https://github.com/phofl)
- Never use an asynchronous `Client` when calling top level compute function ([dask#11790](https://github.com/dask/dask/pull/11790)) [Florian Jetter](https://github.com/fjetter)
- Refactor import tests ([dask#11794](https://github.com/dask/dask/pull/11794)) [Florian Jetter](https://github.com/fjetter)
- Migrate `base.unpack_collections` to `Task` class ([dask#11793](https://github.com/dask/dask/pull/11793)) [Florian Jetter](https://github.com/fjetter)
- Ensure `map_blocks` generates unique tokens ([dask#11792](https://github.com/dask/dask/pull/11792)) [Florian Jetter](https://github.com/fjetter)
- Speed up `normalize_pickle` by 50 percent ([dask#11788](https://github.com/dask/dask/pull/11788)) [Florian Jetter](https://github.com/fjetter)
- Fix divisions calculation with duplicates ([dask#11787](https://github.com/dask/dask/pull/11787)) [Patrick Hoefler](https://github.com/phofl)
- Fix assign align for duplicated divisions ([dask#11786](https://github.com/dask/dask/pull/11786)) [Patrick Hoefler](https://github.com/phofl)
- Ensure concat optimize project does not raise ([dask#11784](https://github.com/dask/dask/pull/11784)) [Florian Jetter](https://github.com/fjetter)
- Add array-expr from_array ([dask#11772](https://github.com/dask/dask/pull/11772)) [Patrick Hoefler](https://github.com/phofl)
- Keep chunksizes consistent in `apply_gufunc` ([dask#11683](https://github.com/dask/dask/pull/11683)) [Patrick Hoefler](https://github.com/phofl)
- Test `dask.dataframe.__all__` ([dask#11782](https://github.com/dask/dask/pull/11782)) [Philipp A.](https://github.com/flying-sheep)
- Add `__all__` to `dask.bag` ([dask#11781](https://github.com/dask/dask/pull/11781)) [Philipp A.](https://github.com/flying-sheep)
- Add test for `dask.array.__all__` ([dask#11780](https://github.com/dask/dask/pull/11780)) [Philipp A.](https://github.com/flying-sheep)
- Bump `JamesIves/github-pages-deploy-action` from 4.7.2 to 4.7.3 ([dask#11777](https://github.com/dask/dask/pull/11777))
- Export `dask.array` members ([dask#11779](https://github.com/dask/dask/pull/11779)) [Philipp A.](https://github.com/flying-sheep)
- Fix `sorted_divisions_locations` with duplicates ([dask#11773](https://github.com/dask/dask/pull/11773)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix small typo in `best-practices.rst` ([dask#11775](https://github.com/dask/dask/pull/11775)) [Sergey Kolesnikov](https://github.com/SCORE1387)
- Allow unknown chunks in `blockwise` `adjust_chunks` ([dask#11769](https://github.com/dask/dask/pull/11769)) [Lindsey Gray](https://github.com/lgray)
- Fix crash in `asarray(..., like=...)` vs. `scipy.sparse` objects ([dask#11755](https://github.com/dask/dask/pull/11755)) [Guido Imperiale](https://github.com/crusaderky)
- Remove flaky optional dependency ([dask#11771](https://github.com/dask/dask/pull/11771)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add support for scipy sparray ([dask#11750](https://github.com/dask/dask/pull/11750)) [Philipp A.](https://github.com/flying-sheep)
- Added `flaky` to tests extra ([dask#11770](https://github.com/dask/dask/pull/11770)) [Tom Augspurger](https://github.com/tomaugspurger)
- Ensure divisions are plain scalars ([dask#11767](https://github.com/dask/dask/pull/11767)) [Tom Augspurger](https://github.com/tomaugspurger)
- Remove divisions code duplication ([dask#11764](https://github.com/dask/dask/pull/11764)) [Florian Jetter](https://github.com/fjetter)
- Ensure divisions not diverging from `npartitions` in Merge ([dask#11762](https://github.com/dask/dask/pull/11762)) [Florian Jetter](https://github.com/fjetter)
- Skip `test_visualize_int_overflow` on windows ([dask#11761](https://github.com/dask/dask/pull/11761)) [Florian Jetter](https://github.com/fjetter)
- Reduce pickle size for tasks ([dask#11687](https://github.com/dask/dask/pull/11687)) [Florian Jetter](https://github.com/fjetter)
- Implement `unify_chunks` and Rechunk ([dask#11692](https://github.com/dask/dask/pull/11692)) [Patrick Hoefler](https://github.com/phofl)
- Fix expression getitem to avoid alignment ([dask#11760](https://github.com/dask/dask/pull/11760)) [Patrick Hoefler](https://github.com/phofl)
- `arange(..., like=x)` embeds the graph of x ([dask#11754](https://github.com/dask/dask/pull/11754)) [Guido Imperiale](https://github.com/crusaderky)
- Simplify `assert_divisions` ([dask#11745](https://github.com/dask/dask/pull/11745)) [Florian Jetter](https://github.com/fjetter)
- Fix Projection logic for Series objects ([dask#11747](https://github.com/dask/dask/pull/11747)) [Patrick Hoefler](https://github.com/phofl)
- Remove bytes as keys ([dask#11757](https://github.com/dask/dask/pull/11757)) [Florian Jetter](https://github.com/fjetter)
- Ensure `map_partitions` returns Series object if function returns scalar ([dask#11756](https://github.com/dask/dask/pull/11756)) [Florian Jetter](https://github.com/fjetter)
- Don’t upload env twice ([dask#11748](https://github.com/dask/dask/pull/11748)) [Patrick Hoefler](https://github.com/phofl)
- Fix badges in readme ([distributed#9029](https://github.com/dask/distributed/pull/9029)) [Florian Jetter](https://github.com/fjetter)
- Properly forward cancellation reason ([distributed#9028](https://github.com/dask/distributed/pull/9028)) [Florian Jetter](https://github.com/fjetter)
- Fix `bokeh` circle ([distributed#9026](https://github.com/dask/distributed/pull/9026)) [Florian Jetter](https://github.com/fjetter)
- Ensure `FileInfo` can be serialized ([distributed#9025](https://github.com/dask/distributed/pull/9025)) [Florian Jetter](https://github.com/fjetter)
- Add ipykernel to skipped modules in code sampling ([distributed#9022](https://github.com/dask/distributed/pull/9022)) [Matthew Rocklin](https://github.com/mrocklin)
- SpecCluster: add option to *not* shut down the scheduler when the cluster is closed ([distributed#9021](https://github.com/dask/distributed/pull/9021)) [Taylor Braun-Jones](https://github.com/nocnokneo)
- Fix CI by using `client.persist(collection)` instead of `collection.persist()` ([distributed#9020](https://github.com/dask/distributed/pull/9020)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add redirect from prefix root to status ([distributed#9015](https://github.com/dask/distributed/pull/9015)) [Isaac](https://github.com/icykip)
- Bump `JamesIves/github-pages-deploy-action` from 4.7.2 to 4.7.3 ([distributed#9018](https://github.com/dask/distributed/pull/9018))
- Remove bytes keys from tests ([distributed#9017](https://github.com/dask/distributed/pull/9017)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2025-2-0"></a>

## 2025.2.0

### Highlights

This release includes a critical fix that fixes a deadlock that can arise
when seceded task are rescheduled, or cancelled and resubmitted, e.g. due
to a worker being lost.

See [distributed#8991](https://github.com/dask/distributed/pull/8991) by [Hendrik Makait](https://github.com/hendrikmakait) for more details.

### Additional changes

- Add big array example ([dask#11744](https://github.com/dask/dask/pull/11744)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix exploding chunksizes in pad for constant padding ([dask#11743](https://github.com/dask/dask/pull/11743)) [Patrick Hoefler](https://github.com/phofl)
- Move optimize method to base class ([dask#11742](https://github.com/dask/dask/pull/11742)) [Florian Jetter](https://github.com/fjetter)
- Add changelog entry for fixed deadlock ([dask#11741](https://github.com/dask/dask/pull/11741)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix graph creation in `dask-expr` `to_delayed` ([dask#11739](https://github.com/dask/dask/pull/11739)) [Patrick Hoefler](https://github.com/phofl)
- Remove culling from delayed optimisation ([dask#11737](https://github.com/dask/dask/pull/11737)) [Patrick Hoefler](https://github.com/phofl)
- Compute meta for from_map on the cluster ([dask#11738](https://github.com/dask/dask/pull/11738)) [Patrick Hoefler](https://github.com/phofl)
- Bugs in `__setitem__` with dask bool mask ([dask#11728](https://github.com/dask/dask/pull/11728)) [Guido Imperiale](https://github.com/crusaderky)
- Implement infrastructure, random, blockwise and Elemwise  ([dask#11689](https://github.com/dask/dask/pull/11689)) [Patrick Hoefler](https://github.com/phofl)
- `array` / `asarray` with both `like=` and `dtype=` ([dask#11733](https://github.com/dask/dask/pull/11733)) [Guido Imperiale](https://github.com/crusaderky)
- Fix annotations warnings test ([dask#11734](https://github.com/dask/dask/pull/11734)) [Patrick Hoefler](https://github.com/phofl)
- Catch warnings when writing to remote storage with to_parquet ([dask#11731](https://github.com/dask/dask/pull/11731)) [Patrick Hoefler](https://github.com/phofl)
- Remove LocalCluster from tests ([dask#11729](https://github.com/dask/dask/pull/11729)) [Patrick Hoefler](https://github.com/phofl)
- Fix partition pruning when using from_array ([dask#11725](https://github.com/dask/dask/pull/11725)) [Patrick Hoefler](https://github.com/phofl)
- Fix concatentation with mixed dtype columns ([dask#11727](https://github.com/dask/dask/pull/11727)) [Patrick Hoefler](https://github.com/phofl)
- `arange`: fix extreme values ([dask#11707](https://github.com/dask/dask/pull/11707)) [Guido Imperiale](https://github.com/crusaderky)
- Graph corruption on scalar `getitem` -> `setitem` ([dask#11723](https://github.com/dask/dask/pull/11723)) [Guido Imperiale](https://github.com/crusaderky)
- Never share buffers after compute() ([dask#11697](https://github.com/dask/dask/pull/11697)) [Guido Imperiale](https://github.com/crusaderky)
- Extract Dask Array from xarray DataArray in from_array ([dask#11712](https://github.com/dask/dask/pull/11712)) [Patrick Hoefler](https://github.com/phofl)
- `arange`: support kwargs ([dask#11710](https://github.com/dask/dask/pull/11710)) [Guido Imperiale](https://github.com/crusaderky)
- Ensure `normalize_token` is threadsafe ([dask#11709](https://github.com/dask/dask/pull/11709)) [Florian Jetter](https://github.com/fjetter)
- Expand advise for instance types and processes ([dask#11705](https://github.com/dask/dask/pull/11705)) [Florian Jetter](https://github.com/fjetter)
- Drop legacy timeseries implementation ([dask#11704](https://github.com/dask/dask/pull/11704)) [Florian Jetter](https://github.com/fjetter)
- Update Dask Cloud Provider documentation to include Nebius as a supported cloud option ([dask#11703](https://github.com/dask/dask/pull/11703)) [Alexander](https://github.com/SalikovAlex)
- Fix `normalize_chunks` when squashing into a single chunk ([dask#11702](https://github.com/dask/dask/pull/11702)) [Patrick Hoefler](https://github.com/phofl)
- Fix positional indexing with `newaxis` ([dask#11699](https://github.com/dask/dask/pull/11699)) [Patrick Hoefler](https://github.com/phofl)
- Set array backend in scipy-sparse-indexing ([dask#11700](https://github.com/dask/dask/pull/11700)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix `value_counts` shuffling strategy ([dask#11698](https://github.com/dask/dask/pull/11698)) [Patrick Hoefler](https://github.com/phofl)
- Disentangle core expression class from dataframe specific code ([dask#11688](https://github.com/dask/dask/pull/11688)) [Patrick Hoefler](https://github.com/phofl)
- Bump `conda-incubator/setup-miniconda` from 3.1.0 to 3.1.1 ([dask#11685](https://github.com/dask/dask/pull/11685))
- Fixup dataframe conversion from array methods ([dask#11684](https://github.com/dask/dask/pull/11684)) [Patrick Hoefler](https://github.com/phofl)
- Remove remaining artifacts of `fastparquet` ([dask#11682](https://github.com/dask/dask/pull/11682)) [Patrick Hoefler](https://github.com/phofl)
- Remove traceback from `sizeof` failure warning ([distributed#9006](https://github.com/dask/distributed/pull/9006)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Hotfix: Ignore negative occupancy ([distributed#9012](https://github.com/dask/distributed/pull/9012)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove expensive tokenization for key uniqueness check ([distributed#9009](https://github.com/dask/distributed/pull/9009)) [Patrick Hoefler](https://github.com/phofl)
- Fix CI for changes in `from_map` ([distributed#9011](https://github.com/dask/distributed/pull/9011)) [Patrick Hoefler](https://github.com/phofl)
- Avoid handling stale long-running messages on scheduler ([distributed#8991](https://github.com/dask/distributed/pull/8991)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump `test_stress` timeout ([distributed#9002](https://github.com/dask/distributed/pull/9002)) [Tom Augspurger](https://github.com/tomaugspurger)
- Poll in `test_rmm_metrics` test ([distributed#9004](https://github.com/dask/distributed/pull/9004)) [Tom Augspurger](https://github.com/tomaugspurger)
- Cache occupancy in `WorkStealing.balance()` ([distributed#9005](https://github.com/dask/distributed/pull/9005)) [Hendrik Makait](https://github.com/hendrikmakait)
- Homogeneous balancing by accounting for in-flight requests ([distributed#9003](https://github.com/dask/distributed/pull/9003)) [Hendrik Makait](https://github.com/hendrikmakait)
- Consistent estimation of task duration between stealing, adaptive and occupancy calculation ([distributed#9000](https://github.com/dask/distributed/pull/9000)) [Hendrik Makait](https://github.com/hendrikmakait)
- Increase default work-stealing interval by 10x ([distributed#8997](https://github.com/dask/distributed/pull/8997)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove occupancy plot from status dashboard ([distributed#8995](https://github.com/dask/distributed/pull/8995)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump `conda-incubator/setup-miniconda` from 3.1.0 to 3.1.1 ([distributed#8990](https://github.com/dask/distributed/pull/8990))

<a id="v2025-1-0"></a>

## 2025.1.0

### Highlights

#### Legacy Dask DataFrame Implementation removed

This release drops the legacy Dask DataFrame implementation. The API with
query planning is now the only available Dask DataFrame implementation.

This enforces the deprecation of the configuration:

```default
dask.config.set({"dataframe.query-planning": False})
```

Dask-Expr was merged into the dask package as well as the dask/dask repository.
It is no longer necessary to install dask-expr separately.

#### Reducing Memory Pressure for Xarray Workloads

Dask introduced a mechanism that is called [root task queuing](https://distributed.dask.org/en/stable/scheduling-policies.html#queuing)
in 2022. This mechanism allows Dask to detect tasks that are reading data from storage
and schedule them defensively to avoid memory pressure on the cluster through overproduction
of these tasks. The underlying mechanism was very fragile and failed for specific types of
computations like opening multiple zarr stores or loading a large number of netcdf files.

The recent changes in Dask’s task graph representation allow for more robust
detection of root tasks. This change makes the detection mechanism independent of the workload
running and is especially beneficial for Xarray workloads.

This results in significantly more memory stability and a reduced memory footprint
for workloads where root task detection was previously failing and makes the expected memory
profile deterministic and independent of the topology of the task graph.

<a id="v2024-12-1"></a>

## 2024.12.1

### Highlights

#### Improved scheduler responsiveness for large task graphs

This release reduces the number of Python object references
related to tracking tasks by the Dask scheduler. This increases
scheduler responsiveness by reducing the time needed to run
garbage collection on the scheduler.

See [dask#8958](https://github.com/dask/dask/issues/8958), [dask#11608](https://github.com/dask/dask/pull/11608), [dask#11600](https://github.com/dask/dask/pull/11600), [dask#11598](https://github.com/dask/dask/pull/11598),
[dask#11597](https://github.com/dask/dask/pull/11597), and [distributed#8963](https://github.com/dask/distributed/pull/8963) from [Hendrik Makait](https://github.com/hendrikmakait) for more details.

### Additional changes

- Fix `map_overlap` bug where rechunking and `trim=False` caused inconsistent chunkings ([dask#11605](https://github.com/dask/dask/pull/11605)) [Patrick Hoefler](https://github.com/phofl)
- Avoid legacy implementation in read-csv ([dask#11603](https://github.com/dask/dask/pull/11603)) [Patrick Hoefler](https://github.com/phofl)
- Remove legacy DataFrame import ([dask#11604](https://github.com/dask/dask/pull/11604)) [Patrick Hoefler](https://github.com/phofl)
- `asarray` ignores `dtype` for array inputs ([dask#11586](https://github.com/dask/dask/pull/11586)) [crusaderky](https://github.com/crusaderky)
- Add back LLM chatbot to Dask docs ([dask#11594](https://github.com/dask/dask/pull/11594)) [dchudz](https://github.com/dchudz)
- Bump `JamesIves/github-pages-deploy-action` from 4.6.9 to 4.7.2 ([dask#11593](https://github.com/dask/dask/pull/11593))
- Migrate dask array creation routines to task spec ([dask#11582](https://github.com/dask/dask/pull/11582)) [James Bourbeau](https://github.com/jrbourbeau)
- Migrate most of dask array random to task spec ([dask#11581](https://github.com/dask/dask/pull/11581)) [James Bourbeau](https://github.com/jrbourbeau)
- Do not use local function in `array.push` ([dask#11576](https://github.com/dask/dask/pull/11576)) [Florian Jetter](https://github.com/fjetter)
- Bump `conda-incubator/setup-miniconda` from 3.0.3 to 3.1.0 ([distributed#8922](https://github.com/dask/distributed/pull/8922))
- Pick random dashboard port in tests ([distributed#8965](https://github.com/dask/distributed/pull/8965)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix formatting for `NoValidWorkerException` message ([distributed#8967](https://github.com/dask/distributed/pull/8967)) [Hendrik Makait](https://github.com/hendrikmakait)
- Support `pynvml>=11.5` in WSL ([distributed#8962](https://github.com/dask/distributed/pull/8962)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bump `JamesIves/github-pages-deploy-action` from 4.6.9 to 4.7.2 ([distributed#8960](https://github.com/dask/distributed/pull/8960))

<a id="v2024-12-0"></a>

## 2024.12.0

### Highlights

#### Python 3.13 Support

This release adds support for Python 3.13. Dask now supports Python 3.10-3.13.

See [dask#11456](https://github.com/dask/dask/pull/11456) and [distributed#8904](https://github.com/dask/distributed/pull/8904) from [Patrick Hoefler](https://github.com/phofl) and [James Bourbeau](https://github.com/jrbourbeau) for more details.

### Additional changes

- Revert “Add LLM chatbot to Dask docs ([dask#11556](https://github.com/dask/dask/pull/11556))” ([dask#11577](https://github.com/dask/dask/pull/11577)) [dchudz](https://github.com/dchudz)
- Automatically rechunk if array in `to_zarr` has irregular chunks ([dask#11553](https://github.com/dask/dask/pull/11553)) [Patrick Hoefler](https://github.com/phofl)
- Blockwise uses `Task` class ([dask#11568](https://github.com/dask/dask/pull/11568)) [Florian Jetter](https://github.com/fjetter)
- Migrate `rechunk` and `reshape` to task spec ([dask#11555](https://github.com/dask/dask/pull/11555)) [Patrick Hoefler](https://github.com/phofl)
- Cache svg-representation for arrays ([dask#11560](https://github.com/dask/dask/pull/11560)) [Deepak Cherian](https://github.com/dcherian)
- Fix empty input for containers ([dask#11571](https://github.com/dask/dask/pull/11571)) [Florian Jetter](https://github.com/fjetter)
- Convert `Bag` graphs to `TaskSpec` graphs during optimization ([dask#11569](https://github.com/dask/dask/pull/11569)) [Florian Jetter](https://github.com/fjetter)
- Add LLM chatbot to Dask docs ([dask#11556](https://github.com/dask/dask/pull/11556)) [dchudz](https://github.com/dchudz)
- Fuse data nodes in linear fusion too ([dask#11549](https://github.com/dask/dask/pull/11549)) [Patrick Hoefler](https://github.com/phofl)
- Migrate slicing code to task spec ([dask#11548](https://github.com/dask/dask/pull/11548)) [Patrick Hoefler](https://github.com/phofl)
- Speed up `ArraySliceDep` tokenization ([dask#11551](https://github.com/dask/dask/pull/11551)) [Patrick Hoefler](https://github.com/phofl)
- Fix fusing of `p2p` barrier tasks ([dask#11543](https://github.com/dask/dask/pull/11543)) [Patrick Hoefler](https://github.com/phofl)
- Remove infra/mentions of GPU CI ([dask#11546](https://github.com/dask/dask/pull/11546)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Temporarily disable gpuCI update CI job ([dask#11545](https://github.com/dask/dask/pull/11545)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `BlockwiseDep` to implement `map_blocks` keywords ([dask#11542](https://github.com/dask/dask/pull/11542)) [Patrick Hoefler](https://github.com/phofl)
- Remove `optimize_slices` ([dask#11538](https://github.com/dask/dask/pull/11538)) [Patrick Hoefler](https://github.com/phofl)
- Make `reshape_blockwise` a noop if shape is the same ([dask#11541](https://github.com/dask/dask/pull/11541)) [Patrick Hoefler](https://github.com/phofl)
- Remove read-only flag from `open_arry` in `open_zarr` ([dask#11539](https://github.com/dask/dask/pull/11539)) [Patrick Hoefler](https://github.com/phofl)
- Implement `linear_fusion` for task spec class ([dask#11525](https://github.com/dask/dask/pull/11525)) [Patrick Hoefler](https://github.com/phofl)
- Remove recursion from `TaskSpec` ([dask#11477](https://github.com/dask/dask/pull/11477)) [Florian Jetter](https://github.com/fjetter)
- Fixup test after dask-expr change ([dask#11536](https://github.com/dask/dask/pull/11536)) [Patrick Hoefler](https://github.com/phofl)
- Bump `codecov/codecov-action` from 3 to 5 ([dask#11532](https://github.com/dask/dask/pull/11532))
- Create dask-expr frame directly without roundtripping ([dask#11529](https://github.com/dask/dask/pull/11529)) [Patrick Hoefler](https://github.com/phofl)
- Add `scikit-image` nightly back to upstream CI ([dask#11530](https://github.com/dask/dask/pull/11530)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `from_dask_dataframe` import ([dask#11528](https://github.com/dask/dask/pull/11528)) [Patrick Hoefler](https://github.com/phofl)
- Ensure that `from_array` creates a copy ([dask#11524](https://github.com/dask/dask/pull/11524)) [Patrick Hoefler](https://github.com/phofl)
- Simplify and improve performance of normalize chunks ([dask#11521](https://github.com/dask/dask/pull/11521)) [Patrick Hoefler](https://github.com/phofl)
- Fix flaky `nanquantile` test ([dask#11518](https://github.com/dask/dask/pull/11518)) [Patrick Hoefler](https://github.com/phofl)
- Fix tests for new `read_only` kwarg in `zarr=3` ([dask#11516](https://github.com/dask/dask/pull/11516)) [Patrick Hoefler](https://github.com/phofl)
- Fix `test_jupyter.py::test_shutsdown_cleanly` ([distributed#8954](https://github.com/dask/distributed/pull/8954)) [Hendrik Makait](https://github.com/hendrikmakait)
- Install `tornado` from `conda-forge` in Python 3.13 CI ([distributed#8951](https://github.com/dask/distributed/pull/8951)) [James Bourbeau](https://github.com/jrbourbeau)
- Restore retire workers API ([distributed#8939](https://github.com/dask/distributed/pull/8939)) [Florian Jetter](https://github.com/fjetter)
- Properly convert finalize dependencies to references ([distributed#8949](https://github.com/dask/distributed/pull/8949)) [Hendrik Makait](https://github.com/hendrikmakait)
- Block fusion for barrier tasks ([distributed#8944](https://github.com/dask/distributed/pull/8944)) [Patrick Hoefler](https://github.com/phofl)
- Remove infra/mentions of GPUCI ([distributed#8946](https://github.com/dask/distributed/pull/8946)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Temporarily disable gpuCI update CI job ([distributed#8945](https://github.com/dask/distributed/pull/8945)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove recursion in task spec ([distributed#8920](https://github.com/dask/distributed/pull/8920)) [Florian Jetter](https://github.com/fjetter)
- Less verbose log messages for remove and register worker ([distributed#8938](https://github.com/dask/distributed/pull/8938)) [Florian Jetter](https://github.com/fjetter)
- Do not log full worker info in `retire_workers` ([distributed#8935](https://github.com/dask/distributed/pull/8935)) [Florian Jetter](https://github.com/fjetter)

<a id="v2024-11-2"></a>

## 2024.11.2

#### NOTE
Versions 2024.11.0 and 2024.11.1 included a critical performance regression
and should be skipped by every user.

### Highlights

#### Legacy Dask DataFrame Deprecated

This release deprecates the legacy Dask DataFrame implementation. The old implementation will
be removed completely in a future release. Users are encourage to switch to the new implementation
now and to report any issues they are facing.

Users are also encourage to check that they are only importing functions from `dask.dataframe`
and not any of the submodules.

#### New quantile methods for Dask Array API

Dask Array added new `quantile` and `nanquantile` methods.
Previously, Dask dispatched to the NumPy implementation, which blocked the GIL
a lot. This caused large slowdowns on workers with more than one tread and could lead
to runtimes over 200s per chunk.

The new `quantile` implementation avoids many of these problems and reduces runtime
to around 1s per chunk independently of the number of threads.

#### Consistent chunksize in Xarray rolling-construct

Using Xarrays `rolling(...).construct(...)` with Dask Arrays led to very large
chunksizes that rarely fit into memory on a single worker.

The underlying operations is a view on the smaller NumPy array, but triggering
a copy of the data will lead to very large memory usage.

```default
import xarray as xr
import dask.array as da

arr = xr.DataArray(
    da.ones((93504, 721, 1440), chunks=("auto", -1, -1)),
    dims=["time", "lat", "longitude"],
)   # Initial chunks are ~128 MiB
arr.rolling(time=30).construct("window_dim")
```

**Previously**

Individual chunks are exploding to 10 GiB, likely causing out of memory errors.

![Individual chunks are exploding to 10 GiB, likely causing out of memory errors.](images/changelog/rolling-construct-exploding-chunks.png)

**Now**

Dask will now automatically split individual chunks into chunks that will have the
same chunksize minus a small tolerance.

![Individual chunks are now roughly the same size](images/changelog/rolling-construct-constant-chunks.png)

#### Improved efficiency of map overlap

`map_overlap` now creates smaller and more efficient graphs to keep task graphs
generally a lot smaller.

The previous version injected a lot of tasks that weren’t necessary, increasing the
number of tasks by a factor of 2-10x of what actually necessary. This caused a lot of
stress on the scheduler.

#### Consistent chunksizes for Einstein summation

Einstein summation historically led to very large chunksizes if applied to more than
one Dask Array. This behavior is inherited from NumPy but led to out of memory errors
on workers:

```default
import dask.array as da
arr = da.random.random((1024, 64, 64, 64, 64), chunks=(256, 16, 16, 16, 16)) # Initial chunks are 128 MiB
result = da.einsum("aijkl,amnop->ijklmnop", arr, arr)
```

**Previously**

Individual chunks are exploding to 32 GiB, very likely causing out of memory errors.

![Individual chunks are exploding to 32 GiB, very likely causing out of memory errors](images/changelog/einstein-exploding-chunks.png)

**Now**

The operation keeps individual chunksizes the same.

![Individual chunks are now roughly the same size](images/changelog/einstein-constant-chunks.png)

### Additional changes

- Add changelog for Dask release ([dask#11502](https://github.com/dask/dask/pull/11502)) [Patrick Hoefler](https://github.com/phofl)
- Minor updates to optional dependencies table ([dask#11503](https://github.com/dask/dask/pull/11503)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `push` for `ffill` like operations ([dask#11501](https://github.com/dask/dask/pull/11501)) [Patrick Hoefler](https://github.com/phofl)
- Remove `func` packing for `TaskSpec` ([dask#11496](https://github.com/dask/dask/pull/11496)) [Florian Jetter](https://github.com/fjetter)
- Make tokenization for `vindex` more efficient ([dask#11493](https://github.com/dask/dask/pull/11493)) [Patrick Hoefler](https://github.com/phofl)
- Cut down runtime of einstein summation test ([dask#11499](https://github.com/dask/dask/pull/11499)) [Patrick Hoefler](https://github.com/phofl)
- Improve test runtime for `test_rot90` ([dask#11498](https://github.com/dask/dask/pull/11498)) [Florian Jetter](https://github.com/fjetter)
- Disable low level optimization for `TaskSpec` in Bags ([dask#11495](https://github.com/dask/dask/pull/11495)) [Florian Jetter](https://github.com/fjetter)
- Add automatic rechunking to sliding-window-view ([dask#11479](https://github.com/dask/dask/pull/11479)) [Patrick Hoefler](https://github.com/phofl)
- Add `load_stored` kwarg to `dask.array.store` ([dask#11465](https://github.com/dask/dask/pull/11465)) [Deepak Cherian](https://github.com/dcherian)
- Fix `quantile` error in two dimensions ([dask#11489](https://github.com/dask/dask/pull/11489)) [Patrick Hoefler](https://github.com/phofl)
- Bump `conda-incubator/setup-miniconda` from 3.0.4 to 3.1.0 ([dask#11490](https://github.com/dask/dask/pull/11490))
- Update `map_blocks` docstring ([dask#11491](https://github.com/dask/dask/pull/11491)) [Patrick Hoefler](https://github.com/phofl)
- Fix `einsum` with empty arrays ([dask#11488](https://github.com/dask/dask/pull/11488)) [Patrick Hoefler](https://github.com/phofl)
- Implement non gil-blocking `quantile` method ([dask#11473](https://github.com/dask/dask/pull/11473)) [Patrick Hoefler](https://github.com/phofl)
- Use internal keyword for trimming in `map_overlap` to reduce graph size ([dask#11486](https://github.com/dask/dask/pull/11486)) [Patrick Hoefler](https://github.com/phofl)
- Minor dask `order` refactor ([dask#11467](https://github.com/dask/dask/pull/11467)) [Florian Jetter](https://github.com/fjetter)
- Remove empty tasks from `map_overlap` ([dask#11483](https://github.com/dask/dask/pull/11483)) [Patrick Hoefler](https://github.com/phofl)
- Fixup auto chunks calculation if single chunk goes below 1 ([dask#11485](https://github.com/dask/dask/pull/11485)) [Patrick Hoefler](https://github.com/phofl)
- Fix CI after pandas upstream changes ([dask#11482](https://github.com/dask/dask/pull/11482)) [Patrick Hoefler](https://github.com/phofl)
- Make sure that `block_id` and `block_info` don’t create extra tasks ([dask#11484](https://github.com/dask/dask/pull/11484)) [Patrick Hoefler](https://github.com/phofl)
- Use repeat to build nearest boundary ([dask#9666](https://github.com/dask/dask/pull/9666)) [Jean-Baptiste Bayle](https://github.com/j2bbayle)
- Remove dead code from `make_blockwise` ([dask#11478](https://github.com/dask/dask/pull/11478)) [Florian Jetter](https://github.com/fjetter)
- Patch auto-chunks calculation for `rioxarray` ([dask#11480](https://github.com/dask/dask/pull/11480)) [Patrick Hoefler](https://github.com/phofl)
- Skip legacy test because of flaky warning ([dask#11475](https://github.com/dask/dask/pull/11475)) [Patrick Hoefler](https://github.com/phofl)
- Unskip a few `dask-expr` tests ([dask#11474](https://github.com/dask/dask/pull/11474)) [Patrick Hoefler](https://github.com/phofl)
- Keep chunk sizes consistent in `einsum` ([dask#11464](https://github.com/dask/dask/pull/11464)) [Patrick Hoefler](https://github.com/phofl)
- Improve how `normalize_chunks` squashes together chunks when “auto” is set ([dask#11468](https://github.com/dask/dask/pull/11468)) [Patrick Hoefler](https://github.com/phofl)
- Fix `resolve_aliases` when multiple aliases are in graph ([dask#11469](https://github.com/dask/dask/pull/11469)) [Patrick Hoefler](https://github.com/phofl)
- Avoid cyclic import in `dask.array` ([dask#11472](https://github.com/dask/dask/pull/11472)) [Hendrik Makait](https://github.com/hendrikmakait)
- Unskip dataframe test ([dask#11471](https://github.com/dask/dask/pull/11471)) [Patrick Hoefler](https://github.com/phofl)
- Improve `dask.order` performance for large graphs ([dask#11466](https://github.com/dask/dask/pull/11466)) [Florian Jetter](https://github.com/fjetter)
- Ensure that `slice(None)` just maps the keys ([dask#11450](https://github.com/dask/dask/pull/11450)) [Patrick Hoefler](https://github.com/phofl)
- Fix `Task.__repr__()` of unpickled object ([dask#11463](https://github.com/dask/dask/pull/11463)) [Peter Andreas Entschev](https://github.com/pentschev)
- Use `TaskSpec` in local dask execution ([dask#11378](https://github.com/dask/dask/pull/11378)) [Florian Jetter](https://github.com/fjetter)
- Adjust accuracy in `test_solve_triangular_vector` ([dask#11461](https://github.com/dask/dask/pull/11461)) [Florian Jetter](https://github.com/fjetter)
- Update Aggregation docstring ([dask#11459](https://github.com/dask/dask/pull/11459)) [Guillaume Eynard-Bontemps](https://github.com/guillaumeeb)
- Implement fuse option for `delayed` objects ([dask#11441](https://github.com/dask/dask/pull/11441)) [Patrick Hoefler](https://github.com/phofl)
- Deprecate legacy dask dataframe implementation ([dask#11437](https://github.com/dask/dask/pull/11437)) [Patrick Hoefler](https://github.com/phofl)
- Fix `na` casting behavior for `groupby.agg` with arrow dtypes ([dask#11118](https://github.com/dask/dask/pull/11118)) [Patrick Hoefler](https://github.com/phofl)
- Fix behavior of `keys_in_tasks` for `TaskSpec` nodes ([dask#11445](https://github.com/dask/dask/pull/11445)) [Florian Jetter](https://github.com/fjetter)
- Convert dtype to int instead of np.uint8 for visualizing large task graphs ([dask#11440](https://github.com/dask/dask/pull/11440)) [Patrick Hoefler](https://github.com/phofl)
- Ensure dependencies are not mutated ([dask#11438](https://github.com/dask/dask/pull/11438)) [Florian Jetter](https://github.com/fjetter)
- Full support for task spec in `dask.order` ([dask#11347](https://github.com/dask/dask/pull/11347)) [Florian Jetter](https://github.com/fjetter)
- Remove redundant methods in `P2PBarrierTask` ([distributed#8924](https://github.com/dask/distributed/pull/8924)) [Florian Jetter](https://github.com/fjetter)
- Fix `skipif` condition for `test_tell_workers_when_peers_have_left` ([distributed#8929](https://github.com/dask/distributed/pull/8929)) [Florian Jetter](https://github.com/fjetter)
- Ensure `ConnectionPool` is closed even if network stack swallows `CancelledErrors` ([distributed#8928](https://github.com/dask/distributed/pull/8928)) [Florian Jetter](https://github.com/fjetter)
- Fix flaky `test_server_comms_mark_active_handlers` ([distributed#8927](https://github.com/dask/distributed/pull/8927)) [Florian Jetter](https://github.com/fjetter)
- Make assumption in P2P’s barrier mechanism explicit ([distributed#8926](https://github.com/dask/distributed/pull/8926)) [Hendrik Makait](https://github.com/hendrikmakait)
- Adjust timeouts in Jupyter cli test ([distributed#8925](https://github.com/dask/distributed/pull/8925)) [Florian Jetter](https://github.com/fjetter)
- Add `stimulus_id` to `update_graph` plugin hook ([distributed#8923](https://github.com/dask/distributed/pull/8923)) [Hendrik Makait](https://github.com/hendrikmakait)
- Reduce P2P transfer task overhead ([distributed#8912](https://github.com/dask/distributed/pull/8912)) [Hendrik Makait](https://github.com/hendrikmakait)
- Disable profiler on Python 3.11 ([distributed#8916](https://github.com/dask/distributed/pull/8916)) [Florian Jetter](https://github.com/fjetter)
- Fix `test_restarting_does_not_deadlock` ([distributed#8849](https://github.com/dask/distributed/pull/8849)) [Florian Jetter](https://github.com/fjetter)
- Adjust `popen` timeouts for testing ([distributed#8848](https://github.com/dask/distributed/pull/8848)) [Florian Jetter](https://github.com/fjetter)
- Add retry to shuffle broadcast ([distributed#8900](https://github.com/dask/distributed/pull/8900)) [Florian Jetter](https://github.com/fjetter)
- Fix `test_shuffle_with_array_conversion` ([distributed#8909](https://github.com/dask/distributed/pull/8909)) [Florian Jetter](https://github.com/fjetter)
- Refactor some tests ([distributed#8908](https://github.com/dask/distributed/pull/8908)) [Florian Jetter](https://github.com/fjetter)
- Graduate `dask-expr` from contrib to core project ([distributed#8911](https://github.com/dask/distributed/pull/8911)) [Hendrik Makait](https://github.com/hendrikmakait)
- Skip `test_tell_workers_when_peers_have_left` on py10 ([distributed#8910](https://github.com/dask/distributed/pull/8910)) [Florian Jetter](https://github.com/fjetter)
- Internal cleanup of P2P code ([distributed#8907](https://github.com/dask/distributed/pull/8907)) [Hendrik Makait](https://github.com/hendrikmakait)
- Use `Task` class instead of tuple ([distributed#8797](https://github.com/dask/distributed/pull/8797)) [Florian Jetter](https://github.com/fjetter)
- Increase connect timeout for `test_tell_workers_when_peers_have_left` ([distributed#8906](https://github.com/dask/distributed/pull/8906)) [Florian Jetter](https://github.com/fjetter)
- Remove dispatching in `TaskCollection` ([distributed#8903](https://github.com/dask/distributed/pull/8903)) [Florian Jetter](https://github.com/fjetter)
- Deduplicate requests to scheduler in P2P ([distributed#8899](https://github.com/dask/distributed/pull/8899)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add configurations for rootish taskgroup threshold ([distributed#8898](https://github.com/dask/distributed/pull/8898)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2024-10-0"></a>

## 2024.10.0

### Notable Changes

- Zarr-Python 3 compatibility  ([dask#11388](https://github.com/dask/dask/pull/11388))
- Avoid exponentially increasing taskgraph in overlap ([dask#11423](https://github.com/dask/dask/pull/11423))
- Ensure numba tokenization does not use slow pickle path ([dask#11419](https://github.com/dask/dask/pull/11419))

### Additional changes

- Ensure broadcast_shapes() returns integers, not NumPy scalars. ([dask#11434](https://github.com/dask/dask/pull/11434)) [Martin Yeo](https://github.com/trexfeathers)
- (fix): sparse indexing ([dask#11430](https://github.com/dask/dask/pull/11430)) [Ilan Gold](https://github.com/ilan-gold)
- Ensure that recursively calling tokenize respects ensure_deterministic ([dask#11431](https://github.com/dask/dask/pull/11431)) [Florian Jetter](https://github.com/fjetter)
- Make P2P more configurable ([distributed#8469](https://github.com/dask/distributed/pull/8469)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fit Dashboard worker table to page width ([distributed#8897](https://github.com/dask/distributed/pull/8897)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Raise helpful error when using the wrong plugin base classes ([distributed#8893](https://github.com/dask/distributed/pull/8893)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix url escaping on exceptions dashboard for non-string keys ([distributed#8891](https://github.com/dask/distributed/pull/8891)) [Patrick Hoefler](https://github.com/phofl)
- Add meaningful error for out of disk exception during write ([distributed#8886](https://github.com/dask/distributed/pull/8886)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix binary operations with scalar on the left ([dask-expr#1150](https://github.com/dask/dask-expr/pull/1150)) [Patrick Hoefler](https://github.com/phofl)
- Raise exception when calculating divisons ([dask-expr#1149](https://github.com/dask/dask-expr/pull/1149)) [Patrick Hoefler](https://github.com/phofl)
- Fix merge_asof for single partition ([dask-expr#1145](https://github.com/dask/dask-expr/pull/1145)) [Patrick Hoefler](https://github.com/phofl)
- Improve handling of optional dependencies in analyze and explain ([dask-expr#1146](https://github.com/dask/dask-expr/pull/1146)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix alignment issue with groupby index accessors ([dask-expr#1142](https://github.com/dask/dask-expr/pull/1142)) [Patrick Hoefler](https://github.com/phofl)
- Fix displaying timestamp scalar ([dask-expr#1141](https://github.com/dask/dask-expr/pull/1141)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2024-9-1"></a>

## 2024.9.1

### Highlights

#### Improved adaptive scaling resilience

Adaptive scaling clusters now recover from spurious errors during scaling.

See [distributed#8871](https://github.com/dask/distributed/pull/8871) by [Hendrik Makait](https://github.com/hendrikmakait) for more details.

### Additional changes

- Improve error message for incorrect columns order in meta information ([dask#11393](https://github.com/dask/dask/pull/11393)) [Dmitry Balabka](https://github.com/dbalabka)
- Update gpuCI `RAPIDS_VER` to `24.12` ([dask#11407](https://github.com/dask/dask/pull/11407))
- Bump `jacobtomlinson/gha-anaconda-package-version` from 0.1.3 to 0.1.4 ([dask#11405](https://github.com/dask/dask/pull/11405))
- Switch to using `zarr.open_array` instead of using the `zarr.Array` constructor ([dask#11387](https://github.com/dask/dask/pull/11387)) [Joe Hamman](https://github.com/jhamman)
- Update gpuCI `RAPIDS_VER` to `24.12` ([distributed#8879](https://github.com/dask/distributed/pull/8879))
- Don’t consider scheduler idle while executing `Scheduler.update_graph` ([distributed#8877](https://github.com/dask/distributed/pull/8877)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump `jacobtomlinson/gha-anaconda-package-version` from 0.1.3 to 0.1.4 ([distributed#8878](https://github.com/dask/distributed/pull/8878))
- Support P2P rechunking datetime arrays ([distributed#8875](https://github.com/dask/distributed/pull/8875)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-9-0"></a>

## 2024.9.0

### Highlights

#### Bump Bokeh minimum version to 3.1.0

`bokeh>=3.1.0` is now required for diagnostics and the distributed cluster dashboard.

See [dask#11375](https://github.com/dask/dask/pull/11375) and [distributed#8861](https://github.com/dask/distributed/pull/8861) by [James Bourbeau](https://github.com/jrbourbeau) for more details.

#### Introduce new Task class

Add a `Task` class to replace tuples for task specification.

See [dask#11248](https://github.com/dask/dask/pull/11248) by [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Bump `peter-evans/create-pull-request` from 6 to 7 ([dask#11380](https://github.com/dask/dask/pull/11380))
- Reduce overhead in tokenize ([dask#11373](https://github.com/dask/dask/pull/11373)) [Florian Jetter](https://github.com/fjetter)
- Move `tokenize` to dedicated submodule ([dask#11371](https://github.com/dask/dask/pull/11371)) [Florian Jetter](https://github.com/fjetter)
- Ensure `process_runnables` is not too eager in the presence of multiple splits ([dask#11367](https://github.com/dask/dask/pull/11367)) [Florian Jetter](https://github.com/fjetter)
- Use `np.min_scalar_type` in shuffle ([dask#11369](https://github.com/dask/dask/pull/11369)) [James Bourbeau](https://github.com/jrbourbeau)
- Write indexing arrays into dask graph to reduce size for multiple xarray variables ([dask#11362](https://github.com/dask/dask/pull/11362)) [Patrick Hoefler](https://github.com/phofl)
- Cast indexer to minimal `dtype` in shuffle ([dask#11364](https://github.com/dask/dask/pull/11364)) [Patrick Hoefler](https://github.com/phofl)
- Reduce memory usage of `dask.order` ([dask#11361](https://github.com/dask/dask/pull/11361)) [Florian Jetter](https://github.com/fjetter)
- Bump `JamesIves/github-pages-deploy-action` from 4.6.3 to 4.6.4 ([dask#11366](https://github.com/dask/dask/pull/11366))
- `precommit` autoupdate ([dask#11360](https://github.com/dask/dask/pull/11360)) [Florian Jetter](https://github.com/fjetter)
- Homogeneously schedule P2P’s unpack tasks ([distributed#8873](https://github.com/dask/distributed/pull/8873)) [Hendrik Makait](https://github.com/hendrikmakait)
- Work/fix firewall for localhost ([distributed#8868](https://github.com/dask/distributed/pull/8868)) [Mario Linker](https://github.com/maldag)
- Use new `tokenize` module ([distributed#8858](https://github.com/dask/distributed/pull/8858)) [James Bourbeau](https://github.com/jrbourbeau)
- Point to user code with idempotent plugin warning ([distributed#8856](https://github.com/dask/distributed/pull/8856)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix test nanny timeout ([distributed#8847](https://github.com/dask/distributed/pull/8847)) [Florian Jetter](https://github.com/fjetter)
- Bump JamesIves/github-pages-deploy-action from 4.5.0 to 4.6.4 ([distributed#8853](https://github.com/dask/distributed/pull/8853))
- Speed up `Client.map` by computing `token` only once for `func` and `kwargs` ([distributed#8855](https://github.com/dask/distributed/pull/8855)) [Florian Jetter](https://github.com/fjetter)
- Update `pre-commit` ([distributed#8852](https://github.com/dask/distributed/pull/8852)) [Florian Jetter](https://github.com/fjetter)

<a id="v2024-8-2"></a>

## 2024.8.2

### Highlights

#### Automatic selection of rechunking method

To enable users to rechunk data at larger scales than before, Dask now automatically chooses an appropriate rechunking method when rechunking on a cluster.
This requires no additional configuration and is enabled by default.

Specifically, Dask chooses between task-based and P2P rechunking.
While task-based rechunking has been the previous default, P2P rechunking is beneficial when rechunking requires almost all-to-all communication between the old and new chunks, e.g., when changing between spacial and temporal chunking.
In these cases, P2P rechunking offers constant memory usage and creates smaller task graphs.
As a result, it works for cases where tasks-based rechunking would have previously failed.

To disable automatic selection, users can select their preferred method via the configuration

```default
import dask.config
# Choose either "tasks" or "p2p"
dask.config.set({"array.rechunk.method": "tasks"})
```

or when rechunking

```default
import dask.array as da
arr = da.random.random(size=(1000, 1000, 365), chunks=(-1, -1, "auto"))
# Choose either "tasks" or "p2p"
arr = arr.rechunk(("auto", "auto", -1), method="tasks")
```

See [dask#11337](https://github.com/dask/dask/pull/11337) by [Hendrik Makait](https://github.com/hendrikmakait) for more details.

#### New shuffle API for Dask Arrays

Dask added a shuffle-API to Dask Arrays. This API allows for shuffling the data
along a single dimension. It will ensure that every group of elements along this
dimension are in exactly one chunk. This is a very useful operation for GroupBy-Map
patterns in Xarray. See [`shuffle()`](generated/dask.array.Array.shuffle.md#dask.array.Array.shuffle) for more information
and API signature.

See [dask#11267](https://github.com/dask/dask/pull/11267), [dask#11311](https://github.com/dask/dask/pull/11311) and [dask#11326](https://github.com/dask/dask/pull/11326) by [Patrick Hoefler](https://github.com/phofl) for more details.

#### New blockwise_reshape API for Dask Arrays

The new `blockwise_reshape()` enables an embarassingly parallel
reshaping operation for cases where you don’t care about the order of the underlying
array. It is embarassingly parallel and doesn’t trigger a rechunking operation
under the hood anymore. This is useful when you don’t care about the order of
the resulting Array, i.e. if a reduction is applied to the array or if the reshaping
is only temporary.

```default
arr = da.random.random(size=(100, 100, 48_000), chunks=(1000, 100, 83)
result = reshape_blockwise(arr, (10_000, 48_000))
result.sum()

# or: do something that preserves the shape of each chunk

result = reshape_blockwise(result, (100, 100, 48_000), chunks=arr.chunks)
```

Dask will automatically calculate the resulting chunks if the number of dimensions
is reduced, but you have to specify the resulting chunks if the number of dimensions
is increased.

Reshaping a Dask Array oftentimes creates a very complicated computations with rechunk
operations in between because Dask respect the C ordering of the Array by default. This
ensures that the resulting Dask Array is returned in the same order as the
corresponding NumPy Array. However, this can lead to very inefficient computations.
The `blockwise_reshape` is a lot more efficient than the default implemenation
if you don’t care about the order.

#### WARNING
Blockwise reshape operations are more efficient as the default, but they will
return an Array that is ordered differently. Use with care!

See [dask#11328](https://github.com/dask/dask/pull/11328) by [Patrick Hoefler](https://github.com/phofl) for more details.

#### Mutlidimensional positional indexing keeping chunksizes consistent

Indexing a Dask Array with `vindex()` previously created a single
output chunk along the dimensions that were indexed. `vindex` is commonly used in Xarray
when indexing multiple dimensions in a single step, i.e.:

```default
arr = xr.DataArray(
    da.random.random((100, 100, 100), chunks=(5, 5, 50)),
    dims=['a', "b", "c"],
)
```

Previously, this put the indexed dimensions into a single chunk:

![Size of each individual chunk increases to over 1GB](images/changelog/vindex-memory-increase.png)

Dask now uses an improved algorithm that ensures that the chunksizes are kept consistent:

![Size of each individual chunk increases to over 1GB](images/changelog/vindex-memory-constant.png)

See [dask#11330](https://github.com/dask/dask/pull/11330) by [Patrick Hoefler](https://github.com/phofl) for more details.

### Additional changes

- Add changelog entries for shuffle, `vindex` and `blockwise_reshape` ([dask#11350](https://github.com/dask/dask/pull/11350)) [Patrick Hoefler](https://github.com/phofl)
- Ensure persisted collections are released without GC ([dask#11348](https://github.com/dask/dask/pull/11348)) [Florian Jetter](https://github.com/fjetter)
- Update zoom link for dask meeting ([dask#11357](https://github.com/dask/dask/pull/11357)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add more docstring examples for `normalize_chunks` ([dask#11271](https://github.com/dask/dask/pull/11271)) [Illviljan](https://github.com/Illviljan)
- Choose automatically between tasks-based and p2p rechunking ([dask#11337](https://github.com/dask/dask/pull/11337)) [Hendrik Makait](https://github.com/hendrikmakait)
- Implement blockwise reshaping API for arrays ([dask#11328](https://github.com/dask/dask/pull/11328)) [Patrick Hoefler](https://github.com/phofl)
- Make rechunking in shuffle more intelligent to distribute unevenly if necessary ([dask#11326](https://github.com/dask/dask/pull/11326)) [Patrick Hoefler](https://github.com/phofl)
- Increase visibility of GPU CI updates ([dask#11345](https://github.com/dask/dask/pull/11345)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Update `numpy` and `pyarrow` versions in install docs ([dask#11340](https://github.com/dask/dask/pull/11340)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixup dask and distributed dependencies ([dask#11338](https://github.com/dask/dask/pull/11338)) [Patrick Hoefler](https://github.com/phofl)
- Bump `numpy>=1.24` and `pyarrow>=14.0.1` minimum versions ([dask#11331](https://github.com/dask/dask/pull/11331)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `crick` back to Python 3.11+ CI builds ([dask#11335](https://github.com/dask/dask/pull/11335)) [James Bourbeau](https://github.com/jrbourbeau)
- Preserve chunksizes in `vindex` ([dask#11330](https://github.com/dask/dask/pull/11330)) [Patrick Hoefler](https://github.com/phofl)
- Fix `dask.array.fft` mismatch with Numpy’s interface (add support for norm argument) ([dask#10665](https://github.com/dask/dask/pull/10665)) [joanrue](https://github.com/joanrue)
- Pass additional parameters to `rechunk_p2p` ([dask#11319](https://github.com/dask/dask/pull/11319)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix docstring formatting for `map_overlap` ([dask#11332](https://github.com/dask/dask/pull/11332)) [Tao Xin](https://github.com/Tao-VanJS)
- Fix NumPy overflowing for `prod` on 2.0 ([dask#11327](https://github.com/dask/dask/pull/11327)) [Patrick Hoefler](https://github.com/phofl)
- Ensure `axes` are positive / add tests for negative axes ([dask#10812](https://github.com/dask/dask/pull/10812)) [joanrue](https://github.com/joanrue)
- Fix `map_overlap` with `new_axis` ([dask#11128](https://github.com/dask/dask/pull/11128)) [David Stansby](https://github.com/dstansby)
- Avoid capturing code of `xdist` ([distributed#8846](https://github.com/dask/distributed/pull/8846)) [Florian Jetter](https://github.com/fjetter)
- Reduce memory footprint of culling P2P rechunking ([distributed#8845](https://github.com/dask/distributed/pull/8845)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add tests for choosing default rechunking method ([distributed#8843](https://github.com/dask/distributed/pull/8843)) [Hendrik Makait](https://github.com/hendrikmakait)
- Increase visibility of GPU CI updates ([distributed#8841](https://github.com/dask/distributed/pull/8841)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Bump `test_pause_while_idle` timeout ([distributed#8844](https://github.com/dask/distributed/pull/8844)) [Florian Jetter](https://github.com/fjetter)
- Concatenate small input chunks before P2P rechunking ([distributed#8832](https://github.com/dask/distributed/pull/8832)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove dump cluster from `gen_cluster` ([distributed#8823](https://github.com/dask/distributed/pull/8823)) [Florian Jetter](https://github.com/fjetter)
- Bump `numpy>=1.24` and `pyarrow>=14.0.1` minimum versions ([distributed#8837](https://github.com/dask/distributed/pull/8837)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `PipInstall` plugin on `Worker` ([distributed#8839](https://github.com/dask/distributed/pull/8839)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove more Python 3.10 compatibility code ([distributed#8824](https://github.com/dask/distributed/pull/8824)) [James Bourbeau](https://github.com/jrbourbeau)
- Use task-based rechunking to prechunk along partial boundaries ([distributed#8831](https://github.com/dask/distributed/pull/8831)) [Hendrik Makait](https://github.com/hendrikmakait)
- Ensure `client_desires_keys` does not corrupt `Scheduler` state ([distributed#8827](https://github.com/dask/distributed/pull/8827)) [Florian Jetter](https://github.com/fjetter)
- Bump minimum `cloudpickle` to 3 ([distributed#8836](https://github.com/dask/distributed/pull/8836)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-8-1"></a>

## 2024.8.1

### Highlights

#### Improve output chunksizes for reshaping Dask Arrays

Reshaping a Dask Array oftentimes squashed the dimensions to reshape into a single
chunk. This caused very large output chunks and subsequently a lot of out of memory
errors and performance issues.

```default
arr = da.ones(shape=(1000, 100, 48_000), chunks=(1000, 100, 83))
arr.reshape(1000, 100, 4, 12_000)
```

Previously, this put the last dimension into a single chunk of size 12_000.

![Size of each individual chunk increases to over 1GB](images/changelog/reshape-memory-increase.png)

The new algorithm will ensure that the chunk-size between in- and output is kept
the same. This will avoid large increases in chunk-size and fragmentation of chunks.

![Size of each individual chunk stays the same](images/changelog/reshape-constant-memory.png)

#### Improve scheduling efficiency for Xarray Rechunk-GroupBy-Reduce patterns

The scheduler previously created an inefficient execution graph for Xarray GroupBy-Reduction
patterns that use the cohorts strategy:

```python
import xarray as xr

arr = xr.open_zarr(...)
arr.chunk(time=TimeResampler("ME")).groupby("time.month").mean()
```

An issue in the algorithm that creates the execution order of the task graph
lead to an inefficient execution strategy that accumulates a lot of unnecessary memory on
the cluster. The improvement is very similar to
[the previous ordering improvement in 2024.08.0](#label-xarray-groupby-ordering).

#### Drop support for Python 3.9

This release drops support for Python 3.9 in accordance with NEP 29. Python 3.10
is now the required minimum version to run Dask.

See [dask#11245](https://github.com/dask/dask/pull/11245) and [distributed#8793](https://github.com/dask/distributed/pull/8793) by [Patrick Hoefler](https://github.com/phofl) for more details.

### Additional changes

- Ensure `pickle` does not change tokens ([dask#11320](https://github.com/dask/dask/pull/11320)) [Florian Jetter](https://github.com/fjetter)
- Add changelog entry for `reshape` and ordering improvements ([dask#11324](https://github.com/dask/dask/pull/11324)) [Patrick Hoefler](https://github.com/phofl)
- Rename `chunksize-tolerance` option ([dask#11317](https://github.com/dask/dask/pull/11317)) [Patrick Hoefler](https://github.com/phofl)
- Upgrade gpuCI and fix Dask Array failures with “cupy” backend ([dask#11309](https://github.com/dask/dask/pull/11309)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Implement automatic rechunking for `shuffle` ([dask#11311](https://github.com/dask/dask/pull/11311)) [Patrick Hoefler](https://github.com/phofl)
- Ensure we test against `numpy` 2 in CI ([dask#11182](https://github.com/dask/dask/pull/11182)) [James Bourbeau](https://github.com/jrbourbeau)
- Revert “Test ordering on distributed scheduler ([dask#11310](https://github.com/dask/dask/pull/11310))” ([dask#11321](https://github.com/dask/dask/pull/11321)) [Florian Jetter](https://github.com/fjetter)
- Test ordering on distributed scheduler ([dask#11310](https://github.com/dask/dask/pull/11310)) [Florian Jetter](https://github.com/fjetter)
- Add tests to cover more cases of new `reshape` implementation ([dask#11313](https://github.com/dask/dask/pull/11313)) [Patrick Hoefler](https://github.com/phofl)
- Order: Choose better target for branches with multiple leaf nodes ([dask#11303](https://github.com/dask/dask/pull/11303)) [Patrick Hoefler](https://github.com/phofl)
- Order: Ensure runnable tasks are certainly runnable ([dask#11305](https://github.com/dask/dask/pull/11305)) [Florian Jetter](https://github.com/fjetter)
- Fix upstream `numpy` build ([dask#11304](https://github.com/dask/dask/pull/11304)) [Patrick Hoefler](https://github.com/phofl)
- Make `shuffle` a no-op if possible ([dask#11291](https://github.com/dask/dask/pull/11291)) [Patrick Hoefler](https://github.com/phofl)
- Keep `chunksize` consistent in `reshape` ([dask#11273](https://github.com/dask/dask/pull/11273)) [Patrick Hoefler](https://github.com/phofl)
- Enable slicing with only one unknown chunk ([dask#11301](https://github.com/dask/dask/pull/11301)) [Patrick Hoefler](https://github.com/phofl)
- Link to `dask` vs `spark` benchmarks on Dask docs ([dask#11289](https://github.com/dask/dask/pull/11289)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Fix slicing for masked arrays ([dask#11300](https://github.com/dask/dask/pull/11300)) [Patrick Hoefler](https://github.com/phofl)
- Array: fix `asarray` for array input with `dtype` ([dask#11288](https://github.com/dask/dask/pull/11288)) [Lucas Colley](https://github.com/lucascolley)
- Add `numpy` constants to array api ([dask#11287](https://github.com/dask/dask/pull/11287)) [Lucas Colley](https://github.com/lucascolley)
- Ignore typing of return value ([dask#11286](https://github.com/dask/dask/pull/11286)) [Patrick Hoefler](https://github.com/phofl)
- Remove automatic resizing in reshape ([dask#11269](https://github.com/dask/dask/pull/11269)) [Patrick Hoefler](https://github.com/phofl)
- API: expose `np` dtypes in `dask.array` namespace ([dask#11178](https://github.com/dask/dask/pull/11178)) [Lucas Colley](https://github.com/lucascolley)
- Reduce frequency of unmanaged memory use warning ([distributed#8834](https://github.com/dask/distributed/pull/8834)) [Patrick Hoefler](https://github.com/phofl)
- Update gpuCI `RAPIDS_VER` to `24.10` ([distributed#8786](https://github.com/dask/distributed/pull/8786))
- Avoid `RuntimeError: dictionary changed size during iteration` in `Server._shift_counters()` ([distributed#8828](https://github.com/dask/distributed/pull/8828)) [Hendrik Makait](https://github.com/hendrikmakait)
- Improve concurrent close for scheduler ([distributed#8829](https://github.com/dask/distributed/pull/8829)) [Hendrik Makait](https://github.com/hendrikmakait)
- MINOR: Extract truncation logic out of partial concatenation in P2P rechunking ([distributed#8826](https://github.com/dask/distributed/pull/8826)) [Hendrik Makait](https://github.com/hendrikmakait)
- avoid excessive attribute access overhead for `remove_from_task_prefix_count` ([distributed#8821](https://github.com/dask/distributed/pull/8821)) [Florian Jetter](https://github.com/fjetter)
- Avoid key validation if validation is disabled ([distributed#8822](https://github.com/dask/distributed/pull/8822)) [Florian Jetter](https://github.com/fjetter)
- Log `worker_client` event ([distributed#8819](https://github.com/dask/distributed/pull/8819)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-8-0"></a>

## 2024.8.0

### Highlights

#### Improve efficiency and performance of slicing with positional indexers

Performance improvement for slicing a Dask Array with a positional indexer.
Random access patterns are now more stable and produce easier-to-use results.

```python
x[slice(None), [1, 1, 3, 6, 3, 4, 5]]
```

Using a positional indexer was previously prone to drastically increasing the
number of output chunks and generating a very large task graph. This has been
fixed with a more efficient algorithm.

The new algorithm will keep the chunk-sizes along the axis that is indexed
the same to avoid fragmentation of chunks or a large increase in chunk-size.

See [dask#11262](https://github.com/dask/dask/pull/11262) and [dask#11267](https://github.com/dask/dask/pull/11267) by [Patrick Hoefler](https://github.com/phofl) for more details and performance
benchmarks.

<a id="label-xarray-groupby-ordering"></a>

#### Improve scheduling efficiency for Xarray GroupBy-Reduce patterns

The scheduler previously created an inefficient execution graph for Xarray GroupBy-Reduction
patterns like:

```python
import xarray as xr

arr = xr.open_zarr(...)
arr.groupby("time.month").mean()
```

An issue in the algorithm that creates the execution order of the task graph
lead to an inefficient execution strategy that accumulates a lot of unneceessary memory on
the cluster.

![Memory keeps accumulating on the cluster when running an embarassingly parallel
operation.](images/changelog/dask-order-growing-memory.png)

The operation itself is embarassingly parallel. Using the proper execution strategy
the scheduler can now execute the operation with constant memory, avoiding spilling
and allowing us to scale to larger datasets.

![Same operation is running with constant memory usage for the whole computation
and can scale for bigger datasets.](images/changelog/dask-order-constant-memory.png)

See [distributed#8818](https://github.com/dask/distributed/pull/8818) by [Patrick Hoefler](https://github.com/phofl) for more details and examples.

### Additional changes

- Add changelog for dask order patch ([dask#11278](https://github.com/dask/dask/pull/11278)) [Patrick Hoefler](https://github.com/phofl)
- Add regression test for `xarray` map reduce ([dask#11277](https://github.com/dask/dask/pull/11277)) [Florian Jetter](https://github.com/fjetter)
- Add changelog entry for `take` ([dask#11274](https://github.com/dask/dask/pull/11274)) [Patrick Hoefler](https://github.com/phofl)
- Revert “order: remove data task graph normalization” ([dask#11276](https://github.com/dask/dask/pull/11276)) [Patrick Hoefler](https://github.com/phofl)
- Use the shuffle algorithm for `take` ([dask#11267](https://github.com/dask/dask/pull/11267)) [Patrick Hoefler](https://github.com/phofl)
- Implement task-based array shuffle ([dask#11262](https://github.com/dask/dask/pull/11262)) [Patrick Hoefler](https://github.com/phofl)
- Remove data task graph normalization ([dask#11263](https://github.com/dask/dask/pull/11263)) [Florian Jetter](https://github.com/fjetter)
- Update zoom link for monthly meeting ([dask#11265](https://github.com/dask/dask/pull/11265)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Update data loading section of best practices ([dask#11247](https://github.com/dask/dask/pull/11247)) [Patrick Hoefler](https://github.com/phofl)
- Match default `chunksize` in docstring to actual default set in code ([dask#11254](https://github.com/dask/dask/pull/11254)) [Bernhard Raml](https://github.com/SwamyDev)
- Fixup casting error in `pandas` 3 ([dask#11250](https://github.com/dask/dask/pull/11250)) [Patrick Hoefler](https://github.com/phofl)
- Skip new warning from `pandas` ([dask#11249](https://github.com/dask/dask/pull/11249)) [Patrick Hoefler](https://github.com/phofl)
- Fix `pandas` nightly bugs ([dask#11244](https://github.com/dask/dask/pull/11244)) [Patrick Hoefler](https://github.com/phofl)
- Run graph normalisation after dask order ([distributed#8818](https://github.com/dask/distributed/pull/8818)) [Patrick Hoefler](https://github.com/phofl)
- Update large graph size warning to remove scatter recommendation ([distributed#8815](https://github.com/dask/distributed/pull/8815)) [Patrick Hoefler](https://github.com/phofl)
- Fail tasks exceeding `no-workers-timeout` ([distributed#8806](https://github.com/dask/distributed/pull/8806)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix exception handling for `NannyPlugin.setup` and `NannyPlugin.teardown` ([distributed#8811](https://github.com/dask/distributed/pull/8811)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix exception handling for `WorkerPlugin.setup` and `WorkerPlugin.teardown` ([distributed#8810](https://github.com/dask/distributed/pull/8810)) [Hendrik Makait](https://github.com/hendrikmakait)
- typo fix ([distributed#8812](https://github.com/dask/distributed/pull/8812)) [alex-rakowski](https://github.com/alex-rakowski)
- Fix `if` / `else` for `send_recv_from_rpc` ([distributed#8809](https://github.com/dask/distributed/pull/8809)) [Patrick Hoefler](https://github.com/phofl)
- Ensure that adaptive only stops once ([distributed#8807](https://github.com/dask/distributed/pull/8807)) [Hendrik Makait](https://github.com/hendrikmakait)
- Reduce noise from GC-related logging ([distributed#8804](https://github.com/dask/distributed/pull/8804)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove unused `delete_interval` and  `synchronize_worker_interval` from `Scheduler` ([distributed#8801](https://github.com/dask/distributed/pull/8801)) [Hendrik Makait](https://github.com/hendrikmakait)
- Change log level for Compute Failed log message ([distributed#8802](https://github.com/dask/distributed/pull/8802)) [Patrick Hoefler](https://github.com/phofl)
- Add Prometheus metric for time spent on GC ([distributed#8803](https://github.com/dask/distributed/pull/8803)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add Prometheus metrics for `dask_worker_{added|removed}_total` ([distributed#8798](https://github.com/dask/distributed/pull/8798)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add log event for `worker-ttl-timed-out` ([distributed#8800](https://github.com/dask/distributed/pull/8800)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add Prometheus metrics for `dask_client_connections_{added|removed}_total` ([distributed#8799](https://github.com/dask/distributed/pull/8799)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix `PackageInstall` plugin ([distributed#8794](https://github.com/dask/distributed/pull/8794)) [Hendrik Makait](https://github.com/hendrikmakait)
- Make stealing more robust ([distributed#8788](https://github.com/dask/distributed/pull/8788)) [Hendrik Makait](https://github.com/hendrikmakait)
- Leave a warning about future instantiation ([distributed#8782](https://github.com/dask/distributed/pull/8782)) [Florian Jetter](https://github.com/fjetter)

<a id="v2024-7-1"></a>

## 2024.7.1

### Highlights

#### More resilient distributed lock

[`distributed.Lock`](futures.md#distributed.Lock) is now resilient to worker failures.
Previously deadlocks were possible in cases where a lock-holding worker
was lost and/or failed to release the lock due to an error.

See [distributed#8770](https://github.com/dask/distributed/pull/8770) by [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Remove and warn of persist usage ([dask#11237](https://github.com/dask/dask/pull/11237)) [Patrick Hoefler](https://github.com/phofl)
- Preserve `timestamp` unit during `meta` creation ([dask#11233](https://github.com/dask/dask/pull/11233)) [Patrick Hoefler](https://github.com/phofl)
- Ensure that `dask-expr` `DataFrames` are optimized when put into `delayed` ([dask#11231](https://github.com/dask/dask/pull/11231)) [Patrick Hoefler](https://github.com/phofl)
- Fixes for `d` freq deprecation in `pandas=3` ([dask#11228](https://github.com/dask/dask/pull/11228)) [James Bourbeau](https://github.com/jrbourbeau)
- bump approx threshold for `test_quantile` ([dask#10720](https://github.com/dask/dask/pull/10720)) [Florian Jetter](https://github.com/fjetter)
- Bump `xarray-contrib/issue-from-pytest-log` from 1.2.8 to 1.3.0 ([dask#11221](https://github.com/dask/dask/pull/11221))
- Bump `JamesIves/github-pages-deploy-action` from 4.6.1 to 4.6.3 ([dask#11222](https://github.com/dask/dask/pull/11222))
- Ensure `Lock` always register with scheduler ([distributed#8781](https://github.com/dask/distributed/pull/8781)) [Florian Jetter](https://github.com/fjetter)
- Temporarily pin `setuptools < 71` ([distributed#8785](https://github.com/dask/distributed/pull/8785)) [James Bourbeau](https://github.com/jrbourbeau)
- Restore `len()` on `TaskPrefix` ([distributed#8783](https://github.com/dask/distributed/pull/8783)) [Hendrik Makait](https://github.com/hendrikmakait)
- Avoid false positives for `p2p-failed` log event ([distributed#8777](https://github.com/dask/distributed/pull/8777)) [Hendrik Makait](https://github.com/hendrikmakait)
- Expose paused and retired workers separately in prometheus ([distributed#8613](https://github.com/dask/distributed/pull/8613)) [Patrick Hoefler](https://github.com/phofl)
- Creating transitions-failures log event ([distributed#8776](https://github.com/dask/distributed/pull/8776)) [alex-rakowski](https://github.com/alex-rakowski)
- Implement HLG layer for P2P rechunking ([distributed#8751](https://github.com/dask/distributed/pull/8751)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add another test for a possible deadlock scenario caused by ([distributed#8703](https://github.com/dask/distributed/pull/8703)) ([distributed#8769](https://github.com/dask/distributed/pull/8769)) [Hendrik Makait](https://github.com/hendrikmakait)
- Raise an error if compute on persisted collection with released futures ([distributed#8764](https://github.com/dask/distributed/pull/8764)) [Florian Jetter](https://github.com/fjetter)
- Re-raise `P2PConsistencyError` from failed P2P tasks ([distributed#8748](https://github.com/dask/distributed/pull/8748)) [Hendrik Makait](https://github.com/hendrikmakait)
- Robuster faster tests memory sampler ([distributed#8758](https://github.com/dask/distributed/pull/8758)) [Florian Jetter](https://github.com/fjetter)
- Fix `scheduler_bokeh::test_shuffling` ([distributed#8766](https://github.com/dask/distributed/pull/8766)) [Florian Jetter](https://github.com/fjetter)
- Increase timeouts for `pubsub::test_client_worker` ([distributed#8765](https://github.com/dask/distributed/pull/8765)) [Florian Jetter](https://github.com/fjetter)
- Factor out async taskgroup ([distributed#8756](https://github.com/dask/distributed/pull/8756)) [Florian Jetter](https://github.com/fjetter)
- Don’t sort keys lexicographically in worker table ([distributed#8753](https://github.com/dask/distributed/pull/8753)) [Florian Jetter](https://github.com/fjetter)
- Use `functools.cache` instead of `functools.lru_cache` for extremely often called functions ([distributed#8762](https://github.com/dask/distributed/pull/8762)) [Jonas Dedden](https://github.com/jonded94)
- Robuster deeply nested structures ([distributed#8730](https://github.com/dask/distributed/pull/8730)) [Florian Jetter](https://github.com/fjetter)
- Adding HLG to MAP ([distributed#8740](https://github.com/dask/distributed/pull/8740)) [alex-rakowski](https://github.com/alex-rakowski)
- Add close worker button to worker info page ([distributed#8742](https://github.com/dask/distributed/pull/8742)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-7-0"></a>

## 2024.7.0

### Highlights

#### Drop support for pandas 1.x

This release drops support for `pandas<2`. `pandas` 2.0
is now the required minimum version to run Dask DataFrame.

The mimimum version of `partd` was also raised to 1.4.0. Versions before 1.4
are not compatible with `pandas` 2.

See [dask#11199](https://github.com/dask/dask/pull/11199) by [Patrick Hoefler](https://github.com/phofl) for more details.

#### Publish-subscribe APIs deprecated

`distributed.Pub` and `distributed.Sub` have been deprecated and will be removed
in a future release. Please switch to [`distributed.Client.log_event()`](futures.md#distributed.Client.log_event) and [`distributed.Worker.log_event()`](deploying-python-advanced.md#distributed.Worker.log_event)
instead.

See [distributed#8724](https://github.com/dask/distributed/pull/8724) by [Hendrik Makait](https://github.com/hendrikmakait) for more details.

### Additional changes

- Only count data that is in memory for `xarray` `sizeof` ([dask#11206](https://github.com/dask/dask/pull/11206)) [Florian Jetter](https://github.com/fjetter)
- Fix `botocore` re-raising error ([dask#11209](https://github.com/dask/dask/pull/11209)) [Patrick Hoefler](https://github.com/phofl)
- Update Coiled links in documentation ([dask#11211](https://github.com/dask/dask/pull/11211)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add some array-expr methods ([dask#11210](https://github.com/dask/dask/pull/11210)) [Patrick Hoefler](https://github.com/phofl)
- Fix `quantile` for arrow dtypes ([dask#11202](https://github.com/dask/dask/pull/11202)) [Patrick Hoefler](https://github.com/phofl)
- Add utility to verify optional dependencies ([dask#11205](https://github.com/dask/dask/pull/11205)) [Patrick Hoefler](https://github.com/phofl)
- Implement array expression switch ([dask#11203](https://github.com/dask/dask/pull/11203)) [Patrick Hoefler](https://github.com/phofl)
- Remove no longer supported `ipython` reference ([dask#11196](https://github.com/dask/dask/pull/11196)) [Patrick Hoefler](https://github.com/phofl)
- Remove `from_delayed` references ([dask#11195](https://github.com/dask/dask/pull/11195)) [Patrick Hoefler](https://github.com/phofl)
- Add other IO connectors to docs ([dask#11189](https://github.com/dask/dask/pull/11189)) [Patrick Hoefler](https://github.com/phofl)
- Fix `assert_eq` import from `cudf` ([distributed#8747](https://github.com/dask/distributed/pull/8747)) [James Bourbeau](https://github.com/jrbourbeau)
- Log traceback upon task error ([distributed#8746](https://github.com/dask/distributed/pull/8746)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update system monitor when polling Prometheus metrics ([distributed#8745](https://github.com/dask/distributed/pull/8745)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump `pandas` to 2.0 in `mindeps` build ([distributed#8743](https://github.com/dask/distributed/pull/8743)) [James Bourbeau](https://github.com/jrbourbeau)
- Refactor event logging functionality into broker ([distributed#8731](https://github.com/dask/distributed/pull/8731)) [Hendrik Makait](https://github.com/hendrikmakait)
- Drop support for pandas 1.X ([distributed#8741](https://github.com/dask/distributed/pull/8741)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove `is_python_shutting_down` ([distributed#8492](https://github.com/dask/distributed/pull/8492)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix `test_task_state_instance_are_garbage_collected` ([distributed#8735](https://github.com/dask/distributed/pull/8735)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix floating-point inaccuracy ([distributed#8736](https://github.com/dask/distributed/pull/8736)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix `pynvml` handles ([distributed#8693](https://github.com/dask/distributed/pull/8693)) [Benjamin Zaitlen](https://github.com/quasiben)
- `get_ip`: handle getting `0.0.0.0` ([distributed#8712](https://github.com/dask/distributed/pull/8712)) [Adam Williamson](https://github.com/AdamWill)
- Remove `FutureWarning` in `test_task_state_instance_are_garbage_collected` ([distributed#8734](https://github.com/dask/distributed/pull/8734)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix `mindeps`-testing on CI ([distributed#8728](https://github.com/dask/distributed/pull/8728)) [Hendrik Makait](https://github.com/hendrikmakait)
- Extract tests related to event-logging into separate file ([distributed#8733](https://github.com/dask/distributed/pull/8733)) [Hendrik Makait](https://github.com/hendrikmakait)
- Use safer context for `ProcessPoolExecutor` ([distributed#8715](https://github.com/dask/distributed/pull/8715)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Cache URL encoding of worker addresses in dashboard ([distributed#8725](https://github.com/dask/distributed/pull/8725)) [Florian Jetter](https://github.com/fjetter)
- More robust `bokeh` `test_shuffling` ([distributed#8727](https://github.com/dask/distributed/pull/8727)) [Florian Jetter](https://github.com/fjetter)
- Fix type in actor docs ([distributed#8711](https://github.com/dask/distributed/pull/8711)) [Sultan Orazbayev](https://github.com/SultanOrazbayev)
- More useful warning if a plugin type is provided instead of instance ([distributed#8689](https://github.com/dask/distributed/pull/8689)) [Florian Jetter](https://github.com/fjetter)
- Improve error on cancelled tasks due to disconnect ([distributed#8705](https://github.com/dask/distributed/pull/8705)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix wait condition on `test_forget_errors` ([distributed#8714](https://github.com/dask/distributed/pull/8714)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Skip `test_deadlock_dependency_of_queued_released` ([distributed#8723](https://github.com/dask/distributed/pull/8723)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix `test_quiet_client_close` ([distributed#8722](https://github.com/dask/distributed/pull/8722)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix cleanup iteration in `save_sys_modules` ([distributed#8713](https://github.com/dask/distributed/pull/8713)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Add quotes to missing `bokeh` installation commands ([distributed#8717](https://github.com/dask/distributed/pull/8717)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-6-2"></a>

## 2024.6.2

This is a patch release to update an issue with `dask` and `distributed`
version pinning in the 2024.6.1 release.

### Additional changes

- Get docs build passing ([dask#11184](https://github.com/dask/dask/pull/11184)) [James Bourbeau](https://github.com/jrbourbeau)
- `profile._f_lineno`: handle `next_line` being `None` in Python 3.13 ([dask#8710](https://github.com/dask/dask/pull/8710)) [Adam Williamson](https://github.com/AdamWill)

<a id="v2024-6-1"></a>

## 2024.6.1

### Highlights

This release includes a critical fix that fixes a deadlock that can arise
when dependencies of root-ish tasks are rescheduled, e.g. due to a worker being
lost.

See [distributed#8703](https://github.com/dask/distributed/pull/8703) by [Hendrik Makait](https://github.com/hendrikmakait) for more details.

### Additional changes

- Cache global query-planning config ([dask#11183](https://github.com/dask/dask/pull/11183)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Python 3.13 fixes ([dask#11185](https://github.com/dask/dask/pull/11185)) [Adam Williamson](https://github.com/AdamWill)
- Fix `test_map_freq_to_period_start` for `pandas=3` ([dask#11181](https://github.com/dask/dask/pull/11181)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump release-drafter/release-drafter from 5 to 6 ([distributed#8699](https://github.com/dask/distributed/pull/8699))

<a id="v2024-6-0"></a>

## 2024.6.0

### Highlights

#### memmap array tokenization

Tokenizing `memmap` arrays will now avoid materializing the array into memory.

See [dask#11161](https://github.com/dask/dask/pull/11161) by [Florian Jetter](https://github.com/fjetter) for more details.

### Additional changes

- Fix `test_dt_accessor` with query planning disabled ([dask#11177](https://github.com/dask/dask/pull/11177)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `packaging.version.Version` ([dask#11171](https://github.com/dask/dask/pull/11171)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove deprecated `dask.compatibility` module ([dask#11172](https://github.com/dask/dask/pull/11172)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure compatibility for `xarray.NamedArray` ([dask#11168](https://github.com/dask/dask/pull/11168)) [Hendrik Makait](https://github.com/hendrikmakait)
- Estimate sizes of `xarray` collections ([dask#11166](https://github.com/dask/dask/pull/11166)) [Florian Jetter](https://github.com/fjetter)
- Add section about futures and variables ([dask#11164](https://github.com/dask/dask/pull/11164)) [Florian Jetter](https://github.com/fjetter)
- Update docs for combined Dask community meeting info ([dask#11159](https://github.com/dask/dask/pull/11159)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Avoid rounding error in `test_prometheus_collect_count_total_by_cost_multipliers` ([distributed#8687](https://github.com/dask/distributed/pull/8687)) [Hendrik Makait](https://github.com/hendrikmakait)
- Log key collision count in `update_graph` log event ([distributed#8692](https://github.com/dask/distributed/pull/8692)) [Hendrik Makait](https://github.com/hendrikmakait)
- Automate GitHub Releases when new tags are pushed ([distributed#8626](https://github.com/dask/distributed/pull/8626)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix log event with multiple topics ([distributed#8691](https://github.com/dask/distributed/pull/8691)) [Hendrik Makait](https://github.com/hendrikmakait)
- Rename `safe` to `expected` in `Scheduler.remove_worker` ([distributed#8686](https://github.com/dask/distributed/pull/8686)) [Hendrik Makait](https://github.com/hendrikmakait)
- Log event during failure ([distributed#8663](https://github.com/dask/distributed/pull/8663)) [Hendrik Makait](https://github.com/hendrikmakait)
- Eagerly update aggregate statistics for `TaskPrefix` instead of calculating them on-demand ([distributed#8681](https://github.com/dask/distributed/pull/8681)) [Hendrik Makait](https://github.com/hendrikmakait)
- Improve graph submission time for P2P rechunking by avoiding unpack recursion into indices ([distributed#8672](https://github.com/dask/distributed/pull/8672)) [Florian Jetter](https://github.com/fjetter)
- Add safe keyword to `remove-worker` event ([distributed#8647](https://github.com/dask/distributed/pull/8647)) [alex-rakowski](https://github.com/alex-rakowski)
- Improved errors and reduced logging for P2P RPC calls ([distributed#8666](https://github.com/dask/distributed/pull/8666)) [Hendrik Makait](https://github.com/hendrikmakait)
- Adjust P2P tests for `dask-expr` ([distributed#8662](https://github.com/dask/distributed/pull/8662)) [Hendrik Makait](https://github.com/hendrikmakait)
- Iterate over copy of `Server.digests_total_since_heartbeat` to avoid `RuntimeError` ([distributed#8670](https://github.com/dask/distributed/pull/8670)) [Hendrik Makait](https://github.com/hendrikmakait)
- Log task state in Compute Failed ([distributed#8668](https://github.com/dask/distributed/pull/8668)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add Prometheus gauge for task groups ([distributed#8661](https://github.com/dask/distributed/pull/8661)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix too strict assertion in shuffle code for `pandas` subclasses ([distributed#8667](https://github.com/dask/distributed/pull/8667)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Reduce noise from erring tasks that are not supposed to be running ([distributed#8664](https://github.com/dask/distributed/pull/8664)) [Hendrik Makait](https://github.com/hendrikmakait)

<a id="v2024-5-2"></a>

## 2024.5.2

This release primarily contains minor bug fixes.

### Additional changes

- Fix nightly Zarr installation in CI ([dask#11151](https://github.com/dask/dask/pull/11151)) [James Bourbeau](https://github.com/jrbourbeau)
- Add python 3.11 build to GPU CI ([dask#11135](https://github.com/dask/dask/pull/11135)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Update gpuCI `RAPIDS_VER` to `24.08` ([dask#11141](https://github.com/dask/dask/pull/11141))
- Update `test_groupby_grouper_dispatch` ([dask#11144](https://github.com/dask/dask/pull/11144)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bump `JamesIves/github-pages-deploy-action` from 4.6.0 to 4.6.1 ([dask#11136](https://github.com/dask/dask/pull/11136))
- Unskip `test_array_function_sparse` with new `sparse` release ([dask#11139](https://github.com/dask/dask/pull/11139)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_parse_dates_multi_column` on `pandas=3` ([dask#11132](https://github.com/dask/dask/pull/11132)) [James Bourbeau](https://github.com/jrbourbeau)
- Don’t draft release notes for tagged commits ([dask#11138](https://github.com/dask/dask/pull/11138)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Reduce task group count for partial P2P rechunks ([distributed#8655](https://github.com/dask/distributed/pull/8655)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update gpuCI `RAPIDS_VER` to `24.08` ([distributed#8652](https://github.com/dask/distributed/pull/8652))
- Submit collections metadata to scheduler ([distributed#8612](https://github.com/dask/distributed/pull/8612)) [Florian Jetter](https://github.com/fjetter)
- Fix indent in code example in `task-launch.rst` ([distributed#8650](https://github.com/dask/distributed/pull/8650)) [Ray Bell](https://github.com/raybellwaves)
- Avoid multiple `WorkerState` sphinx error ([distributed#8643](https://github.com/dask/distributed/pull/8643)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-5-1"></a>

## 2024.5.1

### Highlights

#### NumPy 2.0 support

This release contains compatibility updates for the upcoming NumPy 2.0 release.

See [dask#11096](https://github.com/dask/dask/pull/11096) by [Benjamin Zaitlen](https://github.com/quasiben)  and [dask#11106](https://github.com/dask/dask/pull/11106) by [James Bourbeau](https://github.com/jrbourbeau) for more details.

#### Increased Zarr store support

This release contains adds support for `MutableMapping`-backed Zarr stores like
`zarr.storage.DirectoryStore`, etc.

See [dask#10422](https://github.com/dask/dask/pull/10422) by [Greg M. Fleishman](https://github.com/GFleishman) for more details.

### Additional changes

- Minor updates to ML page ([dask#11129](https://github.com/dask/dask/pull/11129)) [James Bourbeau](https://github.com/jrbourbeau)
- Skip failing `sparse` test on 0.15.2 ([dask#11131](https://github.com/dask/dask/pull/11131)) [James Bourbeau](https://github.com/jrbourbeau)
- Make sure nightly `pyarrow` is installed in upstream CI build ([dask#11121](https://github.com/dask/dask/pull/11121)) [James Bourbeau](https://github.com/jrbourbeau)
- Add initial draft of ML overview document ([dask#11114](https://github.com/dask/dask/pull/11114)) [Matthew Rocklin](https://github.com/mrocklin)
- Test query-planning in gpuCI ([dask#11060](https://github.com/dask/dask/pull/11060)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid `pytest` error when skipping NumPy 2.0 tests  ([dask#11110](https://github.com/dask/dask/pull/11110)) [James Bourbeau](https://github.com/jrbourbeau)
- Use nightly `h5py` in upstream CI build ([dask#11108](https://github.com/dask/dask/pull/11108)) [James Bourbeau](https://github.com/jrbourbeau)
- Use nightly `scikit-image` in upstream CI build ([dask#11107](https://github.com/dask/dask/pull/11107)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump `actions/checkout` from 4.1.4 to 4.1.5 ([dask#11105](https://github.com/dask/dask/pull/11105))
- Enable parquet append tests after fix ([dask#11104](https://github.com/dask/dask/pull/11104)) [Patrick Hoefler](https://github.com/phofl)
- Skip `fastparquet` tests for `numpy` 2 ([dask#11103](https://github.com/dask/dask/pull/11103)) [Patrick Hoefler](https://github.com/phofl)
- Fix misspelling found by codespell ([dask#11097](https://github.com/dask/dask/pull/11097)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix doc build ([dask#11099](https://github.com/dask/dask/pull/11099)) [Patrick Hoefler](https://github.com/phofl)
- Clean up `percentiles_summary` logic ([dask#11094](https://github.com/dask/dask/pull/11094)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Apply `ruff/flake8-implicit-str-concat` rule ISC001 ([dask#11098](https://github.com/dask/dask/pull/11098)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Fix clocks on Windows with Python 3.13 ([distributed#8642](https://github.com/dask/distributed/pull/8642)) [Victor Stinner](https://github.com/vstinner)
- Fix “Print host info” CI step on Mac OS (arm64) ([distributed#8638](https://github.com/dask/distributed/pull/8638)) [Hendrik Makait](https://github.com/hendrikmakait)

<a id="v2024-5-0"></a>

## 2024.5.0

### Highlights

This release primarily contains minor bugfixes.

### Additional changes

- Don’t link to `click` intersphinx dev version ([dask#11091](https://github.com/dask/dask/pull/11091)) [M Bussonnier](https://github.com/Carreau)
- Fix API doc links for some `dask-expr` expressions ([dask#11092](https://github.com/dask/dask/pull/11092)) [Patrick Hoefler](https://github.com/phofl)
- Add `dask-expr` to upstream build ([dask#11086](https://github.com/dask/dask/pull/11086)) [Patrick Hoefler](https://github.com/phofl)
- Add `melt` support when `query-planning` is enabled ([dask#11088](https://github.com/dask/dask/pull/11088)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Skip dataframe/product when in `numpy` 2 envs ([dask#11089](https://github.com/dask/dask/pull/11089)) [Benjamin Zaitlen](https://github.com/quasiben)
- Add plots to illustrate what the optimizer does ([dask#11072](https://github.com/dask/dask/pull/11072)) [Patrick Hoefler](https://github.com/phofl)
- Fixup `pandas` upstream tests ([dask#11085](https://github.com/dask/dask/pull/11085)) [Patrick Hoefler](https://github.com/phofl)
- Bump `conda-incubator/setup-miniconda` from 3.0.3 to 3.0.4 ([dask#11084](https://github.com/dask/dask/pull/11084))
- Bump `actions/checkout` from 4.1.3 to 4.1.4 ([dask#11083](https://github.com/dask/dask/pull/11083))
- Fix CI after `pytest` changes ([dask#11082](https://github.com/dask/dask/pull/11082)) [Patrick Hoefler](https://github.com/phofl)
- Fixup tests for more efficient `dask-expr` implementation ([dask#11071](https://github.com/dask/dask/pull/11071)) [Patrick Hoefler](https://github.com/phofl)
- Generalize `clear_known_categories` utility ([dask#11059](https://github.com/dask/dask/pull/11059)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bump `JamesIves/github-pages-deploy-action` from 4.5.0 to 4.6.0 ([dask#11062](https://github.com/dask/dask/pull/11062))
- Bump `release-drafter/release-drafter` from 5 to 6 ([dask#11063](https://github.com/dask/dask/pull/11063))
- Bump `actions/checkout` from 4.1.2 to 4.1.3 ([dask#11061](https://github.com/dask/dask/pull/11061))
- Update GPU CI `RAPIDS_VER` to 24.06, disable query planning  ([dask#11045](https://github.com/dask/dask/pull/11045)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Move tests ([distributed#8631](https://github.com/dask/distributed/pull/8631)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump `actions/checkout` from 4.1.2 to 4.1.3 ([distributed#8628](https://github.com/dask/distributed/pull/8628))

<a id="v2024-4-2"></a>

## 2024.4.2

### Highlights

#### Trivial Merge Implementation

The Query Optimizer will inspect quires to determine if a `merge(...)` or
`groupby(...).apply(...)` requires a shuffle. A shuffle can be avoided, if the
DataFrame was shuffled on the same columns in a previous step without any operations
in between that change the partitioning layout or the relevant values in each
partition.

```python
>>> result = df.merge(df2, on="a")
>>> result = result.merge(df3, on="a")
```

The Query optimizer will identify that `result` was previously shuffled on `"a"` as
well and thus only shuffle `df3` in the second merge operation before doing a blockwise
merge.

#### Auto-partitioning in `read_parquet`

The Query Optimizer will automatically repartition datasets read from Parquet files
if individual partitions are too small. This will reduce the number of partitions in
consequentially also the size of the task graph.

The Optimizer aims to produce partitions of at least 75MB and will combine multiple files
together if necessary to reach this threshold. The value can be configured by using

```python
>>> dask.config.set({"dataframe.parquet.minimum-partition-size": 100_000_000})
```

The value is given in bytes. The default threshold is relatively conservative to avoid
memory issues on worker nodes with a relatively small amount of memory per thread.

### Additional changes

- Add GitHub Releases automation ([dask#11057](https://github.com/dask/dask/pull/11057)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add changelog entries for new release ([dask#11058](https://github.com/dask/dask/pull/11058)) [Patrick Hoefler](https://github.com/phofl)
- Reinstate try/except block in `_bind_property` ([dask#11049](https://github.com/dask/dask/pull/11049)) [Lawrence Mitchell](https://github.com/wence-)
- Fix link for query planning docs ([dask#11054](https://github.com/dask/dask/pull/11054)) [Patrick Hoefler](https://github.com/phofl)
- Add config parameter for parquet file size ([dask#11052](https://github.com/dask/dask/pull/11052)) [Patrick Hoefler](https://github.com/phofl)
- Update `percentile` docstring ([dask#11053](https://github.com/dask/dask/pull/11053)) [Abel Aoun](https://github.com/bzah)
- Add docs for query optimizer ([dask#11043](https://github.com/dask/dask/pull/11043)) [Patrick Hoefler](https://github.com/phofl)
- Assignment of np.ma.masked to obect-type Array ([dask#9627](https://github.com/dask/dask/pull/9627)) [David Hassell](https://github.com/davidhassell)
- Don’t error if `dask_expr` is not installed ([dask#11048](https://github.com/dask/dask/pull/11048)) [Simon Høxbro Hansen](https://github.com/Hoxbro)
- Adjust `test_set_index` for “cudf” backend ([dask#11029](https://github.com/dask/dask/pull/11029)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use `to/from_legacy_dataframe` instead of `to/from_dask_dataframe` ([dask#11025](https://github.com/dask/dask/pull/11025)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Tokenize bag `groupby` keys ([dask#10734](https://github.com/dask/dask/pull/10734)) [Charles Stern](https://github.com/cisaacstern)
- Add lazy “cudf” registration for p2p-related dispatch functions ([dask#11040](https://github.com/dask/dask/pull/11040)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Collect `memray` profiles on exception ([distributed#8625](https://github.com/dask/distributed/pull/8625)) [Florian Jetter](https://github.com/fjetter)
- Ensure `inproc` properly emulates serialization protocol ([distributed#8622](https://github.com/dask/distributed/pull/8622)) [Florian Jetter](https://github.com/fjetter)
- Relax test stats profiling2 ([distributed#8621](https://github.com/dask/distributed/pull/8621)) [Florian Jetter](https://github.com/fjetter)
- Restart workers when `worker-ttl` expires ([distributed#8538](https://github.com/dask/distributed/pull/8538)) [crusaderky](https://github.com/crusaderky)
- Use `monotonic` for deadline test ([distributed#8620](https://github.com/dask/distributed/pull/8620)) [Florian Jetter](https://github.com/fjetter)
- Fix race condition for published futures with annotations ([distributed#8577](https://github.com/dask/distributed/pull/8577)) [Florian Jetter](https://github.com/fjetter)
- Scatter by worker instead of `worker` -> `nthreads` ([distributed#8590](https://github.com/dask/distributed/pull/8590)) [Miles](https://github.com/milesgranger)
- Send log-event if worker is restarted because of memory pressure ([distributed#8617](https://github.com/dask/distributed/pull/8617)) [Patrick Hoefler](https://github.com/phofl)
- Do not print xfailed tests in CI ([distributed#8619](https://github.com/dask/distributed/pull/8619)) [Florian Jetter](https://github.com/fjetter)
- ensure workers are not downscaled when participating in p2p ([distributed#8610](https://github.com/dask/distributed/pull/8610)) [Florian Jetter](https://github.com/fjetter)
- Run against stable `fsspec` ([distributed#8615](https://github.com/dask/distributed/pull/8615)) [Florian Jetter](https://github.com/fjetter)

<a id="v2024-4-1"></a>

## 2024.4.1

This is a minor bugfix release that that fixes an error when importing
`dask.dataframe` with Python 3.11.9.

See [dask#11035](https://github.com/dask/dask/pull/11035) and [dask#11039](https://github.com/dask/dask/pull/11039) from [Richard (Rick) Zamora](https://github.com/rjzamora) for details.

### Additional changes

- Remove skips for named aggregations ([dask#11036](https://github.com/dask/dask/pull/11036)) [Patrick Hoefler](https://github.com/phofl)
- Don’t deep-copy read-only buffers on unpickle ([distributed#8609](https://github.com/dask/distributed/pull/8609)) [crusaderky](https://github.com/crusaderky)
- Add `dask-expr` to `dask` conda recipe ([distributed#8601](https://github.com/dask/distributed/pull/8601)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2024-4-0"></a>

## 2024.4.0

### Highlights

#### Query planning fixes

This release contains a variety of bugfixes in Dask DataFrame’s new
query planner.

#### GPU metric dashboard fixes

GPU memory and utilization dashboard functionality has been restored.
Previously these plots were unintentionally left blank.

See [distributed#8572](https://github.com/dask/distributed/pull/8572) from [Benjamin Zaitlen](https://github.com/quasiben) for details.

### Additional changes

- Build nightlies on tag releases ([dask#11014](https://github.com/dask/dask/pull/11014)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Remove `xfail` tracebacks from test suite ([dask#11028](https://github.com/dask/dask/pull/11028)) [Patrick Hoefler](https://github.com/phofl)
- Fix CI for upstream `pandas` changes ([dask#11027](https://github.com/dask/dask/pull/11027)) [Patrick Hoefler](https://github.com/phofl)
- Fix `value_counts` raising if branch exists of nans only ([dask#11023](https://github.com/dask/dask/pull/11023)) [Patrick Hoefler](https://github.com/phofl)
- Enable custom expressions in `dask_cudf` ([dask#11013](https://github.com/dask/dask/pull/11013)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Raise `ImportError` instead of `ValueError` when `dask-expr` cannot be imported ([dask#11007](https://github.com/dask/dask/pull/11007)) [James Lamb](https://github.com/jameslamb)
- Add HypersSpy to `ecosystem.rst` ([dask#11008](https://github.com/dask/dask/pull/11008)) [Jonas Lähnemann](https://github.com/jlaehne)
- Add Hugging Face `hf://` to the list of `fsspec` compatible remote services ([dask#11012](https://github.com/dask/dask/pull/11012)) [Quentin Lhoest](https://github.com/lhoestq)
- Bump `actions/checkout` from 4.1.1 to 4.1.2 ([dask#11009](https://github.com/dask/dask/pull/11009))
- Refresh documentation for annotations and spans ([distributed#8593](https://github.com/dask/distributed/pull/8593)) [crusaderky](https://github.com/crusaderky)
- Fixup deprecation warning from `pandas` ([distributed#8564](https://github.com/dask/distributed/pull/8564)) [Patrick Hoefler](https://github.com/phofl)
- Add Python 3.11 to GPU CI matrix ([distributed#8598](https://github.com/dask/distributed/pull/8598)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Deadline to use a monotonic timer ([distributed#8597](https://github.com/dask/distributed/pull/8597)) [crusaderky](https://github.com/crusaderky)
- Update gpuCI `RAPIDS_VER` to `24.06` ([distributed#8588](https://github.com/dask/distributed/pull/8588))
- Refactor `restart()` and `restart_workers()` ([distributed#8550](https://github.com/dask/distributed/pull/8550)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/checkout` from 4.1.1 to 4.1.2 ([distributed#8587](https://github.com/dask/distributed/pull/8587))
- Fix `bokeh` deprecations ([distributed#8594](https://github.com/dask/distributed/pull/8594)) [Miles](https://github.com/milesgranger)
- Fix flaky test: `test_shutsdown_cleanly` ([distributed#8582](https://github.com/dask/distributed/pull/8582)) [Miles](https://github.com/milesgranger)
- Include type in failed `sizeof` warning ([distributed#8580](https://github.com/dask/distributed/pull/8580)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2024-3-1"></a>

## 2024.3.1

This is a minor release that primarily demotes an exception to a warning if
`dask-expr` is not installed when upgrading.

### Additional changes

- Only warn if `dask-expr` is not installed ([dask#11003](https://github.com/dask/dask/pull/11003)) [Florian Jetter](https://github.com/fjetter)
- Fix typos found by codespell ([dask#10993](https://github.com/dask/dask/pull/10993)) [Dimitri Papadopoulos Orfanos](https://github.com/DimitriPapadopoulos)
- Extra CI job with `dask-expr` disabled ([distributed#8583](https://github.com/dask/distributed/pull/8583)) [crusaderky](https://github.com/crusaderky)
- Fix worker dashboard proxy ([distributed#8528](https://github.com/dask/distributed/pull/8528)) [Miles](https://github.com/milesgranger)
- Fix flaky `test_restart_waits_for_new_workers` ([distributed#8573](https://github.com/dask/distributed/pull/8573)) [crusaderky](https://github.com/crusaderky)
- Fix flaky `test_raise_on_incompatible_partitions` ([distributed#8571](https://github.com/dask/distributed/pull/8571)) [crusaderky](https://github.com/crusaderky)

<a id="v2024-3-0"></a>

## 2024.3.0

Released on March 11, 2024

### Highlights

#### Query planning

This release is enabling query planning by default for all users of
`dask.dataframe`.

The query planning functionality represents a rewrite of the `DataFrame` using
`dask-expr`. This is a drop-in replacement and we expect that most users will
not have to adjust any of their code.
Any feedback can be reported on the Dask [issue tracker](https://github.com/dask/dask/issues) or on the [query planning feedback issue](https://github.com/dask/dask/issues/10995).

If you are encountering any issues you are still able to opt-out by setting

```python
>>> import dask
>>> dask.config.set({'dataframe.query-planning': False})
```

#### Sunset of Pandas 1.X support

The new query planning backend is requiring at least pandas `2.0`. This pandas
version will automatically be installed if you are installing from conda or if
you are installing using dask[complete] or dask[dataframe] from pip.

The legacy DataFrame implementation is still supporting pandas `1.X` if you
install `dask` without extras.

### Additional changes

- Update tests for pandas nightlies with dask-expr ([dask#10989](https://github.com/dask/dask/pull/10989)) [Patrick Hoefler](https://github.com/phofl)
- Use dask-expr docs as main reference docs for DataFrames ([dask#10990](https://github.com/dask/dask/pull/10990)) [Patrick Hoefler](https://github.com/phofl)
- Adjust from_array test for dask-expr ([dask#10988](https://github.com/dask/dask/pull/10988)) [Patrick Hoefler](https://github.com/phofl)
- Unskip `to_delayed` test ([dask#10985](https://github.com/dask/dask/pull/10985)) [Patrick Hoefler](https://github.com/phofl)
- Bump conda-incubator/setup-miniconda from 3.0.1 to 3.0.3 ([dask#10978](https://github.com/dask/dask/pull/10978))
- Fix bug when enabling dask-expr ([dask#10977](https://github.com/dask/dask/pull/10977)) [Patrick Hoefler](https://github.com/phofl)
- Update docs and requirements for dask-expr and remove warning ([dask#10976](https://github.com/dask/dask/pull/10976)) [Patrick Hoefler](https://github.com/phofl)
- Fix numpy 2 compatibility with ogrid usage ([dask#10929](https://github.com/dask/dask/pull/10929)) [David Hoese](https://github.com/djhoese)
- Turn on dask-expr switch ([dask#10967](https://github.com/dask/dask/pull/10967)) [Patrick Hoefler](https://github.com/phofl)
- Force initializing the random seed with the same byte order interpret… ([dask#10970](https://github.com/dask/dask/pull/10970)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Use correct encoding for line terminator when reading CSV ([dask#10972](https://github.com/dask/dask/pull/10972)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- perf: do not unnecessarily recalculate input/output indices in \_optimize_blockwise ([dask#10966](https://github.com/dask/dask/pull/10966)) [Lindsey Gray](https://github.com/lgray)
- Adjust tests for string option in dask-expr ([dask#10968](https://github.com/dask/dask/pull/10968)) [Patrick Hoefler](https://github.com/phofl)
- Adjust tests for array conversion in dask-expr ([dask#10973](https://github.com/dask/dask/pull/10973)) [Patrick Hoefler](https://github.com/phofl)
- TST: Fix sizeof tests on 32bit ([dask#10971](https://github.com/dask/dask/pull/10971)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- TST: Add missing skip for pyarrow ([dask#10969](https://github.com/dask/dask/pull/10969)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Implement dask-expr conversion for `bag.to_dataframe` ([dask#10963](https://github.com/dask/dask/pull/10963)) [Patrick Hoefler](https://github.com/phofl)
- Fix dask-expr import errors ([dask#10964](https://github.com/dask/dask/pull/10964)) [Miles](https://github.com/milesgranger)
- Clean up Sphinx documentation for `dask.config` ([dask#10959](https://github.com/dask/dask/pull/10959)) [crusaderky](https://github.com/crusaderky)
- Use stdlib `importlib.metadata` on Python 3.12+ ([dask#10955](https://github.com/dask/dask/pull/10955)) [wim glenn](https://github.com/wimglenn)
- Cast partitioning_index to smaller size ([dask#10953](https://github.com/dask/dask/pull/10953)) [Florian Jetter](https://github.com/fjetter)
- Reuse dask/dask groupby Aggregation ([dask#10952](https://github.com/dask/dask/pull/10952)) [Patrick Hoefler](https://github.com/phofl)
- ensure tokens on futures are unique ([distributed#8569](https://github.com/dask/distributed/pull/8569)) [Florian Jetter](https://github.com/fjetter)
- Don’t obfuscate fine performance metrics failures ([distributed#8568](https://github.com/dask/distributed/pull/8568)) [crusaderky](https://github.com/crusaderky)
- Mark shuffle fast tasks in dask-expr ([distributed#8563](https://github.com/dask/distributed/pull/8563)) [crusaderky](https://github.com/crusaderky)
- Weigh gilknocker Prometheus metric by duration ([distributed#8558](https://github.com/dask/distributed/pull/8558)) [crusaderky](https://github.com/crusaderky)
- Fix scheduler transition error on memory->erred ([distributed#8549](https://github.com/dask/distributed/pull/8549)) [Hendrik Makait](https://github.com/hendrikmakait)
- Make CI happy again ([distributed#8560](https://github.com/dask/distributed/pull/8560)) [Miles](https://github.com/milesgranger)
- Fix flaky test_Future_release_sync ([distributed#8562](https://github.com/dask/distributed/pull/8562)) [crusaderky](https://github.com/crusaderky)
- Fix flaky test_flaky_connect_recover_with_retry ([distributed#8556](https://github.com/dask/distributed/pull/8556)) [Hendrik Makait](https://github.com/hendrikmakait)
- typing tweaks in scheduler.py ([distributed#8551](https://github.com/dask/distributed/pull/8551)) [crusaderky](https://github.com/crusaderky)
- Bump conda-incubator/setup-miniconda from 3.0.2 to 3.0.3 ([distributed#8553](https://github.com/dask/distributed/pull/8553))
- Install dask-expr on CI ([distributed#8552](https://github.com/dask/distributed/pull/8552)) [Hendrik Makait](https://github.com/hendrikmakait)
- P2P shuffle can drop partition column before writing to disk ([distributed#8531](https://github.com/dask/distributed/pull/8531)) [Hendrik Makait](https://github.com/hendrikmakait)
- Better logging for worker removal ([distributed#8517](https://github.com/dask/distributed/pull/8517)) [crusaderky](https://github.com/crusaderky)
- Add indicator support to merge ([distributed#8539](https://github.com/dask/distributed/pull/8539)) [Patrick Hoefler](https://github.com/phofl)
- Bump conda-incubator/setup-miniconda from 3.0.1 to 3.0.2 ([distributed#8535](https://github.com/dask/distributed/pull/8535))
- Avoid iteration error when getting module path ([distributed#8533](https://github.com/dask/distributed/pull/8533)) [James Bourbeau](https://github.com/jrbourbeau)
- Ignore stdlib threading module in code collection ([distributed#8532](https://github.com/dask/distributed/pull/8532)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix excessive logging on P2P retry ([distributed#8511](https://github.com/dask/distributed/pull/8511)) [Hendrik Makait](https://github.com/hendrikmakait)
- Prevent typos in retire_workers parameters ([distributed#8524](https://github.com/dask/distributed/pull/8524)) [crusaderky](https://github.com/crusaderky)
- Cosmetic cleanup of test_steal (backport from #8185) ([distributed#8509](https://github.com/dask/distributed/pull/8509)) [crusaderky](https://github.com/crusaderky)
- Fix flaky test_compute_per_key ([distributed#8521](https://github.com/dask/distributed/pull/8521)) [crusaderky](https://github.com/crusaderky)
- Fix flaky test_no_workers_timeout_queued ([distributed#8523](https://github.com/dask/distributed/pull/8523)) [crusaderky](https://github.com/crusaderky)

<a id="v2024-2-1"></a>

## 2024.2.1

Released on February 23, 2024

### Highlights

#### Allow silencing dask.DataFrame deprecation warning

The last release contained a `DeprecationWarning` that alerts users to an
upcoming switch of `dask.dafaframe` to use the new backend with support for
query planning (see also [dask#10934](https://github.com/dask/dask/issues/10934)).

This `DeprecationWarning` is triggered in import of the `dask.dataframe`
module and the community raised concerns about this being to verbose.

It is now possible to silence this warning

```default
# via Python
>>> dask.config.set({'dataframe.query-planning-warning': False})

# via CLI
dask config set dataframe.query-planning-warning False
```

See [dask#10936](https://github.com/dask/dask/pull/10936) and [dask#10925](https://github.com/dask/dask/pull/10925) from [Miles](https://github.com/milesgranger) for details.

#### More robust distributed scheduler for rare key collisions

Blockwise fusion optimization can cause a task key collision that is not being
handled properly by the distributed scheduler (see [dask#9888](https://github.com/dask/dask/issues/9888)). Users will
typically notice this by seeing one of various internal exceptions that cause a
system deadlock or critical failure. While this issue could not be fixed, the
scheduler now implements a mechanism that should mitigate most occurences and
issues a warning if the issue is detected.

See [distributed#8185](https://github.com/dask/distributed/pull/8185) from [crusaderky](https://github.com/crusaderky) and [Florian Jetter](https://github.com/fjetter) for details.

Over the course of this, various improvements to `tokenization` have been
implemented. See [dask#10913](https://github.com/dask/dask/pull/10913), [dask#10884](https://github.com/dask/dask/pull/10884), [dask#10919](https://github.com/dask/dask/pull/10919), [dask#10896](https://github.com/dask/dask/pull/10896) and
primarily [dask#10883](https://github.com/dask/dask/pull/10883) from  [crusaderky](https://github.com/crusaderky) for more details.

#### More robust adaptive scaling on large clusters

Adaptive scaling could previously lose data during downscaling if many tasks had
to be moved. This typically, but not exclusively, occured on large clusters and
would manifest as a recomputation of tasks and could cause clusters to oscillate
between up- and downscaling without ever finishing.

See [distributed#8522](https://github.com/dask/distributed/pull/8522) from [crusaderky](https://github.com/crusaderky) for more details.

### Additional changes

- Remove flaky fastparquet test ([dask#10948](https://github.com/dask/dask/pull/10948)) [Patrick Hoefler](https://github.com/phofl)
- Enable Aggregation from dask-expr ([dask#10947](https://github.com/dask/dask/pull/10947)) [Patrick Hoefler](https://github.com/phofl)
- Update tests for assign change in dask-expr ([dask#10944](https://github.com/dask/dask/pull/10944)) [Patrick Hoefler](https://github.com/phofl)
- Adjust for pandas large string change ([dask#10942](https://github.com/dask/dask/pull/10942)) [Patrick Hoefler](https://github.com/phofl)
- Fix flaky test_describe_empty ([dask#10943](https://github.com/dask/dask/pull/10943)) [crusaderky](https://github.com/crusaderky)
- Use Python 3.12 as reference environment ([dask#10939](https://github.com/dask/dask/pull/10939)) [crusaderky](https://github.com/crusaderky)
- [Cosmetic] Clean up temp paths in test_config.py ([dask#10938](https://github.com/dask/dask/pull/10938)) [crusaderky](https://github.com/crusaderky)
- [CLI] `dask config set` and `dask config find` updates. ([dask#10930](https://github.com/dask/dask/pull/10930)) [Miles](https://github.com/milesgranger)
- combine_first when a chunk is full of NaNs ([dask#10932](https://github.com/dask/dask/pull/10932)) [crusaderky](https://github.com/crusaderky)
- Correctly parse lowercase true/false config from CLI ([dask#10926](https://github.com/dask/dask/pull/10926)) [crusaderky](https://github.com/crusaderky)
- `dask config get` fix when printing None values ([dask#10927](https://github.com/dask/dask/pull/10927)) [crusaderky](https://github.com/crusaderky)
- query-planning can’t be None ([dask#10928](https://github.com/dask/dask/pull/10928)) [crusaderky](https://github.com/crusaderky)
- Add `dask config set` ([dask#10921](https://github.com/dask/dask/pull/10921)) [Miles](https://github.com/milesgranger)
- Make nunique faster again ([dask#10922](https://github.com/dask/dask/pull/10922)) [Patrick Hoefler](https://github.com/phofl)
- Clean up some Cython warnings handling ([dask#10924](https://github.com/dask/dask/pull/10924)) [crusaderky](https://github.com/crusaderky)
- Bump pre-commit/action from 3.0.0 to 3.0.1 ([dask#10920](https://github.com/dask/dask/pull/10920))
- Raise and avoid data loss of meta provided to P2P shuffle is wrong ([distributed#8520](https://github.com/dask/distributed/pull/8520)) [Florian Jetter](https://github.com/fjetter)
- Fix gpuci: np.product is deprecated ([distributed#8518](https://github.com/dask/distributed/pull/8518)) [crusaderky](https://github.com/crusaderky)
- Update gpuCI `RAPIDS_VER` to `24.04` ([distributed#8471](https://github.com/dask/distributed/pull/8471))
- Unpin ipywidgets on Python 3.12 ([distributed#8516](https://github.com/dask/distributed/pull/8516)) [crusaderky](https://github.com/crusaderky)
- Keep old dependencies on run_spec collision ([distributed#8512](https://github.com/dask/distributed/pull/8512)) [crusaderky](https://github.com/crusaderky)
- Trivial mypy fix ([distributed#8513](https://github.com/dask/distributed/pull/8513)) [crusaderky](https://github.com/crusaderky)
- Ensure large payload can be serialized and sent over comms ([distributed#8507](https://github.com/dask/distributed/pull/8507)) [Florian Jetter](https://github.com/fjetter)
- Allow large graph warning threshold to be configured ([distributed#8508](https://github.com/dask/distributed/pull/8508)) [Florian Jetter](https://github.com/fjetter)
- Tokenization-related test tweaks (backport from #8185) ([distributed#8499](https://github.com/dask/distributed/pull/8499)) [crusaderky](https://github.com/crusaderky)
- Tweaks to `update_graph` (backport from #8185) ([distributed#8498](https://github.com/dask/distributed/pull/8498)) [crusaderky](https://github.com/crusaderky)
- AMM: test incremental retirements ([distributed#8501](https://github.com/dask/distributed/pull/8501)) [crusaderky](https://github.com/crusaderky)
- Suppress dask-expr warning in CI ([distributed#8505](https://github.com/dask/distributed/pull/8505)) [crusaderky](https://github.com/crusaderky)
- Ignore dask-expr warning in CI ([distributed#8504](https://github.com/dask/distributed/pull/8504)) [James Bourbeau](https://github.com/jrbourbeau)
- Improve tests for P2P stable ordering ([distributed#8458](https://github.com/dask/distributed/pull/8458)) [Hendrik Makait](https://github.com/hendrikmakait)
- Bump pre-commit/action from 3.0.0 to 3.0.1 ([distributed#8503](https://github.com/dask/distributed/pull/8503))

<a id="v2024-2-0"></a>

## 2024.2.0

Released on February 9, 2024

### Highlights

#### Deprecate Dask DataFrame implementation

The current Dask DataFrame implementation is deprecated.
In a future release, Dask DataFrame will use new implementation that
contains several improvements including a logical query planning.
The user-facing DataFrame API will remain unchanged.

The new implementation is already available and can be enabled by
installing the `dask-expr` library:

```bash
$ pip install dask-expr
```

and turning the query planning option on:

```python
>>> import dask
>>> dask.config.set({'dataframe.query-planning': True})
>>> import dask.dataframe as dd
```

API documentation for the new implementation is available at
[https://docs.dask.org/en/stable/dataframe-api.html](https://docs.dask.org/en/stable/dataframe-api.html)

Any feedback can be reported on the Dask issue tracker
[https://github.com/dask/dask/issues](https://github.com/dask/dask/issues)

See [dask#10912](https://github.com/dask/dask/pull/10912) from [Patrick Hoefler](https://github.com/phofl) for details.

#### Improved tokenization

This release contains several improvements to Dask’s object tokenization logic.
More objects now produce deterministic tokens, which can lead to improved performance
through caching of intermediate results.

See [dask#10898](https://github.com/dask/dask/pull/10898), [dask#10904](https://github.com/dask/dask/pull/10904), [dask#10876](https://github.com/dask/dask/pull/10876), [dask#10874](https://github.com/dask/dask/pull/10874), and [dask#10865](https://github.com/dask/dask/pull/10865) from [crusaderky](https://github.com/crusaderky) for details.

### Additional changes

- Fix inplace modification on read-only arrays for string conversion ([dask#10886](https://github.com/dask/dask/pull/10886)) [Patrick Hoefler](https://github.com/phofl)
- Add changelog entry for `dask-expr` ([dask#10915](https://github.com/dask/dask/pull/10915)) [Patrick Hoefler](https://github.com/phofl)
- Fix `leftsemi` merge for `cudf` ([dask#10914](https://github.com/dask/dask/pull/10914)) [Patrick Hoefler](https://github.com/phofl)
- Slight update to `dask-expr` warning ([dask#10916](https://github.com/dask/dask/pull/10916)) [James Bourbeau](https://github.com/jrbourbeau)
- Improve performance for `groupby.nunique` ([dask#10910](https://github.com/dask/dask/pull/10910)) [Patrick Hoefler](https://github.com/phofl)
- Add configuration for `leftsemi` merges in `dask-expr` ([dask#10908](https://github.com/dask/dask/pull/10908)) [Patrick Hoefler](https://github.com/phofl)
- Adjust assign test for `dask-expr` ([dask#10907](https://github.com/dask/dask/pull/10907)) [Patrick Hoefler](https://github.com/phofl)
- Avoid `pytest.warns` in `test_to_datetime` for GPU CI ([dask#10902](https://github.com/dask/dask/pull/10902)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update deployment options in docs homepage ([dask#10901](https://github.com/dask/dask/pull/10901)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix typo in dataframe docs ([dask#10900](https://github.com/dask/dask/pull/10900)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump `peter-evans/create-pull-request` from 5 to 6 ([dask#10894](https://github.com/dask/dask/pull/10894))
- Fix mimesis API `>=13.1.0` - use `random.randint` ([dask#10888](https://github.com/dask/dask/pull/10888)) [Miles](https://github.com/milesgranger)
- Adjust invalid test ([dask#10897](https://github.com/dask/dask/pull/10897)) [Patrick Hoefler](https://github.com/phofl)
- Pickle `da.argwhere` and `da.count_nonzero` ([dask#10885](https://github.com/dask/dask/pull/10885)) [crusaderky](https://github.com/crusaderky)
- Fix `dask-expr` tests after singleton pr ([dask#10892](https://github.com/dask/dask/pull/10892)) [Patrick Hoefler](https://github.com/phofl)
- Set lower bound version for `s3fs` ([dask#10889](https://github.com/dask/dask/pull/10889)) [Miles](https://github.com/milesgranger)
- Add a couple of `dask-expr` fixes for new parquet cache ([dask#10880](https://github.com/dask/dask/pull/10880)) [Florian Jetter](https://github.com/fjetter)
- Update deployment documentation ([dask#10882](https://github.com/dask/dask/pull/10882)) [Matthew Rocklin](https://github.com/mrocklin)
- Start with `dask-expr` doc build ([dask#10879](https://github.com/dask/dask/pull/10879)) [Patrick Hoefler](https://github.com/phofl)
- Test tokenization of static and class methods ([dask#10872](https://github.com/dask/dask/pull/10872)) [crusaderky](https://github.com/crusaderky)
- Add `distributed.print` and `distributed.warn` to API docs ([dask#10878](https://github.com/dask/dask/pull/10878)) [James Bourbeau](https://github.com/jrbourbeau)
- Run macos ci on M1 architecture ([dask#10877](https://github.com/dask/dask/pull/10877)) [Patrick Hoefler](https://github.com/phofl)
- Update tests for `dask-expr` ([dask#10838](https://github.com/dask/dask/pull/10838)) [Patrick Hoefler](https://github.com/phofl)
- Update parquet tests to align with `dask-expr` fixes ([dask#10851](https://github.com/dask/dask/pull/10851)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix regression in `test_graph_manipulation` ([dask#10873](https://github.com/dask/dask/pull/10873)) [crusaderky](https://github.com/crusaderky)
- Adjust `pytest` errors for dask-expr ci ([dask#10871](https://github.com/dask/dask/pull/10871)) [Patrick Hoefler](https://github.com/phofl)
- Set upper bound version for `numba` when `pandas<2.1` ([dask#10890](https://github.com/dask/dask/pull/10890)) [Miles](https://github.com/milesgranger)
- Deprecate `method` parameter in `DataFrame.fillna` ([dask#10846](https://github.com/dask/dask/pull/10846)) [Miles](https://github.com/milesgranger)
- Remove warning filter from `pyproject.toml` ([dask#10867](https://github.com/dask/dask/pull/10867)) [Patrick Hoefler](https://github.com/phofl)
- Skip `test_append_with_partition` for fastparquet ([dask#10828](https://github.com/dask/dask/pull/10828)) [Patrick Hoefler](https://github.com/phofl)
- Fix `pytest` 8 issues ([dask#10868](https://github.com/dask/dask/pull/10868)) [Patrick Hoefler](https://github.com/phofl)
- Adjust test for support of median in `Groupby.aggregate` in `dask-expr` (2/2) ([dask#10870](https://github.com/dask/dask/pull/10870)) [Hendrik Makait](https://github.com/hendrikmakait)
- Allow length of ascending to be larger than one in `sort_values` ([dask#10864](https://github.com/dask/dask/pull/10864)) [Florian Jetter](https://github.com/fjetter)
- Allow other message raised in Python 3.9 ([dask#10862](https://github.com/dask/dask/pull/10862)) [Hendrik Makait](https://github.com/hendrikmakait)
- Don’t crash when getting computation code in pathological cases ([distributed#8502](https://github.com/dask/distributed/pull/8502)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump `peter-evans/create-pull-request` from 5 to 6 ([distributed#8494](https://github.com/dask/distributed/pull/8494))
- fix test of `cudf` spilling metrics ([distributed#8478](https://github.com/dask/distributed/pull/8478)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Upgrade to `pytest` 8 ([distributed#8482](https://github.com/dask/distributed/pull/8482)) [crusaderky](https://github.com/crusaderky)
- Fix `test_two_consecutive_clients_share_results` ([distributed#8484](https://github.com/dask/distributed/pull/8484)) [crusaderky](https://github.com/crusaderky)
- Client word mix-up ([distributed#8481](https://github.com/dask/distributed/pull/8481)) [templiert](https://github.com/templiert)

<a id="v2024-1-1"></a>

## 2024.1.1

Released on January 26, 2024

### Highlights

#### Pandas 2.2 and Scipy 1.12 support

This release contains compatibility updates for the latest `pandas` and `scipy` releases.

See [dask#10834](https://github.com/dask/dask/pull/10834), [dask#10849](https://github.com/dask/dask/pull/10849), [dask#10845](https://github.com/dask/dask/pull/10845), and [distributed#8474](https://github.com/dask/distributed/pull/8474) from [crusaderky](https://github.com/crusaderky) for details.

#### Deprecations

- Deprecate `convert_dtype` in `apply` ([dask#10827](https://github.com/dask/dask/pull/10827)) [Miles](https://github.com/milesgranger)
- Deprecate `axis` in `DataFrame.rolling` ([dask#10803](https://github.com/dask/dask/pull/10803)) [Miles](https://github.com/milesgranger)
- Deprecate `out=` and `dtype=` parameter in most DataFrame methods ([dask#10800](https://github.com/dask/dask/pull/10800)) [crusaderky](https://github.com/crusaderky)
- Deprecate `axis` in `groupby` cumulative transformers ([dask#10796](https://github.com/dask/dask/pull/10796)) [Miles](https://github.com/milesgranger)
- Rename `shuffle` to `shuffle_method` in remaining methods ([dask#10797](https://github.com/dask/dask/pull/10797)) [Miles](https://github.com/milesgranger)

### Additional changes

- Add recommended deployment options to deployment docs ([dask#10866](https://github.com/dask/dask/pull/10866)) [James Bourbeau](https://github.com/jrbourbeau)
- Improve `_agg_finalize` to confirm to output expectation ([dask#10835](https://github.com/dask/dask/pull/10835)) [Hendrik Makait](https://github.com/hendrikmakait)
- Implement deterministic tokenization for hlg ([dask#10817](https://github.com/dask/dask/pull/10817)) [Patrick Hoefler](https://github.com/phofl)
- Refactor: move tests for `tokenize()` to its own module ([dask#10863](https://github.com/dask/dask/pull/10863)) [crusaderky](https://github.com/crusaderky)
- Update DataFrame examples section ([dask#10856](https://github.com/dask/dask/pull/10856)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily pin `mimesis<13.1.0` ([dask#10860](https://github.com/dask/dask/pull/10860)) [James Bourbeau](https://github.com/jrbourbeau)
- Trivial cosmetic tweaks to `_testing.py` ([dask#10857](https://github.com/dask/dask/pull/10857)) [crusaderky](https://github.com/crusaderky)
- Unskip and adjust tests for `groupby`-aggregate with `median` using `dask-expr` ([dask#10832](https://github.com/dask/dask/pull/10832)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix test for `sizeof(pd.MultiIndex)` in upstream CI ([dask#10850](https://github.com/dask/dask/pull/10850)) [crusaderky](https://github.com/crusaderky)
- `numpy` 2.0: fix slicing by `uint64` array ([dask#10854](https://github.com/dask/dask/pull/10854)) [crusaderky](https://github.com/crusaderky)
- Rename `numpy` version constants to match `pandas` ([dask#10843](https://github.com/dask/dask/pull/10843)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/cache` from 3 to 4 ([dask#10852](https://github.com/dask/dask/pull/10852))
- Update gpuCI `RAPIDS_VER` to `24.04` ([dask#10841](https://github.com/dask/dask/pull/10841))
- Fix deprecations in doctest ([dask#10844](https://github.com/dask/dask/pull/10844)) [crusaderky](https://github.com/crusaderky)
- Changed `dtype` arithmetics in `numpy` 2.x ([dask#10831](https://github.com/dask/dask/pull/10831)) [crusaderky](https://github.com/crusaderky)
- Adjust tests for `median` support in `dask-expr` ([dask#10839](https://github.com/dask/dask/pull/10839)) [Patrick Hoefler](https://github.com/phofl)
- Adjust tests for `median` support in `groupby-aggregate` in `dask-expr` ([dask#10840](https://github.com/dask/dask/pull/10840)) [Hendrik Makait](https://github.com/hendrikmakait)
- `numpy` 2.x: fix `std()` on `MaskedArray` ([dask#10837](https://github.com/dask/dask/pull/10837)) [crusaderky](https://github.com/crusaderky)
- Fail `dask-expr` ci if tests fail ([dask#10829](https://github.com/dask/dask/pull/10829)) [Patrick Hoefler](https://github.com/phofl)
- Activate `query_planning` when exporting tests ([dask#10833](https://github.com/dask/dask/pull/10833)) [Patrick Hoefler](https://github.com/phofl)
- Expose dataframe tests ([dask#10830](https://github.com/dask/dask/pull/10830)) [Patrick Hoefler](https://github.com/phofl)
- `numpy` 2: deprecations in n-dimensional `fft` functions ([dask#10821](https://github.com/dask/dask/pull/10821)) [crusaderky](https://github.com/crusaderky)
- Generalize `CreationDispatch` for `dask-expr` ([dask#10794](https://github.com/dask/dask/pull/10794)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Remove circular import when `dask-expr` enabled ([dask#10824](https://github.com/dask/dask/pull/10824)) [Miles](https://github.com/milesgranger)
- Minor[CI]: `publish-test-results` not marked as failed ([dask#10825](https://github.com/dask/dask/pull/10825)) [Miles](https://github.com/milesgranger)
- Fix more tests to use `pytest.warns()` ([dask#10818](https://github.com/dask/dask/pull/10818)) [Michał Górny](https://github.com/mgorny)
- `np.unique()`: inverse is shaped in `numpy` 2 ([dask#10819](https://github.com/dask/dask/pull/10819)) [crusaderky](https://github.com/crusaderky)
- Pin `test_split_adaptive_files` to `pyarrow` engine ([dask#10820](https://github.com/dask/dask/pull/10820)) [Patrick Hoefler](https://github.com/phofl)
- Adjust remaining tests in `dask/dask` ([dask#10813](https://github.com/dask/dask/pull/10813)) [Patrick Hoefler](https://github.com/phofl)
- Restrict test to Arrow only ([dask#10814](https://github.com/dask/dask/pull/10814)) [Patrick Hoefler](https://github.com/phofl)
- Filter warnings from `std` test ([dask#10815](https://github.com/dask/dask/pull/10815)) [Patrick Hoefler](https://github.com/phofl)
- Adjust mostly indexing tests ([dask#10790](https://github.com/dask/dask/pull/10790)) [Patrick Hoefler](https://github.com/phofl)
- Updates to deployment docs ([dask#10778](https://github.com/dask/dask/pull/10778)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Unblock documentation build ([dask#10807](https://github.com/dask/dask/pull/10807)) [Miles](https://github.com/milesgranger)
- Adjust `test_to_datetime` for `dask-expr` compatibility [Hendrik Makait](https://github.com/hendrikmakait)
- Upstream CI tweaks ([dask#10806](https://github.com/dask/dask/pull/10806)) [crusaderky](https://github.com/crusaderky)
- Improve tests for `to_numeric` ([dask#10804](https://github.com/dask/dask/pull/10804)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix test-report cache key indent ([dask#10798](https://github.com/dask/dask/pull/10798)) [Miles](https://github.com/milesgranger)
- Add test-report workflow ([dask#10783](https://github.com/dask/dask/pull/10783)) [Miles](https://github.com/milesgranger)
- Handle matrix subclass serialization ([distributed#8480](https://github.com/dask/distributed/pull/8480)) [Florian Jetter](https://github.com/fjetter)
- Use smallest data type for partition column in P2P ([distributed#8479](https://github.com/dask/distributed/pull/8479)) [Florian Jetter](https://github.com/fjetter)
- `pandas` 2.2: fix `test_dataframe_groupby_tasks` ([distributed#8475](https://github.com/dask/distributed/pull/8475)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/cache` from 3 to 4 ([distributed#8477](https://github.com/dask/distributed/pull/8477))
- `pandas` 2.2 vs. `pyarrow` 14: deprecated `DatetimeTZBlock` ([distributed#8476](https://github.com/dask/distributed/pull/8476)) [crusaderky](https://github.com/crusaderky)
- `pandas` 2.2.0: Deprecated frequency alias `M` in favor of `ME` ([distributed#8473](https://github.com/dask/distributed/pull/8473)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix docs build ([distributed#8472](https://github.com/dask/distributed/pull/8472)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix P2P-based joins with explicit `npartitions` ([distributed#8470](https://github.com/dask/distributed/pull/8470)) [Hendrik Makait](https://github.com/hendrikmakait)
- Ignore `dask-expr` in `test_report.py` script ([distributed#8464](https://github.com/dask/distributed/pull/8464)) [Miles](https://github.com/milesgranger)
- Nit: hardcode Python version in test report environment ([distributed#8462](https://github.com/dask/distributed/pull/8462)) [crusaderky](https://github.com/crusaderky)
- Change `test_report.py` - skip bad artifacts in `dask/dask` ([distributed#8461](https://github.com/dask/distributed/pull/8461)) [Miles](https://github.com/milesgranger)
- Replace all occurrences of `sys.is_finalizing` ([distributed#8449](https://github.com/dask/distributed/pull/8449)) [Florian Jetter](https://github.com/fjetter)

<a id="v2024-1-0"></a>

## 2024.1.0

Released on January 12, 2024

### Highlights

#### Partial rechunks within P2P

P2P rechunking now utilizes the relationships between input and output chunks.
For situations that do not require all-to-all data transfer, this may significantly
reduce the runtime and memory/disk footprint. It also enables task culling.

See [distributed#8330](https://github.com/dask/distributed/pull/8330) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Fastparquet engine deprecated

The `fastparquet` Parquet engine has been deprecated. Users should migrate to the `pyarrow`
engine by [installing PyArrow](https://arrow.apache.org/docs/python/install.html) and removing
`engine="fastparquet"` in `read_parquet` or `to_parquet` calls.

See [dask#10743](https://github.com/dask/dask/pull/10743) from [crusaderky](https://github.com/crusaderky) for details.

#### Improved serialization for arbitrary data

This release improves serialization robustness for arbitrary data. Previously there were
some cases where serialization could fail for non-`msgpack` serializable data.
In those cases we now fallback to using `pickle`.

See [dask#8447](https://github.com/dask/dask/pull/8447) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Additional deprecations

- Deprecate `shuffle` keyword in favour of `shuffle_method` for DataFrame methods ([dask#10738](https://github.com/dask/dask/pull/10738)) [Hendrik Makait](https://github.com/hendrikmakait)
- Deprecate automatic argument inference in `repartition` ([dask#10691](https://github.com/dask/dask/pull/10691)) [Patrick Hoefler](https://github.com/phofl)
- Deprecate `compute` parameter in `set_index` ([dask#10784](https://github.com/dask/dask/pull/10784)) [Miles](https://github.com/milesgranger)
- Deprecate `inplace` in `eval` ([dask#10785](https://github.com/dask/dask/pull/10785)) [Miles](https://github.com/milesgranger)
- Deprecate `Series.view` ([dask#10754](https://github.com/dask/dask/pull/10754)) [Miles](https://github.com/milesgranger)
- Deprecate `npartitions="auto"` for `set_index` & `sort_values` ([dask#10750](https://github.com/dask/dask/pull/10750)) [Miles](https://github.com/milesgranger)

### Additional changes

- Avoid shortcut in tasks shuffle that let to data loss ([dask#10763](https://github.com/dask/dask/pull/10763)) [Patrick Hoefler](https://github.com/phofl)
- Ignore data tasks when ordering ([dask#10706](https://github.com/dask/dask/pull/10706)) [Florian Jetter](https://github.com/fjetter)
- Add `get_dummies` from `dask-expr` ([dask#10791](https://github.com/dask/dask/pull/10791)) [Patrick Hoefler](https://github.com/phofl)
- Adjust IO tests for `dask-expr` migration ([dask#10776](https://github.com/dask/dask/pull/10776)) [Patrick Hoefler](https://github.com/phofl)
- Remove deprecation warning about `sort` and `split_out` in `groupby` ([dask#10788](https://github.com/dask/dask/pull/10788)) [Patrick Hoefler](https://github.com/phofl)
- Address `pandas` deprecations ([dask#10789](https://github.com/dask/dask/pull/10789)) [Patrick Hoefler](https://github.com/phofl)
- Import `distributed` only once in `get_scheduler` ([dask#10771](https://github.com/dask/dask/pull/10771)) [Florian Jetter](https://github.com/fjetter)
- Simplify GitHub actions ([dask#10781](https://github.com/dask/dask/pull/10781)) [crusaderky](https://github.com/crusaderky)
- Add unit test overview ([dask#10769](https://github.com/dask/dask/pull/10769)) [Miles](https://github.com/milesgranger)
- Clean up redundant bits in CI ([dask#10768](https://github.com/dask/dask/pull/10768)) [crusaderky](https://github.com/crusaderky)
- Update tests for `ufunc` ([dask#10773](https://github.com/dask/dask/pull/10773)) [Patrick Hoefler](https://github.com/phofl)
- Use `pytest.mark.skipif(DASK_EXPR_ENABLED)` ([dask#10774](https://github.com/dask/dask/pull/10774)) [crusaderky](https://github.com/crusaderky)
- Adjust shuffle tests for `dask-expr` ([dask#10759](https://github.com/dask/dask/pull/10759)) [Patrick Hoefler](https://github.com/phofl)
- Fix some deprecation warnings from `pandas` ([dask#10749](https://github.com/dask/dask/pull/10749)) [Patrick Hoefler](https://github.com/phofl)
- Adjust shuffle tests for `dask-expr` ([dask#10762](https://github.com/dask/dask/pull/10762)) [Patrick Hoefler](https://github.com/phofl)
- Update `pre-commit` ([dask#10767](https://github.com/dask/dask/pull/10767)) [Hendrik Makait](https://github.com/hendrikmakait)
- Clean up config switches in CI ([dask#10766](https://github.com/dask/dask/pull/10766)) [crusaderky](https://github.com/crusaderky)
- Improve exception for `validate_key` ([dask#10765](https://github.com/dask/dask/pull/10765)) [Hendrik Makait](https://github.com/hendrikmakait)
- Handle `datetimeindexes` in `set_index` with unknown divisions ([dask#10757](https://github.com/dask/dask/pull/10757)) [Patrick Hoefler](https://github.com/phofl)
- Add hashing for decimals ([dask#10758](https://github.com/dask/dask/pull/10758)) [Patrick Hoefler](https://github.com/phofl)
- Review tests for `is_monotonic` ([dask#10756](https://github.com/dask/dask/pull/10756)) [crusaderky](https://github.com/crusaderky)
- Change argument order in `value_counts_aggregate` ([dask#10751](https://github.com/dask/dask/pull/10751)) [Patrick Hoefler](https://github.com/phofl)
- Adjust some groupby tests for `dask-expr` ([dask#10752](https://github.com/dask/dask/pull/10752)) [Patrick Hoefler](https://github.com/phofl)
- Restrict mimesis to `< 12` for 3.9 build ([dask#10755](https://github.com/dask/dask/pull/10755)) [Patrick Hoefler](https://github.com/phofl)
- Don’t evaluate config in skip condition ([dask#10753](https://github.com/dask/dask/pull/10753)) [Patrick Hoefler](https://github.com/phofl)
- Adjust some tests to be compatible with `dask-expr` ([dask#10714](https://github.com/dask/dask/pull/10714)) [Patrick Hoefler](https://github.com/phofl)
- Make `dask.array.utils` functions more generic to other Dask Arrays ([dask#10676](https://github.com/dask/dask/pull/10676)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove duplciate “single machine” section ([dask#10747](https://github.com/dask/dask/pull/10747)) [Matthew Rocklin](https://github.com/mrocklin)
- Tweak ORC `engine=` parameter ([dask#10746](https://github.com/dask/dask/pull/10746)) [crusaderky](https://github.com/crusaderky)
- Add pandas 3.0 deprecations and migration prep for `dask-expr` ([dask#10723](https://github.com/dask/dask/pull/10723)) [Miles](https://github.com/milesgranger)
- Add task graph animation to docs homepage ([dask#10730](https://github.com/dask/dask/pull/10730)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Use new Xarray logo ([dask#10729](https://github.com/dask/dask/pull/10729)) [James Bourbeau](https://github.com/jrbourbeau)
- Update tab styling on “10 Minutes to Dask” page ([dask#10728](https://github.com/dask/dask/pull/10728)) [James Bourbeau](https://github.com/jrbourbeau)
- Update environment file upload step in CI ([dask#10726](https://github.com/dask/dask/pull/10726)) [James Bourbeau](https://github.com/jrbourbeau)
- Don’t duplicate unobserved categories in GroupBy.nunqiue if `split_out>1` ([dask#10716](https://github.com/dask/dask/pull/10716)) [Patrick Hoefler](https://github.com/phofl)
- Changelog entry for `dask.order` update ([dask#10715](https://github.com/dask/dask/pull/10715)) [Florian Jetter](https://github.com/fjetter)
- Relax redundant-key check in `_check_dsk` ([dask#10701](https://github.com/dask/dask/pull/10701)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `test_report.py` ([distributed#8459](https://github.com/dask/distributed/pull/8459)) [Miles](https://github.com/milesgranger)
- Revert `pickle` change ([distributed#8456](https://github.com/dask/distributed/pull/8456)) [Florian Jetter](https://github.com/fjetter)
- Adapt `test_report.py` to support `dask/dask` repository ([distributed#8450](https://github.com/dask/distributed/pull/8450)) [Miles](https://github.com/milesgranger)
- Maintain stable ordering for P2P shuffling ([distributed#8453](https://github.com/dask/distributed/pull/8453)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add no worker timeout for scheduler ([distributed#8371](https://github.com/dask/distributed/pull/8371)) [FTang21](https://github.com/FTang21)
- Allow tests workflow to be dispatched manually by maintainers ([distributed#8445](https://github.com/dask/distributed/pull/8445)) [Erik Sundell](https://github.com/consideRatio)
- Make scheduler-related transition functionality private ([distributed#8448](https://github.com/dask/distributed/pull/8448)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update `pre-commit` hooks ([distributed#8444](https://github.com/dask/distributed/pull/8444)) [Hendrik Makait](https://github.com/hendrikmakait)
- Do not always check if `__main__ in result` when pickling ([distributed#8443](https://github.com/dask/distributed/pull/8443)) [Florian Jetter](https://github.com/fjetter)
- Delegate `wait_for_workers` to cluster instances only when implemented ([distributed#8441](https://github.com/dask/distributed/pull/8441)) [Erik Sundell](https://github.com/consideRatio)
- Extend sleep in `test_pandas` ([distributed#8440](https://github.com/dask/distributed/pull/8440)) [Julian Gilbey](https://github.com/juliangilbey)
- Avoid deprecated `shuffle` keyword ([distributed#8439](https://github.com/dask/distributed/pull/8439)) [Hendrik Makait](https://github.com/hendrikmakait)
- Shuffle metrics 4/4: Remove bespoke diagnostics ([distributed#8367](https://github.com/dask/distributed/pull/8367)) [crusaderky](https://github.com/crusaderky)
- Do not run `gilknocker` in testsuite ([distributed#8423](https://github.com/dask/distributed/pull/8423)) [Florian Jetter](https://github.com/fjetter)
- Tweak `abstractmethods` ([distributed#8427](https://github.com/dask/distributed/pull/8427)) [crusaderky](https://github.com/crusaderky)
- Shuffle metrics 3/4: Capture background metrics ([distributed#8366](https://github.com/dask/distributed/pull/8366)) [crusaderky](https://github.com/crusaderky)
- Shuffle metrics 2/4: Add background metrics ([distributed#8365](https://github.com/dask/distributed/pull/8365)) [crusaderky](https://github.com/crusaderky)
- Shuffle metrics 1/4: Add foreground metrics ([distributed#8364](https://github.com/dask/distributed/pull/8364)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/upload-artifact` from 3 to 4 ([distributed#8420](https://github.com/dask/distributed/pull/8420))
- Fix `test_merge_p2p_shuffle_reused_dataframe_with_different_parameters` ([distributed#8422](https://github.com/dask/distributed/pull/8422)) [Hendrik Makait](https://github.com/hendrikmakait)
- Expand `Client.upload_file` docs example ([distributed#8313](https://github.com/dask/distributed/pull/8313)) [Miles](https://github.com/milesgranger)
- Improve logging in P2P’s scheduler plugin ([distributed#8410](https://github.com/dask/distributed/pull/8410)) [Hendrik Makait](https://github.com/hendrikmakait)
- Re-enable `test_decide_worker_coschedule_order_neighbors` ([distributed#8402](https://github.com/dask/distributed/pull/8402)) [Florian Jetter](https://github.com/fjetter)
- Add cuDF spilling statistics to RMM/GPU memory plot ([distributed#8148](https://github.com/dask/distributed/pull/8148)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix inconsistent hashing for Nanny-spawned workers ([distributed#8400](https://github.com/dask/distributed/pull/8400)) [Charles Stern](https://github.com/cisaacstern)
- Do not allow workers to downscale if they are running long-running tasks (e.g. `worker_client`) ([distributed#7481](https://github.com/dask/distributed/pull/7481)) [Florian Jetter](https://github.com/fjetter)
- Fix flaky `test_subprocess_cluster_does_not_depend_on_logging` ([distributed#8417](https://github.com/dask/distributed/pull/8417)) [crusaderky](https://github.com/crusaderky)

<a id="v2023-12-1"></a>

## 2023.12.1

Released on December 15, 2023

### Highlights

#### Logical Query Planning now available for Dask DataFrames

Dask DataFrames are now much more performant by using a logical query planner.
This feature is currently off by default, but can be turned on with:

```python
dask.config.set({"dataframe.query-planning": True})
```

You also need to have `dask-expr` installed:

```bash
pip install dask-expr
```

We’ve seen promising performance improvements so far, see
[this blog post](https://blog.coiled.io/blog/dask-expr-tpch-dask.html)
and [these regularly updated benchmarks](https://tpch.coiled.io)  for more information.
A more detailed explanation of how the query optimizer works can be found in
[this blog post](https://blog.coiled.io/blog/dask-expr-introduction.html).

This feature is still under active development
and the [API](https://github.com/dask-contrib/dask-expr#api-coverage) isn’t stable yet,
so breaking changes can occur. We expect to make the query optimizer the default early next year.

See [dask#10634](https://github.com/dask/dask/pull/10634) from [Patrick Hoefler](https://github.com/phofl) for details.

#### Dtype inference in `read_parquet`

`read_parquet` will now infer the Arrow types `pa.date32()`, `pa.date64()` and
`pa.decimal()` as a `ArrowDtype` in pandas. These dtypes are backed by the
original Arrow array, and thus avoid the conversion to NumPy object. Additionally,
`read_parquet` will no longer infer nested and binary types as strings, they will
be stored in NumPy object arrays.

See [dask#10698](https://github.com/dask/dask/pull/10698) and [dask#10705](https://github.com/dask/dask/pull/10705) from [Patrick Hoefler](https://github.com/phofl) for details.

#### Scheduling improvements to reduce memory usage

This release includes a major rewrite to a core part of our scheduling logic. It
includes a new approach to the topological sorting algorithm in `dask.order`
which determines the order in which tasks are run. Improper ordering is known to
be a major contributor to too large cluster memory pressure.

Updates in this release fix a couple of performance regressions that were introduced
in the release `2023.10.0` (see [dask#10535](https://github.com/dask/dask/pull/10535)). Generally, computations should now
be much more eager to release data if it is no longer required in memory.

See [dask#10660](https://github.com/dask/dask/pull/10660), [dask#10697](https://github.com/dask/dask/pull/10697) from [Florian Jetter](https://github.com/fjetter) for details.

#### Improved P2P-based merging robustness and performance

This release contains several updates that fix a possible deadlock introduced in
2023.9.2 and improve the robustness of P2P-based merging when the cluster is
dynamically scaling up.

See [distributed#8415](https://github.com/dask/distributed/pull/8415), [distributed#8416](https://github.com/dask/distributed/pull/8416), and [distributed#8414](https://github.com/dask/distributed/pull/8414) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Removed disabling pickle option

The `distributed.scheduler.pickle` configuration option is no longer supported.
As of the 2023.4.0 release, `pickle` is used to transmit task graphs, so can no
longer be disabled. We now raise an informative error when `distributed.scheduler.pickle`
is set to `False`.

See [distributed#8401](https://github.com/dask/distributed/pull/8401) from [Florian Jetter](https://github.com/fjetter) for details.

### Additional changes

- Add changelog entry for recent P2P merge fixes ([dask#10712](https://github.com/dask/dask/pull/10712)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update DataFrame page ([dask#10710](https://github.com/dask/dask/pull/10710)) [Matthew Rocklin](https://github.com/mrocklin)
- Add changelog entry for `dask-expr` switch ([dask#10704](https://github.com/dask/dask/pull/10704)) [Patrick Hoefler](https://github.com/phofl)
- Improve changelog entry for `PipInstall` changes ([dask#10711](https://github.com/dask/dask/pull/10711)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove PR labeler ([dask#10709](https://github.com/dask/dask/pull/10709)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `.__wrapped__` to `Delayed` object ([dask#10695](https://github.com/dask/dask/pull/10695)) [Andrew S. Rosen](https://github.com/Andrew-S-Rosen)
- Bump `actions/labeler` from 4.3.0 to 5.0.0 ([dask#10689](https://github.com/dask/dask/pull/10689))
- Bump `actions/stale` from 8 to 9 ([dask#10690](https://github.com/dask/dask/pull/10690))
- [Dask.order] Remove non-runnable leaf nodes from ordering ([dask#10697](https://github.com/dask/dask/pull/10697)) [Florian Jetter](https://github.com/fjetter)
- Update installation docs ([dask#10699](https://github.com/dask/dask/pull/10699)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix software environment link in docs ([dask#10700](https://github.com/dask/dask/pull/10700)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid converting non-strings to arrow strings for read_parquet ([dask#10692](https://github.com/dask/dask/pull/10692)) [Patrick Hoefler](https://github.com/phofl)
- Bump `xarray-contrib/issue-from-pytest-log` from 1.2.7 to 1.2.8 ([dask#10687](https://github.com/dask/dask/pull/10687))
- Fix `tokenize` for `pd.DateOffset` ([dask#10664](https://github.com/dask/dask/pull/10664)) [jochenott](https://github.com/jochenott)
- Bugfix for writing empty array to zarr ([dask#10506](https://github.com/dask/dask/pull/10506)) [Ben](https://github.com/benjaminhduncan)
- Docs update, fixup styling, mention free ([dask#10679](https://github.com/dask/dask/pull/10679)) [Matthew Rocklin](https://github.com/mrocklin)
- Update deployment docs ([dask#10680](https://github.com/dask/dask/pull/10680)) [Matthew Rocklin](https://github.com/mrocklin)
- Dask.order rewrite using a critical path approach ([dask#10660](https://github.com/dask/dask/pull/10660)) [Florian Jetter](https://github.com/fjetter)
- Avoid substituting keys that occur multiple times ([dask#10646](https://github.com/dask/dask/pull/10646)) [Florian Jetter](https://github.com/fjetter)
- Add missing image to docs ([dask#10694](https://github.com/dask/dask/pull/10694)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump `actions/setup-python` from 4 to 5 ([dask#10688](https://github.com/dask/dask/pull/10688))
- Update landing page ([dask#10674](https://github.com/dask/dask/pull/10674)) [Matthew Rocklin](https://github.com/mrocklin)
- Make meta check simpler in dispatch ([dask#10638](https://github.com/dask/dask/pull/10638)) [Patrick Hoefler](https://github.com/phofl)
- Pin PR Labeler ([dask#10675](https://github.com/dask/dask/pull/10675)) [Matthew Rocklin](https://github.com/mrocklin)
- Reorganize docs index a bit ([dask#10669](https://github.com/dask/dask/pull/10669)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump `actions/setup-java` from 3 to 4 ([dask#10667](https://github.com/dask/dask/pull/10667))
- Bump `conda-incubator/setup-miniconda` from 2.2.0 to 3.0.1 ([dask#10668](https://github.com/dask/dask/pull/10668))
- Bump `xarray-contrib/issue-from-pytest-log` from 1.2.6 to 1.2.7 ([dask#10666](https://github.com/dask/dask/pull/10666))
- Fix `test_categorize_info` with nightly `pyarrow` ([dask#10662](https://github.com/dask/dask/pull/10662)) [James Bourbeau](https://github.com/jrbourbeau)
- Rewrite `test_subprocess_cluster_does_not_depend_on_logging` ([distributed#8409](https://github.com/dask/distributed/pull/8409)) [Hendrik Makait](https://github.com/hendrikmakait)
- Avoid `RecursionError` when failing to pickle key in `SpillBuffer` and using `tblib=3` ([distributed#8404](https://github.com/dask/distributed/pull/8404)) [Hendrik Makait](https://github.com/hendrikmakait)
- Allow tasks to override `is_rootish` heuristic  ([distributed#8412](https://github.com/dask/distributed/pull/8412)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove GPU executor ([distributed#8399](https://github.com/dask/distributed/pull/8399)) [Hendrik Makait](https://github.com/hendrikmakait)
- Do not rely on logging for subprocess cluster ([distributed#8398](https://github.com/dask/distributed/pull/8398)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update gpuCI `RAPIDS_VER` to `24.02` ([distributed#8384](https://github.com/dask/distributed/pull/8384))
- Bump `actions/setup-python` from 4 to 5 ([distributed#8396](https://github.com/dask/distributed/pull/8396))
- Ensure output chunks in P2P rechunking are distributed homogeneously ([distributed#8207](https://github.com/dask/distributed/pull/8207)) [Florian Jetter](https://github.com/fjetter)
- Trivial: fix typo ([distributed#8395](https://github.com/dask/distributed/pull/8395)) [crusaderky](https://github.com/crusaderky)
- Bump `JamesIves/github-pages-deploy-action` from 4.4.3 to 4.5.0 ([distributed#8387](https://github.com/dask/distributed/pull/8387))
- Bump `conda-incubator/setup-miniconda from` 3.0.0 to 3.0.1 ([distributed#8388](https://github.com/dask/distributed/pull/8388))

<a id="v2023-12-0"></a>

## 2023.12.0

Released on December 1, 2023

### Highlights

#### PipInstall restart and environment variables

The `distributed.PipInstall` plugin now has more robust restart logic and also supports
[environment variables](https://pip.pypa.io/en/stable/reference/requirements-file-format/#using-environment-variables).

Below shows how users can use the `distributed.PipInstall` plugin and a `TOKEN` environment
variable to securely install a package from a private repository:

```python
from dask.distributed import PipInstall
plugin = PipInstall(packages=["private_package@git+https://${TOKEN}@github.com/dask/private_package.git])
client.register_plugin(plugin)
```

See [distributed#8374](https://github.com/dask/distributed/pull/8374), [distributed#8357](https://github.com/dask/distributed/pull/8357), and [distributed#8343](https://github.com/dask/distributed/pull/8343) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Bokeh 3.3.0 compatibility

This release contains compatibility updates for using `bokeh>=3.3.0` with proxied Dask dashboards.
Previously the contents of dashboard plots wouldn’t be displayed.

See [distributed#8347](https://github.com/dask/distributed/pull/8347) and [distributed#8381](https://github.com/dask/distributed/pull/8381) from [Jacob Tomlinson](https://github.com/jacobtomlinson) for details.

### Additional changes

- Add `network` marker to `test_pyarrow_filesystem_option_real_data` ([dask#10653](https://github.com/dask/dask/pull/10653)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bump GPU CI to CUDA 11.8 ([dask#10656](https://github.com/dask/dask/pull/10656)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Tokenize `pandas` offsets deterministically ([dask#10643](https://github.com/dask/dask/pull/10643)) [Patrick Hoefler](https://github.com/phofl)
- Add tokenize `pd.NA` functionality ([dask#10640](https://github.com/dask/dask/pull/10640)) [Patrick Hoefler](https://github.com/phofl)
- Update gpuCI `RAPIDS_VER` to `24.02` ([dask#10636](https://github.com/dask/dask/pull/10636))
- Fix precision handling in `array.linalg.norm` ([dask#10556](https://github.com/dask/dask/pull/10556)) [joanrue](https://github.com/joanrue)
- Add `axis` argument to `DataFrame.clip` and `Series.clip` ([dask#10616](https://github.com/dask/dask/pull/10616)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update changelog entry for in-memory rechunking ([dask#10630](https://github.com/dask/dask/pull/10630)) [Florian Jetter](https://github.com/fjetter)
- Fix flaky `test_resources_reset_after_cancelled_task` ([distributed#8373](https://github.com/dask/distributed/pull/8373)) [crusaderky](https://github.com/crusaderky)
- Bump GPU CI to CUDA 11.8 ([distributed#8376](https://github.com/dask/distributed/pull/8376)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Bump `conda-incubator/setup-miniconda` from 2.2.0 to 3.0.0 ([distributed#8372](https://github.com/dask/distributed/pull/8372))
- Add debug logs to P2P scheduler plugin ([distributed#8358](https://github.com/dask/distributed/pull/8358)) [Hendrik Makait](https://github.com/hendrikmakait)
- `O(1)` access for `/info/task/` endpoint ([distributed#8363](https://github.com/dask/distributed/pull/8363)) [crusaderky](https://github.com/crusaderky)
- Remove stringification from shuffle annotations ([distributed#8362](https://github.com/dask/distributed/pull/8362)) [crusaderky](https://github.com/crusaderky)
- Don’t cast `int` metrics to `float` ([distributed#8361](https://github.com/dask/distributed/pull/8361)) [crusaderky](https://github.com/crusaderky)
- Drop asyncio TCP backend ([distributed#8355](https://github.com/dask/distributed/pull/8355)) [Florian Jetter](https://github.com/fjetter)
- Add offload support to `context_meter.add_callback` ([distributed#8360](https://github.com/dask/distributed/pull/8360)) [crusaderky](https://github.com/crusaderky)
- Test that `sync()` propagates contextvars ([distributed#8354](https://github.com/dask/distributed/pull/8354)) [crusaderky](https://github.com/crusaderky)
- `captured_context_meter` ([distributed#8352](https://github.com/dask/distributed/pull/8352)) [crusaderky](https://github.com/crusaderky)
- `context_meter.clear_callbacks` ([distributed#8353](https://github.com/dask/distributed/pull/8353)) [crusaderky](https://github.com/crusaderky)
- Use `@log_errors` decorator ([distributed#8351](https://github.com/dask/distributed/pull/8351)) [crusaderky](https://github.com/crusaderky)
- Fix `test_statistical_profiling_cycle` ([distributed#8356](https://github.com/dask/distributed/pull/8356)) [Florian Jetter](https://github.com/fjetter)
- Shuffle: don’t parse dask.config at every RPC ([distributed#8350](https://github.com/dask/distributed/pull/8350)) [crusaderky](https://github.com/crusaderky)
- Replace `Client.register_plugin` s `idempotent` argument with `.idempotent` attribute on plugins ([distributed#8342](https://github.com/dask/distributed/pull/8342)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix test report generation ([distributed#8346](https://github.com/dask/distributed/pull/8346)) [Hendrik Makait](https://github.com/hendrikmakait)
- Install `pyarrow-hotfix` on `mindeps-pandas` CI ([distributed#8344](https://github.com/dask/distributed/pull/8344)) [Hendrik Makait](https://github.com/hendrikmakait)
- Reduce memory usage of scheduler process - optimize `scheduler.py::TaskState` class ([distributed#8331](https://github.com/dask/distributed/pull/8331)) [Miles](https://github.com/milesgranger)
- Bump `pre-commit` linters ([distributed#8340](https://github.com/dask/distributed/pull/8340)) [crusaderky](https://github.com/crusaderky)
- Update cuDF test with explicit `dtype=object` ([distributed#8339](https://github.com/dask/distributed/pull/8339)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix `Cluster` / `SpecCluster` calls to async close methods ([distributed#8327](https://github.com/dask/distributed/pull/8327)) [Peter Andreas Entschev](https://github.com/pentschev)

<a id="v2023-11-0"></a>

## 2023.11.0

Released on November 10, 2023

### Highlights

#### Zero-copy P2P Array Rechunking

Users should see significant performance improvements when using in-memory P2P array rechunking.
This is due to no longer copying underlying data buffers.

Below shows a simple example where we compare performance of different rechunking methods.

```python
shape = (30_000, 6_000, 150) # 201.17 GiB
input_chunks = (60, -1, -1) # 411.99 MiB
output_chunks = (-1, 6, -1) # 205.99 MiB

arr = da.random.random(size, chunks=input_chunks)
with dask.config.set({
    "array.rechunk.method": "p2p",
    "distributed.p2p.disk": True,
}):
    (
      da.random.random(size, chunks=input_chunks)
      .rechunk(output_chunks)
      .sum()
      .compute()
    )
```

![A comparison of rechunking performance between the different methods
tasks, p2p with disk and p2p without disk on different cluster sizes. The
graph shows that p2p without disk is up to 60% faster than the default
tasks based approach.](images/changelog/2023110-rechunking-disk-perf.png)

See [distributed#8282](https://github.com/dask/distributed/pull/8282), [distributed#8318](https://github.com/dask/distributed/pull/8318), [distributed#8321](https://github.com/dask/distributed/pull/8321) from [crusaderky](https://github.com/crusaderky) and
([distributed#8322](https://github.com/dask/distributed/pull/8322)) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Deprecating PyArrow <14.0.1

`pyarrow<14.0.1` usage is deprecated starting in this release. It’s recommended for all users to upgrade their
version of `pyarrow` or install `pyarrow-hotfix`. See [this CVE](https://www.cve.org/CVERecord?id=CVE-2023-47248)
for full details.

See [dask#10622](https://github.com/dask/dask/pull/10622) from [Florian Jetter](https://github.com/fjetter) for details.

#### Improved PyArrow filesystem for Parquet

Using `filesystem="arrow"` when reading Parquet datasets now properly inferrs the correct cloud region
when accessing remote, cloud-hosted data.

See [dask#10590](https://github.com/dask/dask/pull/10590) from [Richard (Rick) Zamora](https://github.com/rjzamora) for details.

#### Improve Type Reconciliation in P2P Shuffling

See [distributed#8332](https://github.com/dask/distributed/pull/8332) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

### Additional changes

- Fix sporadic failure of `test_dataframe::test_quantile` ([dask#10625](https://github.com/dask/dask/pull/10625)) [Miles](https://github.com/milesgranger)
- Bump minimum `click` to `>=8.1` ([dask#10623](https://github.com/dask/dask/pull/10623)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Refactor `test_quantile` ([dask#10620](https://github.com/dask/dask/pull/10620)) [Miles](https://github.com/milesgranger)
- Avoid `PerformanceWarning` for fragmented DataFrame ([dask#10621](https://github.com/dask/dask/pull/10621)) [Patrick Hoefler](https://github.com/phofl)
- Generalize computation of `NEW_*_VER` in GPU CI updating workflow ([dask#10610](https://github.com/dask/dask/pull/10610)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Switch to newer GPU CI images ([dask#10608](https://github.com/dask/dask/pull/10608)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Remove double slash in `fsspec` tests ([dask#10605](https://github.com/dask/dask/pull/10605)) [Mario Šaško](https://github.com/mariosasko)
- Reenable `test_ucx_config_w_env_var` ([distributed#8272](https://github.com/dask/distributed/pull/8272)) [Peter Andreas Entschev](https://github.com/pentschev)
- Don’t share `host_array` when receiving from network ([distributed#8308](https://github.com/dask/distributed/pull/8308)) [crusaderky](https://github.com/crusaderky)
- Generalize computation of `NEW_*_VER` in GPU CI updating workflow ([distributed#8319](https://github.com/dask/distributed/pull/8319)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Switch to newer GPU CI images ([distributed#8316](https://github.com/dask/distributed/pull/8316)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Minor updates to shuffle dashboard ([distributed#8315](https://github.com/dask/distributed/pull/8315)) [Matthew Rocklin](https://github.com/mrocklin)
- Don’t use `bytearray().join` ([distributed#8312](https://github.com/dask/distributed/pull/8312)) [crusaderky](https://github.com/crusaderky)
- Reuse identical shuffles in P2P hash join ([distributed#8306](https://github.com/dask/distributed/pull/8306)) [Hendrik Makait](https://github.com/hendrikmakait)

<a id="v2023-10-1"></a>

## 2023.10.1

Released on October 27, 2023

### Highlights

#### Python 3.12

This release adds official support for Python 3.12.

See [dask#10544](https://github.com/dask/dask/pull/10544) and [distributed#8223](https://github.com/dask/distributed/pull/8223) from [Thomas Grainger](https://github.com/graingert) for details.

### Additional changes

- Avoid splitting parquet files to row groups as aggressively ([dask#10600](https://github.com/dask/dask/pull/10600)) [Matthew Rocklin](https://github.com/mrocklin)
- Speed up `normalize_chunks` for common case ([dask#10579](https://github.com/dask/dask/pull/10579)) [Martin Durant](https://github.com/martindurant)
- Use Python 3.11 for upstream and doctests CI build ([dask#10596](https://github.com/dask/dask/pull/10596)) [Thomas Grainger](https://github.com/graingert)
- Bump `actions/checkout` from 4.1.0 to 4.1.1 ([dask#10592](https://github.com/dask/dask/pull/10592))
- Switch to PyTables `HEAD` ([dask#10580](https://github.com/dask/dask/pull/10580)) [Thomas Grainger](https://github.com/graingert)
- Remove `numpy.core` warning filter, link to issue on `pyarrow` caused `BlockManager` warning ([dask#10571](https://github.com/dask/dask/pull/10571)) [Thomas Grainger](https://github.com/graingert)
- Unignore and fix deprecated freq aliases ([dask#10577](https://github.com/dask/dask/pull/10577)) [Thomas Grainger](https://github.com/graingert)
- Move `register_assert_rewrite` earlier in `conftest` to fix warnings ([dask#10578](https://github.com/dask/dask/pull/10578)) [Thomas Grainger](https://github.com/graingert)
- Upgrade `versioneer` to 0.29 ([dask#10575](https://github.com/dask/dask/pull/10575)) [Thomas Grainger](https://github.com/graingert)
- change `test_concat_categorical` to be non-strict ([dask#10574](https://github.com/dask/dask/pull/10574)) [Thomas Grainger](https://github.com/graingert)
- Enable SciPy tests with NumPy 2.0 [Thomas Grainger](https://github.com/graingert)
- Enable tests for scikit-image with NumPy 2.0 ([dask#10569](https://github.com/dask/dask/pull/10569)) [Thomas Grainger](https://github.com/graingert)
- Fix upstream build ([dask#10549](https://github.com/dask/dask/pull/10549)) [Thomas Grainger](https://github.com/graingert)
- Add optimized code paths for `drop_duplicates` ([dask#10542](https://github.com/dask/dask/pull/10542)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support `cudf` backend in `dd.DataFrame.sort_values` ([dask#10551](https://github.com/dask/dask/pull/10551)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Rename “GIL Contention” to just GIL in chart labels ([distributed#8305](https://github.com/dask/distributed/pull/8305)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump `actions/checkout` from 4.1.0 to 4.1.1 ([distributed#8299](https://github.com/dask/distributed/pull/8299))
- Fix dashboard ([distributed#8293](https://github.com/dask/distributed/pull/8293)) [Hendrik Makait](https://github.com/hendrikmakait)
- `@log_errors` for async tasks ([distributed#8294](https://github.com/dask/distributed/pull/8294)) [crusaderky](https://github.com/crusaderky)
- Annotations and better tests for serialize_bytes ([distributed#8300](https://github.com/dask/distributed/pull/8300)) [crusaderky](https://github.com/crusaderky)
- Temporarily xfail `test_decide_worker_coschedule_order_neighbors` to unblock CI ([distributed#8298](https://github.com/dask/distributed/pull/8298)) [James Bourbeau](https://github.com/jrbourbeau)
- Skip `xdist` and `matplotlib` in code samples ([distributed#8290](https://github.com/dask/distributed/pull/8290)) [Matthew Rocklin](https://github.com/mrocklin)
- Use `numpy._core` on `numpy>=2.dev0` ([distributed#8291](https://github.com/dask/distributed/pull/8291)) [Thomas Grainger](https://github.com/graingert)
- Fix calculation of `MemoryShardsBuffer.bytes_read` ([distributed#8289](https://github.com/dask/distributed/pull/8289)) [crusaderky](https://github.com/crusaderky)
- Allow P2P to store data in-memory ([distributed#8279](https://github.com/dask/distributed/pull/8279)) [Hendrik Makait](https://github.com/hendrikmakait)
- Upgrade `versioneer` to 0.29 ([distributed#8288](https://github.com/dask/distributed/pull/8288)) [Thomas Grainger](https://github.com/graingert)
- Allow `ResourceLimiter` to be unlimited ([distributed#8276](https://github.com/dask/distributed/pull/8276)) [Hendrik Makait](https://github.com/hendrikmakait)
- Run `pre-commit` autoupdate ([distributed#8281](https://github.com/dask/distributed/pull/8281)) [Thomas Grainger](https://github.com/graingert)
- Annotate instance variables for P2P layers ([distributed#8280](https://github.com/dask/distributed/pull/8280)) [Hendrik Makait](https://github.com/hendrikmakait)
- Remove worker gracefully should not mark tasks as suspicious ([distributed#8234](https://github.com/dask/distributed/pull/8234)) [Thomas Grainger](https://github.com/graingert)
- Add signal handling to `dask spec` ([distributed#8261](https://github.com/dask/distributed/pull/8261)) [Thomas Grainger](https://github.com/graingert)
- Add typing for `sync` ([distributed#8275](https://github.com/dask/distributed/pull/8275)) [Hendrik Makait](https://github.com/hendrikmakait)
- Better annotations for shuffle offload ([distributed#8277](https://github.com/dask/distributed/pull/8277)) [crusaderky](https://github.com/crusaderky)
- Test minimum versions for p2p shuffle ([distributed#8270](https://github.com/dask/distributed/pull/8270)) [crusaderky](https://github.com/crusaderky)
- Run coverage on test failures ([distributed#8269](https://github.com/dask/distributed/pull/8269)) [crusaderky](https://github.com/crusaderky)
- Use `aiohttp` with extensions ([distributed#8274](https://github.com/dask/distributed/pull/8274)) [Thomas Grainger](https://github.com/graingert)

<a id="v2023-10-0"></a>

## 2023.10.0

Released on October 13, 2023

### Highlights

#### Reduced memory pressure for multi array reductions

This release contains major updates to Dask’s task graph scheduling logic.
The updates here significantly reduce memory pressure on array reductions.
We anticipate this will have a strong impact on the array computing community.

See [dask#10535](https://github.com/dask/dask/pull/10535) from [Florian Jetter](https://github.com/fjetter) for details.

#### Improved P2P shuffling robustness

There are several updates (listed below) that make P2P shuffling much more
robust and less likely to fail.

See [distributed#8262](https://github.com/dask/distributed/pull/8262), [distributed#8264](https://github.com/dask/distributed/pull/8264), [distributed#8242](https://github.com/dask/distributed/pull/8242), [distributed#8244](https://github.com/dask/distributed/pull/8244),
and [distributed#8235](https://github.com/dask/distributed/pull/8235) from [Hendrik Makait](https://github.com/hendrikmakait) and [distributed#8124](https://github.com/dask/distributed/pull/8124) from
[Charles Blackmon-Luca](https://github.com/charlesbluca) for details.

#### Reduced scheduler CPU load for large graphs

Users should see reduced CPU load on their scheduler when computing
large task graphs.

See [distributed#8238](https://github.com/dask/distributed/pull/8238) and [dask#10547](https://github.com/dask/dask/pull/10547) from [Florian Jetter](https://github.com/fjetter) and
[distributed#8240](https://github.com/dask/distributed/pull/8240) from [crusaderky](https://github.com/crusaderky) for details.

### Additional changes

- Dispatch the `partd.Encode` class used for disk-based shuffling ([dask#10552](https://github.com/dask/dask/pull/10552)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add documentation for hive partitioning ([dask#10454](https://github.com/dask/dask/pull/10454)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add typing to `dask.order` ([dask#10553](https://github.com/dask/dask/pull/10553)) [Florian Jetter](https://github.com/fjetter)
- Allow passing `index_col=False` in `dd.read_csv` ([dask#9961](https://github.com/dask/dask/pull/9961)) [Michael Leslie](https://github.com/michaeldleslie)
- Tighten `HighLevelGraph` annotations ([dask#10524](https://github.com/dask/dask/pull/10524)) [crusaderky](https://github.com/crusaderky)
- Support for latest `ipykernel`/`ipywidgets` ([distributed#8253](https://github.com/dask/distributed/pull/8253)) [crusaderky](https://github.com/crusaderky)
- Check minimal `pyarrow` version for P2P merge ([distributed#8266](https://github.com/dask/distributed/pull/8266)) [Hendrik Makait](https://github.com/hendrikmakait)
- Support for Python 3.12 ([distributed#8223](https://github.com/dask/distributed/pull/8223)) [Thomas Grainger](https://github.com/graingert)
- Use `memoryview.nbytes` when warning on large graph send ([distributed#8268](https://github.com/dask/distributed/pull/8268)) [crusaderky](https://github.com/crusaderky)
- Run tests without `gilknocker` ([distributed#8263](https://github.com/dask/distributed/pull/8263)) [crusaderky](https://github.com/crusaderky)
- Disable ipv6 on MacOS CI ([distributed#8254](https://github.com/dask/distributed/pull/8254)) [crusaderky](https://github.com/crusaderky)
- Clean up redundant minimum versions ([distributed#8251](https://github.com/dask/distributed/pull/8251)) [crusaderky](https://github.com/crusaderky)
- Clean up use of `BARRIER_PREFIX` in scheduler plugin ([distributed#8252](https://github.com/dask/distributed/pull/8252)) [crusaderky](https://github.com/crusaderky)
- Improve shuffle run handling in P2P’s worker plugin ([distributed#8245](https://github.com/dask/distributed/pull/8245)) [Hendrik Makait](https://github.com/hendrikmakait)
- Explicitly set `charset=utf-8` ([distributed#8250](https://github.com/dask/distributed/pull/8250)) [crusaderky](https://github.com/crusaderky)
- Typing tweaks to [distributed#8239](https://github.com/dask/distributed/pull/8239) ([distributed#8247](https://github.com/dask/distributed/pull/8247)) [crusaderky](https://github.com/crusaderky)
- Simplify scheduler assertion ([distributed#8246](https://github.com/dask/distributed/pull/8246)) [crusaderky](https://github.com/crusaderky)
- Improve typing ([distributed#8239](https://github.com/dask/distributed/pull/8239)) [Hendrik Makait](https://github.com/hendrikmakait)
- Respect cgroups v2 “low” memory limit ([distributed#8243](https://github.com/dask/distributed/pull/8243)) [Samantha Hughes](https://github.com/shughes-uk)
- Fix `PackageInstall` by making it a scheduler plugin ([distributed#8142](https://github.com/dask/distributed/pull/8142)) [Hendrik Makait](https://github.com/hendrikmakait)
- Xfail `test_ucx_config_w_env_var` ([distributed#8241](https://github.com/dask/distributed/pull/8241)) [crusaderky](https://github.com/crusaderky)
- `SpecCluster` resilience to broken workers ([distributed#8233](https://github.com/dask/distributed/pull/8233)) [crusaderky](https://github.com/crusaderky)
- Suppress `SpillBuffer` stack traces for cancelled tasks ([distributed#8232](https://github.com/dask/distributed/pull/8232)) [crusaderky](https://github.com/crusaderky)
- Update annotations after stringification changes ([distributed#8195](https://github.com/dask/distributed/pull/8195)) [crusaderky](https://github.com/crusaderky)
- Reduce max recursion depth of profile ([distributed#8224](https://github.com/dask/distributed/pull/8224)) [crusaderky](https://github.com/crusaderky)
- Offload deeply nested objects ([distributed#8214](https://github.com/dask/distributed/pull/8214)) [crusaderky](https://github.com/crusaderky)
- Fix flaky `test_close_connections` ([distributed#8231](https://github.com/dask/distributed/pull/8231)) [crusaderky](https://github.com/crusaderky)
- Fix flaky `test_popen_timeout` ([distributed#8229](https://github.com/dask/distributed/pull/8229)) [crusaderky](https://github.com/crusaderky)
- Fix flaky `test_adapt_then_manual` ([distributed#8228](https://github.com/dask/distributed/pull/8228)) [crusaderky](https://github.com/crusaderky)
- Prevent collisions in `SpillBuffer` ([distributed#8226](https://github.com/dask/distributed/pull/8226)) [crusaderky](https://github.com/crusaderky)
- Allow `retire_workers` to run concurrently ([distributed#8056](https://github.com/dask/distributed/pull/8056)) [Florian Jetter](https://github.com/fjetter)
- Fix HTML repr for `TaskState` objects ([distributed#8188](https://github.com/dask/distributed/pull/8188)) [Florian Jetter](https://github.com/fjetter)
- Fix `AttributeError` for `builtin_function_or_method` in `profile.py` ([distributed#8181](https://github.com/dask/distributed/pull/8181)) [Florian Jetter](https://github.com/fjetter)
- Fix flaky `test_spans` (v2) ([distributed#8222](https://github.com/dask/distributed/pull/8222)) [crusaderky](https://github.com/crusaderky)

<a id="v2023-9-3"></a>

## 2023.9.3

Released on September 29, 2023

### Highlights

#### Restore previous configuration override behavior

The 2023.9.2 release introduced an unintentional breaking change in
how configuration options are overriden in `dask.config.get` with
the `override_with=` keyword (see [dask#10519](https://github.com/dask/dask/issues/10519)).
This release restores the previous behavior.

See [dask#10521](https://github.com/dask/dask/pull/10521) from [crusaderky](https://github.com/crusaderky) for details.

#### Complex dtypes in Dask Array reductions

This release includes improved support for using common reductions
in Dask Array (e.g. `var`, `std`, `moment`) with complex dtypes.

See [dask#10009](https://github.com/dask/dask/pull/10009) from [wkrasnicki](https://github.com/wkrasnicki) for details.

### Additional changes

- Bump `actions/checkout` from 4.0.0 to 4.1.0 ([dask#10532](https://github.com/dask/dask/pull/10532))
- Match `pandas` reverting `apply` deprecation ([dask#10531](https://github.com/dask/dask/pull/10531)) [James Bourbeau](https://github.com/jrbourbeau)
- Update gpuCI `RAPIDS_VER` to `23.12` ([dask#10526](https://github.com/dask/dask/pull/10526))
- Temporarily skip failing tests with `fsspec==2023.9.1` ([dask#10520](https://github.com/dask/dask/pull/10520)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2023-9-2"></a>

## 2023.9.2

Released on September 15, 2023

### Highlights

#### P2P shuffling now raises when outdated PyArrow is installed

Previously the default shuffling method would silently fallback from P2P
to task-based shuffling if an older version of `pyarrow` was installed.
Now we raise an informative error with the minimum required `pyarrow`
version for P2P instead of silently falling back.

See [dask#10496](https://github.com/dask/dask/pull/10496) from [Hendrik Makait](https://github.com/hendrikmakait) for details.

#### Deprecation cycle for admin.traceback.shorten

The 2023.9.0 release modified the `admin.traceback.shorten` configuration option
without introducing a deprecation cycle. This resulted in failures to create Dask
clusters in some cases. This release introduces a deprecation cycle for this configuration
change.

See [dask#10509](https://github.com/dask/dask/pull/10509) from [crusaderky](https://github.com/crusaderky) for details.

### Additional changes

- Avoid materializing all iterators in `delayed` tasks ([dask#10498](https://github.com/dask/dask/pull/10498)) [James Bourbeau](https://github.com/jrbourbeau)
- Overhaul deprecations system in `dask.config` ([dask#10499](https://github.com/dask/dask/pull/10499)) [crusaderky](https://github.com/crusaderky)
- Remove unnecessary check in `timeseries` ([dask#10447](https://github.com/dask/dask/pull/10447)) [Patrick Hoefler](https://github.com/phofl)
- Use `register_plugin` in tests ([dask#10503](https://github.com/dask/dask/pull/10503)) [James Bourbeau](https://github.com/jrbourbeau)
- Make `preserve_index` explicit in `pyarrow_schema_dispatch` ([dask#10501](https://github.com/dask/dask/pull/10501)) [Hendrik Makait](https://github.com/hendrikmakait)
- Add `**kwargs` support for `pyarrow_schema_dispatch` ([dask#10500](https://github.com/dask/dask/pull/10500)) [Hendrik Makait](https://github.com/hendrikmakait)
- Centralize and type `no_default` ([dask#10495](https://github.com/dask/dask/pull/10495)) [crusaderky](https://github.com/crusaderky)

<a id="v2023-9-1"></a>

## 2023.9.1

Released on September 6, 2023

#### NOTE
This is a hotfix release that fixes a P2P shuffling bug introduced in the 2023.9.0
release (see [dask#10493](https://github.com/dask/dask/pull/10493)).

### Enhancements

- Stricter data type for dask keys ([dask#10485](https://github.com/dask/dask/pull/10485)) [crusaderky](https://github.com/crusaderky)
- Special handling for `None` in `DASK_` environment variables ([dask#10487](https://github.com/dask/dask/pull/10487)) [crusaderky](https://github.com/crusaderky)

### Bug Fixes

- Fix `_partitions` `dtype` in `meta` for `DataFrame.set_index` and `DataFrame.sort_values` ([dask#10493](https://github.com/dask/dask/pull/10493)) [Hendrik Makait](https://github.com/hendrikmakait)
- Handle `cached_property` decorators in `derived_from` ([dask#10490](https://github.com/dask/dask/pull/10490)) [Lawrence Mitchell](https://github.com/wence-)

### Maintenance

- Bump `actions/checkout` from 3.6.0 to 4.0.0 ([dask#10492](https://github.com/dask/dask/pull/10492))
- Simplify some tests that `import distributed` ([dask#10484](https://github.com/dask/dask/pull/10484)) [crusaderky](https://github.com/crusaderky)

<a id="v2023-9-0"></a>

## 2023.9.0

Released on September 1, 2023

### Bug Fixes

- Remove support for `np.int64` in keys ([dask#10483](https://github.com/dask/dask/pull/10483)) [crusaderky](https://github.com/crusaderky)
- Fix `_partitions` `dtype` in `meta` for shuffling ([dask#10462](https://github.com/dask/dask/pull/10462)) [Hendrik Makait](https://github.com/hendrikmakait)
- Don’t use exception hooks to shorten tracebacks ([dask#10456](https://github.com/dask/dask/pull/10456)) [crusaderky](https://github.com/crusaderky)

### Documentation

- Add `p2p` shuffle option to DataFrame docs ([dask#10477](https://github.com/dask/dask/pull/10477)) [Patrick Hoefler](https://github.com/phofl)

### Maintenance

- Skip failing tests for `pandas=2.1.0` ([dask#10488](https://github.com/dask/dask/pull/10488)) [Patrick Hoefler](https://github.com/phofl)
- Update tests for `pandas=2.1.0` ([dask#10439](https://github.com/dask/dask/pull/10439)) [Patrick Hoefler](https://github.com/phofl)
- Enable `pytest-timeout` ([dask#10482](https://github.com/dask/dask/pull/10482)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/checkout` from 3.5.3 to 3.6.0 ([dask#10470](https://github.com/dask/dask/pull/10470))

<a id="v2023-8-1"></a>

## 2023.8.1

Released on August 18, 2023

### Enhancements

- Adding support for cgroup v2 to `cpu_count` ([dask#10419](https://github.com/dask/dask/pull/10419)) [Johan Olsson](https://github.com/johanols)
- Support multi-column `groupby` with `sort=True` and `split_out>1` ([dask#10425](https://github.com/dask/dask/pull/10425)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `DataFrame.enforce_runtime_divisions` method ([dask#10404](https://github.com/dask/dask/pull/10404)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Enable file `mode="x"` with a `single_file=True` for Dask DataFrame `to_csv` ([dask#10443](https://github.com/dask/dask/pull/10443)) [Genevieve Buckley](https://github.com/GenevieveBuckley)

### Bug Fixes

- Fix `ValueError` when running `to_csv` in append mode with `single_file` as `True` ([dask#10441](https://github.com/dask/dask/pull/10441)) [Ben](https://github.com/benjaminhduncan)

### Maintenance

- Add default `types_mapper` to `from_pyarrow_table_dispatch` for `pandas` ([dask#10446](https://github.com/dask/dask/pull/10446)) [Richard (Rick) Zamora](https://github.com/rjzamora)

<a id="v2023-8-0"></a>

## 2023.8.0

Released on August 4, 2023

### Enhancements

- Fix for `make_timeseries` performance regression ([dask#10428](https://github.com/dask/dask/pull/10428)) [Irina Truong](https://github.com/j-bennet)

### Documentation

- Add `distributed.print` to debugging docs ([dask#10435](https://github.com/dask/dask/pull/10435)) [James Bourbeau](https://github.com/jrbourbeau)
- Documenting compatibility of NumPy functions with Dask functions ([dask#9941](https://github.com/dask/dask/pull/9941)) [Chiara Marmo](https://github.com/cmarmo)

### Maintenance

- Use SPDX in `license` metadata ([dask#10437](https://github.com/dask/dask/pull/10437)) [John A Kirkham](https://github.com/jakirkham)
- Require `dask[array]` in `dask[dataframe]` ([dask#10357](https://github.com/dask/dask/pull/10357)) [John A Kirkham](https://github.com/jakirkham)
- Update gpuCI `RAPIDS_VER` to `23.10` ([dask#10427](https://github.com/dask/dask/pull/10427))
- Simplify compatibility code ([dask#10426](https://github.com/dask/dask/pull/10426)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix compatibility variable naming ([dask#10424](https://github.com/dask/dask/pull/10424)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix a few errors with upstream `pandas` and `pyarrow` ([dask#10412](https://github.com/dask/dask/pull/10412)) [Irina Truong](https://github.com/j-bennet)

<a id="v2023-7-1"></a>

## 2023.7.1

Released on July 20, 2023

#### NOTE
This release updates Dask DataFrame to automatically convert
text data using `object` data types to `string[pyarrow]`
if `pandas>=2` and `pyarrow>=12` are installed.

This should result in significantly reduced
memory consumption and increased computation performance in many
workflows that deal with text data.

You can disable this change by setting the `dataframe.convert-string`
configuration value to `False` with

```python
dask.config.set({"dataframe.convert-string": False})
```

### Enhancements

- Convert to `pyarrow` strings if proper dependencies are installed ([dask#10400](https://github.com/dask/dask/pull/10400)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid `repartition` before `shuffle` for `p2p` ([dask#10421](https://github.com/dask/dask/pull/10421)) [Patrick Hoefler](https://github.com/phofl)
- API to generate random Dask DataFrames ([dask#10392](https://github.com/dask/dask/pull/10392)) [Irina Truong](https://github.com/j-bennet)
- Speed up `dask.bag.Bag.random_sample` ([dask#10356](https://github.com/dask/dask/pull/10356)) [crusaderky](https://github.com/crusaderky)
- Raise helpful `ValueError` for invalid time units ([dask#10408](https://github.com/dask/dask/pull/10408)) [Nat Tabris](https://github.com/ntabris)
- Make `repartition` a no-op when divisions match (divisions provided as a list) ([dask#10395](https://github.com/dask/dask/pull/10395)) [Nicolas Grandemange](https://github.com/epizut)

### Bug Fixes

- Use `dataframe.convert-string` in `read_parquet` token ([dask#10411](https://github.com/dask/dask/pull/10411)) [James Bourbeau](https://github.com/jrbourbeau)
- Category `dtype` is lost when concatenating `MultiIndex` ([dask#10407](https://github.com/dask/dask/pull/10407)) [Irina Truong](https://github.com/j-bennet)
- Fix `FutureWarning: The provided callable...` ([dask#10405](https://github.com/dask/dask/pull/10405)) [Irina Truong](https://github.com/j-bennet)
- Enable non-categorical hive-partition columns in `read_parquet` ([dask#10353](https://github.com/dask/dask/pull/10353)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `concat` ignoring `DataFrame` withouth columns ([dask#10359](https://github.com/dask/dask/pull/10359)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2023-7-0"></a>

## 2023.7.0

Released on July 7, 2023

### Enhancements

- Catch exceptions when attempting to load CLI entry points ([dask#10380](https://github.com/dask/dask/pull/10380)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

### Bug Fixes

- Fix typo in `_clean_ipython_traceback` ([dask#10385](https://github.com/dask/dask/pull/10385)) [Alexander Clausen](https://github.com/sk1p)
- Ensure that `df` is immutable after `from_pandas` ([dask#10383](https://github.com/dask/dask/pull/10383)) [Patrick Hoefler](https://github.com/phofl)
- Warn consistently for `inplace` in `Series.rename` ([dask#10313](https://github.com/dask/dask/pull/10313)) [Patrick Hoefler](https://github.com/phofl)

### Documentation

- Add clarification about output shape and reshaping in rechunk documentation ([dask#10377](https://github.com/dask/dask/pull/10377)) [Swayam Patil](https://github.com/Swish78)

### Maintenance

- Simplify `astype` implementation ([dask#10393](https://github.com/dask/dask/pull/10393)) [Patrick Hoefler](https://github.com/phofl)
- Fix `test_first_and_last` to accommodate deprecated `last` ([dask#10373](https://github.com/dask/dask/pull/10373)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `level` to `create_merge_tree` ([dask#10391](https://github.com/dask/dask/pull/10391)) [Patrick Hoefler](https://github.com/phofl)
- Do not derive from `scipy.stats.chisquare` docstring ([dask#10382](https://github.com/dask/dask/pull/10382)) [Doug Davis](https://github.com/douglasdavis)

<a id="v2023-6-1"></a>

## 2023.6.1

Released on June 26, 2023

### Enhancements

- Remove no longer supported `clip_lower` and `clip_upper` ([dask#10371](https://github.com/dask/dask/pull/10371)) [Patrick Hoefler](https://github.com/phofl)
- Support `DataFrame.set_index(..., sort=False)` ([dask#10342](https://github.com/dask/dask/pull/10342)) [Miles](https://github.com/milesgranger)
- Cleanup remote tracebacks ([dask#10354](https://github.com/dask/dask/pull/10354)) [Irina Truong](https://github.com/j-bennet)
- Add dispatching mechanisms for `pyarrow.Table` conversion ([dask#10312](https://github.com/dask/dask/pull/10312)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Choose P2P even if fusion is enabled ([dask#10344](https://github.com/dask/dask/pull/10344)) [Hendrik Makait](https://github.com/hendrikmakait)
- Validate that rechunking is possible earlier in graph generation ([dask#10336](https://github.com/dask/dask/pull/10336)) [Hendrik Makait](https://github.com/hendrikmakait)

### Bug Fixes

- Fix issue with `header` passed to `read_csv` ([dask#10355](https://github.com/dask/dask/pull/10355)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Respect `dropna` and `observed` in `GroupBy.var` and `GroupBy.std` ([dask#10350](https://github.com/dask/dask/pull/10350)) [Patrick Hoefler](https://github.com/phofl)
- Fix `H5FD_lock` error when writing to hdf with distributed client ([dask#10309](https://github.com/dask/dask/pull/10309)) [Irina Truong](https://github.com/j-bennet)
- Fix for `total_mem_usage` of `bag.map()` ([dask#10341](https://github.com/dask/dask/pull/10341)) [Irina Truong](https://github.com/j-bennet)

### Deprecations

- Deprecate `DataFrame.fillna`/`Series.fillna` with `method` ([dask#10349](https://github.com/dask/dask/pull/10349)) [Irina Truong](https://github.com/j-bennet)
- Deprecate `DataFrame.first` and `Series.first` ([dask#10352](https://github.com/dask/dask/pull/10352)) [Irina Truong](https://github.com/j-bennet)

### Maintenance

- Deprecate `numpy.compat` ([dask#10370](https://github.com/dask/dask/pull/10370)) [Irina Truong](https://github.com/j-bennet)
- Fix annotations and spans leaking between threads ([dask#10367](https://github.com/dask/dask/pull/10367)) [Irina Truong](https://github.com/j-bennet)
- Use general kwargs in `pyarrow_table_dispatch` functions ([dask#10364](https://github.com/dask/dask/pull/10364)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Remove unnecessary `try`/`except` in `isna` ([dask#10363](https://github.com/dask/dask/pull/10363)) [Patrick Hoefler](https://github.com/phofl)
- `mypy` support for numpy 1.25 ([dask#10362](https://github.com/dask/dask/pull/10362)) [crusaderky](https://github.com/crusaderky)
- Bump `actions/checkout` from 3.5.2 to 3.5.3 ([dask#10348](https://github.com/dask/dask/pull/10348))
- Restore `numba` in `upstream` build ([dask#10330](https://github.com/dask/dask/pull/10330)) [James Bourbeau](https://github.com/jrbourbeau)
- Update nightly wheel index for `pandas`/`numpy`/`scipy` ([dask#10346](https://github.com/dask/dask/pull/10346)) [Matthew Roeschke](https://github.com/mroeschke)
- Add rechunk config values to yaml ([dask#10343](https://github.com/dask/dask/pull/10343)) [Hendrik Makait](https://github.com/hendrikmakait)

<a id="v2023-6-0"></a>

## 2023.6.0

Released on June 9, 2023

### Enhancements

- Add missing `not in` predicate support to `read_parquet` ([dask#10320](https://github.com/dask/dask/pull/10320)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Bug Fixes

- Fix for incorrect `value_counts` ([dask#10323](https://github.com/dask/dask/pull/10323)) [Irina Truong](https://github.com/j-bennet)
- Update empty `describe` top and freq values ([dask#10319](https://github.com/dask/dask/pull/10319)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Fix hetzner typo ([dask#10332](https://github.com/dask/dask/pull/10332)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Maintenance

- Test with `numba` and `sparse` on Python 3.11 ([dask#10329](https://github.com/dask/dask/pull/10329)) [Thomas Grainger](https://github.com/graingert)
- Remove `numpy.find_common_type` warning ignore ([dask#10311](https://github.com/dask/dask/pull/10311)) [James Bourbeau](https://github.com/jrbourbeau)
- Update gpuCI `RAPIDS_VER` to `23.08` ([dask#10310](https://github.com/dask/dask/pull/10310))

<a id="v2023-5-1"></a>

## 2023.5.1

Released on May 26, 2023

#### NOTE
This release drops support for Python 3.8. As of this release
Dask supports Python 3.9, 3.10, and 3.11.
See [this community issue](https://github.com/dask/community/issues/315)
for more details.

### Enhancements

- Drop Python 3.8 support ([dask#10295](https://github.com/dask/dask/pull/10295)) [Thomas Grainger](https://github.com/graingert)
- Change Dask Bag partitioning scheme to improve cluster saturation ([dask#10294](https://github.com/dask/dask/pull/10294)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Generalize `dd.to_datetime` for GPU-backed collections, introduce `get_meta_library` utility ([dask#9881](https://github.com/dask/dask/pull/9881)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add `na_action` to `DataFrame.map` ([dask#10305](https://github.com/dask/dask/pull/10305)) [Patrick Hoefler](https://github.com/phofl)
- Raise `TypeError` in `DataFrame.nsmallest` and `DataFrame.nlargest` when `columns` is not given ([dask#10301](https://github.com/dask/dask/pull/10301)) [Patrick Hoefler](https://github.com/phofl)
- Improve `sizeof` for `pd.MultiIndex` ([dask#10230](https://github.com/dask/dask/pull/10230)) [Patrick Hoefler](https://github.com/phofl)
- Support duplicated columns in a bunch of `DataFrame` methods ([dask#10261](https://github.com/dask/dask/pull/10261)) [Patrick Hoefler](https://github.com/phofl)
- Add `numeric_only` support to `DataFrame.idxmin` and `DataFrame.idxmax` ([dask#10253](https://github.com/dask/dask/pull/10253)) [Patrick Hoefler](https://github.com/phofl)
- Implement `numeric_only` support for `DataFrame.quantile` ([dask#10259](https://github.com/dask/dask/pull/10259)) [Patrick Hoefler](https://github.com/phofl)
- Add support for `numeric_only=False` in `DataFrame.std` ([dask#10251](https://github.com/dask/dask/pull/10251)) [Patrick Hoefler](https://github.com/phofl)
- Implement `numeric_only=False` for `GroupBy.cumprod` and `GroupBy.cumsum` ([dask#10262](https://github.com/dask/dask/pull/10262)) [Patrick Hoefler](https://github.com/phofl)
- Implement `numeric_only` for `skew` and `kurtosis` ([dask#10258](https://github.com/dask/dask/pull/10258)) [Patrick Hoefler](https://github.com/phofl)
- `mask` and `where` should accept a `callable` ([dask#10289](https://github.com/dask/dask/pull/10289)) [Irina Truong](https://github.com/j-bennet)
- Fix conversion from `Categorical` to `pa.dictionary` in `read_parquet` ([dask#10285](https://github.com/dask/dask/pull/10285)) [Patrick Hoefler](https://github.com/phofl)

### Bug Fixes

- Spurious config on nested annotations ([dask#10318](https://github.com/dask/dask/pull/10318)) [crusaderky](https://github.com/crusaderky)
- Fix rechunking behavior for dimensions with known and unknown chunk sizes ([dask#10157](https://github.com/dask/dask/pull/10157)) [Hendrik Makait](https://github.com/hendrikmakait)
- Enable `drop` to support mismatched partitions ([dask#10300](https://github.com/dask/dask/pull/10300)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `divisions` construction for `to_timestamp` ([dask#10304](https://github.com/dask/dask/pull/10304)) [Patrick Hoefler](https://github.com/phofl)
- pandas `ExtensionDtype` raising in `Series` reduction operations ([dask#10149](https://github.com/dask/dask/pull/10149)) [Patrick Hoefler](https://github.com/phofl)
- Fix regression in `da.random` interface ([dask#10247](https://github.com/dask/dask/pull/10247)) [Eray Aslan](https://github.com/erayaslan)
- `da.coarsen` doesn’t trim an empty chunk in meta ([dask#10281](https://github.com/dask/dask/pull/10281)) [Irina Truong](https://github.com/j-bennet)
- Fix dtype inference for `engine="pyarrow"` in `read_csv` ([dask#10280](https://github.com/dask/dask/pull/10280)) [Patrick Hoefler](https://github.com/phofl)

### Documentation

- Add `meta_from_array` to API docs ([dask#10306](https://github.com/dask/dask/pull/10306)) [Ruth Comer](https://github.com/rcomer)
- Update Coiled links ([dask#10296](https://github.com/dask/dask/pull/10296)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add docs for demo day ([dask#10288](https://github.com/dask/dask/pull/10288)) [Matthew Rocklin](https://github.com/mrocklin)

### Maintenance

- Explicitly install `anaconda-client` from conda-forge when uploading conda nightlies ([dask#10316](https://github.com/dask/dask/pull/10316)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Configure `isort` to add `from __future__ import annotations` ([dask#10314](https://github.com/dask/dask/pull/10314)) [Thomas Grainger](https://github.com/graingert)
- Avoid `pandas` `Series.__getitem__` deprecation in tests ([dask#10308](https://github.com/dask/dask/pull/10308)) [James Bourbeau](https://github.com/jrbourbeau)
- Ignore `numpy.find_common_type` warning from `pandas` ([dask#10307](https://github.com/dask/dask/pull/10307)) [James Bourbeau](https://github.com/jrbourbeau)
- Add test to check that `DataFrame.__setitem__` does not modify `df` inplace ([dask#10223](https://github.com/dask/dask/pull/10223)) [Patrick Hoefler](https://github.com/phofl)
- Clean up default value of `dropna` in `value_counts` ([dask#10299](https://github.com/dask/dask/pull/10299)) [Patrick Hoefler](https://github.com/phofl)
- Add `pytest-cov` to `test` extra ([dask#10271](https://github.com/dask/dask/pull/10271)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2023-5-0"></a>

## 2023.5.0

Released on May 12, 2023

### Enhancements

- Implement `numeric_only=False` for `GroupBy.corr` and `GroupBy.cov` ([dask#10264](https://github.com/dask/dask/pull/10264)) [Patrick Hoefler](https://github.com/phofl)
- Add support for `numeric_only=False` in `DataFrame.var` ([dask#10250](https://github.com/dask/dask/pull/10250)) [Patrick Hoefler](https://github.com/phofl)
- Add `numeric_only` support to `DataFrame.mode` ([dask#10257](https://github.com/dask/dask/pull/10257)) [Patrick Hoefler](https://github.com/phofl)
- Add `DataFrame.map` to `dask.DataFrame` API ([dask#10246](https://github.com/dask/dask/pull/10246)) [Patrick Hoefler](https://github.com/phofl)
- Adjust for `DataFrame.applymap` deprecation and all `NA` `concat` behaviour change ([dask#10245](https://github.com/dask/dask/pull/10245)) [Patrick Hoefler](https://github.com/phofl)
- Enable `numeric_only=False` for `DataFrame.count` ([dask#10234](https://github.com/dask/dask/pull/10234)) [Patrick Hoefler](https://github.com/phofl)
- Disallow array input in mask/where ([dask#10163](https://github.com/dask/dask/pull/10163)) [Irina Truong](https://github.com/j-bennet)
- Support `numeric_only=True` in `GroupBy.corr` and `GroupBy.cov` ([dask#10227](https://github.com/dask/dask/pull/10227)) [Patrick Hoefler](https://github.com/phofl)
- Add `numeric_only` support to `GroupBy.median` ([dask#10236](https://github.com/dask/dask/pull/10236)) [Patrick Hoefler](https://github.com/phofl)
- Support `mimesis=9` in `dask.datasets` ([dask#10241](https://github.com/dask/dask/pull/10241)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `numeric_only` support to `min`, `max` and `prod` ([dask#10219](https://github.com/dask/dask/pull/10219)) [Patrick Hoefler](https://github.com/phofl)
- Add `numeric_only=True` support for `GroupBy.cumsum` and `GroupBy.cumprod` ([dask#10224](https://github.com/dask/dask/pull/10224)) [Patrick Hoefler](https://github.com/phofl)
- Add helper to unpack `numeric_only` keyword ([dask#10228](https://github.com/dask/dask/pull/10228)) [Patrick Hoefler](https://github.com/phofl)

### Bug Fixes

- Fix `clone` + `from_array` failure ([dask#10211](https://github.com/dask/dask/pull/10211)) [crusaderky](https://github.com/crusaderky)
- Fix dataframe reductions for ea dtypes ([dask#10150](https://github.com/dask/dask/pull/10150)) [Patrick Hoefler](https://github.com/phofl)
- Avoid scalar conversion deprecation warning in `numpy=1.25` ([dask#10248](https://github.com/dask/dask/pull/10248)) [James Bourbeau](https://github.com/jrbourbeau)
- Make sure transform output has the same index as input ([dask#10184](https://github.com/dask/dask/pull/10184)) [Irina Truong](https://github.com/j-bennet)
- Fix `corr` and `cov` on a single-row partition ([dask#9756](https://github.com/dask/dask/pull/9756)) [Irina Truong](https://github.com/j-bennet)
- Fix `test_groupby_numeric_only_supported` and `test_groupby_aggregate_categorical_observed` upstream errors  ([dask#10243](https://github.com/dask/dask/pull/10243)) [Irina Truong](https://github.com/j-bennet)

### Documentation

- Clean up futures docs ([dask#10266](https://github.com/dask/dask/pull/10266)) [Matthew Rocklin](https://github.com/mrocklin)
- Add `Index` API reference ([dask#10263](https://github.com/dask/dask/pull/10263)) [hotpotato](https://github.com/hotpotato)

### Maintenance

- Warn when meta is passed to `apply` ([dask#10256](https://github.com/dask/dask/pull/10256)) [Patrick Hoefler](https://github.com/phofl)
- Remove `imageio` version restriction in CI ([dask#10260](https://github.com/dask/dask/pull/10260)) [Patrick Hoefler](https://github.com/phofl)
- Remove unused `DataFrame` variance methods ([dask#10252](https://github.com/dask/dask/pull/10252)) [Patrick Hoefler](https://github.com/phofl)
- Un-`xfail` `test_categories` with `pyarrow` strings and `pyarrow>=12` ([dask#10244](https://github.com/dask/dask/pull/10244)) [Irina Truong](https://github.com/j-bennet)
- Bump gpuCI `PYTHON_VER` 3.8->3.9 ([dask#10233](https://github.com/dask/dask/pull/10233)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2023-4-1"></a>

## 2023.4.1

Released on April 28, 2023

### Enhancements

- Implement `numeric_only` support for `DataFrame.sum` ([dask#10194](https://github.com/dask/dask/pull/10194)) [Patrick Hoefler](https://github.com/phofl)
- Add support for `numeric_only=True` in `GroupBy` operations ([dask#10222](https://github.com/dask/dask/pull/10222)) [Patrick Hoefler](https://github.com/phofl)
- Avoid deep copy in `DataFrame.__setitem__` for `pandas` 1.4 and up ([dask#10221](https://github.com/dask/dask/pull/10221)) [Patrick Hoefler](https://github.com/phofl)
- Avoid calling `Series.apply` with `_meta_nonempty` ([dask#10212](https://github.com/dask/dask/pull/10212)) [Patrick Hoefler](https://github.com/phofl)
- Unpin `sqlalchemy` and fix compatibility issues ([dask#10140](https://github.com/dask/dask/pull/10140)) [Patrick Hoefler](https://github.com/phofl)

### Bug Fixes

- Partially revert default client discovery ([dask#10225](https://github.com/dask/dask/pull/10225)) [Florian Jetter](https://github.com/fjetter)
- Support arrow dtypes in `Index` meta creation ([dask#10170](https://github.com/dask/dask/pull/10170)) [Patrick Hoefler](https://github.com/phofl)
- Repartitioning raises with extension dtype when truncating floats ([dask#10169](https://github.com/dask/dask/pull/10169)) [Patrick Hoefler](https://github.com/phofl)
- Adjust empty `Index` from `fastparquet` to `object` dtype ([dask#10179](https://github.com/dask/dask/pull/10179)) [Patrick Hoefler](https://github.com/phofl)

### Documentation

- Update Kubernetes docs ([dask#10232](https://github.com/dask/dask/pull/10232)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add `DataFrame.reduction` to API docs ([dask#10229](https://github.com/dask/dask/pull/10229)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `DataFrame.persist` to docs and fix links ([dask#10231](https://github.com/dask/dask/pull/10231)) [Patrick Hoefler](https://github.com/phofl)
- Add documentation for `GroupBy.transform` ([dask#10185](https://github.com/dask/dask/pull/10185)) [Irina Truong](https://github.com/j-bennet)
- Fix formatting in random number generation docs ([dask#10189](https://github.com/dask/dask/pull/10189)) [Eray Aslan](https://github.com/erayaslan)

### Maintenance

- Pin imageio to `<2.28` ([dask#10216](https://github.com/dask/dask/pull/10216)) [Patrick Hoefler](https://github.com/phofl)
- Add note about `importlib_metadata` backport ([dask#10207](https://github.com/dask/dask/pull/10207)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `xarray` back to Python 3.11 CI builds ([dask#10200](https://github.com/dask/dask/pull/10200)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `mindeps` build with all optional dependencies ([dask#10161](https://github.com/dask/dask/pull/10161)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Provide proper `like` value for `array_safe` in `percentiles_summary` ([dask#10156](https://github.com/dask/dask/pull/10156)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Avoid re-opening hdf file multiple times in `read_hdf` ([dask#10205](https://github.com/dask/dask/pull/10205)) [Thomas Grainger](https://github.com/graingert)
- Add merge tests on nullable columns ([dask#10071](https://github.com/dask/dask/pull/10071)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix coverage configuration ([dask#10203](https://github.com/dask/dask/pull/10203)) [Thomas Grainger](https://github.com/graingert)
- Remove `is_period_dtype` and `is_sparse_dtype` ([dask#10197](https://github.com/dask/dask/pull/10197)) [Patrick Hoefler](https://github.com/phofl)
- Bump `actions/checkout` from 3.5.0 to 3.5.2 ([dask#10201](https://github.com/dask/dask/pull/10201))
- Avoid deprecated `is_categorical_dtype` from pandas ([dask#10180](https://github.com/dask/dask/pull/10180)) [Patrick Hoefler](https://github.com/phofl)
- Adjust for deprecated `is_interval_dtype` and `is_datetime64tz_dtype` ([dask#10188](https://github.com/dask/dask/pull/10188)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2023-4-0"></a>

## 2023.4.0

Released on April 14, 2023

### Enhancements

- Override old default values in `update_defaults` ([dask#10159](https://github.com/dask/dask/pull/10159)) [Gabe Joseph](https://github.com/gjoseph92)
- Add a CLI command to `list` and `get` a value from dask config ([dask#9936](https://github.com/dask/dask/pull/9936)) [Irina Truong](https://github.com/j-bennet)
- Handle string-based engine argument to `read_json` ([dask#9947](https://github.com/dask/dask/pull/9947)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid deprecated `GroupBy.dtypes` ([dask#10111](https://github.com/dask/dask/pull/10111)) [Irina Truong](https://github.com/j-bennet)

### Bug Fixes

- Revert `grouper`-related changes ([dask#10182](https://github.com/dask/dask/pull/10182)) [Irina Truong](https://github.com/j-bennet)
- `GroupBy.cov` raising for non-numeric grouping column ([dask#10171](https://github.com/dask/dask/pull/10171)) [Patrick Hoefler](https://github.com/phofl)
- Updates for `Index` supporting `numpy` numeric dtypes ([dask#10154](https://github.com/dask/dask/pull/10154)) [Irina Truong](https://github.com/j-bennet)
- Preserve `dtype` for partitioning columns when read with `pyarrow` ([dask#10115](https://github.com/dask/dask/pull/10115)) [Patrick Hoefler](https://github.com/phofl)
- Fix annotations for `to_hdf` ([dask#10123](https://github.com/dask/dask/pull/10123)) [Hendrik Makait](https://github.com/hendrikmakait)
- Handle `None` column name when checking if columns are all numeric ([dask#10128](https://github.com/dask/dask/pull/10128)) [Lawrence Mitchell](https://github.com/wence-)
- Fix `valid_divisions` when passed a `tuple` ([dask#10126](https://github.com/dask/dask/pull/10126)) [Brian Phillips](https://github.com/bphillips-exos)
- Maintain annotations in `DataFrame.categorize` ([dask#10120](https://github.com/dask/dask/pull/10120)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix handling of missing min/max parquet statistics during filtering ([dask#10042](https://github.com/dask/dask/pull/10042)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Deprecations

- Deprecate `use_nullable_dtypes=` and add `dtype_backend=` ([dask#10076](https://github.com/dask/dask/pull/10076)) [Irina Truong](https://github.com/j-bennet)
- Deprecate `convert_dtype` in `Series.apply` ([dask#10133](https://github.com/dask/dask/pull/10133)) [Irina Truong](https://github.com/j-bennet)

### Documentation

- Document `Generator` based random number generation ([dask#10134](https://github.com/dask/dask/pull/10134)) [Eray Aslan](https://github.com/erayaslan)

### Maintenance

- Update `dataframe.convert_string` to `dataframe.convert-string` ([dask#10191](https://github.com/dask/dask/pull/10191)) [Irina Truong](https://github.com/j-bennet)
- Add `python-cityhash` to CI environments ([dask#10190](https://github.com/dask/dask/pull/10190)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Temporarily pin `scikit-image` to fix Windows CI ([dask#10186](https://github.com/dask/dask/pull/10186)) [Patrick Hoefler](https://github.com/phofl)
- Handle pandas deprecation warnings for `to_pydatetime` and `apply` ([dask#10168](https://github.com/dask/dask/pull/10168)) [Patrick Hoefler](https://github.com/phofl)
- Drop `bokeh<3` restriction ([dask#10177](https://github.com/dask/dask/pull/10177)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix failing tests under copy-on-write ([dask#10173](https://github.com/dask/dask/pull/10173)) [Patrick Hoefler](https://github.com/phofl)
- Allow `pyarrow` CI to fail ([dask#10176](https://github.com/dask/dask/pull/10176)) [James Bourbeau](https://github.com/jrbourbeau)
- Switch to `Generator` for random number generation in `dask.array` ([dask#10003](https://github.com/dask/dask/pull/10003)) [Eray Aslan](https://github.com/erayaslan)
- Bump `peter-evans/create-pull-request` from 4 to 5 ([dask#10166](https://github.com/dask/dask/pull/10166))
- Fix flaky `modf` operation in `test_arithmetic` ([dask#10162](https://github.com/dask/dask/pull/10162)) [Irina Truong](https://github.com/j-bennet)
- Temporarily remove `xarray` from CI with `pandas` 2.0 ([dask#10153](https://github.com/dask/dask/pull/10153)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `update_graph` counting logic in `test_default_scheduler_on_worker` ([dask#10145](https://github.com/dask/dask/pull/10145)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix documentation build with `pandas` 2.0 ([dask#10138](https://github.com/dask/dask/pull/10138)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `dask/gpu` from gpuCI update reviewers ([dask#10135](https://github.com/dask/dask/pull/10135)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Update gpuCI `RAPIDS_VER` to `23.06` ([dask#10129](https://github.com/dask/dask/pull/10129))
- Bump `actions/stale` from 6 to 8 ([dask#10121](https://github.com/dask/dask/pull/10121))
- Use declarative `setuptools` ([dask#10102](https://github.com/dask/dask/pull/10102)) [Thomas Grainger](https://github.com/graingert)
- Relax `assert_eq` checks on `Scalar`-like objects ([dask#10125](https://github.com/dask/dask/pull/10125)) [Matthew Rocklin](https://github.com/mrocklin)
- Upgrade readthedocs config to ubuntu 22.04 and Python 3.11 ([dask#10124](https://github.com/dask/dask/pull/10124)) [Thomas Grainger](https://github.com/graingert)
- Bump `actions/checkout` from 3.4.0 to 3.5.0 ([dask#10122](https://github.com/dask/dask/pull/10122))
- Fix `test_null_partition_pyarrow` in `pyarrow` CI build ([dask#10116](https://github.com/dask/dask/pull/10116)) [Irina Truong](https://github.com/j-bennet)
- Drop distributed pack ([dask#9988](https://github.com/dask/dask/pull/9988)) [Florian Jetter](https://github.com/fjetter)
- Make `dask.compatibility` private ([dask#10114](https://github.com/dask/dask/pull/10114)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2023-3-2"></a>

## 2023.3.2

Released on March 24, 2023

### Enhancements

- Deprecate `observed=False` for `groupby` with categoricals ([dask#10095](https://github.com/dask/dask/pull/10095)) [Irina Truong](https://github.com/j-bennet)
- Deprecate `axis=` for some groupby operations ([dask#10094](https://github.com/dask/dask/pull/10094)) [James Bourbeau](https://github.com/jrbourbeau)
- The `axis` keyword in `DataFrame.rolling/Series.rolling` is deprecated ([dask#10110](https://github.com/dask/dask/pull/10110)) [Irina Truong](https://github.com/j-bennet)
- `DataFrame._data` deprecation in `pandas` ([dask#10081](https://github.com/dask/dask/pull/10081)) [Irina Truong](https://github.com/j-bennet)
- Use `importlib_metadata` backport to avoid CLI `UserWarning` ([dask#10070](https://github.com/dask/dask/pull/10070)) [Thomas Grainger](https://github.com/graingert)
- Port option parsing logic from `dask.dataframe.read_parquet` to `to_parquet` ([dask#9981](https://github.com/dask/dask/pull/9981)) [Anton Loukianov](https://github.com/antonl)

### Bug Fixes

- Avoid using `dd.shuffle` in groupby-apply ([dask#10043](https://github.com/dask/dask/pull/10043)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Enable null hive partitions with `pyarrow` parquet engine ([dask#10007](https://github.com/dask/dask/pull/10007)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support unknown shapes in `*_like` functions ([dask#10064](https://github.com/dask/dask/pull/10064)) [Doug Davis](https://github.com/douglasdavis)

### Documentation

- Add `to_backend` methods to API docs ([dask#10093](https://github.com/dask/dask/pull/10093)) [Lawrence Mitchell](https://github.com/wence-)
- Remove broken gpuCI link in developer docs ([dask#10065](https://github.com/dask/dask/pull/10065)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

### Maintenance

- Configure readthedocs sphinx warnings as errors  ([dask#10104](https://github.com/dask/dask/pull/10104)) [Thomas Grainger](https://github.com/graingert)
- Un-`xfail` `test_division_or_partition` with `pyarrow` strings active  ([dask#10108](https://github.com/dask/dask/pull/10108)) [Irina Truong](https://github.com/j-bennet)
- Un-`xfail` `test_different_columns_are_allowed` with `pyarrow` strings active ([dask#10109](https://github.com/dask/dask/pull/10109)) [Irina Truong](https://github.com/j-bennet)
- Restore Entrypoints compatibility ([dask#10113](https://github.com/dask/dask/pull/10113)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Un-`xfail` `test_to_dataframe_optimize_graph` with `pyarrow` strings active ([dask#10087](https://github.com/dask/dask/pull/10087)) [Irina Truong](https://github.com/j-bennet)
- Only run `test_development_guidelines_matches_ci` on editable install ([dask#10106](https://github.com/dask/dask/pull/10106)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Un-`xfail` `test_dataframe_cull_key_dependencies_materialized` with `pyarrow` strings active ([dask#10088](https://github.com/dask/dask/pull/10088)) [Irina Truong](https://github.com/j-bennet)
- Install `mimesis` in CI environments ([dask#10105](https://github.com/dask/dask/pull/10105)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix for no module named `ipykernel` ([dask#10101](https://github.com/dask/dask/pull/10101)) [Irina Truong](https://github.com/j-bennet)
- Fix docs builds by installing `ipykernel` ([dask#10103](https://github.com/dask/dask/pull/10103)) [Thomas Grainger](https://github.com/graingert)
- Allow `pyarrow` build to continue on failures ([dask#10097](https://github.com/dask/dask/pull/10097)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump `actions/checkout` from 3.3.0 to 3.4.0 ([dask#10096](https://github.com/dask/dask/pull/10096))
- Fix `test_set_index_on_empty` with `pyarrow` strings active  ([dask#10054](https://github.com/dask/dask/pull/10054)) [Irina Truong](https://github.com/j-bennet)
- Un-`xfail` `pyarrow` pickling tests ([dask#10082](https://github.com/dask/dask/pull/10082)) [James Bourbeau](https://github.com/jrbourbeau)
- CI environment file cleanup ([dask#10078](https://github.com/dask/dask/pull/10078)) [James Bourbeau](https://github.com/jrbourbeau)
- Un-`xfail` more `pyarrow` tests ([dask#10066](https://github.com/dask/dask/pull/10066)) [Irina Truong](https://github.com/j-bennet)
- Temporarily skip `pyarrow_compat` tests with p\`andas 2.0 ([dask#10063](https://github.com/dask/dask/pull/10063)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_melt` with `pyarrow` strings active ([dask#10052](https://github.com/dask/dask/pull/10052)) [Irina Truong](https://github.com/j-bennet)
- Fix `test_str_accessor` with `pyarrow` strings active ([dask#10048](https://github.com/dask/dask/pull/10048)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_better_errors_object_reductions` with `pyarrow` strings active ([dask#10051](https://github.com/dask/dask/pull/10051)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_loc_with_non_boolean_series` with `pyarrow` strings active ([dask#10046](https://github.com/dask/dask/pull/10046)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_values` with `pyarrow` strings active ([dask#10050](https://github.com/dask/dask/pull/10050)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily `xfail` `test_upstream_packages_installed` ([dask#10047](https://github.com/dask/dask/pull/10047)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2023-3-1"></a>

## 2023.3.1

Released on March 10, 2023

### Enhancements

- Support pyarrow strings in `MultiIndex` ([dask#10040](https://github.com/dask/dask/pull/10040)) [Irina Truong](https://github.com/j-bennet)
- Improved support for `pyarrow` strings ([dask#10000](https://github.com/dask/dask/pull/10000)) [Irina Truong](https://github.com/j-bennet)
- Fix flaky `RuntimeWarning` during array reductions ([dask#10030](https://github.com/dask/dask/pull/10030)) [James Bourbeau](https://github.com/jrbourbeau)
- Extend `complete` extras ([dask#10023](https://github.com/dask/dask/pull/10023)) [James Bourbeau](https://github.com/jrbourbeau)
- Raise an error with `dataframe.convert-string=True` and `pandas<2.0` ([dask#10033](https://github.com/dask/dask/pull/10033)) [Irina Truong](https://github.com/j-bennet)
- Rename shuffle/rechunk config option/kwarg to `method` ([dask#10013](https://github.com/dask/dask/pull/10013)) [James Bourbeau](https://github.com/jrbourbeau)
- Add initial support for converting `pandas` extension dtypes to arrays ([dask#10018](https://github.com/dask/dask/pull/10018)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `randomgen` support ([dask#9987](https://github.com/dask/dask/pull/9987)) [Eray Aslan](https://github.com/erayaslan)

### Bug Fixes

- Skip rechunk when rechunking to the same chunks with unknown sizes ([dask#10027](https://github.com/dask/dask/pull/10027)) [Hendrik Makait](https://github.com/hendrikmakait)
- Custom utility to convert parquet filters to `pyarrow` expression ([dask#9885](https://github.com/dask/dask/pull/9885)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Consider `numpy` scalars and 0d arrays as scalars when padding ([dask#9653](https://github.com/dask/dask/pull/9653)) [Justus Magin](https://github.com/keewis)
- Fix parquet overwrite behavior after an adaptive `read_parquet` operation ([dask#10002](https://github.com/dask/dask/pull/10002)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Add and update docs for Data Transfer section ([dask#10022](https://github.com/dask/dask/pull/10022)) [Miles](https://github.com/milesgranger)

### Maintenance

- Remove stale hive-partitioning code from `pyarrow` parquet engine ([dask#10039](https://github.com/dask/dask/pull/10039)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Increase minimum supported `pyarrow` to 7.0 ([dask#10024](https://github.com/dask/dask/pull/10024)) [James Bourbeau](https://github.com/jrbourbeau)
- Revert “Prepare drop packunpack ([dask#9994](https://github.com/dask/dask/pull/9994)) ([dask#10037](https://github.com/dask/dask/pull/10037)) [Florian Jetter](https://github.com/fjetter)
- Have codecov wait for more builds before reporting ([dask#10031](https://github.com/dask/dask/pull/10031)) [James Bourbeau](https://github.com/jrbourbeau)
- Prepare drop packunpack ([dask#9994](https://github.com/dask/dask/pull/9994)) [Florian Jetter](https://github.com/fjetter)
- Add CI job with `pyarrow` strings turned on ([dask#10017](https://github.com/dask/dask/pull/10017)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_groupby_dropna_with_agg` for `pandas` 2.0 ([dask#10001](https://github.com/dask/dask/pull/10001)) [Irina Truong](https://github.com/j-bennet)
- Fix `test_pickle_roundtrip` for `pandas` 2.0 ([dask#10011](https://github.com/dask/dask/pull/10011)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2023-3-0"></a>

## 2023.3.0

Released on March 1, 2023

### Bug Fixes

- Bag must not pick p2p as shuffle default ([dask#10005](https://github.com/dask/dask/pull/10005)) [Florian Jetter](https://github.com/fjetter)

### Documentation

- Minor follow-up to P2P by default ([dask#10008](https://github.com/dask/dask/pull/10008)) [James Bourbeau](https://github.com/jrbourbeau)

### Maintenance

- Add minimum version to optional `jinja2` dependency ([dask#9999](https://github.com/dask/dask/pull/9999)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2023-2-1"></a>

## 2023.2.1

Released on February 24, 2023

#### NOTE
This release changes the default DataFrame shuffle algorithm to `p2p`
to improve stability and performance. [Learn more here](https://blog.coiled.io/blog/shuffling-large-data-at-constant-memory.html?utm_source=dask-docs&utm_medium=changelog)
and please provide any feedback [on this discussion](https://github.com/dask/distributed/discussions/7509).

If you encounter issues with this new algorithm, please see the [documentation](dataframe-groupby.md#shuffle-methods)
for more information, and how to switch back to the old mode.

### Enhancements

- Enable P2P shuffling by default ([dask#9991](https://github.com/dask/dask/pull/9991)) [Florian Jetter](https://github.com/fjetter)
- P2P rechunking ([dask#9939](https://github.com/dask/dask/pull/9939)) [Hendrik Makait](https://github.com/hendrikmakait)
- Efficient dataframe.convert-string support for read_parquet ([dask#9979](https://github.com/dask/dask/pull/9979)) [Irina Truong](https://github.com/j-bennet)
- Allow p2p shuffle kwarg for DataFrame merges ([dask#9900](https://github.com/dask/dask/pull/9900)) [Florian Jetter](https://github.com/fjetter)
- Change `split_row_groups` default to “infer” ([dask#9637](https://github.com/dask/dask/pull/9637)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add option for converting string data to use `pyarrow` strings ([dask#9926](https://github.com/dask/dask/pull/9926)) [James Bourbeau](https://github.com/jrbourbeau)
- Add support for multi-column `sort_values` ([dask#8263](https://github.com/dask/dask/pull/8263)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- `Generator` based random-number generation in\`\`dask.array\`\` ([dask#9038](https://github.com/dask/dask/pull/9038)) [Eray Aslan](https://github.com/erayaslan)
- Support `numeric_only` for simple groupby aggregations for `pandas` 2.0 compatibility ([dask#9889](https://github.com/dask/dask/pull/9889)) [Irina Truong](https://github.com/j-bennet)

### Bug Fixes

- Fix profilers plot not being aligned to context manager enter time ([dask#9739](https://github.com/dask/dask/pull/9739)) [David Hoese](https://github.com/djhoese)
- Relax dask.dataframe assert_eq type checks ([dask#9989](https://github.com/dask/dask/pull/9989)) [Matthew Rocklin](https://github.com/mrocklin)
- Restore `describe` compatibility for `pandas` 2.0 ([dask#9982](https://github.com/dask/dask/pull/9982)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Improving deploying Dask docs ([dask#9912](https://github.com/dask/dask/pull/9912)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- More docs for `DataFrame.partitions` ([dask#9976](https://github.com/dask/dask/pull/9976)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update docs with more information on default Delayed scheduler ([dask#9903](https://github.com/dask/dask/pull/9903)) [Guillaume Eynard-Bontemps](https://github.com/guillaumeeb)
- Deployment Considerations documentation ([dask#9933](https://github.com/dask/dask/pull/9933)) [Gabe Joseph](https://github.com/gjoseph92)

### Maintenance

- Temporarily rerun flaky tests ([dask#9983](https://github.com/dask/dask/pull/9983)) [James Bourbeau](https://github.com/jrbourbeau)
- Update parsing of FULL_RAPIDS_VER/FULL_UCX_PY_VER ([dask#9990](https://github.com/dask/dask/pull/9990)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Increase minimum supported versions to `pandas=1.3` and `numpy=1.21` ([dask#9950](https://github.com/dask/dask/pull/9950)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `std` to work with `numeric_only` for `pandas` 2.0 ([dask#9960](https://github.com/dask/dask/pull/9960)) [Irina Truong](https://github.com/j-bennet)
- Temporarily `xfail` `test_roundtrip_partitioned_pyarrow_dataset` ([dask#9977](https://github.com/dask/dask/pull/9977)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix copy on write failure in test_idxmaxmin ([dask#9944](https://github.com/dask/dask/pull/9944)) [Patrick Hoefler](https://github.com/phofl)
- Bump `pre-commit` versions ([dask#9955](https://github.com/dask/dask/pull/9955)) [crusaderky](https://github.com/crusaderky)
- Fix `test_groupby_unaligned_index` for `pandas` 2.0 ([dask#9963](https://github.com/dask/dask/pull/9963)) [Irina Truong](https://github.com/j-bennet)
- Un-`xfail` `test_set_index_overlap_2` for `pandas` 2.0 ([dask#9959](https://github.com/dask/dask/pull/9959)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_merge_by_index_patterns` for `pandas` 2.0 ([dask#9930](https://github.com/dask/dask/pull/9930)) [Irina Truong](https://github.com/j-bennet)
- Bump jacobtomlinson/gha-find-replace from 2 to 3 ([dask#9953](https://github.com/dask/dask/pull/9953)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `test_rolling_agg_aggregate` for `pandas` 2.0 compatibility ([dask#9948](https://github.com/dask/dask/pull/9948)) [Irina Truong](https://github.com/j-bennet)
- Bump `black` to `23.1.0` ([dask#9956](https://github.com/dask/dask/pull/9956)) [crusaderky](https://github.com/crusaderky)
- Run GPU tests on python 3.8 & 3.10 ([dask#9940](https://github.com/dask/dask/pull/9940)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix `test_to_timestamp` for `pandas` 2.0 ([dask#9932](https://github.com/dask/dask/pull/9932)) [Irina Truong](https://github.com/j-bennet)
- Fix an error with `groupby` `value_counts` for `pandas` 2.0 compatibility ([dask#9928](https://github.com/dask/dask/pull/9928)) [Irina Truong](https://github.com/j-bennet)
- Config converter: replace all dashes with underscores ([dask#9945](https://github.com/dask/dask/pull/9945)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- CI: use nightly wheel to install pyarrow in upstream test build ([dask#9873](https://github.com/dask/dask/pull/9873)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)

<a id="v2023-2-0"></a>

## 2023.2.0

Released on February 10, 2023

### Enhancements

- Update `numeric_only` default in `quantile` for `pandas` 2.0 ([dask#9854](https://github.com/dask/dask/pull/9854)) [Irina Truong](https://github.com/j-bennet)
- Make `repartition` a no-op when divisions match ([dask#9924](https://github.com/dask/dask/pull/9924)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `datetime_is_numeric` behavior in `describe` for `pandas` 2.0 ([dask#9868](https://github.com/dask/dask/pull/9868)) [Irina Truong](https://github.com/j-bennet)
- Update `value_counts` to return correct name in `pandas` 2.0 ([dask#9919](https://github.com/dask/dask/pull/9919)) [Irina Truong](https://github.com/j-bennet)
- Support new `axis=None` behavior in `pandas` 2.0 for certain reductions ([dask#9867](https://github.com/dask/dask/pull/9867)) [James Bourbeau](https://github.com/jrbourbeau)
- Filter out all-nan `RuntimeWarning` at the chunk level for `nanmin` and `nanmax` ([dask#9916](https://github.com/dask/dask/pull/9916)) [Julia Signell](https://github.com/jsignell)
- Fix numeric `meta_nonempty` index `creation` for `pandas` 2.0 ([dask#9908](https://github.com/dask/dask/pull/9908)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `DataFrame.info()` tests for `pandas` 2.0 ([dask#9909](https://github.com/dask/dask/pull/9909)) [James Bourbeau](https://github.com/jrbourbeau)

### Bug Fixes

- Fix `GroupBy.value_counts` handling for multiple `groupby` columns ([dask#9905](https://github.com/dask/dask/pull/9905)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

### Documentation

- Fix some outdated information/typos in development guide ([dask#9893](https://github.com/dask/dask/pull/9893)) [Patrick Hoefler](https://github.com/phofl)
- Add note about `keep=False` in `drop_duplicates` docstring ([dask#9887](https://github.com/dask/dask/pull/9887)) [Jayesh Manani](https://github.com/jayeshmanani)
- Add `meta` details to dask Array ([dask#9886](https://github.com/dask/dask/pull/9886)) [Jayesh Manani](https://github.com/jayeshmanani)
- Clarify task stream showing more rows than threads ([dask#9906](https://github.com/dask/dask/pull/9906)) [Gabe Joseph](https://github.com/gjoseph92)

### Maintenance

- Fix `test_numeric_column_names` for `pandas` 2.0 ([dask#9937](https://github.com/dask/dask/pull/9937)) [Irina Truong](https://github.com/j-bennet)
- Fix `dask/dataframe/tests/test_utils_dataframe.py` tests for `pandas` 2.0 ([dask#9788](https://github.com/dask/dask/pull/9788)) [James Bourbeau](https://github.com/jrbourbeau)
- Replace `index.is_numeric` with `is_any_real_numeric_dtype` for `pandas` 2.0 compatibility ([dask#9918](https://github.com/dask/dask/pull/9918)) [Irina Truong](https://github.com/j-bennet)
- Avoid `pd.core` import in dask utils ([dask#9907](https://github.com/dask/dask/pull/9907)) [Matthew Roeschke](https://github.com/mroeschke)
- Use label for `upstream` build on pull requests ([dask#9910](https://github.com/dask/dask/pull/9910)) [James Bourbeau](https://github.com/jrbourbeau)
- Broaden exception catching for `sqlalchemy.exc.RemovedIn20Warning` ([dask#9904](https://github.com/dask/dask/pull/9904)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily restrict `sqlalchemy < 2` in CI ([dask#9897](https://github.com/dask/dask/pull/9897)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `isort` version to 5.12.0 ([dask#9895](https://github.com/dask/dask/pull/9895)) [Lawrence Mitchell](https://github.com/wence-)
- Remove unused `skiprows` variable in `read_csv` ([dask#9892](https://github.com/dask/dask/pull/9892)) [Patrick Hoefler](https://github.com/phofl)

<a id="v2023-1-1"></a>

## 2023.1.1

Released on January 27, 2023

### Enhancements

- Add `to_backend` method to `Array` and `_Frame` ([dask#9758](https://github.com/dask/dask/pull/9758)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Small fix for timestamp index divisions in `pandas` 2.0 ([dask#9872](https://github.com/dask/dask/pull/9872)) [Irina Truong](https://github.com/j-bennet)
- Add `numeric_only` to `DataFrame.cov` and `DataFrame.corr` ([dask#9787](https://github.com/dask/dask/pull/9787)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixes related to `group_keys` default change in `pandas` 2.0 ([dask#9855](https://github.com/dask/dask/pull/9855)) [Irina Truong](https://github.com/j-bennet)
- `infer_datetime_format` compatibility for `pandas` 2.0  ([dask#9783](https://github.com/dask/dask/pull/9783)) [James Bourbeau](https://github.com/jrbourbeau)

### Bug Fixes

- Fix serialization bug in `BroadcastJoinLayer` ([dask#9871](https://github.com/dask/dask/pull/9871)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Satisfy `broadcast` argument in `DataFrame.merge` ([dask#9852](https://github.com/dask/dask/pull/9852)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `pyarrow` parquet columns statistics computation ([dask#9772](https://github.com/dask/dask/pull/9772)) [aywandji](https://github.com/aywandji)

### Documentation

- Fix “duplicate explicit target name” docs warning ([dask#9863](https://github.com/dask/dask/pull/9863)) [Chiara Marmo](https://github.com/cmarmo)
- Fix code formatting issue in “Defining a new collection backend” docs ([dask#9864](https://github.com/dask/dask/pull/9864)) [Chiara Marmo](https://github.com/cmarmo)
- Update dashboard documentation for memory plot ([dask#9768](https://github.com/dask/dask/pull/9768)) [Jayesh Manani](https://github.com/jayeshmanani)
- Add docs section about `no-worker` tasks ([dask#9839](https://github.com/dask/dask/pull/9839)) [Florian Jetter](https://github.com/fjetter)

### Maintenance

- Additional updates for detecting a `distributed` scheduler  ([dask#9890](https://github.com/dask/dask/pull/9890)) [James Bourbeau](https://github.com/jrbourbeau)
- Update gpuCI `RAPIDS_VER` to `23.04` ([dask#9876](https://github.com/dask/dask/pull/9876))
- Reverse precedence between collection and `distributed` default ([dask#9869](https://github.com/dask/dask/pull/9869)) [Florian Jetter](https://github.com/fjetter)
- Update `xarray-contrib/issue-from-pytest-log` to version 1.2.6 ([dask#9865](https://github.com/dask/dask/pull/9865)) [James Bourbeau](https://github.com/jrbourbeau)
- Dont require dask config shuffle default ([dask#9826](https://github.com/dask/dask/pull/9826)) [Florian Jetter](https://github.com/fjetter)
- Un-`xfail` `datetime64` Parquet roundtripping tests for new `fastparquet` ([dask#9811](https://github.com/dask/dask/pull/9811)) [James Bourbeau](https://github.com/jrbourbeau)
- Add option to manually run `upstream` CI build ([dask#9853](https://github.com/dask/dask/pull/9853)) [James Bourbeau](https://github.com/jrbourbeau)
- Use custom timeout in CI builds ([dask#9844](https://github.com/dask/dask/pull/9844)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `kwargs` from `make_blockwise_graph` ([dask#9838](https://github.com/dask/dask/pull/9838)) [Florian Jetter](https://github.com/fjetter)
- Ignore warnings on `persist` call in `test_setitem_extended_API_2d_mask` ([dask#9843](https://github.com/dask/dask/pull/9843)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix running S3 tests locally ([dask#9833](https://github.com/dask/dask/pull/9833)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2023-1-0"></a>

## 2023.1.0

Released on January 13, 2023

### Enhancements

- Use `distributed` default clients even if no config is set ([dask#9808](https://github.com/dask/dask/pull/9808)) [Florian Jetter](https://github.com/fjetter)
- Implement `ma.where` and `ma.nonzero` ([dask#9760](https://github.com/dask/dask/pull/9760)) [Erik Holmgren](https://github.com/Holmgren825)
- Update `zarr` store creation functions ([dask#9790](https://github.com/dask/dask/pull/9790)) [Ryan Abernathey](https://github.com/rabernat)
- `iteritems` compatibility for `pandas` 2.0 ([dask#9785](https://github.com/dask/dask/pull/9785)) [James Bourbeau](https://github.com/jrbourbeau)
- Accurate `sizeof` for `pandas` `string[python]` dtype ([dask#9781](https://github.com/dask/dask/pull/9781)) [crusaderky](https://github.com/crusaderky)
- Deflate `sizeof()` of duplicate references to pandas object types ([dask#9776](https://github.com/dask/dask/pull/9776)) [crusaderky](https://github.com/crusaderky)
- `GroupBy.__getitem__` compatibility for `pandas` 2.0 ([dask#9779](https://github.com/dask/dask/pull/9779)) [James Bourbeau](https://github.com/jrbourbeau)
- `append` compatibility for `pandas` 2.0 ([dask#9750](https://github.com/dask/dask/pull/9750)) [James Bourbeau](https://github.com/jrbourbeau)
- `get_dummies` compatibility for `pandas` 2.0 ([dask#9752](https://github.com/dask/dask/pull/9752)) [James Bourbeau](https://github.com/jrbourbeau)
- `is_monotonic` compatibility for `pandas` 2.0 ([dask#9751](https://github.com/dask/dask/pull/9751)) [James Bourbeau](https://github.com/jrbourbeau)
- `numpy=1.24` compatability  ([dask#9777](https://github.com/dask/dask/pull/9777)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Remove duplicated `encoding` kwarg in docstring for `to_json` ([dask#9796](https://github.com/dask/dask/pull/9796)) [Sultan Orazbayev](https://github.com/SultanOrazbayev)
- Mention `SubprocessCluster` in `LocalCluster` documentation ([dask#9784](https://github.com/dask/dask/pull/9784)) [Hendrik Makait](https://github.com/hendrikmakait)
- Move Prometheus docs to `dask/distributed` ([dask#9761](https://github.com/dask/dask/pull/9761)) [crusaderky](https://github.com/crusaderky)

### Maintenance

- Temporarily ignore `RuntimeWarning` in `test_setitem_extended_API_2d_mask` ([dask#9828](https://github.com/dask/dask/pull/9828)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix flaky `test_threaded.py::test_interrupt` ([dask#9827](https://github.com/dask/dask/pull/9827)) [Hendrik Makait](https://github.com/hendrikmakait)
- Update `xarray-contrib/issue-from-pytest-log` in `upstream` report ([dask#9822](https://github.com/dask/dask/pull/9822)) [James Bourbeau](https://github.com/jrbourbeau)
- `pip` install dask on gpuCI builds ([dask#9816](https://github.com/dask/dask/pull/9816)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Bump `actions/checkout` from 3.2.0 to 3.3.0 ([dask#9815](https://github.com/dask/dask/pull/9815))
- Resolve `sqlalchemy` import failures in `mindeps` testing ([dask#9809](https://github.com/dask/dask/pull/9809)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Ignore `sqlalchemy.exc.RemovedIn20Warning` ([dask#9801](https://github.com/dask/dask/pull/9801)) [Thomas Grainger](https://github.com/graingert)
- `xfail` `datetime64` Parquet roundtripping tests for `pandas` 2.0 ([dask#9786](https://github.com/dask/dask/pull/9786)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `sqlachemy` 1.3 compatibility ([dask#9695](https://github.com/dask/dask/pull/9695)) [McToel](https://github.com/McToel)
- Reduce size of expected DoK sparse matrix ([dask#9775](https://github.com/dask/dask/pull/9775)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Remove executable flag from `dask/dataframe/io/orc/utils.py` ([dask#9774](https://github.com/dask/dask/pull/9774)) [Elliott Sales de Andrade](https://github.com/QuLogic)

<a id="v2022-12-1"></a>

## 2022.12.1

Released on December 16, 2022

### Enhancements

- Support `dtype_backend="pandas|pyarrow"` configuration ([dask#9719](https://github.com/dask/dask/pull/9719)) [James Bourbeau](https://github.com/jrbourbeau)
- Support `cupy.ndarray` to `cudf.DataFrame` dispatching in `dask.dataframe` ([dask#9579](https://github.com/dask/dask/pull/9579)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Make filesystem-backend configurable in `read_parquet` ([dask#9699](https://github.com/dask/dask/pull/9699)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Serialize all `pyarrow` extension arrays efficiently ([dask#9740](https://github.com/dask/dask/pull/9740)) [James Bourbeau](https://github.com/jrbourbeau)

### Bug Fixes

- Fix bug when repartitioning with `tz`-aware datetime index ([dask#9741](https://github.com/dask/dask/pull/9741)) [James Bourbeau](https://github.com/jrbourbeau)
- Partial functions in aggs may have arguments ([dask#9724](https://github.com/dask/dask/pull/9724)) [Irina Truong](https://github.com/j-bennet)
- Add support for simple operation with `pyarrow`-backed extension dtypes ([dask#9717](https://github.com/dask/dask/pull/9717)) [James Bourbeau](https://github.com/jrbourbeau)
- Rename columns correctly in case of `SeriesGroupby` ([dask#9716](https://github.com/dask/dask/pull/9716)) [Lawrence Mitchell](https://github.com/wence-)

### Documentation

- Fix url link typo in collection backend doc ([dask#9748](https://github.com/dask/dask/pull/9748)) [Shawn](https://github.com/chaokunyang)
- Update Prometheus docs ([dask#9696](https://github.com/dask/dask/pull/9696)) [Hendrik Makait](https://github.com/hendrikmakait)

### Maintenance

- Add `zarr` to Python 3.11 CI environment ([dask#9771](https://github.com/dask/dask/pull/9771)) [James Bourbeau](https://github.com/jrbourbeau)
- Add support for Python 3.11 ([dask#9708](https://github.com/dask/dask/pull/9708)) [Thomas Grainger](https://github.com/graingert)
- Bump `actions/checkout` from 3.1.0 to 3.2.0 ([dask#9753](https://github.com/dask/dask/pull/9753))
- Avoid `np.bool8` deprecation warning ([dask#9737](https://github.com/dask/dask/pull/9737)) [James Bourbeau](https://github.com/jrbourbeau)
- Make sure dev packages aren’t overwritten in `upstream` CI build ([dask#9731](https://github.com/dask/dask/pull/9731)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid adding `data.h5` and `mydask.html` files during tests ([dask#9726](https://github.com/dask/dask/pull/9726)) [Thomas Grainger](https://github.com/graingert)

<a id="v2022-12-0"></a>

## 2022.12.0

Released on December 2, 2022

### Enhancements

- Remove statistics-based `set_index` logic from `read_parquet` ([dask#9661](https://github.com/dask/dask/pull/9661)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add support for `use_nullable_dtypes` to `dd.read_parquet` ([dask#9617](https://github.com/dask/dask/pull/9617)) [Ian Rose](https://github.com/ian-r-rose)
- Fix `map_overlap` in order to accept pandas arguments ([dask#9571](https://github.com/dask/dask/pull/9571)) [Fabien Aulaire](https://github.com/faulaire)
- Fix pandas 1.5+ `FutureWarning` in `.str.split(..., expand=True)` ([dask#9704](https://github.com/dask/dask/pull/9704)) [Jacob Hayes](https://github.com/JacobHayes)
- Enable column projection for `groupby` slicing ([dask#9667](https://github.com/dask/dask/pull/9667)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support duplicate column cum-functions ([dask#9685](https://github.com/dask/dask/pull/9685)) [Ben](https://github.com/benjaminhduncan)
- Improve error message for failed backend dispatch call ([dask#9677](https://github.com/dask/dask/pull/9677)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Bug Fixes

- Revise meta creation in arrow parquet engine ([dask#9672](https://github.com/dask/dask/pull/9672)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `da.fft.fft` for array-like inputs ([dask#9688](https://github.com/dask/dask/pull/9688)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `groupby` -aggregation when grouping on an index by name ([dask#9646](https://github.com/dask/dask/pull/9646)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Maintenance

- Avoid `PytestReturnNotNoneWarning` in `test_inheriting_class` ([dask#9707](https://github.com/dask/dask/pull/9707)) [Thomas Grainger](https://github.com/graingert)
- Fix flaky `test_dataframe_aggregations_multilevel` ([dask#9701](https://github.com/dask/dask/pull/9701)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bump `mypy` version ([dask#9697](https://github.com/dask/dask/pull/9697)) [crusaderky](https://github.com/crusaderky)
- Disable dashboard in `test_map_partitions_df_input` ([dask#9687](https://github.com/dask/dask/pull/9687)) [James Bourbeau](https://github.com/jrbourbeau)
- Use latest `xarray-contrib/issue-from-pytest-log` in `upstream` build ([dask#9682](https://github.com/dask/dask/pull/9682)) [James Bourbeau](https://github.com/jrbourbeau)
- `xfail` `ttest_1samp` for upstream `scipy` ([dask#9670](https://github.com/dask/dask/pull/9670)) [James Bourbeau](https://github.com/jrbourbeau)
- Update gpuCI `RAPIDS_VER` to `23.02` ([dask#9678](https://github.com/dask/dask/pull/9678))

<a id="v2022-11-1"></a>

## 2022.11.1

Released on November 18, 2022

### Enhancements

- Restrict `bokeh=3` support ([dask#9673](https://github.com/dask/dask/pull/9673)) [Gabe Joseph](https://github.com/gjoseph92)
- Updates for `fastparquet` evolution ([dask#9650](https://github.com/dask/dask/pull/9650)) [Martin Durant](https://github.com/martindurant)

### Maintenance

- Update `ga-yaml-parser` step in gpuCI updating workflow  ([dask#9675](https://github.com/dask/dask/pull/9675)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Revert `importlib.metadata` workaround ([dask#9658](https://github.com/dask/dask/pull/9658)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `mindeps-distributed` CI build to handle `numpy`/`pandas` not being installed  ([dask#9668](https://github.com/dask/dask/pull/9668)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2022-11-0"></a>

## 2022.11.0

Released on November 15, 2022

### Enhancements

- Generalize `from_dict` implementation to allow usage from other backends ([dask#9628](https://github.com/dask/dask/pull/9628)) [GALI PREM SAGAR](https://github.com/galipremsagar)

### Bug Fixes

- Avoid `pandas` constructors in `dask.dataframe.core` ([dask#9570](https://github.com/dask/dask/pull/9570)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `sort_values` with `Timestamp` data ([dask#9642](https://github.com/dask/dask/pull/9642)) [James Bourbeau](https://github.com/jrbourbeau)
- Generalize array checking and remove `pd.Index` call in `_get_partitions` ([dask#9634](https://github.com/dask/dask/pull/9634)) [Benjamin Zaitlen](https://github.com/quasiben)
- Fix `read_csv` behavior for `header=0` and `names` ([dask#9614](https://github.com/dask/dask/pull/9614)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Update dashboard docs for queuing ([dask#9660](https://github.com/dask/dask/pull/9660)) [Gabe Joseph](https://github.com/gjoseph92)
- Remove `import dask as d` from docstrings ([dask#9644](https://github.com/dask/dask/pull/9644)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix link to partitions docs in `read_parquet` docstring ([dask#9636](https://github.com/dask/dask/pull/9636)) [qheuristics](https://github.com/qheuristics)
- Add API doc links to `array/bag/dataframe` sections ([dask#9630](https://github.com/dask/dask/pull/9630)) [Matthew Rocklin](https://github.com/mrocklin)

### Maintenance

- Use `conda-incubator/setup-miniconda@v2.2.0` ([dask#9662](https://github.com/dask/dask/pull/9662)) [John A Kirkham](https://github.com/jakirkham)
- Allow `bokeh=3` ([dask#9659](https://github.com/dask/dask/pull/9659)) [James Bourbeau](https://github.com/jrbourbeau)
- Run `upstream` build with Python 3.10 ([dask#9655](https://github.com/dask/dask/pull/9655)) [James Bourbeau](https://github.com/jrbourbeau)
- Pin `pyyaml` version in mindeps testing ([dask#9640](https://github.com/dask/dask/pull/9640)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add `pre-commit` to catch `breakpoint()` ([dask#9638](https://github.com/dask/dask/pull/9638)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump `xarray-contrib/issue-from-pytest-log` from 1.1 to 1.2 ([dask#9635](https://github.com/dask/dask/pull/9635))
- Remove `blosc` references ([dask#9625](https://github.com/dask/dask/pull/9625)) [Naty Clementi](https://github.com/ncclementi)
- Upgrade `mypy` and drop unused comments ([dask#9616](https://github.com/dask/dask/pull/9616)) [Hendrik Makait](https://github.com/hendrikmakait)
- Harden `test_repartition_npartitions` ([dask#9585](https://github.com/dask/dask/pull/9585)) [Richard (Rick) Zamora](https://github.com/rjzamora)

<a id="v2022-10-2"></a>

## 2022.10.2

Released on October 31, 2022

This was a hotfix and has no changes in this repository.
The necessary fix was in dask/distributed, but we decided to bump this version
number for consistency.

<a id="v2022-10-1"></a>

## 2022.10.1

Released on October 28, 2022

### Enhancements

- Enable named aggregation syntax ([dask#9563](https://github.com/dask/dask/pull/9563)) [ChrisJar](https://github.com/ChrisJar)
- Add extension dtype support to `set_index` ([dask#9566](https://github.com/dask/dask/pull/9566)) [James Bourbeau](https://github.com/jrbourbeau)
- Redesigning the array HTML repr for clarity ([dask#9519](https://github.com/dask/dask/pull/9519)) [Shingo OKAWA](https://github.com/ognis1205)

### Bug Fixes

- Fix `merge` with emtpy left DataFrame ([dask#9578](https://github.com/dask/dask/pull/9578)) [Ian Rose](https://github.com/ian-r-rose)

### Documentation

- Add note about limiting thread oversubscription by default ([dask#9592](https://github.com/dask/dask/pull/9592)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `sphinx-click` for `dask` CLI ([dask#9589](https://github.com/dask/dask/pull/9589)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix Semaphore API docs ([dask#9584](https://github.com/dask/dask/pull/9584)) [James Bourbeau](https://github.com/jrbourbeau)
- Render meta description in `map_overlap` docstring ([dask#9568](https://github.com/dask/dask/pull/9568)) [James Bourbeau](https://github.com/jrbourbeau)

### Maintenance

- Require Click 7.0+ in Dask ([dask#9595](https://github.com/dask/dask/pull/9595)) [John A Kirkham](https://github.com/jakirkham)
- Temporarily restrict `bokeh<3` ([dask#9607](https://github.com/dask/dask/pull/9607)) [James Bourbeau](https://github.com/jrbourbeau)
- Resolve `importlib`-related failures in `upstream` CI ([dask#9604](https://github.com/dask/dask/pull/9604)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Improve `upstream` CI report ([dask#9603](https://github.com/dask/dask/pull/9603)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `upstream` CI report ([dask#9602](https://github.com/dask/dask/pull/9602)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `setuptools` host dep, add CLI entrypoint ([dask#9600](https://github.com/dask/dask/pull/9600)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- More `Backend` dispatch class type annotations ([dask#9573](https://github.com/dask/dask/pull/9573)) [Ian Rose](https://github.com/ian-r-rose)

<a id="v2022-10-0"></a>

## 2022.10.0

Released on October 14, 2022

### New Features

- Backend library dispatching for IO in Dask-Array and Dask-DataFrame ([dask#9475](https://github.com/dask/dask/pull/9475)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add new CLI that is extensible ([dask#9283](https://github.com/dask/dask/pull/9283)) [Doug Davis](https://github.com/douglasdavis)

### Enhancements

- Groupby median ([dask#9516](https://github.com/dask/dask/pull/9516)) [Ian Rose](https://github.com/ian-r-rose)
- Fix array copy not being a no-op ([dask#9555](https://github.com/dask/dask/pull/9555)) [David Hoese](https://github.com/djhoese)
- Add support for string timedelta in `map_overlap` ([dask#9559](https://github.com/dask/dask/pull/9559)) [Nicolas Grandemange](https://github.com/epizut)
- Shuffle-based groupby for single functions ([dask#9504](https://github.com/dask/dask/pull/9504)) [Ian Rose](https://github.com/ian-r-rose)
- Make `datetime.datetime` tokenize idempotantly ([dask#9532](https://github.com/dask/dask/pull/9532)) [Martin Durant](https://github.com/martindurant)
- Support tokenizing `datetime.time` ([dask#9528](https://github.com/dask/dask/pull/9528)) [Tim Paine](https://github.com/timkpaine)

### Bug Fixes

- Avoid race condition in lazy dispatch registration ([dask#9545](https://github.com/dask/dask/pull/9545)) [James Bourbeau](https://github.com/jrbourbeau)
- Do not allow setitem to `np.nan` for `int` dtype ([dask#9531](https://github.com/dask/dask/pull/9531)) [Doug Davis](https://github.com/douglasdavis)
- Stable demo column projection ([dask#9538](https://github.com/dask/dask/pull/9538)) [Ian Rose](https://github.com/ian-r-rose)
- Ensure `pickle`-able binops in `delayed` ([dask#9540](https://github.com/dask/dask/pull/9540)) [Ian Rose](https://github.com/ian-r-rose)
- Fix project CSV columns when selecting ([dask#9534](https://github.com/dask/dask/pull/9534)) [Martin Durant](https://github.com/martindurant)

### Documentation

- Update Parquet best practice ([dask#9537](https://github.com/dask/dask/pull/9537)) [Matthew Rocklin](https://github.com/mrocklin)

### Maintenance

- Restrict `tiledb-py` version to avoid CI failures  ([dask#9569](https://github.com/dask/dask/pull/9569)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump `actions/github-script` from 3 to 6 ([dask#9564](https://github.com/dask/dask/pull/9564))
- Bump `actions/stale` from 4 to 6 ([dask#9551](https://github.com/dask/dask/pull/9551))
- Bump `peter-evans/create-pull-request` from 3 to 4 ([dask#9550](https://github.com/dask/dask/pull/9550))
- Bump `actions/checkout` from 2 to 3.1.0 ([dask#9552](https://github.com/dask/dask/pull/9552))
- Bump `codecov/codecov-action` from 1 to 3 ([dask#9549](https://github.com/dask/dask/pull/9549))
- Bump `the-coding-turtle/ga-yaml-parser` from 0.1.1 to 0.1.2 ([dask#9553](https://github.com/dask/dask/pull/9553))
- Move dependabot configuration file ([dask#9547](https://github.com/dask/dask/pull/9547)) [James Bourbeau](https://github.com/jrbourbeau)
- Add dependabot for GitHub actions ([dask#9542](https://github.com/dask/dask/pull/9542)) [James Bourbeau](https://github.com/jrbourbeau)
- Run mypy on Windows and Linux ([dask#9530](https://github.com/dask/dask/pull/9530)) [crusaderky](https://github.com/crusaderky)
- Update gpuCI `RAPIDS_VER` to `22.12` ([dask#9524](https://github.com/dask/dask/pull/9524))

<a id="v2022-9-2"></a>

## 2022.9.2

Released on September 30, 2022

### Enhancements

- Remove factorization logic from array auto chunking ([dask#9507](https://github.com/dask/dask/pull/9507)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Add docs on running Dask in a standalone Python script ([dask#9513](https://github.com/dask/dask/pull/9513)) [James Bourbeau](https://github.com/jrbourbeau)
- Clarify custom-graph multiprocessing example ([dask#9511](https://github.com/dask/dask/pull/9511)) [nouman](https://github.com/noumxn)

### Maintenance

- Groupby sort upstream compatibility ([dask#9486](https://github.com/dask/dask/pull/9486)) [Ian Rose](https://github.com/ian-r-rose)

<a id="v2022-9-1"></a>

## 2022.9.1

Released on September 16, 2022

### New Features

- Add `DataFrame` and `Series` `median` methods ([dask#9483](https://github.com/dask/dask/pull/9483)) [James Bourbeau](https://github.com/jrbourbeau)

### Enhancements

- Shuffle `groupby` default ([dask#9453](https://github.com/dask/dask/pull/9453)) [Ian Rose](https://github.com/ian-r-rose)
- Filter by list ([dask#9419](https://github.com/dask/dask/pull/9419)) [Greg Hayes](https://github.com/hayesgb)
- Added `distributed.utils.key_split` functionality to `dask.utils.key_split` ([dask#9464](https://github.com/dask/dask/pull/9464)) [Luke Conibear](https://github.com/lukeconibear)

### Bug Fixes

- Fix overlap so that `set_index` doesn’t drop rows ([dask#9423](https://github.com/dask/dask/pull/9423)) [Julia Signell](https://github.com/jsignell)
- Fix assigning pandas `Series` to column when `ddf.columns.min()` raises ([dask#9485](https://github.com/dask/dask/pull/9485)) [Erik Welch](https://github.com/eriknw)
- Fix metadata comparison `stack_partitions` ([dask#9481](https://github.com/dask/dask/pull/9481)) [James Bourbeau](https://github.com/jrbourbeau)
- Provide default for `split_out` ([dask#9493](https://github.com/dask/dask/pull/9493)) [Lawrence Mitchell](https://github.com/wence-)

### Deprecations

- Allow `split_out` to be `None`, which then defaults to `1` in `groupby().aggregate()` ([dask#9491](https://github.com/dask/dask/pull/9491)) [Ian Rose](https://github.com/ian-r-rose)

### Documentation

- Fixing `enforce_metadata` documentation, not checking for dtypes ([dask#9474](https://github.com/dask/dask/pull/9474)) [Nicolas Grandemange](https://github.com/epizut)
- Fix `it's` –> `its` typo ([dask#9484](https://github.com/dask/dask/pull/9484)) [Nat Tabris](https://github.com/ntabris)

### Maintenance

- Workaround for parquet writing failure using some datetime series but not others ([dask#9500](https://github.com/dask/dask/pull/9500)) [Ian Rose](https://github.com/ian-r-rose)
- Filter out `numeric_only` warnings from `pandas` ([dask#9496](https://github.com/dask/dask/pull/9496)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid `set_index(..., inplace=True)` where not necessary ([dask#9472](https://github.com/dask/dask/pull/9472)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid passing groupby key list of length one ([dask#9495](https://github.com/dask/dask/pull/9495)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `test_groupby_dropna_cudf` based on `cudf` support for `group_keys` ([dask#9482](https://github.com/dask/dask/pull/9482)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `dd.from_bcolz` ([dask#9479](https://github.com/dask/dask/pull/9479)) [James Bourbeau](https://github.com/jrbourbeau)
- Added `flake8-bugbear` to `pre-commit` hooks ([dask#9457](https://github.com/dask/dask/pull/9457)) [Luke Conibear](https://github.com/lukeconibear)
- Bind loop variables in function definitions (`B023`) ([dask#9461](https://github.com/dask/dask/pull/9461)) [Luke Conibear](https://github.com/lukeconibear)
- Added assert for comparisons (`B015`) ([dask#9459](https://github.com/dask/dask/pull/9459)) [Luke Conibear](https://github.com/lukeconibear)
- Set top-level default shell in CI workflows ([dask#9469](https://github.com/dask/dask/pull/9469)) [James Bourbeau](https://github.com/jrbourbeau)
- Removed unused loop control variables (`B007`) ([dask#9458](https://github.com/dask/dask/pull/9458)) [Luke Conibear](https://github.com/lukeconibear)
- Replaced `getattr` calls for constant attributes (`B009`) ([dask#9460](https://github.com/dask/dask/pull/9460)) [Luke Conibear](https://github.com/lukeconibear)
- Pin `libprotobuf` to allow nightly `pyarrow` in the upstream CI build ([dask#9465](https://github.com/dask/dask/pull/9465)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Replaced mutable data structures for default arguments (`B006`) ([dask#9462](https://github.com/dask/dask/pull/9462)) [Luke Conibear](https://github.com/lukeconibear)
- Changed `flake8` mirror and updated version ([dask#9456](https://github.com/dask/dask/pull/9456)) [Luke Conibear](https://github.com/lukeconibear)

<a id="v2022-9-0"></a>

## 2022.9.0

Released on September 2, 2022

### Enhancements

- Enable automatic column projection for `groupby` aggregations ([dask#9442](https://github.com/dask/dask/pull/9442)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Accept superclasses in NEP-13/17 dispatching ([dask#6710](https://github.com/dask/dask/pull/6710)) [Gabe Joseph](https://github.com/gjoseph92)

### Bug Fixes

- Rename `by` columns internally for cumulative operations on the same `by` columns ([dask#9430](https://github.com/dask/dask/pull/9430)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix `get_group` with categoricals ([dask#9436](https://github.com/dask/dask/pull/9436)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix caching-related `MaterializedLayer.cull` performance regression ([dask#9413](https://github.com/dask/dask/pull/9413)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Add maintainer documentation page ([dask#9309](https://github.com/dask/dask/pull/9309)) [James Bourbeau](https://github.com/jrbourbeau)

### Maintenance

- Revert skipped fastparquet test ([dask#9439](https://github.com/dask/dask/pull/9439)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- `tmpfile` does not end files with period on empty extension ([dask#9429](https://github.com/dask/dask/pull/9429)) [Hendrik Makait](https://github.com/hendrikmakait)
- Skip failing fastparquet test with latest release ([dask#9432](https://github.com/dask/dask/pull/9432)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2022-8-1"></a>

## 2022.8.1

Released on August 19, 2022

### New Features

- Implement `ma.*_like functions` ([dask#9378](https://github.com/dask/dask/pull/9378)) [Ruth Comer](https://github.com/rcomer)

### Enhancements

- Fuse compatible annotations ([dask#9402](https://github.com/dask/dask/pull/9402)) [Ian Rose](https://github.com/ian-r-rose)
- Shuffle-based groupby aggregation for high-cardinality groups ([dask#9302](https://github.com/dask/dask/pull/9302)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Unpack `namedtuple` ([dask#9361](https://github.com/dask/dask/pull/9361)) [Hendrik Makait](https://github.com/hendrikmakait)

### Bug Fixes

- Fix `SeriesGroupBy` cumulative functions with `axis=1` ([dask#9377](https://github.com/dask/dask/pull/9377)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Sparse array reductions ([dask#9342](https://github.com/dask/dask/pull/9342)) [Ian Rose](https://github.com/ian-r-rose)
- Fix `make_meta` while using categorical column with index ([dask#9348](https://github.com/dask/dask/pull/9348)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Don’t allow incompatible keywords in `DataFrame.dropna` ([dask#9366](https://github.com/dask/dask/pull/9366)) [Naty Clementi](https://github.com/ncclementi)
- Make `set_index` handle entirely empty dataframes ([dask#8896](https://github.com/dask/dask/pull/8896)) [Julia Signell](https://github.com/jsignell)
- Improve `dataclass` handling in `unpack_collections` ([dask#9345](https://github.com/dask/dask/pull/9345)) [Hendrik Makait](https://github.com/hendrikmakait)
- Fix bag sampling when there are some smaller partitions ([dask#9349](https://github.com/dask/dask/pull/9349)) [Ian Rose](https://github.com/ian-r-rose)
- Add support for empty partitions to `da.min`/`da.max` functions ([dask#9268](https://github.com/dask/dask/pull/9268)) [geraninam](https://github.com/geraninam)

### Documentation

- Clarify that `bind()` etc. regenerate the keys ([dask#9385](https://github.com/dask/dask/pull/9385)) [crusaderky](https://github.com/crusaderky)
- Consolidate dashboard diagnostics documentation ([dask#9357](https://github.com/dask/dask/pull/9357)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Remove outdated `meta` information [Pavithra Eswaramoorthy](https://github.com/pavithraes)

### Maintenance

- Use `entry_points` utility in `sizeof` ([dask#9390](https://github.com/dask/dask/pull/9390)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `entry_points` compatibility utility ([dask#9388](https://github.com/dask/dask/pull/9388)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Upload environment file artifact for each CI build ([dask#9372](https://github.com/dask/dask/pull/9372)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `werkzeug` pin in CI ([dask#9371](https://github.com/dask/dask/pull/9371)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix type annotations for `dd.from_pandas` and `dd.from_delayed` ([dask#9362](https://github.com/dask/dask/pull/9362)) [Jordan Yap](https://github.com/jjyap)

<a id="v2022-8-0"></a>

## 2022.8.0

Released on August 5, 2022

### Enhancements

- Ensure `make_meta` doesn’t hold ref to data ([dask#9354](https://github.com/dask/dask/pull/9354)) [Jim Crist-Harif](https://github.com/jcrist)
- Revise `divisions` logic in `from_pandas` ([dask#9221](https://github.com/dask/dask/pull/9221)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Warn if user sets index with existing index ([dask#9341](https://github.com/dask/dask/pull/9341)) [Julia Signell](https://github.com/jsignell)
- Add `keepdims` keyword for `da.average` ([dask#9332](https://github.com/dask/dask/pull/9332)) [Ruth Comer](https://github.com/rcomer)
- Change `repr` methods to avoid `Layer` materialization ([dask#9289](https://github.com/dask/dask/pull/9289)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Bug Fixes

- Make sure `order` kwarg will not crash the `astype` method ([dask#9317](https://github.com/dask/dask/pull/9317)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Fix bug for `cumsum` on cupy chunked dask arrays ([dask#9320](https://github.com/dask/dask/pull/9320)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Match input and output structure in `_sample_reduce` ([dask#9272](https://github.com/dask/dask/pull/9272)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Include `meta` in array serialization ([dask#9240](https://github.com/dask/dask/pull/9240)) [Frédéric BRIOL](https://github.com/fbriol)
- Fix `Index.memory_usage` ([dask#9290](https://github.com/dask/dask/pull/9290)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix division calculation in `dask.dataframe.io.from_dask_array` ([dask#9282](https://github.com/dask/dask/pull/9282)) [Jordan Yap](https://github.com/jjyap)

### Documentation

- Fow to use kwargs with custom task graphs ([dask#9322](https://github.com/dask/dask/pull/9322)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add note to `da.from_array` about how the order is not preserved ([dask#9346](https://github.com/dask/dask/pull/9346)) [Julia Signell](https://github.com/jsignell)
- Add I/O info for async functions ([dask#9326](https://github.com/dask/dask/pull/9326)) [Logan Norman](https://github.com/lognorman20)
- Tidy up docs snippet for futures IO functions ([dask#9340](https://github.com/dask/dask/pull/9340)) [Julia Signell](https://github.com/jsignell)
- Use consistent variable names for pandas `df` and Dask `ddf` in `dataframe-groupby.rst` ([dask#9304](https://github.com/dask/dask/pull/9304)) [ivojuroro](https://github.com/ivojuroro)
- Switch `js-yaml` for `yaml.js` in config converter ([dask#9306](https://github.com/dask/dask/pull/9306)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

### Maintenance

- Update `da.linalg.solve` for SciPy 1.9.0 compatibility  ([dask#9350](https://github.com/dask/dask/pull/9350)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Update `test_getitem_avoids_large_chunks_missing` ([dask#9347](https://github.com/dask/dask/pull/9347)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix docs title formatting for “Extend `sizeof`” [Doug Davis](https://github.com/douglasdavis)
- Import `loop_in_thread` fixture in tests ([dask#9337](https://github.com/dask/dask/pull/9337)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily `xfail` `test_solve_sym_pos` ([dask#9336](https://github.com/dask/dask/pull/9336)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix small typo in 10 minutes to Dask page ([dask#9329](https://github.com/dask/dask/pull/9329)) [Shaghayegh](https://github.com/Shadimrad)
- Temporarily pin `werkzeug` in CI to avoid test suite hanging  ([dask#9325](https://github.com/dask/dask/pull/9325)) [James Bourbeau](https://github.com/jrbourbeau)
- Add tests for `cupy.angle()` ([dask#9312](https://github.com/dask/dask/pull/9312)) [Peter Andreas Entschev](https://github.com/pentschev)
- Update gpuCI `RAPIDS_VER` to `22.10` ([dask#9314](https://github.com/dask/dask/pull/9314))
- Add `pandas[test]` to `test` extra ([dask#9110](https://github.com/dask/dask/pull/9110)) [Ben Beasley](https://github.com/musicinmybrain)
- Add `bokeh` and `scipy` to `upstream` CI build ([dask#9265](https://github.com/dask/dask/pull/9265)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2022-7-1"></a>

## 2022.7.1

Released on July 22, 2022

### Enhancements

- Return Dask array if all axes are squeezed ([dask#9250](https://github.com/dask/dask/pull/9250)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Make cycle reported by toposort shorter ([dask#9068](https://github.com/dask/dask/pull/9068)) [Erik Welch](https://github.com/eriknw)
- Unknown chunk slicing - raise informative error ([dask#9285](https://github.com/dask/dask/pull/9285)) [Naty Clementi](https://github.com/ncclementi)

### Bug Fixes

- Fix bug in `HighLevelGraph.cull` ([dask#9267](https://github.com/dask/dask/pull/9267)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Sort categories ([dask#9264](https://github.com/dask/dask/pull/9264)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Use `max` (instead of `sum`) for calculating `warnsize` ([dask#9235](https://github.com/dask/dask/pull/9235)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix bug when filtering on partitioned column with pyarrow ([dask#9252](https://github.com/dask/dask/pull/9252)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Updated repartition documentation to add note about `partition_size` ([dask#9288](https://github.com/dask/dask/pull/9288)) [Dylan Stewart](https://github.com/drstewart19)
- Don’t include docs in `Array` methods, just refer to module docs ([dask#9244](https://github.com/dask/dask/pull/9244)) [Julia Signell](https://github.com/jsignell)
- Remove outdated reference to scheduler and worker dashboards ([dask#9278](https://github.com/dask/dask/pull/9278)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Fix a few typos ([dask#9270](https://github.com/dask/dask/pull/9270)) [Tim Gates](https://github.com/timgates42)
- Adds an custom aggregate example using numpy methods ([dask#9260](https://github.com/dask/dask/pull/9260)) [geraninam](https://github.com/geraninam)

### Maintenance

- Add type annotations to `dd.from_pandas` and `dd.from_delayed` ([dask#9237](https://github.com/dask/dask/pull/9237)) [Michael Milton](https://github.com/multimeric)
- Update `calculate_divisions` docstring ([dask#9275](https://github.com/dask/dask/pull/9275)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update `test_plot_multiple` for upcoming `bokeh` release ([dask#9261](https://github.com/dask/dask/pull/9261)) [James Bourbeau](https://github.com/jrbourbeau)
- Add typing to common array properties ([dask#9255](https://github.com/dask/dask/pull/9255)) [Illviljan](https://github.com/Illviljan)

<a id="v2022-7-0"></a>

## 2022.7.0

Released on July 8, 2022

### Enhancements

- Support `pathlib.PurePath` in `normalize_token` ([dask#9229](https://github.com/dask/dask/pull/9229)) [Angus Hollands](https://github.com/agoose77)
- Add `AttributeNotImplementedError` for properties so IPython glob search works ([dask#9231](https://github.com/dask/dask/pull/9231)) [Erik Welch](https://github.com/eriknw)
- `map_overlap`: multiple dataframe handling ([dask#9145](https://github.com/dask/dask/pull/9145)) [Fabien Aulaire](https://github.com/faulaire)
- Read entrypoints in `dask.sizeof` ([dask#7688](https://github.com/dask/dask/pull/7688)) [Angus Hollands](https://github.com/agoose77)

### Bug Fixes

- Fix `TypeError: 'Serialize' object is not subscriptable` when writing parquet dataset with `Client(processes=False)` ([dask#9015](https://github.com/dask/dask/pull/9015)) [Lucas Miguel Ponce](https://github.com/lucasmsp)
- Correct dtypes when `concat` with an empty dataframe ([dask#9193](https://github.com/dask/dask/pull/9193)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)

### Documentation

- Highlight note about persist ([dask#9234](https://github.com/dask/dask/pull/9234)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Update release-procedure to include more detail and helpful commands ([dask#9215](https://github.com/dask/dask/pull/9215)) [Julia Signell](https://github.com/jsignell)
- Better SEO for Futures and Dask vs. Spark pages ([dask#9217](https://github.com/dask/dask/pull/9217)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Maintenance

- Use `math.prod` instead of `np.prod` on lists, tuples, and iters ([dask#9232](https://github.com/dask/dask/pull/9232)) [crusaderky](https://github.com/crusaderky)
- Only import IPython if type checking ([dask#9230](https://github.com/dask/dask/pull/9230)) [Florian Jetter](https://github.com/fjetter)
- Tougher mypy checks ([dask#9206](https://github.com/dask/dask/pull/9206)) [crusaderky](https://github.com/crusaderky)

<a id="v2022-6-1"></a>

## 2022.6.1

Released on June 24, 2022

### Enhancements

- Dask in pyodide ([dask#9053](https://github.com/dask/dask/pull/9053)) [Ian Rose](https://github.com/ian-r-rose)
- Create `dask.utils.show_versions` ([dask#9144](https://github.com/dask/dask/pull/9144)) [Sultan Orazbayev](https://github.com/SultanOrazbayev)
- Better error message for unsupported numpy operations on dask.dataframe objects. ([dask#9201](https://github.com/dask/dask/pull/9201)) [Julia Signell](https://github.com/jsignell)
- Add `allow_rechunk` kwarg to `dask.array.overlap` function ([dask#7776](https://github.com/dask/dask/pull/7776)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add minutes and hours to `dask.utils.format_time` ([dask#9116](https://github.com/dask/dask/pull/9116)) [Matthew Rocklin](https://github.com/mrocklin)
- More retries when writing parquet to remote filesystem ([dask#9175](https://github.com/dask/dask/pull/9175)) [Ian Rose](https://github.com/ian-r-rose)

### Bug Fixes

- Timedelta deterministic hashing ([dask#9213](https://github.com/dask/dask/pull/9213)) [Fabien Aulaire](https://github.com/faulaire)
- Enum deterministic hashing ([dask#9212](https://github.com/dask/dask/pull/9212)) [Fabien Aulaire](https://github.com/faulaire)
- `shuffle_group()`: avoid converting to arrays ([dask#9157](https://github.com/dask/dask/pull/9157)) [Mads R. B. Kristensen](https://github.com/madsbk)

### Deprecations

- Deprecate extra `format_time` utility ([dask#9184](https://github.com/dask/dask/pull/9184)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Better SEO for 10 Minutes to Dask ([dask#9182](https://github.com/dask/dask/pull/9182)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Better SEO for Delayed and Best Practices ([dask#9194](https://github.com/dask/dask/pull/9194)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Include known inconsistency in DataFrame `str.split` accessor docstring  ([dask#9177](https://github.com/dask/dask/pull/9177)) [Richard Pelgrim](https://github.com/rrpelgrim)
- Add `inconsistencies` keyword to `derived_from` ([dask#9192](https://github.com/dask/dask/pull/9192)) [Richard Pelgrim](https://github.com/rrpelgrim)
- Add missing `append` in `delayed` best practices example ([dask#9202](https://github.com/dask/dask/pull/9202)) [Ben](https://github.com/benjaminhduncan)
- Fix indentation in Best Practices ([dask#9196](https://github.com/dask/dask/pull/9196)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add link to [Genevieve Buckley](https://github.com/GenevieveBuckley)’s blog on chunk sizes ([dask#9199](https://github.com/dask/dask/pull/9199)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Update `to_csv` docstring ([dask#9094](https://github.com/dask/dask/pull/9094)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Maintenance

- Update versioneer: change from using `SafeConfigParser` to `ConfigParser` ([dask#9205](https://github.com/dask/dask/pull/9205)) [Thomas A Caswell](https://github.com/tacaswell)
- Remove ipython hack in CI([dask#9200](https://github.com/dask/dask/pull/9200)) [crusaderky](https://github.com/crusaderky)

<a id="v2022-6-0"></a>

## 2022.6.0

Released on June 10, 2022

### Enhancements

- Add feature to show names of layer dependencies in HLG JupyterLab repr ([dask#9081](https://github.com/dask/dask/pull/9081)) [Angelos Omirolis](https://github.com/aomirolis)
- Add arrow schema extraction dispatch ([dask#9169](https://github.com/dask/dask/pull/9169)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Add `sort_results` argument to `assert_eq` ([dask#9130](https://github.com/dask/dask/pull/9130)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Add weeks to `parse_timedelta` ([dask#9168](https://github.com/dask/dask/pull/9168)) [Matthew Rocklin](https://github.com/mrocklin)
- Warn that cloudpickle is not always deterministic ([dask#9148](https://github.com/dask/dask/pull/9148)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Switch parquet default engine ([dask#9140](https://github.com/dask/dask/pull/9140)) [Jim Crist-Harif](https://github.com/jcrist)
- Use deterministic hashing with `_iLocIndexer` /  `_LocIndexer` ([dask#9108](https://github.com/dask/dask/pull/9108)) [Fabien Aulaire](https://github.com/faulaire)
- Enfore consistent schema in `to_parquet` pyarrow ([dask#9131](https://github.com/dask/dask/pull/9131)) [Jim Crist-Harif](https://github.com/jcrist)

### Bug Fixes

- Fix `pyarrow.StringArray` pickle ([dask#9170](https://github.com/dask/dask/pull/9170)) [Jim Crist-Harif](https://github.com/jcrist)
- Fix parallel metadata collection in pyarrow engine ([dask#9165](https://github.com/dask/dask/pull/9165)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Improve `pyarrow` partitioning logic ([dask#9147](https://github.com/dask/dask/pull/9147)) [James Bourbeau](https://github.com/jrbourbeau)
- `pyarrow` 8.0 partitioning fix ([dask#9143](https://github.com/dask/dask/pull/9143)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Better SEO for Installing Dask and Dask DataFrame Best Practices ([dask#9178](https://github.com/dask/dask/pull/9178)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Update logos page in docs ([dask#9167](https://github.com/dask/dask/pull/9167)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add example using pandas Series to `map_partition` doctring ([dask#9161](https://github.com/dask/dask/pull/9161)) [Alex-JG3](https://github.com/Alex-JG3)
- Update docs theme for rebranding ([dask#9160](https://github.com/dask/dask/pull/9160)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Better SEO for docs on Dask DataFrames ([dask#9128](https://github.com/dask/dask/pull/9128)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Maintenance

- Remove ensure_file from recommended practice for downstream libraries ([dask#9171](https://github.com/dask/dask/pull/9171)) [Matthew Rocklin](https://github.com/mrocklin)
- Test round-tripping DataFrame parquet I/O including pyspark ([dask#9156](https://github.com/dask/dask/pull/9156)) [Ian Rose](https://github.com/ian-r-rose)
- Try disabling HDF5 locking ([dask#9154](https://github.com/dask/dask/pull/9154)) [Ian Rose](https://github.com/ian-r-rose)
- Link best practices to DataFrame-parquet ([dask#9150](https://github.com/dask/dask/pull/9150)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix typo in `map_partitions` `func` parameter description ([dask#9149](https://github.com/dask/dask/pull/9149)) [Christopher Akiki](https://github.com/cakiki)
- Un-`xfail` `test_groupby_grouper_dispatch` ([dask#9139](https://github.com/dask/dask/pull/9139)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Temporarily import cleanup fixture from distributed ([dask#9138](https://github.com/dask/dask/pull/9138)) [James Bourbeau](https://github.com/jrbourbeau)
- Simplify partitioning logic in pyarrow parquet engine ([dask#9041](https://github.com/dask/dask/pull/9041)) [Richard (Rick) Zamora](https://github.com/rjzamora)

<a id="v2022-05-2"></a>

## 2022.05.2

Released on May 26, 2022

### Enhancements

- Add a dispatch for  non-pandas `Grouper` objects and use it in `GroupBy` ([dask#9074](https://github.com/dask/dask/pull/9074)) [brandon-b-miller](https://github.com/brandon-b-miller)
- Error if `read_parquet` & `to_parquet` files intersect ([dask#9124](https://github.com/dask/dask/pull/9124)) [Jim Crist-Harif](https://github.com/jcrist)
- Visualize task graphs using `ipycytoscape` ([dask#9091](https://github.com/dask/dask/pull/9091)) [Ian Rose](https://github.com/ian-r-rose)

### Documentation

- Fix various typos ([dask#9126](https://github.com/dask/dask/pull/9126)) [Ryan Russell](https://github.com/ryanrussell)

### Maintenance

- Fix flaky `test_filter_nonpartition_columns` ([dask#9127](https://github.com/dask/dask/pull/9127)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Update gpuCI `RAPIDS_VER` to `22.08` ([dask#9120](https://github.com/dask/dask/pull/9120))
- Include `conftest.py`` in sdists ([dask#9115](https://github.com/dask/dask/pull/9115)) [Ben Beasley](https://github.com/musicinmybrain)

<a id="v2022-05-1"></a>

## 2022.05.1

Released on May 24, 2022

### New Features

- Add `DataFrame.from_dict` classmethod ([dask#9017](https://github.com/dask/dask/pull/9017)) [Matthew Powers](https://github.com/MrPowers)
- Add `from_map` function to Dask DataFrame ([dask#8911](https://github.com/dask/dask/pull/8911)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Enhancements

- Improve `to_parquet` error for appended divisions overlap ([dask#9102](https://github.com/dask/dask/pull/9102)) [Jim Crist-Harif](https://github.com/jcrist)
- Enabled user-defined process-initializer functions ([dask#9087](https://github.com/dask/dask/pull/9087)) [ParticularMiner](https://github.com/ParticularMiner)
- Mention `align_dataframes=False` option in `map_partitions` error ([dask#9075](https://github.com/dask/dask/pull/9075)) [Gabe Joseph](https://github.com/gjoseph92)
- Add kwarg `enforce_ndim` to `dask.array.map_blocks()` ([dask#8865](https://github.com/dask/dask/pull/8865)) [ParticularMiner](https://github.com/ParticularMiner)
- Implement `Series.GroupBy.fillna` / `DataFrame.GroupBy.fillna` methods ([dask#8869](https://github.com/dask/dask/pull/8869)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Allow `fillna` with Dask DataFrame ([dask#8950](https://github.com/dask/dask/pull/8950)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Update error message for assignment with 1-d dask array ([dask#9036](https://github.com/dask/dask/pull/9036)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Collection Protocol ([dask#8674](https://github.com/dask/dask/pull/8674)) [Doug Davis](https://github.com/douglasdavis)
- Patch around `pandas` `ArrowStringArray` pickling ([dask#9024](https://github.com/dask/dask/pull/9024)) [Jim Crist-Harif](https://github.com/jcrist)
- Band-aid for `compute_as_if_collection` ([dask#8998](https://github.com/dask/dask/pull/8998)) [Ian Rose](https://github.com/ian-r-rose)
- Add `p2p` shuffle option ([dask#8836](https://github.com/dask/dask/pull/8836)) [Matthew Rocklin](https://github.com/mrocklin)

### Bug Fixes

- Fixup column projection with no columns ([dask#9106](https://github.com/dask/dask/pull/9106)) [Jim Crist-Harif](https://github.com/jcrist)
- Blockwise cull NumPy `dtype` ([dask#9100](https://github.com/dask/dask/pull/9100)) [Ian Rose](https://github.com/ian-r-rose)
- Fix column-projection bug in `from_map` ([dask#9078](https://github.com/dask/dask/pull/9078)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Prevent nulls in index for non-numeric dtypes ([dask#8963](https://github.com/dask/dask/pull/8963)) [Jorge López](https://github.com/jorloplaz)
- Fix `is_monotonic` methods for more than 8 partitions ([dask#9019](https://github.com/dask/dask/pull/9019)) [Julia Signell](https://github.com/jsignell)
- Handle enumerate and generator inputs to `from_map` ([dask#9066](https://github.com/dask/dask/pull/9066)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Revert `is_dask_collection`; back to previous implementation ([dask#9062](https://github.com/dask/dask/pull/9062)) [Doug Davis](https://github.com/douglasdavis)
- Fix `Blockwise.clone` does not handle iterable literal arguments correctly ([dask#8979](https://github.com/dask/dask/pull/8979)) [JSKenyon](https://github.com/jskenyon)
- Array `setitem` hardmask ([dask#9027](https://github.com/dask/dask/pull/9027)) [David Hassell](https://github.com/davidhassell)
- Fix overlapping divisions error on append ([dask#8997](https://github.com/dask/dask/pull/8997)) [Ian Rose](https://github.com/ian-r-rose)

### Deprecations

- Add pre-deprecation warnings for  `read_parquet` kwargs `chunksize` and `aggregate_files` ([dask#9052](https://github.com/dask/dask/pull/9052)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Document `map_partitions` handling of `args` vs `kwargs`, usage of `partition_info` ([dask#9084](https://github.com/dask/dask/pull/9084)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Update custom collection documentation (leverage new collection protocol) ([dask#9097](https://github.com/dask/dask/pull/9097)) [Doug Davis](https://github.com/douglasdavis)
- Better SEO for docs on creating and storing Dask DataFrames ([dask#9098](https://github.com/dask/dask/pull/9098)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Clarify chunking in `imread` docstring ([dask#9082](https://github.com/dask/dask/pull/9082)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Rearrange docs TOC ([dask#9001](https://github.com/dask/dask/pull/9001)) [Matthew Rocklin](https://github.com/mrocklin)
- Corrected `map_blocks()` docstring for kwarg `enforce_ndim` ([dask#9071](https://github.com/dask/dask/pull/9071)) [ParticularMiner](https://github.com/ParticularMiner)
- Update DataFrame SQL docs references to other libraries ([dask#9077](https://github.com/dask/dask/pull/9077)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Update page on creating and storing Dask DataFrames ([dask#9025](https://github.com/dask/dask/pull/9025)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Maintenance

- Include `NUMPY_LICENSE.txt` in license files ([dask#9113](https://github.com/dask/dask/pull/9113)) [Ben Beasley](https://github.com/musicinmybrain)
- Increase retries when installing nightly `pandas` ([dask#9103](https://github.com/dask/dask/pull/9103)) [James Bourbeau](https://github.com/jrbourbeau)
- Force nightly `pyarrow` in the upstream build ([dask#9095](https://github.com/dask/dask/pull/9095)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Improve object handling & testing of `ensure_unicode` ([dask#9059](https://github.com/dask/dask/pull/9059)) [John A Kirkham](https://github.com/jakirkham)
- Force nightly `pyarrow` in the upstream build ([dask#8993](https://github.com/dask/dask/pull/8993)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Additional check on `is_dask_collection` ([dask#9054](https://github.com/dask/dask/pull/9054)) [Doug Davis](https://github.com/douglasdavis)
- Update `ensure_bytes` ([dask#9050](https://github.com/dask/dask/pull/9050)) [John A Kirkham](https://github.com/jakirkham)
- Add end of file pre-commit hook ([dask#9045](https://github.com/dask/dask/pull/9045)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `codespell` pre-commit hook ([dask#9040](https://github.com/dask/dask/pull/9040)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove the HDFS tests ([dask#9039](https://github.com/dask/dask/pull/9039)) [Jim Crist-Harif](https://github.com/jcrist)
- Fix flaky `test_reductions_2D` ([dask#9037](https://github.com/dask/dask/pull/9037)) [Jim Crist-Harif](https://github.com/jcrist)
- Prevent codecov from notifying of failure too soon ([dask#9031](https://github.com/dask/dask/pull/9031)) [Jim Crist-Harif](https://github.com/jcrist)
- Only test on Python 3.9 on macos ([dask#9029](https://github.com/dask/dask/pull/9029)) [Jim Crist-Harif](https://github.com/jcrist)
- Update `to_timedelta` default unit ([dask#9010](https://github.com/dask/dask/pull/9010)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)

<a id="v2022-05-0"></a>

## 2022.05.0

Released on May 2, 2022

### Highlights

This is a bugfix release for [this issue](https://github.com/dask/distributed/issues/6255).

### Documentation

- Add highlights section to 2022.04.2 release notes ([dask#9012](https://github.com/dask/dask/pull/9012)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2022-04-2"></a>

## 2022.04.2

Released on April 29, 2022

### Highlights

This release includes several deprecations/breaking API changes to
`dask.dataframe.read_parquet` and `dask.dataframe.to_parquet`:

- `to_parquet` no longer writes `_metadata` files by default. If you want to
  write a `_metadata` file, you can pass in `write_metadata_file=True`.
- `read_parquet` now defaults to `split_row_groups=False`, which results in one
  Dask dataframe partition per parquet file when reading in a parquet dataset.
  If you’re working with large parquet files you may need to set
  `split_row_groups=True` to reduce your partition size.
- `read_parquet` no longer calculates divisions by default. If you require
  `read_parquet` to return dataframes with known divisions, please set
  `calculate_divisions=True`.
- `read_parquet` has deprecated the `gather_statistics` keyword argument.
  Please use the `calculate_divisions` keyword argument instead.
- `read_parquet` has deprecated the `require_extensions` keyword argument.
  Please use the `parquet_file_extension` keyword argument instead.

### New Features

- Add `removeprefix` and `removesuffix` as `StringMethods` ([dask#8912](https://github.com/dask/dask/pull/8912)) [Jorge López](https://github.com/jorloplaz)

### Enhancements

- Call `fs.invalidate_cache` in `to_parquet` ([dask#8994](https://github.com/dask/dask/pull/8994)) [Jim Crist-Harif](https://github.com/jcrist)
- Change `to_parquet` default to `write_metadata_file=None` ([dask#8988](https://github.com/dask/dask/pull/8988)) [Jim Crist-Harif](https://github.com/jcrist)
- Let arg reductions pass `keepdims` ([dask#8926](https://github.com/dask/dask/pull/8926)) [Julia Signell](https://github.com/jsignell)
- Change `split_row_groups` default to `False` in `read_parquet` ([dask#8981](https://github.com/dask/dask/pull/8981)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Improve `NotImplementedError` message for `da.reshape` ([dask#8987](https://github.com/dask/dask/pull/8987)) [Jim Crist-Harif](https://github.com/jcrist)
- Simplify `to_parquet` compute path ([dask#8982](https://github.com/dask/dask/pull/8982)) [Jim Crist-Harif](https://github.com/jcrist)
- Raise an error if you try to use `vindex` with a Dask object ([dask#8945](https://github.com/dask/dask/pull/8945)) [Julia Signell](https://github.com/jsignell)
- Avoid `pre_buffer=True` when a precache method is specified ([dask#8957](https://github.com/dask/dask/pull/8957)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `from_dask_array` uses `blockwise` instead of merging graphs ([dask#8889](https://github.com/dask/dask/pull/8889)) [Bryan Weber](https://github.com/bryanwweber)
- Use `pre_buffer=True` for “pyarrow” Parquet engine ([dask#8952](https://github.com/dask/dask/pull/8952)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Bug Fixes

- Handle `dtype=None` correctly in `da.full` ([dask#8954](https://github.com/dask/dask/pull/8954)) [Tom White](https://github.com/tomwhite)
- Fix `dask-sql` bug caused by `blockwise` fusion ([dask#8989](https://github.com/dask/dask/pull/8989)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `to_parquet` errors for non-string column names ([dask#8990](https://github.com/dask/dask/pull/8990)) [Jim Crist-Harif](https://github.com/jcrist)
- Make sure `da.roll` works even if shape is 0 ([dask#8925](https://github.com/dask/dask/pull/8925)) [Julia Signell](https://github.com/jsignell)
- Fix recursion error issue with `set_index` ([dask#8967](https://github.com/dask/dask/pull/8967)) [Paul Hobson](https://github.com/phobson)
- Stringify `BlockwiseDepDict` mapping values when `produces_keys=True` ([dask#8972](https://github.com/dask/dask/pull/8972)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use DataFram\`eIOLayer in `DataFrame.from_delayed` ([dask#8852](https://github.com/dask/dask/pull/8852)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Check that values for the `in` predicate in `read_parquet` are correct ([dask#8846](https://github.com/dask/dask/pull/8846)) [Bryan Weber](https://github.com/bryanwweber)
- Fix bug for reduction of zero dimensional arrays ([dask#8930](https://github.com/dask/dask/pull/8930)) [Tom White](https://github.com/tomwhite)
- Specify `dtype` when deciding division using `np.linspace` in `read_sql_query` ([dask#8940](https://github.com/dask/dask/pull/8940)) [Cheun Hong](https://github.com/cheunhong)

### Deprecations

- Deprecate `gather_statistics` from `read_parquet` ([dask#8992](https://github.com/dask/dask/pull/8992)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Change `require_extension` to top-level `parquet_file_extension` `read_parquet` kwarg ([dask#8935](https://github.com/dask/dask/pull/8935)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Update `write_metadata_file` discussion in documentation ([dask#8995](https://github.com/dask/dask/pull/8995)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update `DataFrame.merge` docstring ([dask#8966](https://github.com/dask/dask/pull/8966)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Added description for parameter `align_arrays` in `array.blockwise()` ([dask#8977](https://github.com/dask/dask/pull/8977)) [ParticularMiner](https://github.com/ParticularMiner)
- ecommend not to use `map_block(drop_axis=...)` on chunked axes of an array ([dask#8921](https://github.com/dask/dask/pull/8921)) [ParticularMiner](https://github.com/ParticularMiner)
- Add copy button to code snippets in docs ([dask#8956](https://github.com/dask/dask/pull/8956)) [James Bourbeau](https://github.com/jrbourbeau)

### Maintenance

- Pandas 1.5.0 compatibility ([dask#8961](https://github.com/dask/dask/pull/8961)) [Ian Rose](https://github.com/ian-r-rose)
- Add `pytest-timeout` to distributed envs on CI ([dask#8986](https://github.com/dask/dask/pull/8986)) [Julia Signell](https://github.com/jsignell)
- Improve `read_parquet` docstring formatting ([dask#8971](https://github.com/dask/dask/pull/8971)) [Bryan Weber](https://github.com/bryanwweber)
- Remove `pytest.warns(None)` ([dask#8924](https://github.com/dask/dask/pull/8924)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Document Python 3.10 as supported ([dask#8976](https://github.com/dask/dask/pull/8976)) [Eray Aslan](https://github.com/erayaslan)
- `parse_timedelta` option to enforce explicit unit ([dask#8969](https://github.com/dask/dask/pull/8969)) [crusaderky](https://github.com/crusaderky)
- `mypy` compatibility ([dask#8854](https://github.com/dask/dask/pull/8854)) [Paul Hobson](https://github.com/phobson)
- Add a docs page for Dask & Parquet ([dask#8899](https://github.com/dask/dask/pull/8899)) [Jim Crist-Harif](https://github.com/jcrist)
- Adds configuration to ignore revs in blame ([dask#8933](https://github.com/dask/dask/pull/8933)) [Bryan Weber](https://github.com/bryanwweber)

<a id="v2022-04-1"></a>

## 2022.04.1

Released on April 15, 2022

### New Features

- Add missing NumPy ufuncs: `abs`, `left_shift`, `right_shift`, `positive`. ([dask#8920](https://github.com/dask/dask/pull/8920)) [Tom White](https://github.com/tomwhite)

### Enhancements

- Avoid collecting parquet metadata in pyarrow when `write_metadata_file=False` ([dask#8906](https://github.com/dask/dask/pull/8906)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Better error for failed wildcard path in `dd.read_csv()` (fixes #8878) ([dask#8908](https://github.com/dask/dask/pull/8908)) [Roger Filmyer](https://github.com/rfilmyer)
- Return `da.Array` rather than `dd.Series` for non-ufunc elementwise functions on `dd.Series` ([dask#8558](https://github.com/dask/dask/pull/8558)) [Julia Signell](https://github.com/jsignell)
- Let `get_dummies` use `meta` computation in `map_partitions` ([dask#8898](https://github.com/dask/dask/pull/8898)) [Julia Signell](https://github.com/jsignell)
- Masked scalars input to `da.from_array` ([dask#8895](https://github.com/dask/dask/pull/8895)) [David Hassell](https://github.com/davidhassell)
- Raise `ValueError` in `merge_asof` for duplicate `kwargs` ([dask#8861](https://github.com/dask/dask/pull/8861)) [Bryan Weber](https://github.com/bryanwweber)

### Bug Fixes

- Make `is_monotonic` work when some partitions are empty ([dask#8897](https://github.com/dask/dask/pull/8897)) [Julia Signell](https://github.com/jsignell)
- Fix custom getter in `da.from_array` when `inline_array=False` ([dask#8903](https://github.com/dask/dask/pull/8903)) [Ian Rose](https://github.com/ian-r-rose)
- Correctly handle dict-specification for rechunk. ([dask#8859](https://github.com/dask/dask/pull/8859)) [Richard](https://github.com/richarms)
- Fix `merge_asof`: drop index column if `left_on == right_on` ([dask#8874](https://github.com/dask/dask/pull/8874)) [Gil Forsyth](https://github.com/gforsyth)

### Deprecations

- Warn users that `engine='auto'` will change in future ([dask#8907](https://github.com/dask/dask/pull/8907)) [Jim Crist-Harif](https://github.com/jcrist)
- Remove `pyarrow-legacy` engine from parquet API ([dask#8835](https://github.com/dask/dask/pull/8835)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Add note on missing parameter `out` for `dask.array.dot` ([dask#8913](https://github.com/dask/dask/pull/8913)) [Francesco Andreuzzi](https://github.com/fAndreuzzi)
- Update `DataFrame.query` docstring ([dask#8890](https://github.com/dask/dask/pull/8890)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)

### Maintenance

- Don’t test `da.prod` on large integer data ([dask#8893](https://github.com/dask/dask/pull/8893)) [Jim Crist-Harif](https://github.com/jcrist)
- Add `network` marks to tests that fail without an internet connection ([dask#8881](https://github.com/dask/dask/pull/8881)) [Paul Hobson](https://github.com/phobson)
- Fix gpuCI GHA version ([dask#8891](https://github.com/dask/dask/pull/8891)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- `xfail`/`skip` some flaky `distributed` tests ([dask#8887](https://github.com/dask/dask/pull/8887)) [Jim Crist-Harif](https://github.com/jcrist)
- Remove unused (deprecated) code from `ArrowDatasetEngine` ([dask#8885](https://github.com/dask/dask/pull/8885)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add mild typing to common utils functions, part 2 ([dask#8867](https://github.com/dask/dask/pull/8867)) [crusaderky](https://github.com/crusaderky)
- Documentation of Limitation of `sample()` ([dask#8858](https://github.com/dask/dask/pull/8858)) [Nadiem Sissouno](https://github.com/sissnad)

<a id="v2022-04-0"></a>

## 2022.04.0

Released on April 1, 2022

#### NOTE
This is the first release with support for Python 3.10

### New Features

- Add Python 3.10 support ([dask#8566](https://github.com/dask/dask/pull/8566)) [James Bourbeau](https://github.com/jrbourbeau)

### Enhancements

- Add check on `dtype.itemsize` in order to produce a useful error ([dask#8860](https://github.com/dask/dask/pull/8860)) [Davide Gavio](https://github.com/davidegavio)
- Add mild typing to common utils functions ([dask#8848](https://github.com/dask/dask/pull/8848)) [Matthew Rocklin](https://github.com/mrocklin)
- Add sanity checks to `divisions` `setter` ([dask#8806](https://github.com/dask/dask/pull/8806)) [Jim Crist-Harif](https://github.com/jcrist)
- Use `Blockwise` and `map_partitions` for more tasks ([dask#8831](https://github.com/dask/dask/pull/8831)) [Bryan Weber](https://github.com/bryanwweber)

### Bug Fixes

- Fix `dataframe.merge_asof` to preserve `right_on` column ([dask#8857](https://github.com/dask/dask/pull/8857)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Fix “Buffer dtype mismatch” for pandas >= 1.3 on 32bit ([dask#8851](https://github.com/dask/dask/pull/8851)) [Ben Greiner](https://github.com/bnavigator)
- Fix slicing fusion by altering `SubgraphCallable` `getter` ([dask#8827](https://github.com/dask/dask/pull/8827)) [Ian Rose](https://github.com/ian-r-rose)

### Deprecations

- Remove support for PyPy ([dask#8863](https://github.com/dask/dask/pull/8863)) [James Bourbeau](https://github.com/jrbourbeau)
- Drop `setuptools` at runtime ([dask#8855](https://github.com/dask/dask/pull/8855)) [crusaderky](https://github.com/crusaderky)
- Remove `dataframe.tseries.resample.getnanos` ([dask#8834](https://github.com/dask/dask/pull/8834)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)

### Documentation

- Organize diagnostic and performance docs ([dask#8871](https://github.com/dask/dask/pull/8871)) [Naty Clementi](https://github.com/ncclementi)
- Add image to explain `drop_axis` option of `map_blocks` ([dask#8868](https://github.com/dask/dask/pull/8868)) [ParticularMiner](https://github.com/ParticularMiner)

### Maintenance

- Update gpuCI `RAPIDS_VER` to `22.06` ([dask#8828](https://github.com/dask/dask/pull/8828))
- Restore `test_parquet` in http ([dask#8850](https://github.com/dask/dask/pull/8850)) [Bryan Weber](https://github.com/bryanwweber)
- Simplify gpuCI updating workflow ([dask#8849](https://github.com/dask/dask/pull/8849)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2022-03-0"></a>

## 2022.03.0

Released on March 18, 2022

### New Features

- Bag: add implementation for reservoir sampling ([dask#7636](https://github.com/dask/dask/pull/7636)) [Daniel Mesejo-León](https://github.com/mesejo)
- Add `ma.count` to Dask array ([dask#8785](https://github.com/dask/dask/pull/8785)) [David Hassell](https://github.com/davidhassell)
- Change `to_parquet` default to `compression="snappy"` ([dask#8814](https://github.com/dask/dask/pull/8814)) [Jim Crist-Harif](https://github.com/jcrist)
- Add `weights` parameter to `dask.array.reduction` ([dask#8805](https://github.com/dask/dask/pull/8805)) [David Hassell](https://github.com/davidhassell)
- Add `ddf.compute_current_divisions` to get divisions on a sorted index or column ([dask#8517](https://github.com/dask/dask/pull/8517)) [Julia Signell](https://github.com/jsignell)

### Enhancements

- Pass `__name__` and `__doc__` through on DelayedLeaf ([dask#8820](https://github.com/dask/dask/pull/8820)) [Leo Gao](https://github.com/leogao2)
- Raise exception for not implemented merge `how` option ([dask#8818](https://github.com/dask/dask/pull/8818)) [Naty Clementi](https://github.com/ncclementi)
- Move `Bag.map_partitions` to `Blockwise` ([dask#8646](https://github.com/dask/dask/pull/8646)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Improve error messages for malformed config files ([dask#8801](https://github.com/dask/dask/pull/8801)) [Jim Crist-Harif](https://github.com/jcrist)
- Revise column-projection optimization to capture common dask-sql patterns ([dask#8692](https://github.com/dask/dask/pull/8692)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Useful error for empty divisions ([dask#8789](https://github.com/dask/dask/pull/8789)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Scipy 1.8.0 compat: copy private classes into dask/array/stats.py ([dask#8694](https://github.com/dask/dask/pull/8694)) [Julia Signell](https://github.com/jsignell)
- Raise warning when using multiple types of schedulers where one is `distributed` ([dask#8700](https://github.com/dask/dask/pull/8700)) [Pedro Silva](https://github.com/ppsbs)

### Bug Fixes

- Fix bug in applying != filter in `read_parquet` ([dask#8824](https://github.com/dask/dask/pull/8824)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `set_index` when directly passed a dask Index ([dask#8680](https://github.com/dask/dask/pull/8680)) [Paul Hobson](https://github.com/phobson)
- Quick fix for unbounded memory usage in tensordot  ([dask#7980](https://github.com/dask/dask/pull/7980)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- If hdf file is empty, don’t fail on meta creation ([dask#8809](https://github.com/dask/dask/pull/8809)) [Julia Signell](https://github.com/jsignell)
- Update `clone_key("x")` to retain prefix ([dask#8792](https://github.com/dask/dask/pull/8792)) [crusaderky](https://github.com/crusaderky)
- Fix “physical” column bug in pyarrow-based `read_parquet` ([dask#8775](https://github.com/dask/dask/pull/8775)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix `groupby.shift` bug caused by unsorted partitions after shuffle ([dask#8782](https://github.com/dask/dask/pull/8782)) [kori73](https://github.com/kori73)
- Fix serialization bug ([dask#8786](https://github.com/dask/dask/pull/8786)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Deprecations

- Bump diagnostics bokeh dependency to 2.4.2 ([dask#8791](https://github.com/dask/dask/pull/8791)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Deprecate `bcolz` support ([dask#8754](https://github.com/dask/dask/pull/8754)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)
- Finish making `map_overlap` default boundary `kwarg` `'none'` ([dask#8743](https://github.com/dask/dask/pull/8743)) [Genevieve Buckley](https://github.com/GenevieveBuckley)

### Documentation

- Custom collection example docs fix ([dask#8807](https://github.com/dask/dask/pull/8807)) [Doug Davis](https://github.com/douglasdavis)
- Add `Series.str`, `Series.dt`, and `Series.cat` accessors to docs ([dask#8757](https://github.com/dask/dask/pull/8757)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Fix docstring for `ddf.compute_current_divisions` ([dask#8793](https://github.com/dask/dask/pull/8793)) [Julia Signell](https://github.com/jsignell)
- Dashboard docs on /status page  ([dask#8648](https://github.com/dask/dask/pull/8648)) [Naty Clementi](https://github.com/ncclementi)
- Clarify divisions `kwarg` in repartition docstring ([dask#8781](https://github.com/dask/dask/pull/8781)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Update Docker images to use ghcr.io ([dask#8774](https://github.com/dask/dask/pull/8774)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

### Maintenance

- Reduce gpuci `pytest` parallelism ([dask#8826](https://github.com/dask/dask/pull/8826)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- `absolufy-imports` - No relative imports - PEP8 ([dask#8796](https://github.com/dask/dask/pull/8796)) [Julia Signell](https://github.com/jsignell)
- Tidy up `assert_eq` calls in array tests ([dask#8812](https://github.com/dask/dask/pull/8812)) [Julia Signell](https://github.com/jsignell)
- Avoid `pytest.warns(None)` ([dask#8718](https://github.com/dask/dask/pull/8718)) [LSturtew](https://github.com/LSturtew)
- Fix `test_describe_empty` to work without global `-Werror` ([dask#8291](https://github.com/dask/dask/pull/8291)) [Michał Górny](https://github.com/mgorny)
- Temporarily xfail graphviz tests on windows ([dask#8794](https://github.com/dask/dask/pull/8794)) [Jim Crist-Harif](https://github.com/jcrist)
- Use `packaging.parse` for `md5` compatibility ([dask#8763](https://github.com/dask/dask/pull/8763)) [James Bourbeau](https://github.com/jrbourbeau)
- Make `tokenize` work in a FIPS 140-2 environment ([dask#8762](https://github.com/dask/dask/pull/8762)) [Jim Crist-Harif](https://github.com/jcrist)
- Label issues and PRs on open with ‘needs triage’ ([dask#8761](https://github.com/dask/dask/pull/8761)) [Julia Signell](https://github.com/jsignell)
- Add some extra test coverage ([dask#8302](https://github.com/dask/dask/pull/8302)) [lrjball](https://github.com/lrjball)
- Specify action version and change from `pull_request_target` to `pull_request` ([dask#8767](https://github.com/dask/dask/pull/8767)) [Julia Signell](https://github.com/jsignell)
- Make scheduler `kwarg` pass though to sub functions in `da.assert_eq` ([dask#8755](https://github.com/dask/dask/pull/8755)) [Julia Signell](https://github.com/jsignell)

<a id="v2022-02-1"></a>

## 2022.02.1

Released on February 25, 2022

### New Features

- Add  aggregate functions `first` and `last` to `dask.dataframe.pivot_table` ([dask#8649](https://github.com/dask/dask/pull/8649)) [Knut Nordanger](https://github.com/nordange)
- Add `std()` support for `datetime64` `dtype` for pandas-like objects ([dask#8523](https://github.com/dask/dask/pull/8523)) [Ben Glossner](https://github.com/bglossner)
- Add materialized task counts to `HighLevelGraph` and `Layer` html reprs ([dask#8589](https://github.com/dask/dask/pull/8589)) [kori73](https://github.com/kori73)

### Enhancements

- Do not allow iterating a `DataFrameGroupBy` ([dask#8696](https://github.com/dask/dask/pull/8696)) [Bryan Weber](https://github.com/bryanwweber)
- Fix missing newline after `info()` call on empty `DataFrame` ([dask#8727](https://github.com/dask/dask/pull/8727)) [Naty Clementi](https://github.com/ncclementi)
- Add `groupby.compute` as a not implemented method ([dask#8734](https://github.com/dask/dask/pull/8734)) [Dranaxel](https://github.com/Dranaxel)
- Improve multi dataframe join performance ([dask#8740](https://github.com/dask/dask/pull/8740)) [Holden Karau](https://github.com/holdenk)
- Include `bool` type for `Index` ([dask#8732](https://github.com/dask/dask/pull/8732)) [Naty Clementi](https://github.com/ncclementi)
- Allow `ArrowDatasetEngine` subclass to override pandas->arrow conversion also for partitioned write ([dask#8741](https://github.com/dask/dask/pull/8741)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Increase performance of k-diagonal extraction in `da.diag()` and `da.diagonal()` ([dask#8689](https://github.com/dask/dask/pull/8689)) [ParticularMiner](https://github.com/ParticularMiner)
- Change `linspace` creation to match numpy when num equal to 0 ([dask#8676](https://github.com/dask/dask/pull/8676)) [Peter](https://github.com/peterpandelidis)
- Tokenize `dataclasses` ([dask#8557](https://github.com/dask/dask/pull/8557)) [Gabe Joseph](https://github.com/gjoseph92)
- Update `tokenize` to treat `dict` and `kwargs` differently ([dask#8655](https://github.com/dask/dask/pull/8655)) [James Bourbeau](https://github.com/jrbourbeau)

### Bug Fixes

- Fix bug in `dask.array.roll()` for roll-shifts that match the size of the input array ([dask#8723](https://github.com/dask/dask/pull/8723)) [ParticularMiner](https://github.com/ParticularMiner)
- Fix for `normalize_function` `dataclass` methods ([dask#8527](https://github.com/dask/dask/pull/8527)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Fix rechunking with zero-size-chunks ([dask#8703](https://github.com/dask/dask/pull/8703)) [ParticularMiner](https://github.com/ParticularMiner)
- Move creation of `sqlalchemy` connection for picklability ([dask#8745](https://github.com/dask/dask/pull/8745)) [Julia Signell](https://github.com/jsignell)

### Deprecations

- Drop Python 3.7 ([dask#8572](https://github.com/dask/dask/pull/8572)) [James Bourbeau](https://github.com/jrbourbeau)
- Deprecate `iteritems` ([dask#8660](https://github.com/dask/dask/pull/8660)) [James Bourbeau](https://github.com/jrbourbeau)
- Deprecate `dataframe.tseries.resample.getnanos` ([dask#8752](https://github.com/dask/dask/pull/8752)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add deprecation warning for pyarrow-legacy engine ([dask#8758](https://github.com/dask/dask/pull/8758)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Update link typos in changelog ([dask#8717](https://github.com/dask/dask/pull/8717)) [James Bourbeau](https://github.com/jrbourbeau)
- Clarify `dask.visualize` docstring ([dask#8710](https://github.com/dask/dask/pull/8710)) [Dranaxel](https://github.com/Dranaxel)
- Update Docker example to use current best practices ([dask#8731](https://github.com/dask/dask/pull/8731)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Update docs to include `distributed.Client.preload` ([dask#8679](https://github.com/dask/dask/pull/8679)) [Bryan Weber](https://github.com/bryanwweber)
- Document monthly social meeting ([dask#8595](https://github.com/dask/dask/pull/8595)) [Thomas Grainger](https://github.com/graingert)
- Add docs for Gen2 access with RBAC/ACL i.e. security principal ([dask#8748](https://github.com/dask/dask/pull/8748)) [Martin Thøgersen](https://github.com/th0ger)
- Use Dask configuration extension from `dask-sphinx-theme` ([dask#8751](https://github.com/dask/dask/pull/8751)) [Benjamin Zaitlen](https://github.com/quasiben)

### Maintenance

- Unpin `coverage` in CI ([dask#8690](https://github.com/dask/dask/pull/8690)) [James Bourbeau](https://github.com/jrbourbeau)
- Add manual trigger for running test suite ([dask#8716](https://github.com/dask/dask/pull/8716)) [James Bourbeau](https://github.com/jrbourbeau)
- Xfail `scheduler_HLG_unpack_import`; flaky test ([dask#8724](https://github.com/dask/dask/pull/8724)) [Mike McCarty](https://github.com/mmccarty)
- Temporarily remove `scipy` upstream CI build ([dask#8725](https://github.com/dask/dask/pull/8725)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump pre-release version to be greater than stable releases ([dask#8728](https://github.com/dask/dask/pull/8728)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Move custom sort function logic to internal `sort_values` ([dask#8571](https://github.com/dask/dask/pull/8571)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Pin `cloudpickle` and `scipy` in docs requirements ([dask#8737](https://github.com/dask/dask/pull/8737)) [Julia Signell](https://github.com/jsignell)
- Make the labeler not delete labels, and look for the docs at the right spot ([dask#8746](https://github.com/dask/dask/pull/8746)) [Julia Signell](https://github.com/jsignell)
- Fix docs build warnings ([dask#8432](https://github.com/dask/dask/pull/8432)) [Kristopher Overholt](https://github.com/koverholt)
- Update test status badge ([dask#8747](https://github.com/dask/dask/pull/8747)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix parquet `test_pandas_timestamp_overflow_pyarrow` test ([dask#8733](https://github.com/dask/dask/pull/8733)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Only run PR builds on changes to relevant files ([dask#8756](https://github.com/dask/dask/pull/8756)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2022-02-0"></a>

## 2022.02.0

Released on February 11, 2022

#### NOTE
This is the last release with support for Python 3.7

### New Features

- Add `region` to `to_zarr` when using existing array ([dask#8590](https://github.com/dask/dask/pull/8590)) [Chris Roat](https://github.com/ChrisRoat)
- Add `engine_kwargs` support to `dask.dataframe.to_sql` ([dask#8609](https://github.com/dask/dask/pull/8609)) [Amir Kadivar](https://github.com/amirkdv)
- Add `include_path_column` arg to `read_json` ([dask#8603](https://github.com/dask/dask/pull/8603)) [Bryan Weber](https://github.com/bryanwweber)
- Add `expand_dims` to Dask array ([dask#8687](https://github.com/dask/dask/pull/8687)) [Tom White](https://github.com/tomwhite)

### Enhancements

- Add scheduler option to `assert_eq` utilities ([dask#8610](https://github.com/dask/dask/pull/8610)) [Xinrong Meng](https://github.com/xinrong-databricks)
- Fix eye inconsistency with NumPy for `dtype=None` ([dask#8685](https://github.com/dask/dask/pull/8685)) [Tom White](https://github.com/tomwhite)
- Fix concatenate inconsistency with NumPy for `axis=None` ([dask#8686](https://github.com/dask/dask/pull/8686)) [Tom White](https://github.com/tomwhite)
- Type annotations, part 1 ([dask#8295](https://github.com/dask/dask/pull/8295)) [crusaderky](https://github.com/crusaderky)
- Really allow any iterable to be passed as a `meta` ([dask#8629](https://github.com/dask/dask/pull/8629)) [Julia Signell](https://github.com/jsignell)
- Use `map_partitions` (Blockwise) in `to_parquet` ([dask#8487](https://github.com/dask/dask/pull/8487)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Bug Fixes

- Result of reducing an array should not depend on its chunk-structure ([dask#8637](https://github.com/dask/dask/pull/8637)) [ParticularMiner](https://github.com/ParticularMiner)
- Pass place-holder metadata to `map_partitions` in ACA code path ([dask#8643](https://github.com/dask/dask/pull/8643)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Deprecations

- Deprecate `is_monotonic` ([dask#8653](https://github.com/dask/dask/pull/8653)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove some deprecations ([dask#8605](https://github.com/dask/dask/pull/8605)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Add Domino Data Lab to Hosted / managed Dask clusters ([dask#8675](https://github.com/dask/dask/pull/8675)) [Ray Bell](https://github.com/raybellwaves)
- Fix inter-linking and remove deprecated function ([dask#8715](https://github.com/dask/dask/pull/8715)) [Julia Signell](https://github.com/jsignell)
- Fix imbalanced backticks. ([dask#8693](https://github.com/dask/dask/pull/8693)) [Matthias Bussonnier](https://github.com/Carreau)
- Add documentation for high level graph visualization ([dask#8483](https://github.com/dask/dask/pull/8483)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Update documentation of `ProgressBar` `out` parameter ([dask#8604](https://github.com/dask/dask/pull/8604)) [Pedro Silva](https://github.com/ppsbs)
- Improve documentation of `dask.config.set` ([dask#8705](https://github.com/dask/dask/pull/8705)) [crusaderky](https://github.com/crusaderky)
- Revert mention to `mypy` among type checkers ([dask#8699](https://github.com/dask/dask/pull/8699)) [crusaderky](https://github.com/crusaderky)

### Maintenance

- Update warning handling in `get_dummies` tests ([dask#8651](https://github.com/dask/dask/pull/8651)) [James Bourbeau](https://github.com/jrbourbeau)
- Add a github changelog template ([dask#8714](https://github.com/dask/dask/pull/8714)) [Julia Signell](https://github.com/jsignell)
- Update year in LICENSE.txt ([dask#8665](https://github.com/dask/dask/pull/8665)) [David Hoese](https://github.com/djhoese)
- Update `pre-commit` version  ([dask#8691](https://github.com/dask/dask/pull/8691)) [James Bourbeau](https://github.com/jrbourbeau)
- Include `scipy` in upstream CI build ([dask#8681](https://github.com/dask/dask/pull/8681)) [James Bourbeau](https://github.com/jrbourbeau)
- Temporarily pin `scipy < 1.8.0` in CI ([dask#8683](https://github.com/dask/dask/pull/8683)) [James Bourbeau](https://github.com/jrbourbeau)
- Pin `scipy` to less than 1.8.0 in GPU CI ([dask#8698](https://github.com/dask/dask/pull/8698)) [Julia Signell](https://github.com/jsignell)
- Avoid `pytest.warns(None)` in `test_multi.py` ([dask#8678](https://github.com/dask/dask/pull/8678)) [James Bourbeau](https://github.com/jrbourbeau)
- Update GHA concurrent job cancellation ([dask#8652](https://github.com/dask/dask/pull/8652)) [James Bourbeau](https://github.com/jrbourbeau)
- Make `test__get_paths` robust to `site.PREFIXES` being set ([dask#8644](https://github.com/dask/dask/pull/8644)) [James Bourbeau](https://github.com/jrbourbeau)
- Bump gpuCI PYTHON_VER to 3.9 ([dask#8642](https://github.com/dask/dask/pull/8642)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2022-01-1"></a>

## 2022.01.1

Released on January 28, 2022

### New Features

- Add `dask.dataframe.series.view()` ([dask#8533](https://github.com/dask/dask/pull/8533)) [Pavithra Eswaramoorthy](https://github.com/pavithraes)

### Enhancements

- Update `tz` for `fastparquet` + `pandas` 1.4.0 ([dask#8626](https://github.com/dask/dask/pull/8626)) [Martin Durant](https://github.com/martindurant)
- Cleaning up misc tests for `pandas` compat ([dask#8623](https://github.com/dask/dask/pull/8623)) [Julia Signell](https://github.com/jsignell)
- Moving to `SQLAlchemy >= 1.4` ([dask#8158](https://github.com/dask/dask/pull/8158)) [McToel](https://github.com/McToel)
- Pandas compat: Filter sparse warnings ([dask#8621](https://github.com/dask/dask/pull/8621)) [Julia Signell](https://github.com/jsignell)
- Fail if `meta` is not a `pandas` object ([dask#8563](https://github.com/dask/dask/pull/8563)) [Julia Signell](https://github.com/jsignell)
- Use `fsspec.parquet` module for better remote-storage `read_parquet` performance ([dask#8339](https://github.com/dask/dask/pull/8339)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Move DataFrame ACA aggregations to HLG ([dask#8468](https://github.com/dask/dask/pull/8468)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add optional information about originating function call in `DataFrameIOLayer` ([dask#8453](https://github.com/dask/dask/pull/8453)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Blockwise array creation redux ([dask#7417](https://github.com/dask/dask/pull/7417)) [Ian Rose](https://github.com/ian-r-rose)
- Refactor config default search path retrieval ([dask#8573](https://github.com/dask/dask/pull/8573)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `optimize_graph` flag to `Bag.to_dataframe` function ([dask#8486](https://github.com/dask/dask/pull/8486)) [Maxim Lippeveld](https://github.com/MaximLippeveld)
- Make sure that delayed output operations still return lists of paths ([dask#8498](https://github.com/dask/dask/pull/8498)) [Julia Signell](https://github.com/jsignell)
- Pandas compat: Fix `to_frame` `name` to not pass `None` ([dask#8554](https://github.com/dask/dask/pull/8554)) [Julia Signell](https://github.com/jsignell)
- Pandas compat: Fix `axis=None` warning ([dask#8555](https://github.com/dask/dask/pull/8555)) [Julia Signell](https://github.com/jsignell)
- Expand Dask YAML config search directories ([dask#8531](https://github.com/dask/dask/pull/8531)) [abergou](https://github.com/abergou)

### Bug Fixes

- Fix `groupby.cumsum` with series grouped by index ([dask#8588](https://github.com/dask/dask/pull/8588)) [Julia Signell](https://github.com/jsignell)
- Fix `derived_from` for `pandas` methods ([dask#8612](https://github.com/dask/dask/pull/8612)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Enforce boolean `ascending` for `sort_values` ([dask#8440](https://github.com/dask/dask/pull/8440)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix parsing of `__setitem__` indices ([dask#8601](https://github.com/dask/dask/pull/8601)) [David Hassell](https://github.com/davidhassell)
- Avoid divide by zero in slicing  ([dask#8597](https://github.com/dask/dask/pull/8597)) [Doug Davis](https://github.com/douglasdavis)

### Deprecations

- Downgrade `meta` error in ([dask#8563](https://github.com/dask/dask/pull/8563)) to warning ([dask#8628](https://github.com/dask/dask/pull/8628)) [Julia Signell](https://github.com/jsignell)
- Pandas compat: Deprecate `append` when `pandas >= 1.4.0` ([dask#8617](https://github.com/dask/dask/pull/8617)) [Julia Signell](https://github.com/jsignell)

### Documentation

- Replace outdated `columns` argument with `meta` in DataFrame constructor ([dask#8614](https://github.com/dask/dask/pull/8614)) [kori73](https://github.com/kori73)
- Refactor deploying docs ([dask#8602](https://github.com/dask/dask/pull/8602)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

### Maintenance

- Pin `coverage` in CI ([dask#8631](https://github.com/dask/dask/pull/8631)) [James Bourbeau](https://github.com/jrbourbeau)
- Move `cached_cumsum` imports to be from `dask.utils` ([dask#8606](https://github.com/dask/dask/pull/8606)) [James Bourbeau](https://github.com/jrbourbeau)
- Update gpuCI `RAPIDS_VER` to `22.04` ([dask#8600](https://github.com/dask/dask/pull/8600))
- Update cocstring for `from_delayed` function  ([dask#8576](https://github.com/dask/dask/pull/8576)) [Kirito1397](https://github.com/Kirito1397)
- Handle `plot_width` / `plot_height` deprecations ([dask#8544](https://github.com/dask/dask/pull/8544)) [Bryan Van de Ven](https://github.com/bryevdv)
- Remove unnecessary `pyyaml` `importorskip` ([dask#8562](https://github.com/dask/dask/pull/8562)) [James Bourbeau](https://github.com/jrbourbeau)
- Specify scheduler in DataFrame `assert_eq` ([dask#8559](https://github.com/dask/dask/pull/8559)) [Gabe Joseph](https://github.com/gjoseph92)

<a id="v2022-01-0"></a>

## 2022.01.0

Released on January 14, 2022

### New Features

- Add `groupby.shift` method ([dask#8522](https://github.com/dask/dask/pull/8522)) [kori73](https://github.com/kori73)
- Add `DataFrame.nunique` ([dask#8479](https://github.com/dask/dask/pull/8479)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Add `da.ndim` to match `np.ndim` ([dask#8502](https://github.com/dask/dask/pull/8502)) [Julia Signell](https://github.com/jsignell)

### Enhancements

- Only show `percentile` `interpolation=` keyword warning if NumPy version >= 1.22 ([dask#8564](https://github.com/dask/dask/pull/8564)) [Julia Signell](https://github.com/jsignell)
- Raise `PerformanceWarning` when `limit` and `"array.slicing.split-large-chunks"` are `None` ([dask#8511](https://github.com/dask/dask/pull/8511)) [Julia Signell](https://github.com/jsignell)
- Define `normalize_seq` function at import time ([dask#8521](https://github.com/dask/dask/pull/8521)) [Illviljan](https://github.com/Illviljan)
- Ensure that divisions are alway tuples ([dask#8393](https://github.com/dask/dask/pull/8393)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Allow a callable scheduler for `bag.groupby` ([dask#8492](https://github.com/dask/dask/pull/8492)) [Julia Signell](https://github.com/jsignell)
- Save Zarr arrays with dask-on-ray scheduler ([dask#8472](https://github.com/dask/dask/pull/8472)) [TnTo](https://github.com/TnTo)
- Make byte blocks more even in `read_bytes` ([dask#8459](https://github.com/dask/dask/pull/8459)) [Martin Durant](https://github.com/martindurant)
- Improved the efficiency of `matmul()` by completely removing concatenation ([dask#8423](https://github.com/dask/dask/pull/8423)) [ParticularMiner](https://github.com/ParticularMiner)
- Limit max chunk size when reshaping dask arrays ([dask#8124](https://github.com/dask/dask/pull/8124)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Changes for fastparquet superthrift ([dask#8470](https://github.com/dask/dask/pull/8470)) [Martin Durant](https://github.com/martindurant)

### Bug Fixes

- Fix boolean indices in array assignment ([dask#8538](https://github.com/dask/dask/pull/8538)) [David Hassell](https://github.com/davidhassell)
- Detect default `dtype` on array-likes ([dask#8501](https://github.com/dask/dask/pull/8501)) [aeisenbarth](https://github.com/aeisenbarth)
- Fix `optimize_blockwise` bug for duplicate dependency names ([dask#8542](https://github.com/dask/dask/pull/8542)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update warnings for `DataFrame.GroupBy.apply` and transform ([dask#8507](https://github.com/dask/dask/pull/8507)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Track HLG layer name in `Delayed` ([dask#8452](https://github.com/dask/dask/pull/8452)) [Gabe Joseph](https://github.com/gjoseph92)
- Fix single item `nanmin` and `nanmax` reductions ([dask#8484](https://github.com/dask/dask/pull/8484)) [Julia Signell](https://github.com/jsignell)
- Make `read_csv` with `comment` `kwarg` work even if there is a comment in the header ([dask#8433](https://github.com/dask/dask/pull/8433)) [Julia Signell](https://github.com/jsignell)

### Deprecations

- Replace `interpolation` with `method` and `method` with `internal_method` ([dask#8525](https://github.com/dask/dask/pull/8525)) [Julia Signell](https://github.com/jsignell)
- Remove daily stock demo utility ([dask#8477](https://github.com/dask/dask/pull/8477)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Add a join example in docs that be run with copy/paste ([dask#8520](https://github.com/dask/dask/pull/8520)) [kori73](https://github.com/kori73)
- Mention dashboard link in config ([dask#8510](https://github.com/dask/dask/pull/8510)) [Ray Bell](https://github.com/raybellwaves)
- Fix changelog section hyperlinks ([dask#8534](https://github.com/dask/dask/pull/8534)) [Aneesh Nema](https://github.com/aneeshnema)
- Hyphenate “single-machine scheduler” for consistency ([dask#8519](https://github.com/dask/dask/pull/8519)) [Deepyaman Datta](https://github.com/deepyaman)
- Normalize whitespace in doctests in `slicing.py` ([dask#8512](https://github.com/dask/dask/pull/8512)) [Maren Westermann](https://github.com/marenwestermann)
- Best practices storage line typo ([dask#8529](https://github.com/dask/dask/pull/8529)) [Michael Delgado](https://github.com/delgadom)
- Update figures ([dask#8401](https://github.com/dask/dask/pull/8401)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Remove `pyarrow`-only reference from `split_row_groups` in `read_parquet` docstring ([dask#8490](https://github.com/dask/dask/pull/8490)) [Naty Clementi](https://github.com/ncclementi)

### Maintenance

- Remove obsolete `LocalFileSystem` tests that fail for `fsspec>=2022.1.0` ([dask#8565](https://github.com/dask/dask/pull/8565)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Tweak: “RuntimeWarning: invalid value encountered in reciprocal” ([dask#8561](https://github.com/dask/dask/pull/8561)) [crusaderky](https://github.com/crusaderky)
- Fix `skipna=None` for `DataFrame.sem` ([dask#8556](https://github.com/dask/dask/pull/8556)) [Julia Signell](https://github.com/jsignell)
- Fix `PANDAS_GT_140` ([dask#8552](https://github.com/dask/dask/pull/8552)) [Julia Signell](https://github.com/jsignell)
- Collections with HLG must always implement `__dask_layers__` ([dask#8548](https://github.com/dask/dask/pull/8548)) [crusaderky](https://github.com/crusaderky)
- Work around race condition in `import llvmlite` ([dask#8550](https://github.com/dask/dask/pull/8550)) [crusaderky](https://github.com/crusaderky)
- Set a minimum version for `pyyaml` ([dask#8545](https://github.com/dask/dask/pull/8545)) [Gaurav Sheni](https://github.com/gsheni)
- Adding `nodefaults` to environments to fix `tiledb` + mac issue ([dask#8505](https://github.com/dask/dask/pull/8505)) [Julia Signell](https://github.com/jsignell)
- Set ceiling for `setuptools` ([dask#8509](https://github.com/dask/dask/pull/8509)) [Julia Signell](https://github.com/jsignell)
- Add workflow / recipe to generate Dask nightlies ([dask#8469](https://github.com/dask/dask/pull/8469)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Bump gpuCI `CUDA_VER` to 11.5 ([dask#8489](https://github.com/dask/dask/pull/8489)) [Charles Blackmon-Luca](https://github.com/charlesbluca)

<a id="v2021-12-0"></a>

## 2021.12.0

Released on December 10, 2021

### New Features

- Add `Series` and `Index` `is_monotonic*` methods ([dask#8304](https://github.com/dask/dask/pull/8304)) [Daniel Mesejo-León](https://github.com/mesejo)

### Enhancements

- Blockwise `map_partitions` with `partition_info` ([dask#8310](https://github.com/dask/dask/pull/8310)) [Gabe Joseph](https://github.com/gjoseph92)
- Better error message for length of array with unknown chunk sizes ([dask#8436](https://github.com/dask/dask/pull/8436)) [Doug Davis](https://github.com/douglasdavis)
- Use `by` instead of `index` internally on the Groupby class ([dask#8441](https://github.com/dask/dask/pull/8441)) [Julia Signell](https://github.com/jsignell)
- Allow custom sort functions for `sort_values` ([dask#8345](https://github.com/dask/dask/pull/8345)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add warning to `read_parquet` when statistics and partitions are misaligned ([dask#8416](https://github.com/dask/dask/pull/8416)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support `where` argument in ufuncs ([dask#8253](https://github.com/dask/dask/pull/8253)) [mihir](https://github.com/ek234)
- Make visualize more consistent with compute ([dask#8328](https://github.com/dask/dask/pull/8328)) [JSKenyon](https://github.com/jskenyon)

### Bug Fixes

- Fix `map_blocks` not using own arguments in `name` generation ([dask#8462](https://github.com/dask/dask/pull/8462)) [David Hoese](https://github.com/djhoese)
- Fix for index error with reading empty parquet file ([dask#8410](https://github.com/dask/dask/pull/8410)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Fix nullable-dtype error when writing partitioned parquet data ([dask#8400](https://github.com/dask/dask/pull/8400)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix CSV header bug ([dask#8413](https://github.com/dask/dask/pull/8413)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix empty chunk causes exception in `nanmin`/`nanmax` ([dask#8375](https://github.com/dask/dask/pull/8375)) [Boaz Mohar](https://github.com/boazmohar)

### Deprecations

- Deprecate `token` keyword argument to `map_blocks` ([dask#8464](https://github.com/dask/dask/pull/8464)) [James Bourbeau](https://github.com/jrbourbeau)
- Deprecation warning for default value of boundary kwarg in `map_overlap` ([dask#8397](https://github.com/dask/dask/pull/8397)) [Genevieve Buckley](https://github.com/GenevieveBuckley)

### Documentation

- Clarify `block_info` documentation ([dask#8425](https://github.com/dask/dask/pull/8425)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Output from alt text sprint ([dask#8456](https://github.com/dask/dask/pull/8456)) [Sarah Charlotte Johnson](https://github.com/scharlottej13)
- Update talks and presentations ([dask#8370](https://github.com/dask/dask/pull/8370)) [Naty Clementi](https://github.com/ncclementi)
- Update Anaconda link in “Paid support” section of docs ([dask#8427](https://github.com/dask/dask/pull/8427)) [Martin Durant](https://github.com/martindurant)
- Fixed broken `dask-gateway` link in `ecosystem.rst` ([dask#8424](https://github.com/dask/dask/pull/8424)) [ofirr](https://github.com/ofirr)
- Fix CuPy doctest error ([dask#8412](https://github.com/dask/dask/pull/8412)) [Genevieve Buckley](https://github.com/GenevieveBuckley)

### Maintenance

- Bump Bokeh min version to 2.1.1 ([dask#8431](https://github.com/dask/dask/pull/8431)) [Bryan Van de Ven](https://github.com/bryevdv)
- Fix following `fsspec=2021.11.1` release ([dask#8428](https://github.com/dask/dask/pull/8428)) [Martin Durant](https://github.com/martindurant)
- Add `dask/ml.py` to pytest exclude list ([dask#8414](https://github.com/dask/dask/pull/8414)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Update gpuCI `RAPIDS_VER` to `22.02` ([dask#8394](https://github.com/dask/dask/pull/8394))
- Unpin `graphviz`  and improve package management in environment-3.7 ([dask#8411](https://github.com/dask/dask/pull/8411)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-11-2"></a>

## 2021.11.2

Released on November 19, 2021

- Only run gpuCI bump script daily ([dask#8404](https://github.com/dask/dask/pull/8404)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Actually ignore index when asked in `assert_eq` ([dask#8396](https://github.com/dask/dask/pull/8396)) [Gabe Joseph](https://github.com/gjoseph92)
- Ensure single-partition join `divisions` is `tuple` ([dask#8389](https://github.com/dask/dask/pull/8389)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Try to make divisions behavior clearer ([dask#8379](https://github.com/dask/dask/pull/8379)) [Julia Signell](https://github.com/jsignell)
- Fix typo in `set_index` `partition_size` parameter description ([dask#8384](https://github.com/dask/dask/pull/8384)) [FredericOdermatt](https://github.com/FredericOdermatt)
- Use `blockwise` in `single_partition_join` ([dask#8341](https://github.com/dask/dask/pull/8341)) [Gabe Joseph](https://github.com/gjoseph92)
- Use more explicit keyword arguments ([dask#8354](https://github.com/dask/dask/pull/8354)) [Boaz Mohar](https://github.com/boazmohar)
- Fix `.loc` of DataFrame with nullable boolean `dtype` ([dask#8368](https://github.com/dask/dask/pull/8368)) [Marco Rossi](https://github.com/m-rossi)
- Parameterize shuffle implementation in tests ([dask#8250](https://github.com/dask/dask/pull/8250)) [Ian Rose](https://github.com/ian-r-rose)
- Remove some doc build warnings ([dask#8369](https://github.com/dask/dask/pull/8369)) [Boaz Mohar](https://github.com/boazmohar)
- Include properties in array API docs ([dask#8356](https://github.com/dask/dask/pull/8356)) [Julia Signell](https://github.com/jsignell)
- Fix Zarr for upstream ([dask#8367](https://github.com/dask/dask/pull/8367)) [Julia Signell](https://github.com/jsignell)
- Pin `graphviz` to avoid issue with windows and Python 3.7 ([dask#8365](https://github.com/dask/dask/pull/8365)) [Julia Signell](https://github.com/jsignell)
- Import `graphviz.Diagraph` from top of module, not from `dot` ([dask#8363](https://github.com/dask/dask/pull/8363)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-11-1"></a>

## 2021.11.1

Released on November 8, 2021

Patch release to update `distributed` dependency to version `2021.11.1`.

<a id="v2021-11-0"></a>

## 2021.11.0

Released on November 5, 2021

- Fx `required_extension` behavior in `read_parquet` ([dask#8351](https://github.com/dask/dask/pull/8351)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `align_dataframes` to `map_partitions` to broadcast a dataframe passed as an arg ([dask#6628](https://github.com/dask/dask/pull/6628)) [Julia Signell](https://github.com/jsignell)
- Better handling for arrays/series of keys in `dask.dataframe.loc` ([dask#8254](https://github.com/dask/dask/pull/8254)) [Julia Signell](https://github.com/jsignell)
- Point users to Discourse ([dask#8332](https://github.com/dask/dask/pull/8332)) [Ian Rose](https://github.com/ian-r-rose)
- Add `name_function` option to `to_parquet` ([dask#7682](https://github.com/dask/dask/pull/7682)) [Matthew Powers](https://github.com/MrPowers)
- Get rid of `environment-latest.yml` and update to Python 3.9 ([dask#8275](https://github.com/dask/dask/pull/8275)) [Julia Signell](https://github.com/jsignell)
- Require newer `s3fs` in CI ([dask#8336](https://github.com/dask/dask/pull/8336)) [James Bourbeau](https://github.com/jrbourbeau)
- Groupby Rolling ([dask#8176](https://github.com/dask/dask/pull/8176)) [Julia Signell](https://github.com/jsignell)
- Add more ordering diagnostics to `dask.visualize` ([dask#7992](https://github.com/dask/dask/pull/7992)) [Erik Welch](https://github.com/eriknw)
- Use `HighLevelGraph` optimizations for `delayed` ([dask#8316](https://github.com/dask/dask/pull/8316)) [Ian Rose](https://github.com/ian-r-rose)
- `demo_tuples` produces malformed `HighLevelGraph` ([dask#8325](https://github.com/dask/dask/pull/8325)) [crusaderky](https://github.com/crusaderky)
- Dask calendar should show events in local time ([dask#8312](https://github.com/dask/dask/pull/8312)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Fix flaky `test_interrupt` ([dask#8314](https://github.com/dask/dask/pull/8314)) [crusaderky](https://github.com/crusaderky)
- Deprecate `AxisError` ([dask#8305](https://github.com/dask/dask/pull/8305)) [crusaderky](https://github.com/crusaderky)
- Fix name of cuDF in extension documentation. ([dask#8311](https://github.com/dask/dask/pull/8311)) [Vyas Ramasubramani](https://github.com/vyasr)
- Add single eq operator (=) to parquet filters  ([dask#8300](https://github.com/dask/dask/pull/8300)) [Ayush Dattagupta](https://github.com/ayushdg)
- Improve support for Spark output in `read_parquet` ([dask#8274](https://github.com/dask/dask/pull/8274)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `dask.ml` module ([dask#6384](https://github.com/dask/dask/pull/6384)) [Matthew Rocklin](https://github.com/mrocklin)
- CI fixups ([dask#8298](https://github.com/dask/dask/pull/8298)) [James Bourbeau](https://github.com/jrbourbeau)
- Make slice errors match NumPy ([dask#8248](https://github.com/dask/dask/pull/8248)) [Julia Signell](https://github.com/jsignell)
- Fix API docs misrendering with new sphinx theme ([dask#8296](https://github.com/dask/dask/pull/8296)) [Julia Signell](https://github.com/jsignell)
- Replace `block` property with `blockview` for array-like operations on blocks ([dask#8242](https://github.com/dask/dask/pull/8242)) [Davis Bennett](https://github.com/d-v-b)
- Deprecate `file_path` and make it possible to save  from within a notebook ([dask#8283](https://github.com/dask/dask/pull/8283)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-10-0"></a>

## 2021.10.0

Released on October 22, 2021

- `da.store` to create well-formed `HighLevelGraph` ([dask#8261](https://github.com/dask/dask/pull/8261)) [crusaderky](https://github.com/crusaderky)
- CI: force nightly `pyarrow` in the upstream build ([dask#8281](https://github.com/dask/dask/pull/8281)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Remove `chest` ([dask#8279](https://github.com/dask/dask/pull/8279)) [James Bourbeau](https://github.com/jrbourbeau)
- Skip doctests if optional dependencies are not installed ([dask#8258](https://github.com/dask/dask/pull/8258)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Update `tmpdir` and `tmpfile` context manager docstrings ([dask#8270](https://github.com/dask/dask/pull/8270)) [Daniel Mesejo-León](https://github.com/mesejo)
- Unregister callbacks in doctests ([dask#8276](https://github.com/dask/dask/pull/8276)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix typo in docs ([dask#8277](https://github.com/dask/dask/pull/8277)) [JoranDox](https://github.com/JoranDox)
- Stale label GitHub action ([dask#8244](https://github.com/dask/dask/pull/8244)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Client-shutdown method appears twice ([dask#8273](https://github.com/dask/dask/pull/8273)) [German Shiklov](https://github.com/Jeremaiha-xmetix)
- Add pre-commit to test requirements ([dask#8257](https://github.com/dask/dask/pull/8257)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Refactor `read_metadata` in `fastparquet` engine ([dask#8092](https://github.com/dask/dask/pull/8092)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support `Path` objects in `from_zarr` ([dask#8266](https://github.com/dask/dask/pull/8266)) [Samuel Gaist](https://github.com/sgaist)
- Make nested redirects work ([dask#8272](https://github.com/dask/dask/pull/8272)) [Julia Signell](https://github.com/jsignell)
- Set `memory_usage` to `True` if `verbose` is `True` in info ([dask#8222](https://github.com/dask/dask/pull/8222)) [Kinshuk Dua](https://github.com/kinshukdua)
- Remove individual API doc pages from sphinx toctree ([dask#8238](https://github.com/dask/dask/pull/8238)) [James Bourbeau](https://github.com/jrbourbeau)
- Ignore whitespace in gufunc `signature` ([dask#8267](https://github.com/dask/dask/pull/8267)) [James Bourbeau](https://github.com/jrbourbeau)
- Add workflow to update gpuCI ([dask#8215](https://github.com/dask/dask/pull/8215)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- `DataFrame.head` shouldn’t warn when there’s one partition ([dask#8091](https://github.com/dask/dask/pull/8091)) [Pankaj Patil](https://github.com/Patil2099)
- Ignore arrow doctests if `pyarrow` not installed ([dask#8256](https://github.com/dask/dask/pull/8256)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Fix `debugging.html` redirect ([dask#8251](https://github.com/dask/dask/pull/8251)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix null sorting for single partition dataframes ([dask#8225](https://github.com/dask/dask/pull/8225)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Fix `setup.html` redirect ([dask#8249](https://github.com/dask/dask/pull/8249)) [Florian Jetter](https://github.com/fjetter)
- Run `pyupgrade` in CI ([dask#8246](https://github.com/dask/dask/pull/8246)) [crusaderky](https://github.com/crusaderky)
- Fix label typo in upstream CI build ([dask#8237](https://github.com/dask/dask/pull/8237)) [James Bourbeau](https://github.com/jrbourbeau)
- Add support for “dependent” columns in DataFrame.assign ([dask#8086](https://github.com/dask/dask/pull/8086)) [Suriya Senthilkumar](https://github.com/suriya-it19)
- add NumPy array of Dask keys to `Array` ([dask#7922](https://github.com/dask/dask/pull/7922)) [Davis Bennett](https://github.com/d-v-b)
- Remove unnecessary `dask.multiprocessing` import in docs ([dask#8240](https://github.com/dask/dask/pull/8240)) [Ray Bell](https://github.com/raybellwaves)
- Adjust retrieving `_max_workers` from `Executor` ([dask#8228](https://github.com/dask/dask/pull/8228)) [John A Kirkham](https://github.com/jakirkham)
- Update function signatures in `delayed` best practices docs ([dask#8231](https://github.com/dask/dask/pull/8231)) [Vũ Trung Đức](https://github.com/vutrungduc7593)
- Docs reoganization ([dask#7984](https://github.com/dask/dask/pull/7984)) [Julia Signell](https://github.com/jsignell)
- Fix `df.quantile` on all missing data ([dask#8129](https://github.com/dask/dask/pull/8129)) [Julia Signell](https://github.com/jsignell)
- Add `tokenize.ensure-deterministic` config option ([dask#7413](https://github.com/dask/dask/pull/7413)) [Hristo Georgiev](https://github.com/hristog)
- Use `inclusive` rather than `closed` with `pandas>=1.4.0` and `pd.date_range` ([dask#8213](https://github.com/dask/dask/pull/8213)) [Julia Signell](https://github.com/jsignell)
- Add `dask-gateway`, Coiled, and Saturn-Cloud to list of Dask setup tools ([dask#7814](https://github.com/dask/dask/pull/7814)) [Kristopher Overholt](https://github.com/koverholt)
- Ensure existing futures get passed as deps when serializing `HighLevelGraph` layers ([dask#8199](https://github.com/dask/dask/pull/8199)) [Jim Crist-Harif](https://github.com/jcrist)
- Make sure that the divisions of the single partition merge is left ([dask#8162](https://github.com/dask/dask/pull/8162)) [Julia Signell](https://github.com/jsignell)
- Refactor `read_metadata` in `pyarrow` parquet engines ([dask#8072](https://github.com/dask/dask/pull/8072)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support negative `drop_axis` in `map_blocks` and `map_overlap` ([dask#8192](https://github.com/dask/dask/pull/8192)) [Gregory R. Lee](https://github.com/grlee77)
- Fix upstream tests ([dask#8205](https://github.com/dask/dask/pull/8205)) [Julia Signell](https://github.com/jsignell)
- Add support for scalar item assignment by Series ([dask#8195](https://github.com/dask/dask/pull/8195)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add some basic examples to doc strings on `dask.bag` `all`, `any`, `count` methods ([dask#7630](https://github.com/dask/dask/pull/7630)) [Nathan Danielsen](https://github.com/ndanielsen)
- Don’t have upstream report depend on commit message ([dask#8202](https://github.com/dask/dask/pull/8202)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure upstream CI cron job runs ([dask#8200](https://github.com/dask/dask/pull/8200)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `pytest.param` to properly label param-specific GPU tests ([dask#8197](https://github.com/dask/dask/pull/8197)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add `test_set_index` to tests ran on gpuCI ([dask#8198](https://github.com/dask/dask/pull/8198)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Suppress `tmpfile` OSError ([dask#8191](https://github.com/dask/dask/pull/8191)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `s.isna` instead of `pd.isna(s)` in `set_partitions_pre` (fix cudf CI) ([dask#8193](https://github.com/dask/dask/pull/8193)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Open an issue for `test-upstream` failures ([dask#8067](https://github.com/dask/dask/pull/8067)) [Wallace Reis](https://github.com/wreis)
- Fix `to_parquet` bug in call to `pyarrow.parquet.read_metadata` ([dask#8186](https://github.com/dask/dask/pull/8186)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add handling for null values in `sort_values` ([dask#8167](https://github.com/dask/dask/pull/8167)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Bump `RAPIDS_VER` for gpuCI ([dask#8184](https://github.com/dask/dask/pull/8184)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Dispatch walks MRO for lazily registered handlers ([dask#8185](https://github.com/dask/dask/pull/8185)) [Jim Crist-Harif](https://github.com/jcrist)
- Configure SSHCluster instructions ([dask#8181](https://github.com/dask/dask/pull/8181)) [Ray Bell](https://github.com/raybellwaves)
- Preserve `HighLevelGraphs` in `DataFrame.from_delayed` ([dask#8174](https://github.com/dask/dask/pull/8174)) [Gabe Joseph](https://github.com/gjoseph92)
- Deprecate `inplace` argument for Dask series renaming ([dask#8136](https://github.com/dask/dask/pull/8136)) [Marcel Coetzee](https://github.com/marcelned)
- Fix rolling for compatibility with `pandas > 1.3.0` ([dask#8150](https://github.com/dask/dask/pull/8150)) [Julia Signell](https://github.com/jsignell)
- Raise error when `setitem` on unknown chunks ([dask#8166](https://github.com/dask/dask/pull/8166)) [Julia Signell](https://github.com/jsignell)
- Include divisions when doing `Index.to_series` ([dask#8165](https://github.com/dask/dask/pull/8165)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-09-1"></a>

## 2021.09.1

Released on September 21, 2021

- Fix `groupby` for future pandas ([dask#8151](https://github.com/dask/dask/pull/8151)) [Julia Signell](https://github.com/jsignell)
- Remove warning filters in tests that are no longer needed ([dask#8155](https://github.com/dask/dask/pull/8155)) [Julia Signell](https://github.com/jsignell)
- Add link to diagnostic visualize function in local diagnostic docs ([dask#8157](https://github.com/dask/dask/pull/8157)) [David Hoese](https://github.com/djhoese)
- Add `datetime_is_numeric` to `dataframe.describe` ([dask#7719](https://github.com/dask/dask/pull/7719)) [Julia Signell](https://github.com/jsignell)
- Remove references to `pd.Int64Index` in anticipation of deprecation ([dask#8144](https://github.com/dask/dask/pull/8144)) [Julia Signell](https://github.com/jsignell)
- Use `loc` if needed for series `__get_item__` ([dask#7953](https://github.com/dask/dask/pull/7953)) [Julia Signell](https://github.com/jsignell)
- Specifically ignore warnings on mean for empty slices ([dask#8125](https://github.com/dask/dask/pull/8125)) [Julia Signell](https://github.com/jsignell)
- Skip `groupby` `nunique` test for pandas >= 1.3.3 ([dask#8142](https://github.com/dask/dask/pull/8142)) [Julia Signell](https://github.com/jsignell)
- Implement `ascending` arg for `sort_values` ([dask#8130](https://github.com/dask/dask/pull/8130)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Replace `operator.getitem`  ([dask#8015](https://github.com/dask/dask/pull/8015)) [Naty Clementi](https://github.com/ncclementi)
- Deprecate `zero_broadcast_dimensions` and `homogeneous_deepmap` ([dask#8134](https://github.com/dask/dask/pull/8134)) [SnkSynthesis](https://github.com/SnkSynthesis)
- Add error if `drop_index` is negative ([dask#8064](https://github.com/dask/dask/pull/8064)) [neel iyer](https://github.com/spiyer99)
- Allow `scheduler` to be an `Executor` ([dask#8112](https://github.com/dask/dask/pull/8112)) [John A Kirkham](https://github.com/jakirkham)
- Handle `asarray`/`asanyarray` cases where `like` is a `dask.Array` ([dask#8128](https://github.com/dask/dask/pull/8128)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix `index_col` duplication if `index_col` is type `str` ([dask#7661](https://github.com/dask/dask/pull/7661)) [McToel](https://github.com/McToel)
- Add `dtype` and `order` to `asarray` and `asanyarray` definitions ([dask#8106](https://github.com/dask/dask/pull/8106)) [Julia Signell](https://github.com/jsignell)
- Deprecate `dask.dataframe.Series.__contains__` ([dask#7914](https://github.com/dask/dask/pull/7914)) [Julia Signell](https://github.com/jsignell)
- Fix edge case with `like`-arrays in `_wrapped_qr` ([dask#8122](https://github.com/dask/dask/pull/8122)) [Peter Andreas Entschev](https://github.com/pentschev)
- Deprecate `boundary_slice` kwarg: `kind` for pandas compat ([dask#8037](https://github.com/dask/dask/pull/8037)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-09-0"></a>

## 2021.09.0

Released on September 3, 2021

- Fewer open files ([dask#7303](https://github.com/dask/dask/pull/7303)) [Julia Signell](https://github.com/jsignell)
- Add `FileNotFound` to expected http errors ([dask#8109](https://github.com/dask/dask/pull/8109)) [Martin Durant](https://github.com/martindurant)
- Add `DataFrame.sort_values` to API docs ([dask#8107](https://github.com/dask/dask/pull/8107)) [Benjamin Zaitlen](https://github.com/quasiben)
- Change to `dask.order`: be more eager at times ([dask#7929](https://github.com/dask/dask/pull/7929)) [Erik Welch](https://github.com/eriknw)
- Add pytest color to CI ([dask#8090](https://github.com/dask/dask/pull/8090)) [James Bourbeau](https://github.com/jrbourbeau)
- FIX: `make_people` works with `processes` scheduler ([dask#8103](https://github.com/dask/dask/pull/8103)) [Dahn](https://github.com/DahnJ)
- Adds `deep` param to Dataframe copy method and restrict it to `False` ([dask#8068](https://github.com/dask/dask/pull/8068)) [João Paulo Lacerda](https://github.com/jopasdev)
- Fix typo in configuration docs ([dask#8104](https://github.com/dask/dask/pull/8104)) [Robert Hales](https://github.com/robalar)
- Update formatting in `DataFrame.query` docstring ([dask#8100](https://github.com/dask/dask/pull/8100)) [James Bourbeau](https://github.com/jrbourbeau)
- Un-xfail `sparse` tests for 0.13.0 release ([dask#8102](https://github.com/dask/dask/pull/8102)) [James Bourbeau](https://github.com/jrbourbeau)
- Add axes property to DataFrame and Series ([dask#8069](https://github.com/dask/dask/pull/8069)) [Jordan Jensen](https://github.com/dotNomad)
- Add CuPy support in `da.unique` (values only) ([dask#8021](https://github.com/dask/dask/pull/8021)) [Peter Andreas Entschev](https://github.com/pentschev)
- Unit tests for `sparse.zeros_like` (xfailed) ([dask#8093](https://github.com/dask/dask/pull/8093)) [crusaderky](https://github.com/crusaderky)
- Add explicit `like` kwarg support to array creation functions ([dask#8054](https://github.com/dask/dask/pull/8054)) [Peter Andreas Entschev](https://github.com/pentschev)
- Separate Array and DataFrame mindeps builds ([dask#8079](https://github.com/dask/dask/pull/8079)) [James Bourbeau](https://github.com/jrbourbeau)
- Fork out `percentile_dispatch` to `dask.array` ([dask#8083](https://github.com/dask/dask/pull/8083)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Ensure `filepath` exists in `to_parquet` ([dask#8057](https://github.com/dask/dask/pull/8057)) [James Bourbeau](https://github.com/jrbourbeau)
- Update scheduler plugin usage in `test_scheduler_highlevel_graph_unpack_import` ([dask#8080](https://github.com/dask/dask/pull/8080)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `DataFrame.shuffle` to API docs ([dask#8076](https://github.com/dask/dask/pull/8076)) [Martin Fleischmann](https://github.com/martinfleis)
- Order requirements alphabetically ([dask#8073](https://github.com/dask/dask/pull/8073)) [John A Kirkham](https://github.com/jakirkham)

<a id="v2021-08-1"></a>

## 2021.08.1

Released on August 20, 2021

- Add `ignore_metadata_file` option to `read_parquet` (`pyarrow-dataset` and `fastparquet` support only) ([dask#8034](https://github.com/dask/dask/pull/8034)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add reference to `pytest-xdist` in dev docs ([dask#8066](https://github.com/dask/dask/pull/8066)) [Julia Signell](https://github.com/jsignell)
- Include `tz` in meta from `to_datetime` ([dask#8000](https://github.com/dask/dask/pull/8000)) [Julia Signell](https://github.com/jsignell)
- CI Infra Docs ([dask#7985](https://github.com/dask/dask/pull/7985)) [Benjamin Zaitlen](https://github.com/quasiben)
- Include invalid DataFrame key in `assert_eq` check ([dask#8061](https://github.com/dask/dask/pull/8061)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `__class__` when creating DataFrames ([dask#8053](https://github.com/dask/dask/pull/8053)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Use development version of `distributed` in gpuCI build ([dask#7976](https://github.com/dask/dask/pull/7976)) [James Bourbeau](https://github.com/jrbourbeau)
- Ignore whitespace when gufunc `signature` ([dask#8049](https://github.com/dask/dask/pull/8049)) [James Bourbeau](https://github.com/jrbourbeau)
- Move pandas import and percentile dispatch refactor ([dask#8055](https://github.com/dask/dask/pull/8055)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Add colors to represent high level layer types ([dask#7974](https://github.com/dask/dask/pull/7974)) [Freyam Mehta](https://github.com/freyam)
- Upstream instance fix ([dask#8060](https://github.com/dask/dask/pull/8060)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add `dask.widgets` and migrate HTML reprs to `jinja2` ([dask#8019](https://github.com/dask/dask/pull/8019)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Remove `wrap_func_like_safe`, not required with NumPy >= 1.17 ([dask#8052](https://github.com/dask/dask/pull/8052)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix threaded scheduler memory backpressure regression ([dask#8040](https://github.com/dask/dask/pull/8040)) [David Hoese](https://github.com/djhoese)
- Add percentile dispatch ([dask#8029](https://github.com/dask/dask/pull/8029)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Use a publicly documented attribute `obj` in `groupby` rather than private `_selected_obj` ([dask#8038](https://github.com/dask/dask/pull/8038)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Specify module to `import rechunk` from ([dask#8039](https://github.com/dask/dask/pull/8039)) [Illviljan](https://github.com/Illviljan)
- Use `dict` to store data for {nan,}arg{min,max} in certain cases ([dask#8014](https://github.com/dask/dask/pull/8014)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix `blocksize` description formatting in `read_pandas` ([dask#8047](https://github.com/dask/dask/pull/8047)) [Louis Maddox](https://github.com/lmmx)
- Fix “point” -> “pointers” typo in docs ([dask#8043](https://github.com/dask/dask/pull/8043)) [David Chudzicki](https://github.com/dchudz)

<a id="v2021-08-0"></a>

## 2021.08.0

Released on August 13, 2021

- Fix `to_orc` delayed compute behavior ([dask#8035](https://github.com/dask/dask/pull/8035)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Don’t convert to low-level task graph in `compute_as_if_collection` ([dask#7969](https://github.com/dask/dask/pull/7969)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix multifile read for hdf ([dask#8033](https://github.com/dask/dask/pull/8033)) [Julia Signell](https://github.com/jsignell)
- Resolve warning in `distributed` tests ([dask#8025](https://github.com/dask/dask/pull/8025)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `to_orc` collection name ([dask#8024](https://github.com/dask/dask/pull/8024)) [James Bourbeau](https://github.com/jrbourbeau)
- Resolve `skipfooter` problem ([dask#7855](https://github.com/dask/dask/pull/7855)) [Ross](https://github.com/rhjmoore)
- Raise `NotImplementedError` for non-indexable arg passed to `to_datetime` ([dask#7989](https://github.com/dask/dask/pull/7989)) [Doug Davis](https://github.com/douglasdavis)
- Ensure we error on warnings from `distributed` ([dask#8002](https://github.com/dask/dask/pull/8002)) [James Bourbeau](https://github.com/jrbourbeau)
- Added `dict` format in `to_bag` accessories of DataFrame ([dask#7932](https://github.com/dask/dask/pull/7932)) [gurunath](https://github.com/rajagurunath)
- Delayed docs indirect dependencies ([dask#8016](https://github.com/dask/dask/pull/8016)) [aa1371](https://github.com/aa1371)
- Add tooltips to graphviz high-level graphs ([dask#7973](https://github.com/dask/dask/pull/7973)) [Freyam Mehta](https://github.com/freyam)
- Close 2021 User Survey ([dask#8007](https://github.com/dask/dask/pull/8007)) [Julia Signell](https://github.com/jsignell)
- Reorganize CuPy tests into multiple files ([dask#8013](https://github.com/dask/dask/pull/8013)) [Peter Andreas Entschev](https://github.com/pentschev)
- Refactor and Expand Dask-Dataframe ORC API ([dask#7756](https://github.com/dask/dask/pull/7756)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Don’t enforce columns if `enforce=False` ([dask#7916](https://github.com/dask/dask/pull/7916)) [Julia Signell](https://github.com/jsignell)
- Fix `map_overlap` trimming behavior when `drop_axis` is not `None` ([dask#7894](https://github.com/dask/dask/pull/7894)) [Gregory R. Lee](https://github.com/grlee77)
- Mark gpuCI CuPy test as flaky ([dask#7994](https://github.com/dask/dask/pull/7994)) [Peter Andreas Entschev](https://github.com/pentschev)
- Avoid using `Delayed` in `to_csv` and `to_parquet` ([dask#7968](https://github.com/dask/dask/pull/7968)) [Matthew Rocklin](https://github.com/mrocklin)
- Removed redundant `check_dtypes` ([dask#7952](https://github.com/dask/dask/pull/7952)) [gurunath](https://github.com/rajagurunath)
- Use `pytest.warns` instead of raises for checking parquet engine deprecation ([dask#7993](https://github.com/dask/dask/pull/7993)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Bump `RAPIDS_VER` in gpuCI to 21.10 ([dask#7991](https://github.com/dask/dask/pull/7991)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add back `pyarrow-legacy` test coverage for `pyarrow>=5` ([dask#7988](https://github.com/dask/dask/pull/7988)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Allow `pyarrow>=5` in `to_parquet` and `read_parquet` ([dask#7967](https://github.com/dask/dask/pull/7967)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Skip CuPy tests requiring NEP-35 when NumPy < 1.20 is available ([dask#7982](https://github.com/dask/dask/pull/7982)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add `tail` and `head` to `SeriesGroupby` ([dask#7935](https://github.com/dask/dask/pull/7935)) [Daniel Mesejo-León](https://github.com/mesejo)
- Update Zoom link for monthly meeting ([dask#7979](https://github.com/dask/dask/pull/7979)) [James Bourbeau](https://github.com/jrbourbeau)
- Add gpuCI build script ([dask#7966](https://github.com/dask/dask/pull/7966)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Deprecate `daily_stock` utility ([dask#7949](https://github.com/dask/dask/pull/7949)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `distributed.nanny` to configuration reference docs ([dask#7955](https://github.com/dask/dask/pull/7955)) [James Bourbeau](https://github.com/jrbourbeau)
- Require NumPy 1.18+ & Pandas 1.0+ ([dask#7939](https://github.com/dask/dask/pull/7939)) [John A Kirkham](https://github.com/jakirkham)

<a id="v2021-07-2"></a>

## 2021.07.2

Released on July 30, 2021

#### NOTE
This is the last release with support for NumPy 1.17 and pandas 0.25.
Beginning with the next release, NumPy 1.18 and pandas 1.0 will be the minimum
supported versions.

- Add `dask.array` SVG to the HTML Repr ([dask#7886](https://github.com/dask/dask/pull/7886)) [Freyam Mehta](https://github.com/freyam)
- Avoid use of `Delayed` in `to_parquet` ([dask#7958](https://github.com/dask/dask/pull/7958)) [Matthew Rocklin](https://github.com/mrocklin)
- Temporarily pin `pyarrow<5` in CI ([dask#7960](https://github.com/dask/dask/pull/7960)) [James Bourbeau](https://github.com/jrbourbeau)
- Add deprecation warning for top-level `ucx` and `rmm` config values ([dask#7956](https://github.com/dask/dask/pull/7956)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove skips from doctests (4 of 6) ([dask#7865](https://github.com/dask/dask/pull/7865)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Remove skips from doctests (5 of 6) ([dask#7864](https://github.com/dask/dask/pull/7864)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Adds missing prepend/append functionality to `da.diff` ([dask#7946](https://github.com/dask/dask/pull/7946)) [Peter Andreas Entschev](https://github.com/pentschev)
- Change graphviz font family to sans ([dask#7931](https://github.com/dask/dask/pull/7931)) [Freyam Mehta](https://github.com/freyam)
- Fix read-csv name - when path is different, use different name for task ([dask#7942](https://github.com/dask/dask/pull/7942)) [Julia Signell](https://github.com/jsignell)
- Update configuration reference for `ucx` and `rmm` changes ([dask#7943](https://github.com/dask/dask/pull/7943)) [James Bourbeau](https://github.com/jrbourbeau)
- Add meta support to `__setitem__` ([dask#7940](https://github.com/dask/dask/pull/7940)) [Peter Andreas Entschev](https://github.com/pentschev)
- NEP-35 support for `slice_with_int_dask_array` ([dask#7927](https://github.com/dask/dask/pull/7927)) [Peter Andreas Entschev](https://github.com/pentschev)
- Unpin fastparquet in CI ([dask#7928](https://github.com/dask/dask/pull/7928)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove skips from doctests (3 of 6) ([dask#7872](https://github.com/dask/dask/pull/7872)) [Zhengnan Zhao](https://github.com/zzhengnan)

<a id="v2021-07-1"></a>

## 2021.07.1

Released on July 23, 2021

- Make array `assert_eq` check dtype ([dask#7903](https://github.com/dask/dask/pull/7903)) [Julia Signell](https://github.com/jsignell)
- Remove skips from doctests (6 of 6) ([dask#7863](https://github.com/dask/dask/pull/7863)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Remove experimental feature warning from actors docs ([dask#7925](https://github.com/dask/dask/pull/7925)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove skips from doctests (2 of 6) ([dask#7873](https://github.com/dask/dask/pull/7873)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Separate out Array and Bag API ([dask#7917](https://github.com/dask/dask/pull/7917)) [Julia Signell](https://github.com/jsignell)
- Implement lazy `Array.__iter__` ([dask#7905](https://github.com/dask/dask/pull/7905)) [Julia Signell](https://github.com/jsignell)
- Clean up places where we inadvertently iterate over arrays ([dask#7913](https://github.com/dask/dask/pull/7913)) [Julia Signell](https://github.com/jsignell)
- Add `numeric_only` kwarg to DataFrame reductions ([dask#7831](https://github.com/dask/dask/pull/7831)) [Julia Signell](https://github.com/jsignell)
- Add pytest marker for GPU tests ([dask#7876](https://github.com/dask/dask/pull/7876)) [Charles Blackmon-Luca](https://github.com/charlesbluca)
- Add support for `histogram2d` in `dask.array` ([dask#7827](https://github.com/dask/dask/pull/7827)) [Doug Davis](https://github.com/douglasdavis)
- Remove skips from doctests (1 of 6) ([dask#7874](https://github.com/dask/dask/pull/7874)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Add node size scaling to the Graphviz output for the high level graphs ([dask#7869](https://github.com/dask/dask/pull/7869)) [Freyam Mehta](https://github.com/freyam)
- Update old Bokeh links ([dask#7915](https://github.com/dask/dask/pull/7915)) [Bryan Van de Ven](https://github.com/bryevdv)
- Temporarily pin `fastparquet` in CI ([dask#7907](https://github.com/dask/dask/pull/7907)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `dask.array` import to progress bar docs ([dask#7910](https://github.com/dask/dask/pull/7910)) [Fabian Gebhart](https://github.com/fgebhart)
- Use separate files for each DataFrame API function and method ([dask#7890](https://github.com/dask/dask/pull/7890)) [Julia Signell](https://github.com/jsignell)
- Fix `pyarrow-dataset` ordering bug ([dask#7902](https://github.com/dask/dask/pull/7902)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Generalize unique aggregate ([dask#7892](https://github.com/dask/dask/pull/7892)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Raise `NotImplementedError` when using `pd.Grouper` ([dask#7857](https://github.com/dask/dask/pull/7857)) [Ruben van de Geer](https://github.com/rubenvdg)
- Add `aggregate_files` argument to enable multi-file partitions in `read_parquet` ([dask#7557](https://github.com/dask/dask/pull/7557)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Un-`xfail` `test_daily_stock` ([dask#7895](https://github.com/dask/dask/pull/7895)) [James Bourbeau](https://github.com/jrbourbeau)
- Update access configuration docs ([dask#7837](https://github.com/dask/dask/pull/7837)) [Naty Clementi](https://github.com/ncclementi)
- Use packaging for version comparisons ([dask#7820](https://github.com/dask/dask/pull/7820)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Handle infinite loops in `merge_asof` ([dask#7842](https://github.com/dask/dask/pull/7842)) [gerrymanoim](https://github.com/gerrymanoim)

<a id="v2021-07-0"></a>

## 2021.07.0

Released on July 9, 2021

- Include `fastparquet` in upstream CI build ([dask#7884](https://github.com/dask/dask/pull/7884)) [James Bourbeau](https://github.com/jrbourbeau)
- Blockwise: handle non-string constant dependencies  ([dask#7849](https://github.com/dask/dask/pull/7849)) [Mads R. B. Kristensen](https://github.com/madsbk)
- `fastparquet` now supports new time types, including ns precision ([dask#7880](https://github.com/dask/dask/pull/7880)) [Martin Durant](https://github.com/martindurant)
- Avoid `ParquetDataset` API when appending in `ArrowDatasetEngine` ([dask#7544](https://github.com/dask/dask/pull/7544)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add retry logic to `test_shuffle_priority` ([dask#7879](https://github.com/dask/dask/pull/7879)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use strict channel priority in CI ([dask#7878](https://github.com/dask/dask/pull/7878)) [James Bourbeau](https://github.com/jrbourbeau)
- Support nested `dask.distributed` imports ([dask#7866](https://github.com/dask/dask/pull/7866)) [Matthew Rocklin](https://github.com/mrocklin)
- Should check module name only, not the entire directory filepath ([dask#7856](https://github.com/dask/dask/pull/7856)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Updates due to [https://github.com/dask/fastparquet/pull/623](https://github.com/dask/fastparquet/pull/623) ([dask#7875](https://github.com/dask/dask/pull/7875)) [Martin Durant](https://github.com/martindurant)
- `da.eye` fix for `chunks=-1` ([dask#7854](https://github.com/dask/dask/pull/7854)) [Naty Clementi](https://github.com/ncclementi)
- Temporarily xfail `test_daily_stock` ([dask#7858](https://github.com/dask/dask/pull/7858)) [James Bourbeau](https://github.com/jrbourbeau)
- Set priority annotations in `SimpleShuffleLayer` ([dask#7846](https://github.com/dask/dask/pull/7846)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Blockwise: stringify constant key inputs ([dask#7838](https://github.com/dask/dask/pull/7838)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Allow mixing dask and numpy arrays in `@guvectorize` ([dask#6863](https://github.com/dask/dask/pull/6863)) [Julia Signell](https://github.com/jsignell)
- Don’t sample dict result of a shuffle group when calculating its size ([dask#7834](https://github.com/dask/dask/pull/7834)) [Florian Jetter](https://github.com/fjetter)
- Fix scipy tests ([dask#7841](https://github.com/dask/dask/pull/7841)) [Julia Signell](https://github.com/jsignell)
- Deterministically tokenize `datetime.date` ([dask#7836](https://github.com/dask/dask/pull/7836)) [James Bourbeau](https://github.com/jrbourbeau)
- Add `sample_rows` to `read_csv`-like ([dask#7825](https://github.com/dask/dask/pull/7825)) [Martin Durant](https://github.com/martindurant)
- Fix typo in `config.deserialize` docstring ([dask#7830](https://github.com/dask/dask/pull/7830)) [Geoffrey Lentner](https://github.com/glentner)
- Remove warning filter in `test_dataframe_picklable` ([dask#7822](https://github.com/dask/dask/pull/7822)) [James Bourbeau](https://github.com/jrbourbeau)
- Improvements to `histogramdd` (for handling inputs that are sequences-of-arrays). ([dask#7634](https://github.com/dask/dask/pull/7634)) [Doug Davis](https://github.com/douglasdavis)
- Make `PY_VERSION` private ([dask#7824](https://github.com/dask/dask/pull/7824)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2021-06-2"></a>

## 2021.06.2

Released on June 22, 2021

- `layers.py` compare `parts_out` with `set(self.parts_out)` ([dask#7787](https://github.com/dask/dask/pull/7787)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Make `check_meta` understand pandas dtypes better ([dask#7813](https://github.com/dask/dask/pull/7813)) [Julia Signell](https://github.com/jsignell)
- Remove “Educational Resources” doc page ([dask#7818](https://github.com/dask/dask/pull/7818)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2021-06-1"></a>

## 2021.06.1

Released on June 18, 2021

- Replace funding page with ‘Supported By’ section on dask.org ([dask#7817](https://github.com/dask/dask/pull/7817)) [James Bourbeau](https://github.com/jrbourbeau)
- Add initial deprecation utilities ([dask#7810](https://github.com/dask/dask/pull/7810)) [James Bourbeau](https://github.com/jrbourbeau)
- Enforce dtype conservation in ufuncs that explicitly use `dtype=` ([dask#7808](https://github.com/dask/dask/pull/7808)) [Doug Davis](https://github.com/douglasdavis)
- Add Coiled to list of paid support organizations ([dask#7811](https://github.com/dask/dask/pull/7811)) [Kristopher Overholt](https://github.com/koverholt)
- Small tweaks to the HTML repr for `Layer` & `HighLevelGraph` ([dask#7812](https://github.com/dask/dask/pull/7812)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add dark mode support to HLG HTML repr ([dask#7809](https://github.com/dask/dask/pull/7809)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Remove compatibility entries for old distributed ([dask#7801](https://github.com/dask/dask/pull/7801)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Implementation of HTML repr for `HighLevelGraph` layers ([dask#7763](https://github.com/dask/dask/pull/7763)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Update default `blockwise` token to avoid DataFrame column name clash ([dask#6546](https://github.com/dask/dask/pull/6546)) [James Bourbeau](https://github.com/jrbourbeau)
- Use dispatch `concat` for `merge_asof` ([dask#7806](https://github.com/dask/dask/pull/7806)) [Julia Signell](https://github.com/jsignell)
- Fix upstream freq tests ([dask#7795](https://github.com/dask/dask/pull/7795)) [Julia Signell](https://github.com/jsignell)
- Use more context managers from the standard library ([dask#7796](https://github.com/dask/dask/pull/7796)) [James Bourbeau](https://github.com/jrbourbeau)
- Simplify skips in parquet tests ([dask#7802](https://github.com/dask/dask/pull/7802)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Remove check for outdated bokeh ([dask#7804](https://github.com/dask/dask/pull/7804)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- More test coverage uploads ([dask#7799](https://github.com/dask/dask/pull/7799)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove `ImportError` catching from `dask/__init__.py` ([dask#7797](https://github.com/dask/dask/pull/7797)) [James Bourbeau](https://github.com/jrbourbeau)
- Allow `DataFrame.join()` to take a list of DataFrames to merge with ([dask#7578](https://github.com/dask/dask/pull/7578)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Fix maximum recursion depth exception in `dask.array.linspace` ([dask#7667](https://github.com/dask/dask/pull/7667)) [Daniel Mesejo-León](https://github.com/mesejo)
- Fix docs links ([dask#7794](https://github.com/dask/dask/pull/7794)) [Julia Signell](https://github.com/jsignell)
- Initial `da.select()` implementation and test ([dask#7760](https://github.com/dask/dask/pull/7760)) [Gabriel Miretti](https://github.com/gmiretti)
- Layers must implement `get_output_keys` method ([dask#7790](https://github.com/dask/dask/pull/7790)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Don’t include or expect `freq` in divisions ([dask#7785](https://github.com/dask/dask/pull/7785)) [Julia Signell](https://github.com/jsignell)
- A `HighLevelGraph` abstract layer for `map_overlap` ([dask#7595](https://github.com/dask/dask/pull/7595)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Always include kwarg name in `drop` ([dask#7784](https://github.com/dask/dask/pull/7784)) [Julia Signell](https://github.com/jsignell)
- Only rechunk for median if needed ([dask#7782](https://github.com/dask/dask/pull/7782)) [Julia Signell](https://github.com/jsignell)
- Add `add_(prefix|suffix)` to DataFrame and Series ([dask#7745](https://github.com/dask/dask/pull/7745)) [tsuga](https://github.com/tsuga)
- Move `read_hdf` to `Blockwise` ([dask#7625](https://github.com/dask/dask/pull/7625)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Make `Layer.get_output_keys` officially an abstract method ([dask#7775](https://github.com/dask/dask/pull/7775)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Non-dask-arrays and broadcasting in `ravel_multi_index` ([dask#7594](https://github.com/dask/dask/pull/7594)) [Gabe Joseph](https://github.com/gjoseph92)
- Fix for paths ending with “/” in parquet overwrite ([dask#7773](https://github.com/dask/dask/pull/7773)) [Martin Durant](https://github.com/martindurant)
- Fixing calling `.visualize()` with `filename=None` ([dask#7740](https://github.com/dask/dask/pull/7740)) [Freyam Mehta](https://github.com/freyam)
- Generate unique names for `SubgraphCallable` ([dask#7637](https://github.com/dask/dask/pull/7637)) [Bruce Merry](https://github.com/bmerry)
- Pin `fsspec` to `2021.5.0` in CI ([dask#7771](https://github.com/dask/dask/pull/7771)) [James Bourbeau](https://github.com/jrbourbeau)
- Evaluate graph lazily if meta is provided in `from_delayed` ([dask#7769](https://github.com/dask/dask/pull/7769)) [Florian Jetter](https://github.com/fjetter)
- Add `meta` support for `DatetimeTZDtype` ([dask#7627](https://github.com/dask/dask/pull/7627)) [gerrymanoim](https://github.com/gerrymanoim)
- Add dispatch label to automatic PR labeler ([dask#7701](https://github.com/dask/dask/pull/7701)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix HDFS tests ([dask#7752](https://github.com/dask/dask/pull/7752)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-06-0"></a>

## 2021.06.0

Released on June 4, 2021

- Remove abstract tokens from graph keys in `rewrite_blockwise` ([dask#7721](https://github.com/dask/dask/pull/7721)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Ensure correct column order in csv `project_columns` ([dask#7761](https://github.com/dask/dask/pull/7761)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Renamed inner loop variables to avoid duplication ([dask#7741](https://github.com/dask/dask/pull/7741)) [Boaz Mohar](https://github.com/boazmohar)
- Do not return delayed object from `to_zarr` ([dask#7738](https://github.com/dask/dask/pull/7738)) Chris Roat
- Array: correct number of outputs in `apply_gufunc` ([dask#7669](https://github.com/dask/dask/pull/7669)) [Gabe Joseph](https://github.com/gjoseph92)
- Rewrite `da.fromfunction` with `da.blockwise` ([dask#7704](https://github.com/dask/dask/pull/7704)) [John A Kirkham](https://github.com/jakirkham)
- Rename `make_meta_util` to `make_meta` ([dask#7743](https://github.com/dask/dask/pull/7743)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Repartition before shuffle if the requested partitions are less than input partitions ([dask#7715](https://github.com/dask/dask/pull/7715)) [Vibhu Jawa](https://github.com/VibhuJawa)
- Blockwise: handle constant key inputs ([dask#7734](https://github.com/dask/dask/pull/7734)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Added raise to `apply_gufunc` ([dask#7744](https://github.com/dask/dask/pull/7744)) [Boaz Mohar](https://github.com/boazmohar)
- Show failing tests summary in CI ([dask#7735](https://github.com/dask/dask/pull/7735)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- `sizeof` sets in Python 3.9 ([dask#7739](https://github.com/dask/dask/pull/7739)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Warn if using pandas datetimelike string in `dataframe.__getitem__` ([dask#7749](https://github.com/dask/dask/pull/7749)) [Julia Signell](https://github.com/jsignell)
- Highlight the `client.dashboard_link` ([dask#7747](https://github.com/dask/dask/pull/7747)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Easier link for subscribing to the Google calendar ([dask#7733](https://github.com/dask/dask/pull/7733)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Automatically show graph visualization in Jupyter notebooks ([dask#7716](https://github.com/dask/dask/pull/7716)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add `autofunction` for `unify_chunks` in API docs ([dask#7730](https://github.com/dask/dask/pull/7730)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2021-05-1"></a>

## 2021.05.1

Released on May 28, 2021

- Pandas compatibility ([dask#7712](https://github.com/dask/dask/pull/7712)) [Julia Signell](https://github.com/jsignell)
- Fix `optimize_dataframe_getitem` bug ([dask#7698](https://github.com/dask/dask/pull/7698)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update `make_meta` import in docs ([dask#7713](https://github.com/dask/dask/pull/7713)) [Benjamin Zaitlen](https://github.com/quasiben)
- Implement `da.searchsorted` ([dask#7696](https://github.com/dask/dask/pull/7696)) [Tom White](https://github.com/tomwhite)
- Fix format string in error message ([dask#7706](https://github.com/dask/dask/pull/7706)) [Jiaming Yuan](https://github.com/trivialfis)
- Fix `read_sql_table` returning wrong result for single column loads ([dask#7572](https://github.com/dask/dask/pull/7572)) [c-thiel](https://github.com/c-thiel)
- Add slack join link in `support.rst` ([dask#7679](https://github.com/dask/dask/pull/7679)) [Naty Clementi](https://github.com/ncclementi)
- Remove unused alphabet variable ([dask#7700](https://github.com/dask/dask/pull/7700)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix meta creation incase of `object` ([dask#7586](https://github.com/dask/dask/pull/7586)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Add dispatch for `union_categoricals` ([dask#7699](https://github.com/dask/dask/pull/7699)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Consolidate array `Dispatch` objects ([dask#7505](https://github.com/dask/dask/pull/7505)) [James Bourbeau](https://github.com/jrbourbeau)
- Move DataFrame `dispatch.registers` to their own file ([dask#7503](https://github.com/dask/dask/pull/7503)) [Julia Signell](https://github.com/jsignell)
- Fix delayed with `dataclasses` where `init=False` ([dask#7656](https://github.com/dask/dask/pull/7656)) [Julia Signell](https://github.com/jsignell)
- Allow a column to be named `divisions` ([dask#7605](https://github.com/dask/dask/pull/7605)) [Julia Signell](https://github.com/jsignell)
- Stack nd array with unknown chunks ([dask#7562](https://github.com/dask/dask/pull/7562)) [Chris Roat](https://github.com/ChrisRoat)
- Promote the 2021 Dask User Survey ([dask#7694](https://github.com/dask/dask/pull/7694)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Fix typo in `DataFrame.set_index()` ([dask#7691](https://github.com/dask/dask/pull/7691)) [James Lamb](https://github.com/jameslamb)
- Cleanup array API reference links ([dask#7684](https://github.com/dask/dask/pull/7684)) [David Hoese](https://github.com/djhoese)
- Accept `axis` tuple for `flip` to be consistent with NumPy ([dask#7675](https://github.com/dask/dask/pull/7675)) [Andrew Champion](https://github.com/aschampion)
- Bump `pre-commit` hook versions ([dask#7676](https://github.com/dask/dask/pull/7676)) [James Bourbeau](https://github.com/jrbourbeau)
- Cleanup `to_zarr` docstring ([dask#7683](https://github.com/dask/dask/pull/7683)) [David Hoese](https://github.com/djhoese)
- Fix the docstring of `read_orc` ([dask#7678](https://github.com/dask/dask/pull/7678)) [Justus Magin](https://github.com/keewis)
- Doc `ipyparallel` & `mpi4py` `concurrent.futures` ([dask#7665](https://github.com/dask/dask/pull/7665)) [John A Kirkham](https://github.com/jakirkham)
- Update tests to support CuPy 9 ([dask#7671](https://github.com/dask/dask/pull/7671)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix some `HighLevelGraph` documentation inaccuracies ([dask#7662](https://github.com/dask/dask/pull/7662)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Fix spelling in Series `getitem` error message ([dask#7659](https://github.com/dask/dask/pull/7659)) [Maisie Marshall](https://github.com/maisiemarshall)

<a id="v2021-05-0"></a>

## 2021.05.0

Released on May 14, 2021

- Remove deprecated `kind` kwarg to comply with pandas 1.3.0 ([dask#7653](https://github.com/dask/dask/pull/7653)) [Julia Signell](https://github.com/jsignell)
- Fix bug in DataFrame column projection ([dask#7645](https://github.com/dask/dask/pull/7645)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Merge global annotations when packing ([dask#7565](https://github.com/dask/dask/pull/7565)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Avoid `inplace=` in pandas `set_categories` ([dask#7633](https://github.com/dask/dask/pull/7633)) [James Bourbeau](https://github.com/jrbourbeau)
- Change the active-fusion default to `False` for Dask-Dataframe ([dask#7620](https://github.com/dask/dask/pull/7620)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Array: remove extraneous code from `RandomState` ([dask#7487](https://github.com/dask/dask/pull/7487)) [Gabe Joseph](https://github.com/gjoseph92)
- Implement `str.concat` when `others=None` ([dask#7623](https://github.com/dask/dask/pull/7623)) [Daniel Mesejo-León](https://github.com/mesejo)
- Fix `dask.dataframe` in sandboxed environments ([dask#7601](https://github.com/dask/dask/pull/7601)) [Noah D. Brenowitz](https://github.com/nbren12)
- Support for `cupyx.scipy.linalg` ([dask#7563](https://github.com/dask/dask/pull/7563)) [Benjamin Zaitlen](https://github.com/quasiben)
- Move `timeseries` and daily-stock to `Blockwise` ([dask#7615](https://github.com/dask/dask/pull/7615)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix bugs in broadcast join ([dask#7617](https://github.com/dask/dask/pull/7617)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use `Blockwise` for DataFrame IO (parquet, csv, and orc) ([dask#7415](https://github.com/dask/dask/pull/7415)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Adding chunk & type information to Dask `HighLevelGraph` s ([dask#7309](https://github.com/dask/dask/pull/7309)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add `pyarrow` sphinx `intersphinx_mapping` ([dask#7612](https://github.com/dask/dask/pull/7612)) [Ray Bell](https://github.com/raybellwaves)
- Remove skip on test freq ([dask#7608](https://github.com/dask/dask/pull/7608)) [Julia Signell](https://github.com/jsignell)
- Defaults in `read_parquet` parameters ([dask#7567](https://github.com/dask/dask/pull/7567)) [Ray Bell](https://github.com/raybellwaves)
- Remove `ignore_abc_warning` ([dask#7606](https://github.com/dask/dask/pull/7606)) [Julia Signell](https://github.com/jsignell)
- Harden DataFrame merge between column-selection and index ([dask#7575](https://github.com/dask/dask/pull/7575)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Get rid of `ignore_abc` decorator ([dask#7604](https://github.com/dask/dask/pull/7604)) [Julia Signell](https://github.com/jsignell)
- Remove kwarg validation for bokeh ([dask#7597](https://github.com/dask/dask/pull/7597)) [Julia Signell](https://github.com/jsignell)
- Add `loky` example ([dask#7590](https://github.com/dask/dask/pull/7590)) [Naty Clementi](https://github.com/ncclementi)
- Delayed: `nout` when arguments become tasks ([dask#7593](https://github.com/dask/dask/pull/7593)) [Gabe Joseph](https://github.com/gjoseph92)
- Update distributed version in mindep CI build ([dask#7602](https://github.com/dask/dask/pull/7602)) [James Bourbeau](https://github.com/jrbourbeau)
- Support all or no overlap between partition columns and real columns ([dask#7541](https://github.com/dask/dask/pull/7541)) [Richard (Rick) Zamora](https://github.com/rjzamora)

<a id="v2021-04-1"></a>

## 2021.04.1

Released on April 23, 2021

- Handle `Blockwise` HLG pack/unpack for `concatenate=True` ([dask#7455](https://github.com/dask/dask/pull/7455)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `map_partitions`: use tokenized info as name of the `SubgraphCallable` ([dask#7524](https://github.com/dask/dask/pull/7524)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Using `tmp_path` and `tmpdir` to avoid temporary files and directories hanging in the repo ([dask#7592](https://github.com/dask/dask/pull/7592)) [Naty Clementi](https://github.com/ncclementi)
- Contributing to docs (development guide) ([dask#7591](https://github.com/dask/dask/pull/7591)) [Naty Clementi](https://github.com/ncclementi)
- Add more packages to Python 3.9 CI build ([dask#7588](https://github.com/dask/dask/pull/7588)) [James Bourbeau](https://github.com/jrbourbeau)
- Array: Fix NEP-18 dispatching in finalize ([dask#7508](https://github.com/dask/dask/pull/7508)) [Gabe Joseph](https://github.com/gjoseph92)
- Misc fixes for `numpydoc` ([dask#7569](https://github.com/dask/dask/pull/7569)) [Matthias Bussonnier](https://github.com/Carreau)
- Avoid pandas `level=` keyword deprecation ([dask#7577](https://github.com/dask/dask/pull/7577)) [James Bourbeau](https://github.com/jrbourbeau)
- Map e.g. `.repartition(freq="M")` to `.repartition(freq="MS")` ([dask#7504](https://github.com/dask/dask/pull/7504)) [Ruben van de Geer](https://github.com/rubenvdg)
- Remove hash seeding in parallel CI runs ([dask#7128](https://github.com/dask/dask/pull/7128)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Add defaults in parameters in `to_parquet` ([dask#7564](https://github.com/dask/dask/pull/7564)) [Ray Bell](https://github.com/raybellwaves)
- Simplify transpose axes cleanup ([dask#7561](https://github.com/dask/dask/pull/7561)) [Julia Signell](https://github.com/jsignell)
- Make `ValueError in len(index_names) > 1` explicit it’s using `fastparquet` ([dask#7556](https://github.com/dask/dask/pull/7556)) [Ray Bell](https://github.com/raybellwaves)
- Fix `dict`-column appending for `pyarrow` parquet engines ([dask#7527](https://github.com/dask/dask/pull/7527)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add a documentation auto label ([dask#7560](https://github.com/dask/dask/pull/7560)) [Doug Davis](https://github.com/douglasdavis)
- Add `dask.delayed.Delayed` to docs so it can be referenced by other sphinx docs ([dask#7559](https://github.com/dask/dask/pull/7559)) [Doug Davis](https://github.com/douglasdavis)
- Fix upstream `idxmaxmin` for uneven `split_every` ([dask#7538](https://github.com/dask/dask/pull/7538)) [Julia Signell](https://github.com/jsignell)
- Make `normalize_token` for pandas `Series`/`DataFrame` future proof (no direct block access) ([dask#7318](https://github.com/dask/dask/pull/7318)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Redesigned `__setitem__` implementation ([dask#7393](https://github.com/dask/dask/pull/7393)) [David Hassell](https://github.com/davidhassell)
- `histogram`, `histogramdd` improvements (docs; return consistencies) ([dask#7520](https://github.com/dask/dask/pull/7520)) [Doug Davis](https://github.com/douglasdavis)
- Force nightly `pyarrow` in the upstream build ([dask#7530](https://github.com/dask/dask/pull/7530)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Fix Configuration Reference ([dask#7533](https://github.com/dask/dask/pull/7533)) [Benjamin Zaitlen](https://github.com/quasiben)
- Use `.to_parquet` on `dask.dataframe` in doc string ([dask#7528](https://github.com/dask/dask/pull/7528)) [Ray Bell](https://github.com/raybellwaves)
- Avoid double `msgpack` serialization of HLGs ([dask#7525](https://github.com/dask/dask/pull/7525)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Encourage usage of `yaml.safe_load()` in configuration doc ([dask#7529](https://github.com/dask/dask/pull/7529)) [Hristo Georgiev](https://github.com/hristog)
- Fix `reshape` bug. Add relevant test. Fixes #7171. ([dask#7523](https://github.com/dask/dask/pull/7523)) [JSKenyon](https://github.com/jskenyon)
- Support `custom_metadata=` argument in `to_parquet` ([dask#7359](https://github.com/dask/dask/pull/7359)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Clean some documentation warnings ([dask#7518](https://github.com/dask/dask/pull/7518)) [Daniel Mesejo-León](https://github.com/mesejo)
- Getting rid of more docs warnings ([dask#7426](https://github.com/dask/dask/pull/7426)) [Julia Signell](https://github.com/jsignell)
- Added `product` (alias of `prod`) ([dask#7517](https://github.com/dask/dask/pull/7517)) [Freyam Mehta](https://github.com/freyam)
- Fix upstream `__array_ufunc__` tests ([dask#7494](https://github.com/dask/dask/pull/7494)) [Julia Signell](https://github.com/jsignell)
- Escape from `map_overlap` to `map_blocks` if depth is zero ([dask#7481](https://github.com/dask/dask/pull/7481)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add `check_type` to array `assert_eq` ([dask#7491](https://github.com/dask/dask/pull/7491)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-04-0"></a>

## 2021.04.0

Released on April 2, 2021

- Adding support for multidimensional histograms with `dask.array.histogramdd` ([dask#7387](https://github.com/dask/dask/pull/7387)) [Doug Davis](https://github.com/douglasdavis)
- Update docs on number of threads and workers in default `LocalCluster` ([dask#7497](https://github.com/dask/dask/pull/7497)) [cameron16](https://github.com/cameron16)
- Add labels automatically when certain files are touched in a PR ([dask#7506](https://github.com/dask/dask/pull/7506)) [Julia Signell](https://github.com/jsignell)
- Extract `ignore_order` from `kwargs` ([dask#7500](https://github.com/dask/dask/pull/7500)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Only provide installation instructions when distributed is missing ([dask#7498](https://github.com/dask/dask/pull/7498)) [Matthew Rocklin](https://github.com/mrocklin)
- Start adding `isort` ([dask#7370](https://github.com/dask/dask/pull/7370)) [Julia Signell](https://github.com/jsignell)
- Add `ignore_order` parameter in `dd.concat` ([dask#7473](https://github.com/dask/dask/pull/7473)) [Daniel Mesejo-León](https://github.com/mesejo)
- Use powers-of-two when displaying RAM ([dask#7484](https://github.com/dask/dask/pull/7484)) [crusaderky](https://github.com/crusaderky)
- Added License Classifier ([dask#7485](https://github.com/dask/dask/pull/7485)) [Tom Augspurger](https://github.com/tomaugspurger)
- Replace conda with mamba ([dask#7227](https://github.com/dask/dask/pull/7227)) [crusaderky](https://github.com/crusaderky)
- Fix typo in array docs ([dask#7478](https://github.com/dask/dask/pull/7478)) [James Lamb](https://github.com/jameslamb)
- Use `concurrent.futures` in local scheduler ([dask#6322](https://github.com/dask/dask/pull/6322)) [John A Kirkham](https://github.com/jakirkham)

<a id="v2021-03-1"></a>

## 2021.03.1

Released on March 26, 2021

- Add a dispatch for `is_categorical_dtype` to handle non-pandas objects ([dask#7469](https://github.com/dask/dask/pull/7469)) [brandon-b-miller](https://github.com/brandon-b-miller)
- Use `multiprocessing.Pool` in `test_read_text` ([dask#7472](https://github.com/dask/dask/pull/7472)) [John A Kirkham](https://github.com/jakirkham)
- Add missing `meta` kwarg to gufunc class ([dask#7423](https://github.com/dask/dask/pull/7423)) [Peter Andreas Entschev](https://github.com/pentschev)
- Example for memory-mapped Dask array ([dask#7380](https://github.com/dask/dask/pull/7380)) [Dieter Weber](https://github.com/uellue)
- Fix NumPy upstream failures `xfail` pandas and fastparquet failures ([dask#7441](https://github.com/dask/dask/pull/7441)) [Julia Signell](https://github.com/jsignell)
- Fix bug in repartition with freq ([dask#7357](https://github.com/dask/dask/pull/7357)) [Ruben van de Geer](https://github.com/rubenvdg)
- Fix `__array_function__` dispatching for `tril`/`triu` ([dask#7457](https://github.com/dask/dask/pull/7457)) [Peter Andreas Entschev](https://github.com/pentschev)
- Use `concurrent.futures.Executors` in a few tests ([dask#7429](https://github.com/dask/dask/pull/7429)) [John A Kirkham](https://github.com/jakirkham)
- Require NumPy >=1.16 ([dask#7383](https://github.com/dask/dask/pull/7383)) [crusaderky](https://github.com/crusaderky)
- Minor `sort_values` housekeeping ([dask#7462](https://github.com/dask/dask/pull/7462)) [Ryan Williams](https://github.com/ryan-williams)
- Ensure natural sort order in parquet part paths ([dask#7249](https://github.com/dask/dask/pull/7249)) [Ryan Williams](https://github.com/ryan-williams)
- Remove global env mutation upon running `test_config.py` ([dask#7464](https://github.com/dask/dask/pull/7464)) [Hristo Georgiev](https://github.com/hristog)
- Update NumPy intersphinx URL ([dask#7460](https://github.com/dask/dask/pull/7460)) [Gabe Joseph](https://github.com/gjoseph92)
- Add `rot90` ([dask#7440](https://github.com/dask/dask/pull/7440)) [Trevor Manz](https://github.com/manzt)
- Update docs for required package for endpoint ([dask#7454](https://github.com/dask/dask/pull/7454)) [Nick Vazquez](https://github.com/nickvazz)
- Master -> main in `slice_array` docstring ([dask#7453](https://github.com/dask/dask/pull/7453)) [Gabe Joseph](https://github.com/gjoseph92)
- Expand `dask.utils.is_arraylike` docstring ([dask#7445](https://github.com/dask/dask/pull/7445)) [Doug Davis](https://github.com/douglasdavis)
- Simplify `BlockwiseIODeps` importing ([dask#7420](https://github.com/dask/dask/pull/7420)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Update layer annotation packing method ([dask#7430](https://github.com/dask/dask/pull/7430)) [James Bourbeau](https://github.com/jrbourbeau)
- Drop duplicate test in `test_describe_empty` ([dask#7431](https://github.com/dask/dask/pull/7431)) [John A Kirkham](https://github.com/jakirkham)
- Add `Series.dot` method to dataframe module ([dask#7236](https://github.com/dask/dask/pull/7236)) [Madhu94](https://github.com/Madhu94)
- Added df `kurtosis`-method and testing ([dask#7273](https://github.com/dask/dask/pull/7273)) [Jan Borchmann](https://github.com/jborchma)
- Avoid quadratic-time performance for HLG culling ([dask#7403](https://github.com/dask/dask/pull/7403)) [Bruce Merry](https://github.com/bmerry)
- Temporarily skip problematic `sparse` test ([dask#7421](https://github.com/dask/dask/pull/7421)) [James Bourbeau](https://github.com/jrbourbeau)
- Update some CI workflow names ([dask#7422](https://github.com/dask/dask/pull/7422)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix HDFS test ([dask#7418](https://github.com/dask/dask/pull/7418)) [Julia Signell](https://github.com/jsignell)
- Make changelog subtitles match the hierarchy ([dask#7419](https://github.com/dask/dask/pull/7419)) [Julia Signell](https://github.com/jsignell)
- Add support for normalize in `value_counts` ([dask#7342](https://github.com/dask/dask/pull/7342)) [Julia Signell](https://github.com/jsignell)
- Avoid unnecessary imports for HLG Layer unpacking and materialization ([dask#7381](https://github.com/dask/dask/pull/7381)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bincount fix slicing ([dask#7391](https://github.com/dask/dask/pull/7391)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add `sliding_window_view` ([dask#7234](https://github.com/dask/dask/pull/7234)) [Deepak Cherian](https://github.com/dcherian)
- Fix typo in `docs/source/develop.rst` ([dask#7414](https://github.com/dask/dask/pull/7414)) [Hristo Georgiev](https://github.com/hristog)
- Switch documentation builds for PRs to readthedocs ([dask#7397](https://github.com/dask/dask/pull/7397)) [James Bourbeau](https://github.com/jrbourbeau)
- Adds `sort_values` to dask.DataFrame ([dask#7286](https://github.com/dask/dask/pull/7286)) [gerrymanoim](https://github.com/gerrymanoim)
- Pin `sqlalchemy<1.4.0` in CI ([dask#7405](https://github.com/dask/dask/pull/7405)) [James Bourbeau](https://github.com/jrbourbeau)
- Comment fixes ([dask#7215](https://github.com/dask/dask/pull/7215)) [Ryan Williams](https://github.com/ryan-williams)
- Dead code removal / fixes ([dask#7388](https://github.com/dask/dask/pull/7388)) [Ryan Williams](https://github.com/ryan-williams)
- Use single thread for `pa.Table.from_pandas` calls ([dask#7347](https://github.com/dask/dask/pull/7347)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Replace `'container'` with `'image'` ([dask#7389](https://github.com/dask/dask/pull/7389)) [James Lamb](https://github.com/jameslamb)
- DOC hyperlink repartition ([dask#7394](https://github.com/dask/dask/pull/7394)) [Ray Bell](https://github.com/raybellwaves)
- Pass delimiter to `fsspec` in `bag.read_text` ([dask#7349](https://github.com/dask/dask/pull/7349)) [Martin Durant](https://github.com/martindurant)
- Update `read_hdf` default mode to `"r"` ([dask#7039](https://github.com/dask/dask/pull/7039)) [rs9w33](https://github.com/rs9w33)
- Embed literals in `SubgraphCallable` when packing `Blockwise` ([dask#7353](https://github.com/dask/dask/pull/7353)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Update `test_hdf.py` to not reuse file handlers ([dask#7044](https://github.com/dask/dask/pull/7044)) [rs9w33](https://github.com/rs9w33)
- Require additional dependencies: cloudpickle, partd, fsspec, toolz ([dask#7345](https://github.com/dask/dask/pull/7345)) [Julia Signell](https://github.com/jsignell)
- Prepare `Blockwise` + IO infrastructure ([dask#7281](https://github.com/dask/dask/pull/7281)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Remove duplicated imports from `test_slicing.py` ([dask#7365](https://github.com/dask/dask/pull/7365)) [Hristo Georgiev](https://github.com/hristog)
- Add test deps for pip development ([dask#7360](https://github.com/dask/dask/pull/7360)) [Julia Signell](https://github.com/jsignell)
- Support int slicing for non-NumPy arrays ([dask#7364](https://github.com/dask/dask/pull/7364)) [Peter Andreas Entschev](https://github.com/pentschev)
- Automatically cancel previous CI builds ([dask#7348](https://github.com/dask/dask/pull/7348)) [James Bourbeau](https://github.com/jrbourbeau)
- `dask.array.asarray` should handle case where `xarray` class is in top-level namespace ([dask#7335](https://github.com/dask/dask/pull/7335)) [Tom White](https://github.com/tomwhite)
- `HighLevelGraph` length without materializing layers ([dask#7274](https://github.com/dask/dask/pull/7274)) [Gabe Joseph](https://github.com/gjoseph92)
- Drop support for Python 3.6 ([dask#7006](https://github.com/dask/dask/pull/7006)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix fsspec usage in `create_metadata_file` ([dask#7295](https://github.com/dask/dask/pull/7295)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Change default branch from master to main ([dask#7198](https://github.com/dask/dask/pull/7198)) [Julia Signell](https://github.com/jsignell)
- Add Xarray to CI software environment ([dask#7338](https://github.com/dask/dask/pull/7338)) [James Bourbeau](https://github.com/jrbourbeau)
- Update repartition argument name in error text ([dask#7336](https://github.com/dask/dask/pull/7336)) [Eoin Shanaghy](https://github.com/eoinsha)
- Run upstream tests based on commit message ([dask#7329](https://github.com/dask/dask/pull/7329)) [James Bourbeau](https://github.com/jrbourbeau)
- Use `pytest.register_assert_rewrite` on util modules ([dask#7278](https://github.com/dask/dask/pull/7278)) [Bruce Merry](https://github.com/bmerry)
- Add example on using specific chunk sizes in `from_array()` ([dask#7330](https://github.com/dask/dask/pull/7330)) [James Lamb](https://github.com/jameslamb)
- Move NumPy skip into test ([dask#7247](https://github.com/dask/dask/pull/7247)) [Julia Signell](https://github.com/jsignell)

<a id="v2021-03-0"></a>

## 2021.03.0

Released on March 5, 2021

#### NOTE
This is the first release with support for Python 3.9 and the
last release with support for Python 3.6

- Bump minimum version of `distributed` ([dask#7328](https://github.com/dask/dask/pull/7328)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix `percentiles_summary` with `dask_cudf` ([dask#7325](https://github.com/dask/dask/pull/7325)) [Peter Andreas Entschev](https://github.com/pentschev)
- Temporarily revert recent `Array.__setitem__` updates ([dask#7326](https://github.com/dask/dask/pull/7326)) [James Bourbeau](https://github.com/jrbourbeau)
- `Blockwise.clone` ([dask#7312](https://github.com/dask/dask/pull/7312)) [crusaderky](https://github.com/crusaderky)
- NEP-35 duck array update ([dask#7321](https://github.com/dask/dask/pull/7321)) [James Bourbeau](https://github.com/jrbourbeau)
- Don’t allow setting `.name` for array ([dask#7222](https://github.com/dask/dask/pull/7222)) [Julia Signell](https://github.com/jsignell)
- Use nearest interpolation for creating percentiles of integer input ([dask#7305](https://github.com/dask/dask/pull/7305)) [Kyle Barron](https://github.com/kylebarron)
- Test `exp` with CuPy arrays ([dask#7322](https://github.com/dask/dask/pull/7322)) [John A Kirkham](https://github.com/jakirkham)
- Check that computed chunks have right size and dtype ([dask#7277](https://github.com/dask/dask/pull/7277)) [Bruce Merry](https://github.com/bmerry)
- `pytest.mark.flaky` ([dask#7319](https://github.com/dask/dask/pull/7319)) [crusaderky](https://github.com/crusaderky)
- Contributing docs: add note to pull the latest git tags before pip installing Dask ([dask#7308](https://github.com/dask/dask/pull/7308)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Support for Python 3.9 ([dask#7289](https://github.com/dask/dask/pull/7289)) [crusaderky](https://github.com/crusaderky)
- Add broadcast-based merge implementation ([dask#7143](https://github.com/dask/dask/pull/7143)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `split_every` to `graph_manipulation` ([dask#7282](https://github.com/dask/dask/pull/7282)) [crusaderky](https://github.com/crusaderky)
- Typo in optimize docs ([dask#7306](https://github.com/dask/dask/pull/7306)) [Julius Busecke](https://github.com/jbusecke)
- `dask.graph_manipulation` support for `xarray.Dataset` ([dask#7276](https://github.com/dask/dask/pull/7276)) [crusaderky](https://github.com/crusaderky)
- Add plot width and height support for Bokeh 2.3.0 ([dask#7297](https://github.com/dask/dask/pull/7297)) [James Bourbeau](https://github.com/jrbourbeau)
- Add NumPy functions `tri`, `triu_indices`, `triu_indices_from`, `tril_indices`, `tril_indices_from` ([dask#6997](https://github.com/dask/dask/pull/6997)) [Illviljan](https://github.com/Illviljan)
- Remove “cleanup” task in DataFrame on-disk shuffle ([dask#7260](https://github.com/dask/dask/pull/7260)) [Sinclair Target](https://github.com/sinclairtarget)
- Use development version of `distributed` in CI ([dask#7279](https://github.com/dask/dask/pull/7279)) [James Bourbeau](https://github.com/jrbourbeau)
- Moving high level graph pack/unpack Dask  ([dask#7179](https://github.com/dask/dask/pull/7179)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Improve performance of `merge_percentiles` ([dask#7172](https://github.com/dask/dask/pull/7172)) [Ashwin Srinath](https://github.com/shwina)
- DOC: add `dask-sql` and `fugue` ([dask#7129](https://github.com/dask/dask/pull/7129)) [Ray Bell](https://github.com/raybellwaves)
- Example for working with categoricals and parquet ([dask#7085](https://github.com/dask/dask/pull/7085)) [McToel](https://github.com/McToel)
- Adds tree reduction to `bincount` ([dask#7183](https://github.com/dask/dask/pull/7183)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Improve documentation of `name` in `from_array` ([dask#7264](https://github.com/dask/dask/pull/7264)) [Bruce Merry](https://github.com/bmerry)
- Fix `cumsum` for empty partitions ([dask#7230](https://github.com/dask/dask/pull/7230)) [Julia Signell](https://github.com/jsignell)
- Add `map_blocks` example to dask array creation docs ([dask#7221](https://github.com/dask/dask/pull/7221)) [Julia Signell](https://github.com/jsignell)
- Fix performance issue in `dask.graph_manipulation.wait_on()` ([dask#7258](https://github.com/dask/dask/pull/7258)) [crusaderky](https://github.com/crusaderky)
- Replace coveralls with codecov.io ([dask#7246](https://github.com/dask/dask/pull/7246)) [crusaderky](https://github.com/crusaderky)
- Pin to a particular `black` rev in pre-commit ([dask#7256](https://github.com/dask/dask/pull/7256)) [Julia Signell](https://github.com/jsignell)
- Minor typo in documentation: `array-chunks.rst` ([dask#7254](https://github.com/dask/dask/pull/7254)) [Magnus Nord](https://github.com/magnunor)
- Fix bugs in `Blockwise` and `ShuffleLayer` ([dask#7213](https://github.com/dask/dask/pull/7213)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix parquet filtering bug for `"pyarrow-dataset"` with pyarrow-3.0.0 ([dask#7200](https://github.com/dask/dask/pull/7200)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `graph_manipulation` without NumPy ([dask#7243](https://github.com/dask/dask/pull/7243)) [crusaderky](https://github.com/crusaderky)
- Support for NEP-35 ([dask#6738](https://github.com/dask/dask/pull/6738)) [Peter Andreas Entschev](https://github.com/pentschev)
- Avoid running unit tests during doctest CI build ([dask#7240](https://github.com/dask/dask/pull/7240)) [James Bourbeau](https://github.com/jrbourbeau)
- Run doctests on CI ([dask#7238](https://github.com/dask/dask/pull/7238)) [Julia Signell](https://github.com/jsignell)
- Cleanup code quality on set arithmetics ([dask#7196](https://github.com/dask/dask/pull/7196)) [crusaderky](https://github.com/crusaderky)
- Add `dask.array.delete` ([dask#7125](https://github.com/dask/dask/pull/7125)) [Julia Signell](https://github.com/jsignell)
- Unpin graphviz now that new conda-forge recipe is built ([dask#7235](https://github.com/dask/dask/pull/7235)) [Julia Signell](https://github.com/jsignell)
- Don’t use NumPy 1.20 from conda-forge on Mac ([dask#7211](https://github.com/dask/dask/pull/7211)) [crusaderky](https://github.com/crusaderky)
- `map_overlap`: Don’t rechunk axes without overlap ([dask#7233](https://github.com/dask/dask/pull/7233)) [Deepak Cherian](https://github.com/dcherian)
- Pin graphviz to avoid issue with latest conda-forge build ([dask#7232](https://github.com/dask/dask/pull/7232)) [Julia Signell](https://github.com/jsignell)
- Use `html_css_files` in docs for custom CSS ([dask#7220](https://github.com/dask/dask/pull/7220)) [James Bourbeau](https://github.com/jrbourbeau)
- Graph manipulation: `clone`, `bind`, `checkpoint`, `wait_on` ([dask#7109](https://github.com/dask/dask/pull/7109)) [crusaderky](https://github.com/crusaderky)
- Fix handling of filter expressions in parquet `pyarrow-dataset` engine ([dask#7186](https://github.com/dask/dask/pull/7186)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Extend `__setitem__` to more closely match numpy ([dask#7033](https://github.com/dask/dask/pull/7033)) [David Hassell](https://github.com/davidhassell)
- Clean up Python 2 syntax ([dask#7195](https://github.com/dask/dask/pull/7195)) [crusaderky](https://github.com/crusaderky)
- Fix regression in `Delayed._length` ([dask#7194](https://github.com/dask/dask/pull/7194)) [crusaderky](https://github.com/crusaderky)
- `__dask_layers__()` tests and tweaks ([dask#7177](https://github.com/dask/dask/pull/7177)) [crusaderky](https://github.com/crusaderky)
- Properly convert `HighLevelGraph` in multiprocessing scheduler ([dask#7191](https://github.com/dask/dask/pull/7191)) [Jim Crist-Harif](https://github.com/jcrist)
- Don’t fail fast in CI ([dask#7188](https://github.com/dask/dask/pull/7188)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2021-02-0"></a>

## 2021.02.0

Released on February 5, 2021

- Add `percentile` support for NEP-35 ([dask#7162](https://github.com/dask/dask/pull/7162)) [Peter Andreas Entschev](https://github.com/pentschev)
- Added support for `Float64` in column assignment ([dask#7173](https://github.com/dask/dask/pull/7173)) [Nils Braun](https://github.com/nils-braun)
- Coarsen rechunking error ([dask#7127](https://github.com/dask/dask/pull/7127)) [Davis Bennett](https://github.com/d-v-b)
- Fix upstream CI tests ([dask#6896](https://github.com/dask/dask/pull/6896)) [Julia Signell](https://github.com/jsignell)
- Revise `HighLevelGraph` Mapping API ([dask#7160](https://github.com/dask/dask/pull/7160)) [crusaderky](https://github.com/crusaderky)
- Update low-level graph spec to use any hashable for keys ([dask#7163](https://github.com/dask/dask/pull/7163)) [James Bourbeau](https://github.com/jrbourbeau)
- Generically rebuild a collection with different keys ([dask#7142](https://github.com/dask/dask/pull/7142)) [crusaderky](https://github.com/crusaderky)
- Make easier to link issues in PRs ([dask#7130](https://github.com/dask/dask/pull/7130)) [Ray Bell](https://github.com/raybellwaves)
- Add `dask.array.append` ([dask#7146](https://github.com/dask/dask/pull/7146)) [D-Stacks](https://github.com/D-Stacks)
- Allow `dask.array.ravel` to accept `array_like` argument ([dask#7138](https://github.com/dask/dask/pull/7138)) [D-Stacks](https://github.com/D-Stacks)
- Fixes link in array design doc ([dask#7152](https://github.com/dask/dask/pull/7152)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Fix example of using `blockwise` for an outer product ([dask#7119](https://github.com/dask/dask/pull/7119)) [Bruce Merry](https://github.com/bmerry)
- Deprecate `HighlevelGraph.dicts` in favor of `.layers` ([dask#7145](https://github.com/dask/dask/pull/7145)) [Amit Kumar](https://github.com/aktech)
- Align `FastParquetEngine` with pyarrow engines ([dask#7091](https://github.com/dask/dask/pull/7091)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Merge annotations ([dask#7102](https://github.com/dask/dask/pull/7102)) [Ian Rose](https://github.com/ian-r-rose)
- Simplify contents of parts list in `read_parquet` ([dask#7066](https://github.com/dask/dask/pull/7066)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `check_meta(`): use `__class__` when checking DataFrame types ([dask#7099](https://github.com/dask/dask/pull/7099)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Cache several properties ([dask#7104](https://github.com/dask/dask/pull/7104)) [Illviljan](https://github.com/Illviljan)
- Fix parquet `getitem` optimization ([dask#7106](https://github.com/dask/dask/pull/7106)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add cytoolz back to CI environment ([dask#7103](https://github.com/dask/dask/pull/7103)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2021-01-1"></a>

## 2021.01.1

Released on January 22, 2021

- Partially fix `cumprod` ([dask#7089](https://github.com/dask/dask/pull/7089)) [Julia Signell](https://github.com/jsignell)
- Test pandas 1.1.x / 1.2.0 releases and pandas nightly ([dask#6996](https://github.com/dask/dask/pull/6996)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Use assign to avoid `SettingWithCopyWarning` ([dask#7092](https://github.com/dask/dask/pull/7092)) [Julia Signell](https://github.com/jsignell)
- `'mode'` argument passed to `bokeh.output_file()` ([dask#7034](https://github.com/dask/dask/pull/7034)) ([dask#7075](https://github.com/dask/dask/pull/7075)) [patquem](https://github.com/patquem)
- Skip empty partitions when doing `groupby.value_counts` ([dask#7073](https://github.com/dask/dask/pull/7073)) [Julia Signell](https://github.com/jsignell)
- Add error messages to `assert_eq()` ([dask#7083](https://github.com/dask/dask/pull/7083)) [James Lamb](https://github.com/jameslamb)
- Make cached properties read-only ([dask#7077](https://github.com/dask/dask/pull/7077)) [Illviljan](https://github.com/Illviljan)

<a id="v2021-01-0"></a>

## 2021.01.0

Released on January 15, 2021

- `map_partitions` with review comments ([dask#6776](https://github.com/dask/dask/pull/6776)) [Kumar Bharath Prabhu](https://github.com/kumarprabhu1988)
- Make sure that `population` is a real list ([dask#7027](https://github.com/dask/dask/pull/7027)) [Julia Signell](https://github.com/jsignell)
- Propagate `storage_options` in `read_csv` ([dask#7074](https://github.com/dask/dask/pull/7074)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Remove all `BlockwiseIO` code ([dask#7067](https://github.com/dask/dask/pull/7067)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix CI ([dask#7069](https://github.com/dask/dask/pull/7069)) [James Bourbeau](https://github.com/jrbourbeau)
- Add option to control rechunking in `reshape` ([dask#6753](https://github.com/dask/dask/pull/6753)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix `linalg.lstsq` for complex inputs ([dask#7056](https://github.com/dask/dask/pull/7056)) [Johnnie Gray](https://github.com/jcmgray)
- Add `compression='infer'` default to `read_csv` ([dask#6960](https://github.com/dask/dask/pull/6960)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Revert parameter changes in `svd_compressed` #7003 ([dask#7004](https://github.com/dask/dask/pull/7004)) [Eric Czech](https://github.com/eric-czech)
- Skip failing s3 test ([dask#7064](https://github.com/dask/dask/pull/7064)) [Martin Durant](https://github.com/martindurant)
- Revert `BlockwiseIO` ([dask#7048](https://github.com/dask/dask/pull/7048)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add some cross-references to `DataFrame.to_bag()` and `Series.to_bag()` ([dask#7049](https://github.com/dask/dask/pull/7049)) [Rob Malouf](https://github.com/rmalouf)
- Rewrite `matmul` as `blockwise` without contraction/concatenate ([dask#7000](https://github.com/dask/dask/pull/7000)) [Rafal Wojdyla](https://github.com/ravwojdyla)
- Use `functools.cached_property` in `da.shape` ([dask#7023](https://github.com/dask/dask/pull/7023)) [Illviljan](https://github.com/Illviljan)
- Use meta value in series `non_empty` ([dask#6976](https://github.com/dask/dask/pull/6976)) [Julia Signell](https://github.com/jsignell)
- Revert “Temporarly pin sphinx version to 3.3.1 ([dask#7002](https://github.com/dask/dask/pull/7002))” ([dask#7014](https://github.com/dask/dask/pull/7014)) [Rafal Wojdyla](https://github.com/ravwojdyla)
- Revert `python-graphviz` pinning ([dask#7037](https://github.com/dask/dask/pull/7037)) [Julia Signell](https://github.com/jsignell)
- Accidentally committed print statement ([dask#7038](https://github.com/dask/dask/pull/7038)) [Julia Signell](https://github.com/jsignell)
- Pass `dropna` and `observed` in `agg` ([dask#6992](https://github.com/dask/dask/pull/6992)) [Julia Signell](https://github.com/jsignell)
- Add index to `meta` after `.str.split` with expand ([dask#7026](https://github.com/dask/dask/pull/7026)) [Ruben van de Geer](https://github.com/rubenvdg)
- CI: test pyarrow 2.0 and nightly ([dask#7030](https://github.com/dask/dask/pull/7030)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Temporarily pin `python-graphviz` in CI ([dask#7031](https://github.com/dask/dask/pull/7031)) [James Bourbeau](https://github.com/jrbourbeau)
- Underline section in `numpydoc` ([dask#7013](https://github.com/dask/dask/pull/7013)) [Matthias Bussonnier](https://github.com/Carreau)
- Keep normal optimizations when adding custom optimizations ([dask#7016](https://github.com/dask/dask/pull/7016)) [Matthew Rocklin](https://github.com/mrocklin)
- Temporarily pin sphinx version to 3.3.1 ([dask#7002](https://github.com/dask/dask/pull/7002)) [Rafal Wojdyla](https://github.com/ravwojdyla)
- DOC: Misc formatting ([dask#6998](https://github.com/dask/dask/pull/6998)) [Matthias Bussonnier](https://github.com/Carreau)
- Add `inline_array` option to `from_array` ([dask#6773](https://github.com/dask/dask/pull/6773)) [Tom Augspurger](https://github.com/tomaugspurger)
- Revert “Initial pass at blockwise array creation routines ([dask#6931)” (:pr:\`6995](https://github.com/dask/dask/pull/6931)" (:pr:`6995)) [James Bourbeau](https://github.com/jrbourbeau)
- Set `npartitions` in `set_index` ([dask#6978](https://github.com/dask/dask/pull/6978)) [Julia Signell](https://github.com/jsignell)
- Upstream `config` serialization and inheritance ([dask#6987](https://github.com/dask/dask/pull/6987)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Bump the minimum time in `test_minimum_time` ([dask#6988](https://github.com/dask/dask/pull/6988)) [Martin Durant](https://github.com/martindurant)
- Fix pandas `dtype` inference for `read_parquet` ([dask#6985](https://github.com/dask/dask/pull/6985)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid data loss in `set_index` with `sorted=True` ([dask#6980](https://github.com/dask/dask/pull/6980)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Bugfix in `read_parquet` for handling un-named indices with `index=False` ([dask#6969](https://github.com/dask/dask/pull/6969)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use `__class__` when comparing meta data ([dask#6981](https://github.com/dask/dask/pull/6981)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Comparing string versions won’t always work ([dask#6979](https://github.com/dask/dask/pull/6979)) [Rafal Wojdyla](https://github.com/ravwojdyla)
- Fix [dask#6925](https://github.com/dask/dask/pull/6925) ([dask#6982](https://github.com/dask/dask/pull/6982)) [sdementen](https://github.com/sdementen)
- Initial pass at blockwise array creation routines ([dask#6931](https://github.com/dask/dask/pull/6931)) [Ian Rose](https://github.com/ian-r-rose)
- Simplify `has_parallel_type()` ([dask#6927](https://github.com/dask/dask/pull/6927)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Handle annotation unpacking in `BlockwiseIO` ([dask#6934](https://github.com/dask/dask/pull/6934)) [Simon Perkins](https://github.com/sjperkins)
- Avoid deprecated `yield_fixture` in `test_sql.py` ([dask#6968](https://github.com/dask/dask/pull/6968)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Remove bad graph logic in `BlockwiseIO` ([dask#6933](https://github.com/dask/dask/pull/6933)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Get config item if variable is `None` ([dask#6862](https://github.com/dask/dask/pull/6862)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Update `from_pandas` docstring ([dask#6957](https://github.com/dask/dask/pull/6957)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Prevent `fuse_roots` from clobbering annotations ([dask#6955](https://github.com/dask/dask/pull/6955)) [Simon Perkins](https://github.com/sjperkins)

<a id="v2020-12-0"></a>

## 2020.12.0

Released on December 10, 2020

### Highlights

- Switched to [CalVer](https://calver.org/) for versioning scheme.
- Introduced new APIs for `HighLevelGraph` to enable sending high-level representations of
  task graphs to the distributed scheduler.
- Introduced new `HighLevelGraph` layer objects including `BasicLayer`, `Blockwise`,
  `BlockwiseIO`, `ShuffleLayer`, and more.
- Added support for applying custom `Layer`-level annotations like `priority`, `retries`,
  etc. with the `dask.annotations` context manager.
- Updated minimum supported version of pandas to 0.25.0 and NumPy to 1.15.1.
- Support for the `pyarrow.dataset` API to `read_parquet`.
- Several fixes to Dask Array’s SVD.

### All changes

- Make `observed` kwarg optional ([dask#6952](https://github.com/dask/dask/pull/6952)) [Julia Signell](https://github.com/jsignell)
- Min supported pandas 0.25.0 numpy 1.15.1 ([dask#6895](https://github.com/dask/dask/pull/6895)) [Julia Signell](https://github.com/jsignell)
- Make order of categoricals unambiguous ([dask#6949](https://github.com/dask/dask/pull/6949)) [Julia Signell](https://github.com/jsignell)
- Improve “pyarrow-dataset” statistics performance for `read_parquet` ([dask#6918](https://github.com/dask/dask/pull/6918)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `observed` keyword to `groupby` ([dask#6854](https://github.com/dask/dask/pull/6854)) [Julia Signell](https://github.com/jsignell)
- Make sure `include_path_column` works when there are multiple partitions per file ([dask#6911](https://github.com/dask/dask/pull/6911)) [Julia Signell](https://github.com/jsignell)
- Fix: `array.overlap` and `array.map_overlap` block sizes are incorrect when depth is an unsigned bit type ([dask#6909](https://github.com/dask/dask/pull/6909)) [GFleishman](https://github.com/GFleishman)
- Fix syntax error in HLG docs example ([dask#6946](https://github.com/dask/dask/pull/6946)) [Mark](https://github.com/mchi)
- Return a `Bag` from `sample` ([dask#6941](https://github.com/dask/dask/pull/6941)) [Shang Wang](https://github.com/shangw-nvidia)
- Add `ravel_multi_index` ([dask#6939](https://github.com/dask/dask/pull/6939)) [Illviljan](https://github.com/Illviljan)
- Enable parquet metadata collection in parallel ([dask#6921](https://github.com/dask/dask/pull/6921)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid using `_file` in `progressbar` if it is `None` ([dask#6938](https://github.com/dask/dask/pull/6938)) [Mark Harfouche](https://github.com/hmaarrfk)
- Add Zarr to upstream CI build ([dask#6932](https://github.com/dask/dask/pull/6932)) [James Bourbeau](https://github.com/jrbourbeau)
- Introduce `BlockwiseIO` layer ([dask#6878](https://github.com/dask/dask/pull/6878)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Transmit `Layer` Annotations to Scheduler ([dask#6889](https://github.com/dask/dask/pull/6889)) [Simon Perkins](https://github.com/sjperkins)
- Update opportunistic caching page to remove experimental warning ([dask#6926](https://github.com/dask/dask/pull/6926)) [Timost](https://github.com/Timost)
- Allow `pyarrow >2.0.0` ([dask#6772](https://github.com/dask/dask/pull/6772)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Support `pyarrow.dataset` API for `read_parquet` ([dask#6534](https://github.com/dask/dask/pull/6534)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add more informative error message to `da.coarsen` when coarsening factors do not divide shape ([dask#6908](https://github.com/dask/dask/pull/6908)) [Davis Bennett](https://github.com/d-v-b)
- Only run the cron CI on `dask/dask` not forks ([dask#6905](https://github.com/dask/dask/pull/6905)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add `annotations` to `ShuffleLayers` ([dask#6913](https://github.com/dask/dask/pull/6913)) [Matthew Rocklin](https://github.com/mrocklin)
- Temporarily xfail `test_from_s3` ([dask#6915](https://github.com/dask/dask/pull/6915)) [James Bourbeau](https://github.com/jrbourbeau)
- Added dataframe `skew` method ([dask#6881](https://github.com/dask/dask/pull/6881)) [Jan Borchmann](https://github.com/jborchma)
- Fix `dtype` in array `meta` ([dask#6893](https://github.com/dask/dask/pull/6893)) [Julia Signell](https://github.com/jsignell)
- Missing `name` arg in `helm install ...` ([dask#6903](https://github.com/dask/dask/pull/6903)) [Ruben van de Geer](https://github.com/rubenvdg)
- Fix: exception when reading an item with filters ([dask#6901](https://github.com/dask/dask/pull/6901)) [Martin Durant](https://github.com/martindurant)
- Add support for `cupyx` sparse to `dask.array.dot` ([dask#6846](https://github.com/dask/dask/pull/6846)) [Akira Naruse](https://github.com/anaruse)
- Pin array mindeps up a bit to get the tests to pass [test-mindeps] ([dask#6894](https://github.com/dask/dask/pull/6894)) [Julia Signell](https://github.com/jsignell)
- Update/remove pandas and numpy in mindeps ([dask#6888](https://github.com/dask/dask/pull/6888)) [Julia Signell](https://github.com/jsignell)
- Fix `ArrowEngine` bug in use of `clear_known_categories` ([dask#6887](https://github.com/dask/dask/pull/6887)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix documentation about task scheduler ([dask#6879](https://github.com/dask/dask/pull/6879)) [Zhengnan Zhao](https://github.com/zzhengnan)
- Add human relative time formatting utility ([dask#6883](https://github.com/dask/dask/pull/6883)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Possible fix for 6864 `set_index` issue ([dask#6866](https://github.com/dask/dask/pull/6866)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- `BasicLayer`: remove dependency arguments ([dask#6859](https://github.com/dask/dask/pull/6859)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Serialization of `Blockwise` ([dask#6848](https://github.com/dask/dask/pull/6848)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Address `columns=[]` bug ([dask#6871](https://github.com/dask/dask/pull/6871)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid duplicate parquet schema communication ([dask#6841](https://github.com/dask/dask/pull/6841)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `create_metadata_file` utility for existing parquet datasets ([dask#6851](https://github.com/dask/dask/pull/6851)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Improve ordering for workloads with a common terminus ([dask#6779](https://github.com/dask/dask/pull/6779)) [Tom Augspurger](https://github.com/tomaugspurger)
- Stringify utilities ([dask#6852](https://github.com/dask/dask/pull/6852)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Add keyword `overwrite=True` to `to_parquet` to remove dangling files when overwriting a pyarrow `Dataset`. ([dask#6825](https://github.com/dask/dask/pull/6825)) [Greg Hayes](https://github.com/hayesgb)
- Removed `map_tasks()` and `map_basic_layers()` ([dask#6853](https://github.com/dask/dask/pull/6853)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Introduce QR iteration to `svd_compressed` ([dask#6813](https://github.com/dask/dask/pull/6813)) [RogerMoens](https://github.com/RogerMoens)
- `__dask_distributed_pack__()` now takes a `client` argument ([dask#6850](https://github.com/dask/dask/pull/6850)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Use `map_partitions` instead of `delayed` in `set_index` ([dask#6837](https://github.com/dask/dask/pull/6837)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Add doc hit for `as_completed().update(futures)` ([dask#6817](https://github.com/dask/dask/pull/6817)) [manuels](https://github.com/manuels)
- Bump GHA `setup-miniconda` version ([dask#6847](https://github.com/dask/dask/pull/6847)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Remove nans when setting sorted index ([dask#6829](https://github.com/dask/dask/pull/6829)) [Rockwell Weiner](https://github.com/rockwellw)
- Fix transpose of u in SVD ([dask#6799](https://github.com/dask/dask/pull/6799)) [RogerMoens](https://github.com/RogerMoens)
- Migrate to GitHub Actions ([dask#6794](https://github.com/dask/dask/pull/6794)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix sphinx `currentmodule` usage ([dask#6839](https://github.com/dask/dask/pull/6839)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix minimum dependencies CI builds ([dask#6838](https://github.com/dask/dask/pull/6838)) [James Bourbeau](https://github.com/jrbourbeau)
- Avoid graph materialization during `Blockwise` culling ([dask#6815](https://github.com/dask/dask/pull/6815)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fixed typo ([dask#6834](https://github.com/dask/dask/pull/6834)) [Devanshu Desai](https://github.com/devanshuDesai)
- Use `HighLevelGraph.merge` in `collections_to_dsk` ([dask#6836](https://github.com/dask/dask/pull/6836)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Respect `dtype` in svd `compression_matrix` #2849 ([dask#6802](https://github.com/dask/dask/pull/6802)) [RogerMoens](https://github.com/RogerMoens)
- Add blocksize to task name ([dask#6818](https://github.com/dask/dask/pull/6818)) [Julia Signell](https://github.com/jsignell)
- Check for all-NaN partitions ([dask#6821](https://github.com/dask/dask/pull/6821)) [Rockwell Weiner](https://github.com/rockwellw)
- Change “institutional” SQL doc section to point to main SQL doc ([dask#6823](https://github.com/dask/dask/pull/6823)) [Martin Durant](https://github.com/martindurant)
- Fix: `DataFrame.join` doesn’t accept Series as other ([dask#6809](https://github.com/dask/dask/pull/6809)) [David Katz](https://github.com/DavidKatz-il)
- Remove `to_delayed` operations from `to_parquet` ([dask#6801](https://github.com/dask/dask/pull/6801)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Layer annotation docstrings improvements ([dask#6806](https://github.com/dask/dask/pull/6806)) [Simon Perkins](https://github.com/sjperkins)
- Avro reader ([dask#6780](https://github.com/dask/dask/pull/6780)) [Martin Durant](https://github.com/martindurant)
- Rechunk array if smallest chunk size is smaller than depth ([dask#6708](https://github.com/dask/dask/pull/6708)) [Julia Signell](https://github.com/jsignell)
- Add Layer Annotations ([dask#6767](https://github.com/dask/dask/pull/6767)) [Simon Perkins](https://github.com/sjperkins)
- Add “view code” links to documentation ([dask#6793](https://github.com/dask/dask/pull/6793)) [manuels](https://github.com/manuels)
- Add optional IO-subgraph to `Blockwise` Layers ([dask#6715](https://github.com/dask/dask/pull/6715)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add high level graph pack/unpack for distributed ([dask#6786](https://github.com/dask/dask/pull/6786)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Add missing methods of the Dataframe API ([dask#6789](https://github.com/dask/dask/pull/6789)) [Stephannie Jimenez Gacha](https://github.com/steff456)
- Add doc on managing environments ([dask#6778](https://github.com/dask/dask/pull/6778)) [Martin Durant](https://github.com/martindurant)
- HLG: `get_all_external_keys()` ([dask#6774](https://github.com/dask/dask/pull/6774)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Avoid rechunking in reshape with `chunksize=1` ([dask#6748](https://github.com/dask/dask/pull/6748)) [Tom Augspurger](https://github.com/tomaugspurger)
- Try to make categoricals work on join ([dask#6205](https://github.com/dask/dask/pull/6205)) [Julia Signell](https://github.com/jsignell)
- Fix some minor typos and trailing whitespaces in `array-slice.rst` ([dask#6771](https://github.com/dask/dask/pull/6771)) [Magnus Nord](https://github.com/magnunor)
- Bugfix for parquet metadata writes of empty dataframe partitions (pyarrow)  ([dask#6741](https://github.com/dask/dask/pull/6741)) [Callum Noble](https://github.com/callumanoble)
- Document `meta` kwarg in `map_blocks` and `map_overlap`. ([dask#6763](https://github.com/dask/dask/pull/6763)) [Peter Andreas Entschev](https://github.com/pentschev)
- Begin experimenting with parallel prefix scan for `cumsum` and `cumprod` ([dask#6675](https://github.com/dask/dask/pull/6675)) [Erik Welch](https://github.com/eriknw)
- Clarify differences in boolean indexing between dask and numpy arrays ([dask#6764](https://github.com/dask/dask/pull/6764)) [Illviljan](https://github.com/Illviljan)
- Efficient serialization of shuffle layers ([dask#6760](https://github.com/dask/dask/pull/6760)) [James Bourbeau](https://github.com/jrbourbeau)
- Config array optimize to skip fusion and return a HLG ([dask#6751](https://github.com/dask/dask/pull/6751)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Temporarily use `pyarrow<2` in CI ([dask#6759](https://github.com/dask/dask/pull/6759)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix meta for `min`/`max` reductions ([dask#6736](https://github.com/dask/dask/pull/6736)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add 2D possibility to `da.linalg.lstsq` - mirroring numpy ([dask#6749](https://github.com/dask/dask/pull/6749)) [Pascal Bourgault](https://github.com/aulemahal)
- CI: Fixed bug causing flaky test failure in pivot ([dask#6752](https://github.com/dask/dask/pull/6752)) [Tom Augspurger](https://github.com/tomaugspurger)
- Serialization of layers ([dask#6693](https://github.com/dask/dask/pull/6693)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Add `attrs` property to Series/Dataframe ([dask#6742](https://github.com/dask/dask/pull/6742)) [Illviljan](https://github.com/Illviljan)
- Removed Mutable Default Argument ([dask#6747](https://github.com/dask/dask/pull/6747)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Adjust parquet `ArrowEngine` to allow more easy subclass for writing ([dask#6505](https://github.com/dask/dask/pull/6505)) [Joris Van den Bossche](https://github.com/jorisvandenbossche)
- Add `ShuffleStage` HLG Layer ([dask#6650](https://github.com/dask/dask/pull/6650)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Handle literal in `meta_from_array` ([dask#6731](https://github.com/dask/dask/pull/6731)) [Peter Andreas Entschev](https://github.com/pentschev)
- Do balanced rechunking even if chunks are the same ([dask#6735](https://github.com/dask/dask/pull/6735)) [Chris Roat](https://github.com/ChrisRoat)
- Fix docstring `DataFrame.set_index` ([dask#6739](https://github.com/dask/dask/pull/6739)) [Gil Forsyth](https://github.com/gforsyth)
- Ensure `HighLevelGraph` layers always contain `Layer` instances ([dask#6716](https://github.com/dask/dask/pull/6716)) [James Bourbeau](https://github.com/jrbourbeau)
- Map on `HighLevelGraph` Layers ([dask#6689](https://github.com/dask/dask/pull/6689)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Update overlap `*_like` function calls and CuPy tests ([dask#6728](https://github.com/dask/dask/pull/6728)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fixes for `svd` with `__array_function__` ([dask#6727](https://github.com/dask/dask/pull/6727)) [Peter Andreas Entschev](https://github.com/pentschev)
- Added doctest extension for documentation ([dask#6397](https://github.com/dask/dask/pull/6397)) [Jim Circadian](https://github.com/JimCircadian)
- Minor fix to #5628 using @pentschev’s suggestion ([dask#6724](https://github.com/dask/dask/pull/6724)) [John A Kirkham](https://github.com/jakirkham)
- Change type of Dask array when meta type changes ([dask#5628](https://github.com/dask/dask/pull/5628)) [Matthew Rocklin](https://github.com/mrocklin)
- Add `az` ([dask#6719](https://github.com/dask/dask/pull/6719)) [Ray Bell](https://github.com/raybellwaves)
- HLG: `get_dependencies()` of single keys ([dask#6699](https://github.com/dask/dask/pull/6699)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Revert “Revert “Use HighLevelGraph layers everywhere in collections ([dask#6510](https://github.com/dask/dask/pull/6510))” ([dask#6697](https://github.com/dask/dask/pull/6697))” ([dask#6707](https://github.com/dask/dask/pull/6707)) [Tom Augspurger](https://github.com/tomaugspurger)
- Allow `*_like` array creation functions to respect input array type ([dask#6680](https://github.com/dask/dask/pull/6680)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Update `dask-sphinx-theme` version ([dask#6700](https://github.com/dask/dask/pull/6700)) [Gil Forsyth](https://github.com/gforsyth)

<a id="v2-30-0-2020-10-06"></a>

## 2.30.0 / 2020-10-06

### Array

- Allow `rechunk` to evenly split into N chunks ([dask#6420](https://github.com/dask/dask/pull/6420)) [Scott Sievert](https://github.com/stsievert)

<a id="v2-29-0-2020-10-02"></a>

## 2.29.0 / 2020-10-02

### Array

- `_repr_html_`: color sides darker instead of drawing all the lines ([dask#6683](https://github.com/dask/dask/pull/6683)) [Julia Signell](https://github.com/jsignell)
- Removes warning from `nanstd` and `nanvar` ([dask#6667](https://github.com/dask/dask/pull/6667)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Get shape of output from original array - `map_overlap` ([dask#6682](https://github.com/dask/dask/pull/6682)) [Julia Signell](https://github.com/jsignell)
- Replace `np.searchsorted` with `bisect` in indexing ([dask#6669](https://github.com/dask/dask/pull/6669)) [Joachim B Haga](https://github.com/jobh)

### Bag

- Make sure subprocesses have a consistent hash for bag `groupby` ([dask#6660](https://github.com/dask/dask/pull/6660)) [Itamar Turner-Trauring](https://github.com/itamarst)

### Core

- Revert “Use `HighLevelGraph` layers everywhere in collections ([dask#6510](https://github.com/dask/dask/pull/6510))” ([dask#6697](https://github.com/dask/dask/pull/6697)) [Tom Augspurger](https://github.com/tomaugspurger)
- Use `pandas.testing` ([dask#6687](https://github.com/dask/dask/pull/6687)) [John A Kirkham](https://github.com/jakirkham)
- Improve 128-bit floating-point skip in tests ([dask#6676](https://github.com/dask/dask/pull/6676)) [Elliott Sales de Andrade](https://github.com/QuLogic)

### DataFrame

- Allow setting dataframe items using a bool dataframe ([dask#6608](https://github.com/dask/dask/pull/6608)) [Julia Signell](https://github.com/jsignell)

### Documentation

- Fix typo ([dask#6692](https://github.com/dask/dask/pull/6692)) [garanews](https://github.com/garanews)
- Fix a few typos ([dask#6678](https://github.com/dask/dask/pull/6678)) [Pav A](https://github.com/rs2)

<a id="v2-28-0-2020-09-25"></a>

## 2.28.0 / 2020-09-25

### Array

- Partially reverted changes to `Array` indexing that produces large changes.
  This restores the behavior from Dask 2.25.0 and earlier, with a warning
  when large chunks are produced. A configuration option is provided
  to avoid creating the large chunks, see [Efficiency](array-slicing.md#array-slicing-efficiency).
  ([dask#6665](https://github.com/dask/dask/pull/6665)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add `meta` to `to_dask_array` ([dask#6651](https://github.com/dask/dask/pull/6651)) [Kyle Nicholson](https://github.com/kylejn27)
- Fix [dask#6631](https://github.com/dask/dask/pull/6631) and [dask#6611](https://github.com/dask/dask/pull/6611) ([dask#6632](https://github.com/dask/dask/pull/6632)) [Rafal Wojdyla](https://github.com/ravwojdyla)
- Infer object in array reductions ([dask#6629](https://github.com/dask/dask/pull/6629)) [Daniel Saxton](https://github.com/dsaxton)
- Adding `v_based` flag for `svd_flip` ([dask#6658](https://github.com/dask/dask/pull/6658)) [Eric Czech](https://github.com/eric-czech)
- Fix flakey array `mean` ([dask#6656](https://github.com/dask/dask/pull/6656)) [Sam Grayson](https://github.com/charmoniumQ)

### Core

- Removed `dsk` equality check from `SubgraphCallable.__eq__` ([dask#6666](https://github.com/dask/dask/pull/6666)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Use `HighLevelGraph` layers everywhere in collections ([dask#6510](https://github.com/dask/dask/pull/6510)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Adds hash dunder method to `SubgraphCallable` for caching purposes ([dask#6424](https://github.com/dask/dask/pull/6424)) [Andrew Fulton](https://github.com/andrewfulton9)
- Stop writing commented out config files by default ([dask#6647](https://github.com/dask/dask/pull/6647)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Add support for collect list aggregation via `agg` API ([dask#6655](https://github.com/dask/dask/pull/6655)) [Madhur Tandon](https://github.com/madhur-tandon)
- Slightly better error message ([dask#6657](https://github.com/dask/dask/pull/6657)) [Julia Signell](https://github.com/jsignell)

<a id="v2-27-0-2020-09-18"></a>

## 2.27.0 / 2020-09-18

### Array

- Preserve `dtype` in `svd` ([dask#6643](https://github.com/dask/dask/pull/6643)) [Eric Czech](https://github.com/eric-czech)

### Core

- `store()`: create a single HLG layer ([dask#6601](https://github.com/dask/dask/pull/6601)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Add pre-commit CI build ([dask#6645](https://github.com/dask/dask/pull/6645)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `.pre-commit-config` to latest black. ([dask#6641](https://github.com/dask/dask/pull/6641)) [Julia Signell](https://github.com/jsignell)
- Update super usage to remove Python 2 compatibility ([dask#6630](https://github.com/dask/dask/pull/6630)) [Poruri Sai Rahul](https://github.com/rahulporuri)
- Remove u string prefixes ([dask#6633](https://github.com/dask/dask/pull/6633)) [Poruri Sai Rahul](https://github.com/rahulporuri)

### DataFrame

- Improve error message for `to_sql` ([dask#6638](https://github.com/dask/dask/pull/6638)) [Julia Signell](https://github.com/jsignell)
- Use empty list as categories ([dask#6626](https://github.com/dask/dask/pull/6626)) [Julia Signell](https://github.com/jsignell)

### Documentation

- Add `autofunction` to array api docs for more ufuncs ([dask#6644](https://github.com/dask/dask/pull/6644)) [James Bourbeau](https://github.com/jrbourbeau)
- Add a number of missing ufuncs to `dask.array` docs ([dask#6642](https://github.com/dask/dask/pull/6642)) [Ralf Gommers](https://github.com/rgommers)
- Add `HelmCluster` docs ([dask#6290](https://github.com/dask/dask/pull/6290)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2-26-0-2020-09-11"></a>

## 2.26.0 / 2020-09-11

### Array

- Backend-aware dtype inference for single-chunk svd ([dask#6623](https://github.com/dask/dask/pull/6623)) [Eric Czech](https://github.com/eric-czech)
- Make `array.reduction` docstring match for dtype ([dask#6624](https://github.com/dask/dask/pull/6624)) [Martin Durant](https://github.com/martindurant)
- Set lower bound on compression level for `svd_compressed` using rows and cols ([dask#6622](https://github.com/dask/dask/pull/6622)) [Eric Czech](https://github.com/eric-czech)
- Improve SVD consistency and small array handling ([dask#6616](https://github.com/dask/dask/pull/6616)) [Eric Czech](https://github.com/eric-czech)
- Add `svd_flip` #6599 ([dask#6613](https://github.com/dask/dask/pull/6613)) [Eric Czech](https://github.com/eric-czech)
- Handle sequences containing dask Arrays ([dask#6595](https://github.com/dask/dask/pull/6595)) [Gabe Joseph](https://github.com/gjoseph92)
- Avoid large chunks from `getitem` with lists ([dask#6514](https://github.com/dask/dask/pull/6514)) [Tom Augspurger](https://github.com/tomaugspurger)
- Eagerly slice numpy arrays in `from_array` ([dask#6605](https://github.com/dask/dask/pull/6605)) [Deepak Cherian](https://github.com/dcherian)
- Restore ability to pickle dask arrays ([dask#6594](https://github.com/dask/dask/pull/6594)) [Noah D. Brenowitz](https://github.com/nbren12)
- Add SVD support for short-and-fat arrays ([dask#6591](https://github.com/dask/dask/pull/6591)) [Eric Czech](https://github.com/eric-czech)
- Add simple chunk type registry and defer as appropriate to upcast types ([dask#6393](https://github.com/dask/dask/pull/6393)) [Jon Thielen](https://github.com/jthielen)
- Align coarsen chunks by default ([dask#6580](https://github.com/dask/dask/pull/6580)) [Deepak Cherian](https://github.com/dcherian)
- Fixup reshape on unknown dimensions and other testing fixes ([dask#6578](https://github.com/dask/dask/pull/6578)) [Ryan Williams](https://github.com/ryan-williams)

### Core

- Add validation and fixes for `HighLevelGraph` dependencies ([dask#6588](https://github.com/dask/dask/pull/6588)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Fix linting issue ([dask#6598](https://github.com/dask/dask/pull/6598)) [Tom Augspurger](https://github.com/tomaugspurger)
- Skip `bokeh` version 2.0.0 ([dask#6572](https://github.com/dask/dask/pull/6572)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Added bytes/row calculation when using meta ([dask#6585](https://github.com/dask/dask/pull/6585)) [McToel](https://github.com/McToel)
- Handle `min_count` in `Series.sum` / `prod` ([dask#6618](https://github.com/dask/dask/pull/6618)) [Daniel Saxton](https://github.com/dsaxton)
- Update `DataFrame.set_index` docstring ([dask#6549](https://github.com/dask/dask/pull/6549)) [Timost](https://github.com/Timost)
- Always compute 0 and 1 quantiles during quantile calculations ([dask#6564](https://github.com/dask/dask/pull/6564)) [Erik Welch](https://github.com/eriknw)
- Fix wrong path when reading empty csv file ([dask#6573](https://github.com/dask/dask/pull/6573)) [Abdulelah Bin Mahfoodh](https://github.com/abduhbm)

### Documentation

- Doc: Troubleshooting dashboard 404 ([dask#6215](https://github.com/dask/dask/pull/6215)) [Kilian Lieret](https://github.com/klieret)
- Fixup `extraConfig` example ([dask#6625](https://github.com/dask/dask/pull/6625)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update supported Python versions ([dask#6609](https://github.com/dask/dask/pull/6609)) [Julia Signell](https://github.com/jsignell)
- Document dask/daskhub helm chart ([dask#6560](https://github.com/dask/dask/pull/6560)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-25-0-2020-08-28"></a>

## 2.25.0 / 2020-08-28

### Core

- Compare key hashes in `subs()` ([dask#6559](https://github.com/dask/dask/pull/6559)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Rerun with latest `black` release ([dask#6568](https://github.com/dask/dask/pull/6568)) [James Bourbeau](https://github.com/jrbourbeau)
- License update ([dask#6554](https://github.com/dask/dask/pull/6554)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Add gs `read_parquet` example ([dask#6548](https://github.com/dask/dask/pull/6548)) [Ray Bell](https://github.com/raybellwaves)

### Documentation

- Remove version from documentation page names ([dask#6558](https://github.com/dask/dask/pull/6558)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `kubernetes-helm.rst` ([dask#6523](https://github.com/dask/dask/pull/6523)) [David Sheldon](https://github.com/davidsmf)
- Stop 2020 survey ([dask#6547](https://github.com/dask/dask/pull/6547)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-24-0-2020-08-22"></a>

## 2.24.0 / 2020-08-22

### Array

- Fix setting random seed in tests. ([dask#6518](https://github.com/dask/dask/pull/6518)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Support meta in apply gufunc ([dask#6521](https://github.com/dask/dask/pull/6521)) [joshreback](https://github.com/joshreback)
- Replace cupy.sparse with cupyx.scipy.sparse ([dask#6530](https://github.com/dask/dask/pull/6530)) [John A Kirkham](https://github.com/jakirkham)

### Dataframe

- Bump up tolerance for rolling tests ([dask#6502](https://github.com/dask/dask/pull/6502)) [Julia Signell](https://github.com/jsignell)
- Implement DatFrame._\_len_\_ ([dask#6515](https://github.com/dask/dask/pull/6515)) [Tom Augspurger](https://github.com/tomaugspurger)
- Infer arrow schema in to_parquet  (for ArrowEngine\`) ([dask#6490](https://github.com/dask/dask/pull/6490)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Fix parquet test when no pyarrow ([dask#6524](https://github.com/dask/dask/pull/6524)) [Martin Durant](https://github.com/martindurant)
- Remove problematic `filter` arguments in ArrowEngine ([dask#6527](https://github.com/dask/dask/pull/6527)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Avoid schema validation by default in ArrowEngine ([dask#6536](https://github.com/dask/dask/pull/6536)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Core

- Use unpack_collections in make_blockwise_graph ([dask#6517](https://github.com/dask/dask/pull/6517)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Move key_split() from optimization.py to utils.py ([dask#6529](https://github.com/dask/dask/pull/6529)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Make tests run on moto server ([dask#6528](https://github.com/dask/dask/pull/6528)) [Martin Durant](https://github.com/martindurant)

<a id="v2-23-0-2020-08-14"></a>

## 2.23.0 / 2020-08-14

### Array

- Reduce `np.zeros`, `ones`, and `full` array size with broadcasting ([dask#6491](https://github.com/dask/dask/pull/6491)) [Matthias Bussonnier](https://github.com/Carreau)
- Add missing `meta=` for `trim` in `map_overlap` ([dask#6494](https://github.com/dask/dask/pull/6494)) [Peter Andreas Entschev](https://github.com/pentschev)

### Bag

- Bag repartition partition size ([dask#6371](https://github.com/dask/dask/pull/6371)) [joshreback](https://github.com/joshreback)

### Core

- `Scalar.__dask_layers__()` to return `self._name` instead of `self.key` ([dask#6507](https://github.com/dask/dask/pull/6507)) [Mads R. B. Kristensen](https://github.com/madsbk)
- Update dependencies correctly in `fuse_root` optimization ([dask#6508](https://github.com/dask/dask/pull/6508)) [Mads R. B. Kristensen](https://github.com/madsbk)

### DataFrame

- Adds `items` to dataframe ([dask#6503](https://github.com/dask/dask/pull/6503)) [Thomas J. Fan](https://github.com/thomasjpfan)
- Include compression in `write_table` call ([dask#6499](https://github.com/dask/dask/pull/6499)) [Julia Signell](https://github.com/jsignell)
- Fixed warning in `nonempty_series` ([dask#6485](https://github.com/dask/dask/pull/6485)) [Tom Augspurger](https://github.com/tomaugspurger)
- Intelligently determine partitions based on type of first arg ([dask#6479](https://github.com/dask/dask/pull/6479)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix pyarrow `mkdirs` ([dask#6475](https://github.com/dask/dask/pull/6475)) [Julia Signell](https://github.com/jsignell)
- Fix duplicate parquet output in `to_parquet` ([dask#6451](https://github.com/dask/dask/pull/6451)) [michaelnarodovitch](https://github.com/michaelnarodovitch)

### Documentation

- Fix documentation `da.histogram` ([dask#6439](https://github.com/dask/dask/pull/6439)) [Roberto Panai](https://github.com/rpanai)
- Add `agg` `nunique` example ([dask#6404](https://github.com/dask/dask/pull/6404)) [Ray Bell](https://github.com/raybellwaves)
- Fixed a few typos in the SQL docs ([dask#6489](https://github.com/dask/dask/pull/6489)) [Mike McCarty](https://github.com/mmccarty)
- Docs for SQLing ([dask#6453](https://github.com/dask/dask/pull/6453)) [Martin Durant](https://github.com/martindurant)

<a id="v2-22-0-2020-07-31"></a>

## 2.22.0 / 2020-07-31

### Array

- Compatibility for NumPy dtype deprecation ([dask#6430](https://github.com/dask/dask/pull/6430)) [Tom Augspurger](https://github.com/tomaugspurger)

### Core

- Implement `sizeof` for some `bytes`-like objects ([dask#6457](https://github.com/dask/dask/pull/6457)) [John A Kirkham](https://github.com/jakirkham)
- HTTP error for new `fsspec` ([dask#6446](https://github.com/dask/dask/pull/6446)) [Martin Durant](https://github.com/martindurant)
- When `RecursionError` is raised, return uuid from `tokenize` function ([dask#6437](https://github.com/dask/dask/pull/6437)) [Julia Signell](https://github.com/jsignell)
- Install deps of upstream-dev packages ([dask#6431](https://github.com/dask/dask/pull/6431)) [Tom Augspurger](https://github.com/tomaugspurger)
- Use updated link in `setup.cfg` ([dask#6426](https://github.com/dask/dask/pull/6426)) [Zhengnan Zhao](https://github.com/zzhengnan)

### DataFrame

- Add single quotes around column names if strings ([dask#6471](https://github.com/dask/dask/pull/6471)) [Gil Forsyth](https://github.com/gforsyth)
- Refactor `ArrowEngine` for better `read_parquet` performance ([dask#6346](https://github.com/dask/dask/pull/6346)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add `tolist` dispatch ([dask#6444](https://github.com/dask/dask/pull/6444)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Compatibility with pandas 1.1.0rc0 ([dask#6429](https://github.com/dask/dask/pull/6429)) [Tom Augspurger](https://github.com/tomaugspurger)
- Multi value pivot table ([dask#6428](https://github.com/dask/dask/pull/6428)) [joshreback](https://github.com/joshreback)
- Duplicate argument definitions in `to_csv` docstring ([dask#6411](https://github.com/dask/dask/pull/6411)) [Jun Han (Johnson) Ooi](https://github.com/tebesfinwo)

### Documentation

- Add utility to docs to convert YAML config to env vars and back ([dask#6472](https://github.com/dask/dask/pull/6472)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix parameter server rendering ([dask#6466](https://github.com/dask/dask/pull/6466)) [Scott Sievert](https://github.com/stsievert)
- Fixes broken links ([dask#6403](https://github.com/dask/dask/pull/6403)) [Jim Circadian](https://github.com/JimCircadian)
- Complete parameter server implementation in docs ([dask#6449](https://github.com/dask/dask/pull/6449)) [Scott Sievert](https://github.com/stsievert)
- Fix typo ([dask#6436](https://github.com/dask/dask/pull/6436)) [Jack Xiaosong Xu](https://github.com/jackxxu)

<a id="v2-21-0-2020-07-17"></a>

## 2.21.0 / 2020-07-17

### Array

- Correct error message in `array.routines.gradient()` ([dask#6417](https://github.com/dask/dask/pull/6417)) [johnomotani](https://github.com/johnomotani)
- Fix blockwise concatenate for array with some `dimension=1` ([dask#6342](https://github.com/dask/dask/pull/6342)) [Matthias Bussonnier](https://github.com/Carreau)

### Bag

- Fix `bag.take` example ([dask#6418](https://github.com/dask/dask/pull/6418)) [Roberto Panai](https://github.com/rpanai)

### Core

- Groups values in optimization pass should only be graph and keys – not an optimization + keys ([dask#6409](https://github.com/dask/dask/pull/6409)) [Benjamin Zaitlen](https://github.com/quasiben)
- Call custom optimizations once, with `kwargs` provided ([dask#6382](https://github.com/dask/dask/pull/6382)) [Clark Zinzow](https://github.com/clarkzinzow)
- Include `pickle5` for testing on Python 3.7 ([dask#6379](https://github.com/dask/dask/pull/6379)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Correct typo in error message ([dask#6422](https://github.com/dask/dask/pull/6422)) [Tom McTiernan](https://github.com/tmct)
- Use `pytest.warns` to check for `UserWarning` ([dask#6378](https://github.com/dask/dask/pull/6378)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Parse `bytes_per_chunk keyword` from string ([dask#6370](https://github.com/dask/dask/pull/6370)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Numpydoc formatting ([dask#6421](https://github.com/dask/dask/pull/6421)) [Matthias Bussonnier](https://github.com/Carreau)
- Unpin `numpydoc` following 1.1 release ([dask#6407](https://github.com/dask/dask/pull/6407)) [Gil Forsyth](https://github.com/gforsyth)
- Numpydoc formatting ([dask#6402](https://github.com/dask/dask/pull/6402)) [Matthias Bussonnier](https://github.com/Carreau)
- Add instructions for using conda when installing code for development ([dask#6399](https://github.com/dask/dask/pull/6399)) [Ray Bell](https://github.com/raybellwaves)
- Update `visualize` docstrings ([dask#6383](https://github.com/dask/dask/pull/6383)) [Zhengnan Zhao](https://github.com/zzhengnan)

<a id="v2-20-0-2020-07-02"></a>

## 2.20.0 / 2020-07-02

### Array

- Register `sizeof` for numpy zero-strided arrays ([dask#6343](https://github.com/dask/dask/pull/6343)) [Matthias Bussonnier](https://github.com/Carreau)
- Use `concatenate_lookup` in `concatenate` ([dask#6339](https://github.com/dask/dask/pull/6339)) [John A Kirkham](https://github.com/jakirkham)
- Fix rechunking of arrays with some zero-length dimensions ([dask#6335](https://github.com/dask/dask/pull/6335)) [Matthias Bussonnier](https://github.com/Carreau)

### DataFrame

- Dispatch `iloc`` calls to `getitem` ([dask#6355](https://github.com/dask/dask/pull/6355)) [Gil Forsyth](https://github.com/gforsyth)
- Handle unnamed pandas `RangeIndex` in fastparquet engine ([dask#6350](https://github.com/dask/dask/pull/6350)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Preserve index when writing partitioned parquet datasets with pyarrow ([dask#6282](https://github.com/dask/dask/pull/6282)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use `ignore_index` for pandas’ `group_split_dispatch` ([dask#6251](https://github.com/dask/dask/pull/6251)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- Add doc describing argument ([dask#6318](https://github.com/dask/dask/pull/6318)) [asmith26](https://github.com/asmith26)

<a id="v2-19-0-2020-06-19"></a>

## 2.19.0 / 2020-06-19

### Array

- Cast chunk sizes to python int `dtype` ([dask#6326](https://github.com/dask/dask/pull/6326)) [Gil Forsyth](https://github.com/gforsyth)
- Add `shape=None` to `*_like()` array creation functions ([dask#6064](https://github.com/dask/dask/pull/6064)) [Anderson Banihirwe](https://github.com/andersy005)

### Core

- Update expected error msg for protocol difference in fsspec ([dask#6331](https://github.com/dask/dask/pull/6331)) [Gil Forsyth](https://github.com/gforsyth)
- Fix for floats < 1 in `parse_bytes` ([dask#6311](https://github.com/dask/dask/pull/6311)) [Gil Forsyth](https://github.com/gforsyth)
- Fix exception causes all over the codebase ([dask#6308](https://github.com/dask/dask/pull/6308)) [Ram Rachum](https://github.com/cool-RR)
- Fix duplicated tests ([dask#6303](https://github.com/dask/dask/pull/6303)) [James Lamb](https://github.com/jameslamb)
- Remove unused testing function ([dask#6304](https://github.com/dask/dask/pull/6304)) [James Lamb](https://github.com/jameslamb)

### DataFrame

- Add high-level CSV Subgraph ([dask#6262](https://github.com/dask/dask/pull/6262)) [Gil Forsyth](https://github.com/gforsyth)
- Fix `ValueError` when merging an index-only 1-partition dataframe ([dask#6309](https://github.com/dask/dask/pull/6309)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Make `index.map` clear divisions. ([dask#6285](https://github.com/dask/dask/pull/6285)) [Julia Signell](https://github.com/jsignell)

### Documentation

- Add link to 2020 survey ([dask#6328](https://github.com/dask/dask/pull/6328)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update `bag.rst` ([dask#6317](https://github.com/dask/dask/pull/6317)) [Ben Shaver](https://github.com/bpshaver)

<a id="v2-18-1-2020-06-09"></a>

## 2.18.1 / 2020-06-09

### Array

- Don’t try to set name on `full` ([dask#6299](https://github.com/dask/dask/pull/6299)) [Julia Signell](https://github.com/jsignell)
- Histogram: support lazy values for range/bins (another way) ([dask#6252](https://github.com/dask/dask/pull/6252)) [Gabe Joseph](https://github.com/gjoseph92)

### Core

- Fix exception causes in `utils.py` ([dask#6302](https://github.com/dask/dask/pull/6302)) [Ram Rachum](https://github.com/cool-RR)
- Improve performance of `HighLevelGraph` construction ([dask#6293](https://github.com/dask/dask/pull/6293)) [Julia Signell](https://github.com/jsignell)

### Documentation

- Now readthedocs builds unrelased features’ docstrings ([dask#6295](https://github.com/dask/dask/pull/6295)) [Antonio Ercole De Luca](https://github.com/eracle)
- Add `asyncssh` intersphinx mappings ([dask#6298](https://github.com/dask/dask/pull/6298)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

<a id="v2-18-0-2020-06-05"></a>

## 2.18.0 / 2020-06-05

### Array

- Cast slicing index to dask array if same shape as original ([dask#6273](https://github.com/dask/dask/pull/6273)) [Julia Signell](https://github.com/jsignell)
- Fix `stack` error message ([dask#6268](https://github.com/dask/dask/pull/6268)) [Stephanie Gott](https://github.com/stephaniegott)
- `full` & `full_like`: error on non-scalar `fill_value` ([dask#6129](https://github.com/dask/dask/pull/6129)) [Huite](https://github.com/Huite)
- Support for multiple arrays in `map_overlap` ([dask#6165](https://github.com/dask/dask/pull/6165)) [Eric Czech](https://github.com/eric-czech)
- Pad resample divisions so that edges are counted ([dask#6255](https://github.com/dask/dask/pull/6255)) [Julia Signell](https://github.com/jsignell)

### Bag

- Random sampling of k elements from a dask bag #4799 ([dask#6239](https://github.com/dask/dask/pull/6239)) [Antonio Ercole De Luca](https://github.com/eracle)

### DataFrame

- Add `dropna`, `sort`, and `ascending` to `sort_values` ([dask#5880](https://github.com/dask/dask/pull/5880)) [Julia Signell](https://github.com/jsignell)
- Generalize `from_dask_array` ([dask#6263](https://github.com/dask/dask/pull/6263)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Add derived docstring for `SeriesGroupby.nunique` ([dask#6284](https://github.com/dask/dask/pull/6284)) [Julia Signell](https://github.com/jsignell)
- Remove `NotImplementedError` in resample with rule  ([dask#6274](https://github.com/dask/dask/pull/6274)) [Abdulelah Bin Mahfoodh](https://github.com/abduhbm)
- Add `dd.to_sql` ([dask#6038](https://github.com/dask/dask/pull/6038)) [Ryan Williams](https://github.com/ryan-williams)

### Documentation

- Update remote data section ([dask#6258](https://github.com/dask/dask/pull/6258)) [Ray Bell](https://github.com/raybellwaves)

<a id="v2-17-2-2020-05-28"></a>

## 2.17.2 / 2020-05-28

### Core

- Re-add the `complete` extra ([dask#6257](https://github.com/dask/dask/pull/6257)) [Jim Crist-Harif](https://github.com/jcrist)

### DataFrame

- Raise error if `resample` isn’t going to give right answer ([dask#6244](https://github.com/dask/dask/pull/6244)) [Julia Signell](https://github.com/jsignell)

<a id="v2-17-1-2020-05-28"></a>

## 2.17.1 / 2020-05-28

### Array

- Empty array rechunk ([dask#6233](https://github.com/dask/dask/pull/6233)) [Andrew Fulton](https://github.com/andrewfulton9)

### Core

- Make `pyyaml` required ([dask#6250](https://github.com/dask/dask/pull/6250)) [Jim Crist-Harif](https://github.com/jcrist)
- Fix install commands from `ImportError` ([dask#6238](https://github.com/dask/dask/pull/6238)) [Gaurav Sheni](https://github.com/gsheni)
- Remove issue template ([dask#6249](https://github.com/dask/dask/pull/6249)) [Jacob Tomlinson](https://github.com/jacobtomlinson)

### DataFrame

- Pass `ignore_index` to `dd_shuffle` from `DataFrame.shuffle` ([dask#6247](https://github.com/dask/dask/pull/6247)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Cope with missing HDF keys ([dask#6204](https://github.com/dask/dask/pull/6204)) [Martin Durant](https://github.com/martindurant)
- Generalize `describe` & `quantile` apis ([dask#5137](https://github.com/dask/dask/pull/5137)) [GALI PREM SAGAR](https://github.com/galipremsagar)

<a id="v2-17-0-2020-05-26"></a>

## 2.17.0 / 2020-05-26

### Array

- Small improvements to `da.pad` ([dask#6213](https://github.com/dask/dask/pull/6213)) [Mark Boer](https://github.com/mark-boer)
- Return `tuple` if multiple outputs in `dask.array.apply_gufunc`, add test to check for tuple ([dask#6207](https://github.com/dask/dask/pull/6207)) [Kai Mühlbauer](https://github.com/kmuehlbauer)
- Support `stack` with unknown chunksizes ([dask#6195](https://github.com/dask/dask/pull/6195)) [swapna](https://github.com/swapna-pg)

### Bag

- Random Choice on Bags ([dask#6208](https://github.com/dask/dask/pull/6208)) [Antonio Ercole De Luca](https://github.com/eracle)

### Core

- Raise warning `delayed.visualise()` ([dask#6216](https://github.com/dask/dask/pull/6216)) [Amol Umbarkar](https://github.com/mindhash)
- Ensure other pickle arguments work ([dask#6229](https://github.com/dask/dask/pull/6229)) [John A Kirkham](https://github.com/jakirkham)
- Overhaul `fuse()` config ([dask#6198](https://github.com/dask/dask/pull/6198)) [crusaderky](https://github.com/crusaderky)
- Update `dask.order.order` to consider “next” nodes using both FIFO and LIFO ([dask#5872](https://github.com/dask/dask/pull/5872)) [Erik Welch](https://github.com/eriknw)

### DataFrame

- Use 0 as `fill_value` for more agg methods ([dask#6245](https://github.com/dask/dask/pull/6245)) [Julia Signell](https://github.com/jsignell)
- Generalize `rearrange_by_column_tasks` and add `DataFrame.shuffle` ([dask#6066](https://github.com/dask/dask/pull/6066)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Xfail `test_rolling_numba_engine` for newer numba and older pandas ([dask#6236](https://github.com/dask/dask/pull/6236)) [James Bourbeau](https://github.com/jrbourbeau)
- Generalize `fix_overlap` ([dask#6240](https://github.com/dask/dask/pull/6240)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Fix `DataFrame.shape` with no columns ([dask#6237](https://github.com/dask/dask/pull/6237)) [noreentry](https://github.com/noreentry)
- Avoid shuffle when setting a presorted index with overlapping divisions ([dask#6226](https://github.com/dask/dask/pull/6226)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Adjust the Parquet engine classes to allow more easily subclassing ([dask#6211](https://github.com/dask/dask/pull/6211)) [Marius van Niekerk](https://github.com/mariusvniekerk)
- Fix `dd.merge_asof` with `left_on='col'` & `right_index=True` ([dask#6192](https://github.com/dask/dask/pull/6192)) [noreentry](https://github.com/noreentry)
- Disable warning for `concat` ([dask#6210](https://github.com/dask/dask/pull/6210)) [Tung Dang](https://github.com/3cham)
- Move `AUTO_BLOCKSIZE` out of `read_csv` signature ([dask#6214](https://github.com/dask/dask/pull/6214)) [Jim Crist-Harif](https://github.com/jcrist)
- `.loc` indexing with callable ([dask#6185](https://github.com/dask/dask/pull/6185)) [Endre Mark Borza](https://github.com/endremborza)
- Avoid apply in `_compute_sum_of_squares` for groupby std agg ([dask#6186](https://github.com/dask/dask/pull/6186)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Minor correction to `test_parquet` ([dask#6190](https://github.com/dask/dask/pull/6190)) [Brian Larsen](https://github.com/brl0)
- Adhering to the passed pat for delimeter join and fix error message ([dask#6194](https://github.com/dask/dask/pull/6194)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Skip `test_to_parquet_with_get` if no parquet libs available ([dask#6188](https://github.com/dask/dask/pull/6188)) [Scott Sanderson](https://github.com/ssanderson)

### Documentation

- Added documentation for `distributed.Event` class ([dask#6231](https://github.com/dask/dask/pull/6231)) [Nils Braun](https://github.com/nils-braun)
- Doc write to remote ([dask#6124](https://github.com/dask/dask/pull/6124)) [Ray Bell](https://github.com/raybellwaves)

<a id="v2-16-0-2020-05-08"></a>

## 2.16.0 / 2020-05-08

### Array

- Fix array general-reduction name ([dask#6176](https://github.com/dask/dask/pull/6176)) [Nick Evans](https://github.com/nre)
- Replace `dim` with `shape` in `unravel_index` ([dask#6155](https://github.com/dask/dask/pull/6155)) [Julia Signell](https://github.com/jsignell)
- Moment: handle all elements being masked ([dask#5339](https://github.com/dask/dask/pull/5339)) [Gabe Joseph](https://github.com/gjoseph92)

### Core

- Remove Redundant string concatenations in dask code-base ([dask#6137](https://github.com/dask/dask/pull/6137)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Upstream compat ([dask#6159](https://github.com/dask/dask/pull/6159)) [Tom Augspurger](https://github.com/tomaugspurger)
- Ensure `sizeof` of dict and sequences returns an integer ([dask#6179](https://github.com/dask/dask/pull/6179)) [James Bourbeau](https://github.com/jrbourbeau)
- Estimate python collection sizes with random sampling ([dask#6154](https://github.com/dask/dask/pull/6154)) [Florian Jetter](https://github.com/fjetter)
- Update test upstream ([dask#6146](https://github.com/dask/dask/pull/6146)) [Tom Augspurger](https://github.com/tomaugspurger)
- Skip test for mindeps build ([dask#6144](https://github.com/dask/dask/pull/6144)) [Tom Augspurger](https://github.com/tomaugspurger)
- Switch default multiprocessing context to “spawn” ([dask#4003](https://github.com/dask/dask/pull/4003)) [Itamar Turner-Trauring](https://github.com/itamarst)
- Update manifest to include dask-schema ([dask#6140](https://github.com/dask/dask/pull/6140)) [Benjamin Zaitlen](https://github.com/quasiben)

### DataFrame

- Harden inconsistent-schema handling in pyarrow-based `read_parquet` ([dask#6160](https://github.com/dask/dask/pull/6160)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Add compute `kwargs` to methods that write data to disk ([dask#6056](https://github.com/dask/dask/pull/6056)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Fix issue where `unique` returns an index like result from backends ([dask#6153](https://github.com/dask/dask/pull/6153)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Fix internal error in `map_partitions` with collections ([dask#6103](https://github.com/dask/dask/pull/6103)) [Tom Augspurger](https://github.com/tomaugspurger)

### Documentation

- Add phase of computation to index TOC ([dask#6157](https://github.com/dask/dask/pull/6157)) [Benjamin Zaitlen](https://github.com/quasiben)
- Remove unused imports in scheduling script ([dask#6138](https://github.com/dask/dask/pull/6138)) [James Lamb](https://github.com/jameslamb)
- Fix indent ([dask#6147](https://github.com/dask/dask/pull/6147)) [Martin Durant](https://github.com/martindurant)
- Add Tom’s log config example ([dask#6143](https://github.com/dask/dask/pull/6143)) [Martin Durant](https://github.com/martindurant)

<a id="v2-15-0-2020-04-24"></a>

## 2.15.0 / 2020-04-24

### Array

- Update `dask.array.from_array` to warn when passed a Dask collection ([dask#6122](https://github.com/dask/dask/pull/6122)) [James Bourbeau](https://github.com/jrbourbeau)
- Un-numpy like behaviour in `dask.array.pad` ([dask#6042](https://github.com/dask/dask/pull/6042)) [Mark Boer](https://github.com/mark-boer)
- Add support for `repeats=0` in `da.repeat` ([dask#6080](https://github.com/dask/dask/pull/6080)) [James Bourbeau](https://github.com/jrbourbeau)

### Core

- Fix yaml layout for schema ([dask#6132](https://github.com/dask/dask/pull/6132)) [Benjamin Zaitlen](https://github.com/quasiben)
- Configuration Reference ([dask#6069](https://github.com/dask/dask/pull/6069)) [Benjamin Zaitlen](https://github.com/quasiben)
- Add configuration option to turn off task fusion ([dask#6087](https://github.com/dask/dask/pull/6087)) [Matthew Rocklin](https://github.com/mrocklin)
- Skip pyarrow on windows ([dask#6094](https://github.com/dask/dask/pull/6094)) [Tom Augspurger](https://github.com/tomaugspurger)
- Set limit to maximum length of fused key ([dask#6057](https://github.com/dask/dask/pull/6057)) [Lucas Rademaker](https://github.com/lr4d)
- Add test against #6062 ([dask#6072](https://github.com/dask/dask/pull/6072)) [Martin Durant](https://github.com/martindurant)
- Bump checkout action to v2 ([dask#6065](https://github.com/dask/dask/pull/6065)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Generalize categorical calls to support cudf `Categorical` ([dask#6113](https://github.com/dask/dask/pull/6113)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Avoid reading `_metadata` on every worker ([dask#6017](https://github.com/dask/dask/pull/6017)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Use `group_split_dispatch` and `ignore_index` in `apply_concat_apply` ([dask#6119](https://github.com/dask/dask/pull/6119)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Handle new (dtype) pandas metadata with pyarrow ([dask#6090](https://github.com/dask/dask/pull/6090)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Skip `test_partition_on_cats_pyarrow` if pyarrow is not installed ([dask#6112](https://github.com/dask/dask/pull/6112)) [James Bourbeau](https://github.com/jrbourbeau)
- Update DataFrame len to handle columns with the same name ([dask#6111](https://github.com/dask/dask/pull/6111)) [James Bourbeau](https://github.com/jrbourbeau)
- `ArrowEngine` bug fixes and test coverage ([dask#6047](https://github.com/dask/dask/pull/6047)) [Richard (Rick) Zamora](https://github.com/rjzamora)
- Added mode ([dask#5958](https://github.com/dask/dask/pull/5958)) [Adam Lewis](https://github.com/Adam-D-Lewis)

### Documentation

- Update “helm install” for helm 3 usage ([dask#6130](https://github.com/dask/dask/pull/6130)) [JulianWgs](https://github.com/JulianWgs)
- Extend preload documentation ([dask#6077](https://github.com/dask/dask/pull/6077)) [Matthew Rocklin](https://github.com/mrocklin)
- Fixed small typo in DataFrame `map_partitions()` docstring ([dask#6115](https://github.com/dask/dask/pull/6115)) [Eugene Huang](https://github.com/eugeneh101)
- Fix typo: “double” should be times, not plus ([dask#6091](https://github.com/dask/dask/pull/6091)) [David Chudzicki](https://github.com/dchudz)
- Fix first line of `array.random.*` docs ([dask#6063](https://github.com/dask/dask/pull/6063)) [Martin Durant](https://github.com/martindurant)
- Add section about `Semaphore` in distributed ([dask#6053](https://github.com/dask/dask/pull/6053)) [Florian Jetter](https://github.com/fjetter)

<a id="v2-14-0-2020-04-03"></a>

## 2.14.0 / 2020-04-03

### Array

- Added `np.iscomplexobj` implementation ([dask#6045](https://github.com/dask/dask/pull/6045)) [Tom Augspurger](https://github.com/tomaugspurger)

### Core

- Update `test_rearrange_disk_cleanup_with_exception` to pass without cloudpickle installed ([dask#6052](https://github.com/dask/dask/pull/6052)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixed flaky `test-rearrange` ([dask#5977](https://github.com/dask/dask/pull/5977)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Use `_meta_nonempty` for dtype casting in `stack_partitions` ([dask#6061](https://github.com/dask/dask/pull/6061)) [mlondschien](https://github.com/mlondschien)
- Fix bugs in `_metadata` creation and filtering in parquet `ArrowEngine` ([dask#6023](https://github.com/dask/dask/pull/6023)) [Richard (Rick) Zamora](https://github.com/rjzamora)

### Documentation

- DOC: Add name caveats ([dask#6040](https://github.com/dask/dask/pull/6040)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-13-0-2020-03-25"></a>

## 2.13.0 / 2020-03-25

### Array

- Support `dtype` and other keyword arguments in `da.random` ([dask#6030](https://github.com/dask/dask/pull/6030)) [Matthew Rocklin](https://github.com/mrocklin)
- Register support for `cupy` sparse `hstack`/`vstack` ([dask#5735](https://github.com/dask/dask/pull/5735)) [Corey J. Nolet](https://github.com/cjnolet)
- Force `self.name` to `str` in `dask.array` ([dask#6002](https://github.com/dask/dask/pull/6002)) [Chuanzhu Xu](https://github.com/xcz011)

### Bag

- Set `rename_fused_keys` to `None` by default in `bag.optimize` ([dask#6000](https://github.com/dask/dask/pull/6000)) [Lucas Rademaker](https://github.com/lr4d)

### Core

- Copy dict in `to_graphviz` to prevent overwriting ([dask#5996](https://github.com/dask/dask/pull/5996)) [JulianWgs](https://github.com/JulianWgs)
- Stricter pandas `xfail` ([dask#6024](https://github.com/dask/dask/pull/6024)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix CI failures ([dask#6013](https://github.com/dask/dask/pull/6013)) [James Bourbeau](https://github.com/jrbourbeau)
- Update `toolz` to 0.8.2 and use `tlz` ([dask#5997](https://github.com/dask/dask/pull/5997)) [Ryan Grout](https://github.com/groutr)
- Move Windows CI builds to GitHub Actions ([dask#5862](https://github.com/dask/dask/pull/5862)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Improve path-related exceptions in `read_hdf` ([dask#6032](https://github.com/dask/dask/pull/6032)) [psimaj](https://github.com/psimaj)
- Fix `dtype` handling in `dd.concat` ([dask#6006](https://github.com/dask/dask/pull/6006)) [mlondschien](https://github.com/mlondschien)
- Handle cudf’s leftsemi and leftanti joins ([dask#6025](https://github.com/dask/dask/pull/6025)) [Richard J Zamora](https://github.com/rjzamora)
- Remove unused `npartitions` variable in `dd.from_pandas` ([dask#6019](https://github.com/dask/dask/pull/6019)) [Daniel Saxton](https://github.com/dsaxton)
- Added shuffle to `DataFrame.random_split` ([dask#5980](https://github.com/dask/dask/pull/5980)) [petiop](https://github.com/petiop)

### Documentation

- Fix indentation in scheduler-overview docs ([dask#6022](https://github.com/dask/dask/pull/6022)) [Matthew Rocklin](https://github.com/mrocklin)
- Update task graphs in optimize docs ([dask#5928](https://github.com/dask/dask/pull/5928)) [Julia Signell](https://github.com/jsignell)
- Optionally get rid of intermediary boxes in visualize, and add more labels ([dask#5976](https://github.com/dask/dask/pull/5976)) [Julia Signell](https://github.com/jsignell)

<a id="v2-12-0-2020-03-06"></a>

## 2.12.0 / 2020-03-06

### Array

- Improve reuse of temporaries with numpy ([dask#5933](https://github.com/dask/dask/pull/5933)) [Bruce Merry](https://github.com/bmerry)
- Make `map_blocks` with `block_info` produce a `Blockwise` ([dask#5896](https://github.com/dask/dask/pull/5896)) [Bruce Merry](https://github.com/bmerry)
- Optimize `make_blockwise_graph` ([dask#5940](https://github.com/dask/dask/pull/5940)) [Bruce Merry](https://github.com/bmerry)
- Fix axes ordering in `da.tensordot` ([dask#5975](https://github.com/dask/dask/pull/5975)) [Gil Forsyth](https://github.com/gforsyth)
- Adds empty mode to `array.pad` ([dask#5931](https://github.com/dask/dask/pull/5931)) [Thomas J. Fan](https://github.com/thomasjpfan)

### Core

- Remove `toolz.memoize` dependency in `dask.utils` ([dask#5978](https://github.com/dask/dask/pull/5978)) [Ryan Grout](https://github.com/groutr)
- Close pool leaking subprocess ([dask#5979](https://github.com/dask/dask/pull/5979)) [Tom Augspurger](https://github.com/tomaugspurger)
- Pin `numpydoc` to `0.8.0` (fix double autoescape) ([dask#5961](https://github.com/dask/dask/pull/5961)) [Gil Forsyth](https://github.com/gforsyth)
- Register deterministic tokenization for `range` objects ([dask#5947](https://github.com/dask/dask/pull/5947)) [James Bourbeau](https://github.com/jrbourbeau)
- Unpin `msgpack` in CI ([dask#5930](https://github.com/dask/dask/pull/5930)) [JAmes Bourbeau](https://github.com/jrbourbeau)
- Ensure dot results are placed in unique files. ([dask#5937](https://github.com/dask/dask/pull/5937)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Add remaining optional dependencies to Travis 3.8 CI build environment ([dask#5920](https://github.com/dask/dask/pull/5920)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Skip parquet `getitem` optimization for some keys ([dask#5917](https://github.com/dask/dask/pull/5917)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add `ignore_index` argument to `rearrange_by_column` code path ([dask#5973](https://github.com/dask/dask/pull/5973)) [Richard J Zamora](https://github.com/rjzamora)
- Add DataFrame and Series `memory_usage_per_partition` methods ([dask#5971](https://github.com/dask/dask/pull/5971)) [James Bourbeau](https://github.com/jrbourbeau)
- `xfail` test_describe when using Pandas 0.24.2 ([dask#5948](https://github.com/dask/dask/pull/5948)) [James Bourbeau](https://github.com/jrbourbeau)
- Implement `dask.dataframe.to_numeric` ([dask#5929](https://github.com/dask/dask/pull/5929)) [Julia Signell](https://github.com/jsignell)
- Add new error message content when columns are in a different order ([dask#5927](https://github.com/dask/dask/pull/5927)) [Julia Signell](https://github.com/jsignell)
- Use shallow copy for assign operations when possible ([dask#5740](https://github.com/dask/dask/pull/5740)) [Richard J Zamora](https://github.com/rjzamora)

### Documentation

- Changed above to below in `dask.array.triu` docs ([dask#5984](https://github.com/dask/dask/pull/5984)) [Henrik Andersson](https://github.com/hnra)
- Array slicing: fix typo in `slice_with_int_dask_array` error message ([dask#5981](https://github.com/dask/dask/pull/5981)) [Gabe Joseph](https://github.com/gjoseph92)
- Grammar and formatting updates to docstrings ([dask#5963](https://github.com/dask/dask/pull/5963)) [James Lamb](https://github.com/jameslamb)
- Update develop doc with conda option ([dask#5939](https://github.com/dask/dask/pull/5939)) [Ray Bell](https://github.com/raybellwaves)
- Update title of DataFrame extension docs ([dask#5954](https://github.com/dask/dask/pull/5954)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixed typos in documentation ([dask#5962](https://github.com/dask/dask/pull/5962)) [James Lamb](https://github.com/jameslamb)
- Add original class or module as a `kwarg` on `_bind_*` methods ([dask#5946](https://github.com/dask/dask/pull/5946)) [Julia Signell](https://github.com/jsignell)
- Add collect list example ([dask#5938](https://github.com/dask/dask/pull/5938)) [Ray Bell](https://github.com/raybellwaves)
- Update optimization doc for python 3 ([dask#5926](https://github.com/dask/dask/pull/5926)) [Julia Signell](https://github.com/jsignell)

<a id="v2-11-0-2020-02-19"></a>

## 2.11.0 / 2020-02-19

### Array

- Cache result of `Array.shape` ([dask#5916](https://github.com/dask/dask/pull/5916)) [Bruce Merry](https://github.com/bmerry)
- Improve accuracy of `estimate_graph_size` for `rechunk` ([dask#5907](https://github.com/dask/dask/pull/5907)) [Bruce Merry](https://github.com/bmerry)
- Skip rechunk steps that do not alter chunking ([dask#5909](https://github.com/dask/dask/pull/5909)) [Bruce Merry](https://github.com/bmerry)
- Support `dtype` and other `kwargs` in `coarsen` ([dask#5903](https://github.com/dask/dask/pull/5903)) [Matthew Rocklin](https://github.com/mrocklin)
- Push chunk override from `map_blocks` into blockwise ([dask#5895](https://github.com/dask/dask/pull/5895)) [Bruce Merry](https://github.com/bmerry)
- Avoid using `rewrite_blockwise` for a singleton ([dask#5890](https://github.com/dask/dask/pull/5890)) [Bruce Merry](https://github.com/bmerry)
- Optimize `slices_from_chunks` ([dask#5891](https://github.com/dask/dask/pull/5891)) [Bruce Merry](https://github.com/bmerry)
- Avoid unnecessary `__getitem__` in `block()` when chunks have correct dimensionality ([dask#5884](https://github.com/dask/dask/pull/5884)) [Thomas Robitaille](https://github.com/astrofrog)

### Bag

- Add `include_path` option for `dask.bag.read_text` ([dask#5836](https://github.com/dask/dask/pull/5836)) [Yifan Gu](https://github.com/gyf304)
- Fixes `ValueError` in delayed execution of bagged NumPy array ([dask#5828](https://github.com/dask/dask/pull/5828)) [Surya Avala](https://github.com/suryaavala)

### Core

- CI: Pin `msgpack` ([dask#5923](https://github.com/dask/dask/pull/5923)) [Tom Augspurger](https://github.com/tomaugspurger)
- Rename `test_inner` to `test_outer` ([dask#5922](https://github.com/dask/dask/pull/5922)) [Shiva Raisinghani](https://github.com/exemplary-citizen)
- `quote` should quote dicts too ([dask#5905](https://github.com/dask/dask/pull/5905)) [Bruce Merry](https://github.com/bmerry)
- Register a normalizer for literal ([dask#5898](https://github.com/dask/dask/pull/5898)) [Bruce Merry](https://github.com/bmerry)
- Improve layer name synthesis for non-HLGs ([dask#5888](https://github.com/dask/dask/pull/5888)) [Bruce Merry](https://github.com/bmerry)
- Replace flake8 pre-commit-hook with upstream ([dask#5892](https://github.com/dask/dask/pull/5892)) [Julia Signell](https://github.com/jsignell)
- Call pip as a module to avoid warnings ([dask#5861](https://github.com/dask/dask/pull/5861)) [Cyril Shcherbin](https://github.com/shcherbin)
- Close `ThreadPool` at exit ([dask#5852](https://github.com/dask/dask/pull/5852)) [Tom Augspurger](https://github.com/tomaugspurger)
- Remove `dask.dataframe` import in tokenization code ([dask#5855](https://github.com/dask/dask/pull/5855)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Require `pandas>=0.23` ([dask#5883](https://github.com/dask/dask/pull/5883)) [Tom Augspurger](https://github.com/tomaugspurger)
- Remove lambda from dataframe aggregation ([dask#5901](https://github.com/dask/dask/pull/5901)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix exception chaining in `dataframe/__init__.py` ([dask#5882](https://github.com/dask/dask/pull/5882)) [Ram Rachum](https://github.com/cool-RR)
- Add support for reductions on empty dataframes ([dask#5804](https://github.com/dask/dask/pull/5804)) [Shiva Raisinghani](https://github.com/exemplary-citizen)
- Expose `sort=` argument for groupby ([dask#5801](https://github.com/dask/dask/pull/5801)) [Richard J Zamora](https://github.com/rjzamora)
- Add `df.empty` property ([dask#5711](https://github.com/dask/dask/pull/5711)) [rockwellw](https://github.com/rockwellw)
- Use parquet read speed-ups from `fastparquet.api.paths_to_cats`. ([dask#5821](https://github.com/dask/dask/pull/5821)) [Igor Gotlibovych](https://github.com/ig248)

### Documentation

- Deprecate `doc_wraps` ([dask#5912](https://github.com/dask/dask/pull/5912)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update array internal design docs for HighLevelGraph era ([dask#5889](https://github.com/dask/dask/pull/5889)) [Bruce Merry](https://github.com/bmerry)
- Move over dashboard connection docs ([dask#5877](https://github.com/dask/dask/pull/5877)) [Matthew Rocklin](https://github.com/mrocklin)
- Move prometheus docs from distributed.dask.org ([dask#5876](https://github.com/dask/dask/pull/5876)) [Matthew Rocklin](https://github.com/mrocklin)
- Removing duplicated DO block at the end ([dask#5878](https://github.com/dask/dask/pull/5878)) [K.-Michael Aye](https://github.com/michaelaye)
- `map_blocks` see also ([dask#5874](https://github.com/dask/dask/pull/5874)) [Tom Augspurger](https://github.com/tomaugspurger)
- More derived from ([dask#5871](https://github.com/dask/dask/pull/5871)) [Julia Signell](https://github.com/jsignell)
- Fix typo ([dask#5866](https://github.com/dask/dask/pull/5866)) [Yetunde Dada](https://github.com/yetudada)
- Fix typo in `cloud.rst` ([dask#5860](https://github.com/dask/dask/pull/5860)) [Andrew Thomas](https://github.com/amcnicho)
- Add note pointing to code of conduct and diversity statement ([dask#5844](https://github.com/dask/dask/pull/5844)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v2-10-1-2020-01-30"></a>

## 2.10.1 / 2020-01-30

- Fix Pandas 1.0 version comparison ([dask#5851](https://github.com/dask/dask/pull/5851)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix typo in distributed diagnostics documentation ([dask#5841](https://github.com/dask/dask/pull/5841)) [Gerrit Holl](https://github.com/gerritholl)

<a id="v2-10-0-2020-01-28"></a>

## 2.10.0 / 2020-01-28

- Support for pandas 1.0’s new `BooleanDtype` and `StringDtype` ([dask#5815](https://github.com/dask/dask/pull/5815)) [Tom Augspurger](https://github.com/tomaugspurger)
- Compatibility with pandas 1.0’s API breaking changes and deprecations ([dask#5792](https://github.com/dask/dask/pull/5792)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed non-deterministic tokenization of some extension-array backed pandas objects ([dask#5813](https://github.com/dask/dask/pull/5813)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed handling of dataclass class objects in collections ([dask#5812](https://github.com/dask/dask/pull/5812)) [Matteo De Wint](https://github.com/mdwint)
- Fixed resampling with tz-aware dates when one of the endpoints fell in a non-existent time ([dask#5807](https://github.com/dask/dask/pull/5807)) [dfonnegra](https://github.com/dfonnegra)
- Delay initial Zarr dataset creation until the computation occurs ([dask#5797](https://github.com/dask/dask/pull/5797)) [Chris Roat](https://github.com/ChrisRoat)
- Use parquet dataset statistics in more cases with the `pyarrow` engine ([dask#5799](https://github.com/dask/dask/pull/5799)) [Richard J Zamora](https://github.com/rjzamora)
- Fixed exception in `groupby.std()` when some of the keys were large integers ([dask#5737](https://github.com/dask/dask/pull/5737)) [H. Thomson Comer](https://github.com/thomcom)

<a id="v2-9-2-2020-01-16"></a>

## 2.9.2 / 2020-01-16

### Array

- Unify chunks in `broadcast_arrays` ([dask#5765](https://github.com/dask/dask/pull/5765)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- `xfail` CSV encoding tests ([dask#5791](https://github.com/dask/dask/pull/5791)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update order to handle empty dask graph ([dask#5789](https://github.com/dask/dask/pull/5789)) [James Bourbeau](https://github.com/jrbourbeau)
- Redo `dask.order.order` ([dask#5646](https://github.com/dask/dask/pull/5646)) [Erik Welch](https://github.com/eriknw)

### DataFrame

- Add transparent compression for on-disk shuffle with `partd` ([dask#5786](https://github.com/dask/dask/pull/5786)) [Christian Wesp](https://github.com/ChrWesp)
- Fix `repr` for empty dataframes ([dask#5781](https://github.com/dask/dask/pull/5781)) [Shiva Raisinghani](https://github.com/exemplary-citizen)
- Pandas 1.0.0RC0 compat ([dask#5784](https://github.com/dask/dask/pull/5784)) [Tom Augspurger](https://github.com/tomaugspurger)
- Remove buggy assertions ([dask#5783](https://github.com/dask/dask/pull/5783)) [Tom Augspurger](https://github.com/tomaugspurger)
- Pandas 1.0 compat ([dask#5782](https://github.com/dask/dask/pull/5782)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix bug in pyarrow-based `read_parquet` on partitioned datasets ([dask#5777](https://github.com/dask/dask/pull/5777)) [Richard J Zamora](https://github.com/rjzamora)
- Compat for pandas 1.0 ([dask#5779](https://github.com/dask/dask/pull/5779)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix groupby/mean error with with categorical index ([dask#5776](https://github.com/dask/dask/pull/5776)) [Richard J Zamora](https://github.com/rjzamora)
- Support empty partitions when performing cumulative aggregation ([dask#5730](https://github.com/dask/dask/pull/5730)) [Matthew Rocklin](https://github.com/mrocklin)
- `set_index` accepts single-item unnested list ([dask#5760](https://github.com/dask/dask/pull/5760)) [Wes Roach](https://github.com/WesRoach)
- Fixed partitioning in set index for ordered `Categorical` ([dask#5715](https://github.com/dask/dask/pull/5715)) [Tom Augspurger](https://github.com/tomaugspurger)

### Documentation

- Note additional use case for `normalize_token.register` ([dask#5766](https://github.com/dask/dask/pull/5766)) [Thomas A Caswell](https://github.com/tacaswell)
- Update bag `repartition` docstring ([dask#5772](https://github.com/dask/dask/pull/5772)) [Timost](https://github.com/Timost)
- Small typos ([dask#5771](https://github.com/dask/dask/pull/5771)) [Maarten Breddels](https://github.com/maartenbreddels)
- Fix typo in Task Expectations docs ([dask#5767](https://github.com/dask/dask/pull/5767)) [James Bourbeau](https://github.com/jrbourbeau)
- Add docs section on task expectations to graph page ([dask#5764](https://github.com/dask/dask/pull/5764)) [Devin Petersohn](https://github.com/devin-petersohn)

<a id="v2-9-1-2019-12-27"></a>

## 2.9.1 / 2019-12-27

### Array

- Support Array.view with dtype=None ([dask#5736](https://github.com/dask/dask/pull/5736)) [Anderson Banihirwe](https://github.com/andersy005)
- Add dask.array.nanmedian ([dask#5684](https://github.com/dask/dask/pull/5684)) [Deepak Cherian](https://github.com/dcherian)

### Core

- xfail test_temporary_directory on Python 3.8 ([dask#5734](https://github.com/dask/dask/pull/5734)) [James Bourbeau](https://github.com/jrbourbeau)
- Add support for Python 3.8 ([dask#5603](https://github.com/dask/dask/pull/5603)) [James Bourbeau](https://github.com/jrbourbeau)
- Use id to dedupe constants in rewrite_blockwise ([dask#5696](https://github.com/dask/dask/pull/5696)) [Jim Crist](https://github.com/jcrist)

### DataFrame

- Raise error when converting a dask dataframe scalar to a boolean ([dask#5743](https://github.com/dask/dask/pull/5743)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure dataframe groupby-variance is greater than zero ([dask#5728](https://github.com/dask/dask/pull/5728)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix DataFrame._\_iter_\_ ([dask#5719](https://github.com/dask/dask/pull/5719)) [Tom Augspurger](https://github.com/tomaugspurger)
- Support Parquet filters in disjunctive normal form, like PyArrow ([dask#5656](https://github.com/dask/dask/pull/5656)) [Matteo De Wint](https://github.com/mdwint)
- Auto-detect categorical columns in ArrowEngine-based read_parquet ([dask#5690](https://github.com/dask/dask/pull/5690)) [Richard J Zamora](https://github.com/rjzamora)
- Skip parquet getitem optimization tests if no engine found ([dask#5697](https://github.com/dask/dask/pull/5697)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix independent optimization of parquet-getitem ([dask#5613](https://github.com/dask/dask/pull/5613)) [Tom Augspurger](https://github.com/tomaugspurger)

### Documentation

- Update helm config doc ([dask#5750](https://github.com/dask/dask/pull/5750)) [Ray Bell](https://github.com/raybellwaves)
- Link to examples.dask.org in several places ([dask#5733](https://github.com/dask/dask/pull/5733)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add missing “ in performance report example ([dask#5724](https://github.com/dask/dask/pull/5724)) [James Bourbeau](https://github.com/jrbourbeau)
- Resolve several documentation build warnings ([dask#5685](https://github.com/dask/dask/pull/5685)) [James Bourbeau](https://github.com/jrbourbeau)
- add info on performance_report ([dask#5713](https://github.com/dask/dask/pull/5713)) [Benjamin Zaitlen](https://github.com/quasiben)
- Add more docs disclaimers ([dask#5710](https://github.com/dask/dask/pull/5710)) [Julia Signell](https://github.com/jsignell)
- Fix simple typo: wihout -> without ([dask#5708](https://github.com/dask/dask/pull/5708)) [Tim Gates](https://github.com/timgates42)
- Update numpydoc dependency ([dask#5694](https://github.com/dask/dask/pull/5694)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2-9-0-2019-12-06"></a>

## 2.9.0 / 2019-12-06

### Array

- Fix `da.std` to work with NumPy arrays ([dask#5681](https://github.com/dask/dask/pull/5681)) [James Bourbeau](https://github.com/jrbourbeau)

### Core

- Register `sizeof` functions for Numba and RMM ([dask#5668](https://github.com/dask/dask/pull/5668)) [John A Kirkham](https://github.com/jakirkham)
- Update meeting time ([dask#5682](https://github.com/dask/dask/pull/5682)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Modify `dd.DataFrame.drop` to use shallow copy ([dask#5675](https://github.com/dask/dask/pull/5675)) [Richard J Zamora](https://github.com/rjzamora)
- Fix bug in `_get_md_row_groups` ([dask#5673](https://github.com/dask/dask/pull/5673)) [Richard J Zamora](https://github.com/rjzamora)
- Close sqlalchemy engine after querying DB ([dask#5629](https://github.com/dask/dask/pull/5629)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Allow `dd.map_partitions` to not enforce meta ([dask#5660](https://github.com/dask/dask/pull/5660)) [Matthew Rocklin](https://github.com/mrocklin)
- Generalize `concat_unindexed_dataframes` to support cudf-backend ([dask#5659](https://github.com/dask/dask/pull/5659)) [Richard J Zamora](https://github.com/rjzamora)
- Add dataframe resample methods ([dask#5636](https://github.com/dask/dask/pull/5636)) [Benjamin Zaitlen](https://github.com/quasiben)
- Compute length of dataframe as length of first column ([dask#5635](https://github.com/dask/dask/pull/5635)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Doc fixup ([dask#5665](https://github.com/dask/dask/pull/5665)) [James Bourbeau](https://github.com/jrbourbeau)
- Update doc build instructions ([dask#5640](https://github.com/dask/dask/pull/5640)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix ADL link ([dask#5639](https://github.com/dask/dask/pull/5639)) [Ray Bell](https://github.com/raybellwaves)
- Add documentation build ([dask#5617](https://github.com/dask/dask/pull/5617)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2-8-1-2019-11-22"></a>

## 2.8.1 / 2019-11-22

### Array

- Use auto rechunking in `da.rechunk` if no value given ([dask#5605](https://github.com/dask/dask/pull/5605)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Add simple action to activate GH actions ([dask#5619](https://github.com/dask/dask/pull/5619)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Fix “file_path_0” bug in `aggregate_row_groups` ([dask#5627](https://github.com/dask/dask/pull/5627)) [Richard J Zamora](https://github.com/rjzamora)
- Add `chunksize` argument to `read_parquet` ([dask#5607](https://github.com/dask/dask/pull/5607)) [Richard J Zamora](https://github.com/rjzamora)
- Change `test_repartition_npartitions` to support arch64 architecture ([dask#5620](https://github.com/dask/dask/pull/5620)) [ossdev07](https://github.com/ossdev07)
- Categories lost after groupby + agg ([dask#5423](https://github.com/dask/dask/pull/5423)) [Oliver Hofkens](https://github.com/OliverHofkens)
- Fixed relative path issue with parquet metadata file ([dask#5608](https://github.com/dask/dask/pull/5608)) [Nuno Gomes Silva](https://github.com/mgsnuno)
- Enable gpu-backed covariance/correlation in dataframes ([dask#5597](https://github.com/dask/dask/pull/5597)) [Richard J Zamora](https://github.com/rjzamora)

### Documentation

- Fix institutional faq and unknown doc warnings ([dask#5616](https://github.com/dask/dask/pull/5616)) [James Bourbeau](https://github.com/jrbourbeau)
- Add doc for some utils ([dask#5609](https://github.com/dask/dask/pull/5609)) [Tom Augspurger](https://github.com/tomaugspurger)
- Removes `html_extra_path` ([dask#5614](https://github.com/dask/dask/pull/5614)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixed See Also referencence ([dask#5612](https://github.com/dask/dask/pull/5612)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-8-0-2019-11-14"></a>

## 2.8.0 / 2019-11-14

### Array

- Implement complete dask.array.tile function ([dask#5574](https://github.com/dask/dask/pull/5574)) [Bouwe Andela](https://github.com/bouweandela)
- Add median along an axis with automatic rechunking ([dask#5575](https://github.com/dask/dask/pull/5575)) [Matthew Rocklin](https://github.com/mrocklin)
- Allow da.asarray to chunk inputs ([dask#5586](https://github.com/dask/dask/pull/5586)) [Matthew Rocklin](https://github.com/mrocklin)

### Bag

- Use key_split in Bag name ([dask#5571](https://github.com/dask/dask/pull/5571)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Switch Doctests to Py3.7 ([dask#5573](https://github.com/dask/dask/pull/5573)) [Ryan Nazareth](https://github.com/ryankarlos)
- Relax get_colors test to adapt to new Bokeh release ([dask#5576](https://github.com/dask/dask/pull/5576)) [Matthew Rocklin](https://github.com/mrocklin)
- Add dask.blockwise.fuse_roots optimization ([dask#5451](https://github.com/dask/dask/pull/5451)) [Matthew Rocklin](https://github.com/mrocklin)
- Add sizeof implementation for small dicts ([dask#5578](https://github.com/dask/dask/pull/5578)) [Matthew Rocklin](https://github.com/mrocklin)
- Update fsspec, gcsfs, s3fs ([dask#5588](https://github.com/dask/dask/pull/5588)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Add dropna argument to groupby ([dask#5579](https://github.com/dask/dask/pull/5579)) [Richard J Zamora](https://github.com/rjzamora)
- Revert “Remove import of dask_cudf, which is now a part of cudf ([dask#5568](https://github.com/dask/dask/pull/5568))” ([dask#5590](https://github.com/dask/dask/pull/5590)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Add best practice for dask.compute function ([dask#5583](https://github.com/dask/dask/pull/5583)) [Matthew Rocklin](https://github.com/mrocklin)
- Create FUNDING.yml ([dask#5587](https://github.com/dask/dask/pull/5587)) [Gina Helfrich](https://github.com/Dr-G)
- Add screencast for coordination primitives ([dask#5593](https://github.com/dask/dask/pull/5593)) [Matthew Rocklin](https://github.com/mrocklin)
- Move funding to .github repo ([dask#5589](https://github.com/dask/dask/pull/5589)) [Tom Augspurger](https://github.com/tomaugspurger)
- Update calendar link ([dask#5569](https://github.com/dask/dask/pull/5569)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-7-0-2019-11-08"></a>

## 2.7.0 / 2019-11-08

This release drops support for Python 3.5

### Array

- Reuse code for assert_eq util method ([dask#5496](https://github.com/dask/dask/pull/5496)) [Vijayant](https://github.com/VijayantSoni)
- Update da.array to always return a dask array ([dask#5510](https://github.com/dask/dask/pull/5510)) [James Bourbeau](https://github.com/jrbourbeau)
- Skip transpose on trivial inputs ([dask#5523](https://github.com/dask/dask/pull/5523)) [Ryan Abernathey](https://github.com/rabernat)
- Avoid NumPy scalar string representation in tokenize ([dask#5527](https://github.com/dask/dask/pull/5527)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove unnecessary tiledb shape constraint ([dask#5545](https://github.com/dask/dask/pull/5545)) [Norman Barker](https://github.com/normanb)
- Removes bytes from sparse array HTML repr ([dask#5556](https://github.com/dask/dask/pull/5556)) [James Bourbeau](https://github.com/jrbourbeau)

### Core

- Drop Python 3.5 ([dask#5528](https://github.com/dask/dask/pull/5528)) [James Bourbeau](https://github.com/jrbourbeau)
- Update the use of fixtures in distributed tests ([dask#5497](https://github.com/dask/dask/pull/5497)) [Matthew Rocklin](https://github.com/mrocklin)
- Changed deprecated bokeh-port to dashboard-address ([dask#5507](https://github.com/dask/dask/pull/5507)) [darindf](https://github.com/darindf)
- Avoid updating with identical dicts in ensure_dict ([dask#5501](https://github.com/dask/dask/pull/5501)) [James Bourbeau](https://github.com/jrbourbeau)
- Test Upstream ([dask#5516](https://github.com/dask/dask/pull/5516)) [Tom Augspurger](https://github.com/tomaugspurger)
- Accelerate reverse_dict ([dask#5479](https://github.com/dask/dask/pull/5479)) [Ryan Grout](https://github.com/groutr)
- Update test_imports.sh ([dask#5534](https://github.com/dask/dask/pull/5534)) [James Bourbeau](https://github.com/jrbourbeau)
- Support cgroups limits on cpu count in multiprocess and threaded schedulers ([dask#5499](https://github.com/dask/dask/pull/5499)) [Albert DeFusco](https://github.com/AlbertDeFusco)
- Update minimum pyarrow version on CI ([dask#5562](https://github.com/dask/dask/pull/5562)) [James Bourbeau](https://github.com/jrbourbeau)
- Make cloudpickle optional ([dask#5511](https://github.com/dask/dask/pull/5511)) [crusaderky](https://github.com/crusaderky)

### DataFrame

- Add an example of index_col usage ([dask#3072](https://github.com/dask/dask/pull/3072)) [Bruno Bonfils](https://github.com/asyd)
- Explicitly use iloc for row indexing ([dask#5500](https://github.com/dask/dask/pull/5500)) [Krishan Bhasin](https://github.com/KrishanBhasin)
- Accept dask arrays on columns assignemnt ([dask#5224](https://github.com/dask/dask/pull/5224)) Henrique Ribeiro-
- Implement unique and value_counts for SeriesGroupBy ([dask#5358](https://github.com/dask/dask/pull/5358)) [Scott Sievert](https://github.com/stsievert)
- Add sizeof definition for pyarrow tables and columns ([dask#5522](https://github.com/dask/dask/pull/5522)) [Richard J Zamora](https://github.com/rjzamora)
- Enable row-group task partitioning in pyarrow-based read_parquet ([dask#5508](https://github.com/dask/dask/pull/5508)) [Richard J Zamora](https://github.com/rjzamora)
- Removes npartitions=’auto’ from dd.merge docstring ([dask#5531](https://github.com/dask/dask/pull/5531)) [James Bourbeau](https://github.com/jrbourbeau)
- Apply enforce error message shows non-overlapping columns. ([dask#5530](https://github.com/dask/dask/pull/5530)) [Tom Augspurger](https://github.com/tomaugspurger)
- Optimize meta_nonempty for repetitive dtypes ([dask#5553](https://github.com/dask/dask/pull/5553)) [Petio Petrov](https://github.com/petioptrv)
- Remove import of dask_cudf, which is now a part of cudf ([dask#5568](https://github.com/dask/dask/pull/5568)) [Mads R. B. Kristensen](https://github.com/madsbk)

### Documentation

- Make capitalization more consistent in FAQ docs ([dask#5512](https://github.com/dask/dask/pull/5512)) [Matthew Rocklin](https://github.com/mrocklin)
- Add CONTRIBUTING.md ([dask#5513](https://github.com/dask/dask/pull/5513)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Document optional dependencies ([dask#5456](https://github.com/dask/dask/pull/5456)) [Prithvi MK](https://github.com/pmk21)
- Update helm chart docs to reflect new chart repo ([dask#5539](https://github.com/dask/dask/pull/5539)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Add Resampler to API docs ([dask#5551](https://github.com/dask/dask/pull/5551)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix typo in read_sql_table ([dask#5554](https://github.com/dask/dask/pull/5554)) [Eric Dill](https://github.com/ericdill)
- Add adaptive deployments screencast [skip ci] ([dask#5566](https://github.com/dask/dask/pull/5566)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v2-6-0-2019-10-15"></a>

## 2.6.0 / 2019-10-15

### Core

- Call `ensure_dict` on graphs before entering `toolz.merge` ([dask#5486](https://github.com/dask/dask/pull/5486)) [Matthew Rocklin](https://github.com/mrocklin)
- Consolidating hash dispatch functions ([dask#5476](https://github.com/dask/dask/pull/5476)) [Richard J Zamora](https://github.com/rjzamora)

### DataFrame

- Support Python 3.5 in Parquet code ([dask#5491](https://github.com/dask/dask/pull/5491)) [Benjamin Zaitlen](https://github.com/quasiben)
- Avoid identity check in `warn_dtype_mismatch` ([dask#5489](https://github.com/dask/dask/pull/5489)) [Tom Augspurger](https://github.com/tomaugspurger)
- Enable unused groupby tests ([dask#3480](https://github.com/dask/dask/pull/3480)) [Jörg Dietrich](https://github.com/joergdietrich)
- Remove old parquet and bcolz dataframe optimizations ([dask#5484](https://github.com/dask/dask/pull/5484)) [Matthew Rocklin](https://github.com/mrocklin)
- Add getitem optimization for `read_parquet` ([dask#5453](https://github.com/dask/dask/pull/5453)) [Tom Augspurger](https://github.com/tomaugspurger)
- Use `_constructor_sliced` method to determine Series type ([dask#5480](https://github.com/dask/dask/pull/5480)) [Richard J Zamora](https://github.com/rjzamora)
- Fix map(series) for unsorted base series index ([dask#5459](https://github.com/dask/dask/pull/5459)) [Justin Waugh](https://github.com/bluecoconut)
- Fix `KeyError` with Groupby label ([dask#5467](https://github.com/dask/dask/pull/5467)) [Ryan Nazareth](https://github.com/ryankarlos)

### Documentation

- Use Zoom meeting instead of appear.in ([dask#5494](https://github.com/dask/dask/pull/5494)) [Matthew Rocklin](https://github.com/mrocklin)
- Added curated list of resources ([dask#5460](https://github.com/dask/dask/pull/5460)) [Javad](https://github.com/javad94)
- Update SSH docs to include `SSHCluster` ([dask#5482](https://github.com/dask/dask/pull/5482)) [Matthew Rocklin](https://github.com/mrocklin)
- Update “Why Dask?” page ([dask#5473](https://github.com/dask/dask/pull/5473)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix typos in docstrings ([dask#5469](https://github.com/dask/dask/pull/5469)) [garanews](https://github.com/garanews)

<a id="v2-5-2-2019-10-04"></a>

## 2.5.2 / 2019-10-04

### Array

- Correct chunk size logic for asymmetric overlaps ([dask#5449](https://github.com/dask/dask/pull/5449)) [Ben Jeffery](https://github.com/benjeffery)
- Make da.unify_chunks public API ([dask#5443](https://github.com/dask/dask/pull/5443)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Fix dask.dataframe.fillna handling of Scalar object ([dask#5463](https://github.com/dask/dask/pull/5463)) [Zhenqing Li](https://github.com/DigitalPig)

### Documentation

- Remove boxes in Spark comparison page ([dask#5445](https://github.com/dask/dask/pull/5445)) [Matthew Rocklin](https://github.com/mrocklin)
- Add latest presentations ([dask#5446](https://github.com/dask/dask/pull/5446)) [Javad](https://github.com/javad94)
- Update cloud documentation ([dask#5444](https://github.com/dask/dask/pull/5444)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v2-5-0-2019-09-27"></a>

## 2.5.0 / 2019-09-27

### Core

- Add sentinel no_default to get_dependencies task ([dask#5420](https://github.com/dask/dask/pull/5420)) [James Bourbeau](https://github.com/jrbourbeau)
- Update fsspec version ([dask#5415](https://github.com/dask/dask/pull/5415)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove PY2 checks ([dask#5400](https://github.com/dask/dask/pull/5400)) [Jim Crist](https://github.com/jcrist)

### DataFrame

- Add option to not check meta in dd.from_delayed ([dask#5436](https://github.com/dask/dask/pull/5436)) [Christopher J. Wright](https://github.com/CJ-Wright)
- Fix test_timeseries_nulls_in_schema failures with pyarrow master ([dask#5421](https://github.com/dask/dask/pull/5421)) [Richard J Zamora](https://github.com/rjzamora)
- Reduce read_metadata output size in pyarrow/parquet ([dask#5391](https://github.com/dask/dask/pull/5391)) [Richard J Zamora](https://github.com/rjzamora)
- Test numeric edge case for repartition with npartitions. ([dask#5433](https://github.com/dask/dask/pull/5433)) [amerkel2](https://github.com/amerkel2)
- Unxfail pandas-datareader test ([dask#5430](https://github.com/dask/dask/pull/5430)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add DataFrame.pop implementation ([dask#5422](https://github.com/dask/dask/pull/5422)) [Matthew Rocklin](https://github.com/mrocklin)
- Enable merge/set_index for cudf-based dataframes with cupy `values` ([dask#5322](https://github.com/dask/dask/pull/5322)) [Richard J Zamora](https://github.com/rjzamora)
- drop_duplicates support for positional subset parameter ([dask#5410](https://github.com/dask/dask/pull/5410)) [Wes Roach](https://github.com/WesRoach)

### Documentation

- Add screencasts to array, bag, dataframe, delayed, futures and setup  ([dask#5429](https://github.com/dask/dask/pull/5429)) ([dask#5424](https://github.com/dask/dask/pull/5424)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix delimeter parsing documentation ([dask#5428](https://github.com/dask/dask/pull/5428)) [Mahmut Bulut](https://github.com/vertexclique)
- Update overview image ([dask#5404](https://github.com/dask/dask/pull/5404)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2-4-0-2019-09-13"></a>

## 2.4.0 / 2019-09-13

### Array

- Adds explicit `h5py.File` mode ([dask#5390](https://github.com/dask/dask/pull/5390)) [James Bourbeau](https://github.com/jrbourbeau)
- Provides method to compute unknown array chunks sizes ([dask#5312](https://github.com/dask/dask/pull/5312)) [Scott Sievert](https://github.com/stsievert)
- Ignore runtime warning in Array `compute_meta` ([dask#5356](https://github.com/dask/dask/pull/5356)) [estebanag](https://github.com/estebanag)
- Add `_meta` to `Array.__dask_postpersist__` ([dask#5353](https://github.com/dask/dask/pull/5353)) [Benoit Bovy](https://github.com/benbovy)
- Fixup `da.asarray` and `da.asanyarray` for datetime64 dtype and xarray objects ([dask#5334](https://github.com/dask/dask/pull/5334)) [Stephan Hoyer](https://github.com/shoyer)
- Add shape implementation ([dask#5293](https://github.com/dask/dask/pull/5293)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add chunktype to array text repr ([dask#5289](https://github.com/dask/dask/pull/5289)) [James Bourbeau](https://github.com/jrbourbeau)
- Array.random.choice: handle array-like non-arrays ([dask#5283](https://github.com/dask/dask/pull/5283)) [Gabe Joseph](https://github.com/gjoseph92)

### Core

- Remove deprecated code ([dask#5401](https://github.com/dask/dask/pull/5401)) [Jim Crist](https://github.com/jcrist)
- Fix `funcname` when vectorized func has no `__name__` ([dask#5399](https://github.com/dask/dask/pull/5399)) [James Bourbeau](https://github.com/jrbourbeau)
- Truncate `funcname` to avoid long key names ([dask#5383](https://github.com/dask/dask/pull/5383)) [Matthew Rocklin](https://github.com/mrocklin)
- Add support for `numpy.vectorize` in `funcname` ([dask#5396](https://github.com/dask/dask/pull/5396)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixed HDFS upstream test ([dask#5395](https://github.com/dask/dask/pull/5395)) [Tom Augspurger](https://github.com/tomaugspurger)
- Support numbers and None in `parse_bytes`/`timedelta` ([dask#5384](https://github.com/dask/dask/pull/5384)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix tokenizing of subindexes on memmapped numpy arrays ([dask#5351](https://github.com/dask/dask/pull/5351)) [Henry Pinkard](https://github.com/)
- Upstream fixups ([dask#5300](https://github.com/dask/dask/pull/5300)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Allow pandas to cast type of statistics ([dask#5402](https://github.com/dask/dask/pull/5402)) [Richard J Zamora](https://github.com/rjzamora)
- Preserve index dtype after applying `dd.pivot_table` ([dask#5385](https://github.com/dask/dask/pull/5385)) [therhaag](https://github.com/therhaag)
- Implement explode for Series and DataFrame ([dask#5381](https://github.com/dask/dask/pull/5381)) [Arpit Solanki](https://github.com/arpit1997)
- `set_index` on categorical fails with less categories than partitions ([dask#5354](https://github.com/dask/dask/pull/5354)) [Oliver Hofkens](https://github.com/OliverHofkens)
- Support output to a single CSV file ([dask#5304](https://github.com/dask/dask/pull/5304)) [Hongjiu Zhang](https://github.com/hongzmsft)
- Add `groupby().transform()` ([dask#5327](https://github.com/dask/dask/pull/5327)) [Oliver Hofkens](https://github.com/OliverHofkens)
- Adding filter kwarg to pyarrow dataset call ([dask#5348](https://github.com/dask/dask/pull/5348)) [Richard J Zamora](https://github.com/rjzamora)
- Implement and check compression defaults for parquet ([dask#5335](https://github.com/dask/dask/pull/5335)) [Sarah Bird](https://github.com/birdsarah)
- Pass sqlalchemy params to delayed objects ([dask#5332](https://github.com/dask/dask/pull/5332)) [Arpit Solanki](https://github.com/arpit1997)
- Fixing schema handling in arrow-parquet ([dask#5307](https://github.com/dask/dask/pull/5307)) [Richard J Zamora](https://github.com/rjzamora)
- Add support for DF and Series `groupby().idxmin/max()` ([dask#5273](https://github.com/dask/dask/pull/5273)) [Oliver Hofkens](https://github.com/OliverHofkens)
- Add correlation calculation and add test ([dask#5296](https://github.com/dask/dask/pull/5296)) [Benjamin Zaitlen](https://github.com/quasiben)

### Documentation

- Numpy docstring standard has moved ([dask#5405](https://github.com/dask/dask/pull/5405)) [Wes Roach](https://github.com/WesRoach)
- Reference correct NumPy array name ([dask#5403](https://github.com/dask/dask/pull/5403)) [Wes Roach](https://github.com/WesRoach)
- Minor edits to Array chunk documentation ([dask#5372](https://github.com/dask/dask/pull/5372)) [Scott Sievert](https://github.com/stsievert)
- Add methods to API docs ([dask#5387](https://github.com/dask/dask/pull/5387)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add namespacing to configuration example ([dask#5374](https://github.com/dask/dask/pull/5374)) [Matthew Rocklin](https://github.com/mrocklin)
- Add get_task_stream and profile to the diagnostics page ([dask#5375](https://github.com/dask/dask/pull/5375)) [Matthew Rocklin](https://github.com/mrocklin)
- Add best practice to load data with Dask ([dask#5369](https://github.com/dask/dask/pull/5369)) [Matthew Rocklin](https://github.com/mrocklin)
- Update `institutional-faq.rst` ([dask#5345](https://github.com/dask/dask/pull/5345)) [DomHudson](https://github.com/DomHudson)
- Add threads and processes note to the best practices ([dask#5340](https://github.com/dask/dask/pull/5340)) [Matthew Rocklin](https://github.com/mrocklin)
- Update cuDF links ([dask#5328](https://github.com/dask/dask/pull/5328)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixed small typo with parentheses placement ([dask#5311](https://github.com/dask/dask/pull/5311)) [Eugene Huang](https://github.com/eugeneh101)
- Update link in reshape docstring ([dask#5297](https://github.com/dask/dask/pull/5297)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v2-3-0-2019-08-16"></a>

## 2.3.0 / 2019-08-16

### Array

- Raise exception when `from_array` is given a dask array ([dask#5280](https://github.com/dask/dask/pull/5280)) [David Hoese](https://github.com/djhoese)
- Avoid adjusting gufunc’s meta dtype twice ([dask#5274](https://github.com/dask/dask/pull/5274)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add `meta=` keyword to map_blocks and add test with sparse ([dask#5269](https://github.com/dask/dask/pull/5269)) [Matthew Rocklin](https://github.com/mrocklin)
- Add rollaxis and moveaxis ([dask#4822](https://github.com/dask/dask/pull/4822)) [Tobias de Jong](https://github.com/tadejong)
- Always increment old chunk index ([dask#5256](https://github.com/dask/dask/pull/5256)) [James Bourbeau](https://github.com/jrbourbeau)
- Shuffle dask array ([dask#3901](https://github.com/dask/dask/pull/3901)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix ordering when indexing a dask array with a bool dask array ([dask#5151](https://github.com/dask/dask/pull/5151)) [James Bourbeau](https://github.com/jrbourbeau)

### Bag

- Add workaround for memory leaks in bag generators ([dask#5208](https://github.com/dask/dask/pull/5208)) [Marco Neumann](https://github.com/crepererum)

### Core

- Set strict xfail option ([dask#5220](https://github.com/dask/dask/pull/5220)) [James Bourbeau](https://github.com/jrbourbeau)
- test-upstream ([dask#5267](https://github.com/dask/dask/pull/5267)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed HDFS CI failure ([dask#5234](https://github.com/dask/dask/pull/5234)) [Tom Augspurger](https://github.com/tomaugspurger)
- Error nicely if no file size inferred ([dask#5231](https://github.com/dask/dask/pull/5231)) [Jim Crist](https://github.com/jcrist)
- A few changes to `config.set` ([dask#5226](https://github.com/dask/dask/pull/5226)) [Jim Crist](https://github.com/jcrist)
- Fixup black string normalization ([dask#5227](https://github.com/dask/dask/pull/5227)) [Jim Crist](https://github.com/jcrist)
- Pin NumPy in windows tests ([dask#5228](https://github.com/dask/dask/pull/5228)) [Jim Crist](https://github.com/jcrist)
- Ensure parquet tests are skipped if fastparquet and pyarrow not installed ([dask#5217](https://github.com/dask/dask/pull/5217)) [James Bourbeau](https://github.com/jrbourbeau)
- Add fsspec to readthedocs ([dask#5207](https://github.com/dask/dask/pull/5207)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump NumPy and Pandas to 1.17 and 0.25 in CI test ([dask#5179](https://github.com/dask/dask/pull/5179)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Fix `DataFrame.query` docstring (incorrect numexpr API) ([dask#5271](https://github.com/dask/dask/pull/5271)) [Doug Davis](https://github.com/douglasdavis)
- Parquet metadata-handling improvements ([dask#5218](https://github.com/dask/dask/pull/5218)) [Richard J Zamora](https://github.com/rjzamora)
- Improve messaging around sorted parquet columns for index ([dask#5265](https://github.com/dask/dask/pull/5265)) [Martin Durant](https://github.com/martindurant)
- Add `rearrange_by_divisions` and `set_index` support for cudf ([dask#5205](https://github.com/dask/dask/pull/5205)) [Richard J Zamora](https://github.com/rjzamora)
- Fix `groupby.std()` with integer colum names ([dask#5096](https://github.com/dask/dask/pull/5096)) [Nicolas Hug](https://github.com/NicolasHug)
- Add `Series.__iter__` ([dask#5071](https://github.com/dask/dask/pull/5071)) [Blane](https://github.com/BlaneG)
- Generalize `hash_pandas_object` to work for non-pandas backends ([dask#5184](https://github.com/dask/dask/pull/5184)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Add rolling cov ([dask#5154](https://github.com/dask/dask/pull/5154)) [Ivars Geidans](https://github.com/ivarsfg)
- Add columns argument in drop function ([dask#5223](https://github.com/dask/dask/pull/5223)) [Henrique Ribeiro](https://github.com/henriqueribeiro)

### Documentation

- Update institutional FAQ doc ([dask#5277](https://github.com/dask/dask/pull/5277)) [Matthew Rocklin](https://github.com/mrocklin)
- Add draft of institutional FAQ ([dask#5214](https://github.com/dask/dask/pull/5214)) [Matthew Rocklin](https://github.com/mrocklin)
- Make boxes for dask-spark page ([dask#5249](https://github.com/dask/dask/pull/5249)) [Martin Durant](https://github.com/martindurant)
- Add motivation for shuffle docs ([dask#5213](https://github.com/dask/dask/pull/5213)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix links and API entries for best-practices ([dask#5246](https://github.com/dask/dask/pull/5246)) [Martin Durant](https://github.com/martindurant)
- Remove “bytes” (internal data ingestion) doc page ([dask#5242](https://github.com/dask/dask/pull/5242)) [Martin Durant](https://github.com/martindurant)
- Redirect from our local distributed page to distributed.dask.org ([dask#5248](https://github.com/dask/dask/pull/5248)) [Matthew Rocklin](https://github.com/mrocklin)
- Cleanup API page ([dask#5247](https://github.com/dask/dask/pull/5247)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove excess endlines from install docs ([dask#5243](https://github.com/dask/dask/pull/5243)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove item list in phases of computation doc ([dask#5245](https://github.com/dask/dask/pull/5245)) [Martin Durant](https://github.com/martindurant)
- Remove custom graphs from the TOC sidebar ([dask#5241](https://github.com/dask/dask/pull/5241)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove experimental status of custom collections ([dask#5236](https://github.com/dask/dask/pull/5236)) [James Bourbeau](https://github.com/jrbourbeau)
- Adds table of contents to Why Dask? ([dask#5244](https://github.com/dask/dask/pull/5244)) [James Bourbeau](https://github.com/jrbourbeau)
- Moves bag overview to top-level bag page ([dask#5240](https://github.com/dask/dask/pull/5240)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove use-cases in favor of stories.dask.org ([dask#5238](https://github.com/dask/dask/pull/5238)) [Matthew Rocklin](https://github.com/mrocklin)
- Removes redundant TOC information in index.rst ([dask#5235](https://github.com/dask/dask/pull/5235)) [James Bourbeau](https://github.com/jrbourbeau)
- Elevate dashboard in distributed diagnostics documentation ([dask#5239](https://github.com/dask/dask/pull/5239)) [Martin Durant](https://github.com/martindurant)
- Updates “add” layer in HLG docs example ([dask#5237](https://github.com/dask/dask/pull/5237)) [James Bourbeau](https://github.com/jrbourbeau)
- Update GUFunc documentation ([dask#5232](https://github.com/dask/dask/pull/5232)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v2-2-0-2019-08-01"></a>

## 2.2.0 / 2019-08-01

### Array

- Use da.from_array(…, asarray=False) if input follows NEP-18 ([dask#5074](https://github.com/dask/dask/pull/5074)) [Matthew Rocklin](https://github.com/mrocklin)
- Add missing attributes to from_array documentation ([dask#5108](https://github.com/dask/dask/pull/5108)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix meta computation for some reduction functions ([dask#5035](https://github.com/dask/dask/pull/5035)) [Peter Andreas Entschev](https://github.com/pentschev)
- Raise informative error in to_zarr if unknown chunks ([dask#5148](https://github.com/dask/dask/pull/5148)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove invalid pad tests ([dask#5122](https://github.com/dask/dask/pull/5122)) [Tom Augspurger](https://github.com/tomaugspurger)
- Ignore NumPy warnings in compute_meta ([dask#5103](https://github.com/dask/dask/pull/5103)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix kurtosis calc for single dimension input array ([dask#5177](https://github.com/dask/dask/pull/5177)) [@andrethrill](https://github.com/andrethrill)
- Support Numpy 1.17 in tests ([dask#5192](https://github.com/dask/dask/pull/5192)) [Matthew Rocklin](https://github.com/mrocklin)

### Bag

- Supply pool to bag test to resolve intermittent failure ([dask#5172](https://github.com/dask/dask/pull/5172)) [Tom Augspurger](https://github.com/tomaugspurger)

### Core

- Base dask on fsspec ([dask#5064](https://github.com/dask/dask/pull/5064)) ([dask#5121](https://github.com/dask/dask/pull/5121)) [Martin Durant](https://github.com/martindurant)
- Various upstream compatibility fixes ([dask#5056](https://github.com/dask/dask/pull/5056)) [Tom Augspurger](https://github.com/tomaugspurger)
- Make distributed tests optional again. ([dask#5128](https://github.com/dask/dask/pull/5128)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Fix HDFS in dask ([dask#5130](https://github.com/dask/dask/pull/5130)) [Martin Durant](https://github.com/martindurant)
- Ignore some more invalid value warnings. ([dask#5140](https://github.com/dask/dask/pull/5140)) [Elliott Sales de Andrade](https://github.com/QuLogic)

### DataFrame

- Fix pd.MultiIndex size estimate ([dask#5066](https://github.com/dask/dask/pull/5066)) [Brett Naul](https://github.com/bnaul)
- Generalizing has_known_categories ([dask#5090](https://github.com/dask/dask/pull/5090)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Refactor Parquet engine ([dask#4995](https://github.com/dask/dask/pull/4995)) [Richard J Zamora](https://github.com/rjzamora)
- Add divide method to series and dataframe ([dask#5094](https://github.com/dask/dask/pull/5094)) [msbrown47](https://github.com/msbrown47)
- fix flaky partd test ([dask#5111](https://github.com/dask/dask/pull/5111)) [Tom Augspurger](https://github.com/tomaugspurger)
- Adjust is_dataframe_like to adjust for value_counts change ([dask#5143](https://github.com/dask/dask/pull/5143)) [Tom Augspurger](https://github.com/tomaugspurger)
- Generalize rolling windows to support non-Pandas dataframes ([dask#5149](https://github.com/dask/dask/pull/5149)) [Nick Becker](https://github.com/beckernick)
- Avoid unnecessary aggregation in pivot_table ([dask#5173](https://github.com/dask/dask/pull/5173)) [Daniel Saxton](https://github.com/dsaxton)
- Add column names to apply_and_enforce error message ([dask#5180](https://github.com/dask/dask/pull/5180)) [Matthew Rocklin](https://github.com/mrocklin)
- Add schema keyword argument to to_parquet ([dask#5150](https://github.com/dask/dask/pull/5150)) [Sarah Bird](https://github.com/birdsarah)
- Remove recursion error in accessors ([dask#5182](https://github.com/dask/dask/pull/5182)) [Jim Crist](https://github.com/jcrist)
- Allow fastparquet to handle gather_statistics=False for file lists ([dask#5157](https://github.com/dask/dask/pull/5157)) [Richard J Zamora](https://github.com/rjzamora)

### Documentation

- Adds NumFOCUS badge to the README ([dask#5086](https://github.com/dask/dask/pull/5086)) [James Bourbeau](https://github.com/jrbourbeau)
- Update developer docs [ci skip] ([dask#5093](https://github.com/dask/dask/pull/5093)) [Jim Crist](https://github.com/jcrist)
- Document DataFrame.set_index computataion behavior [Natalya Rapstine](https://github.com/natalya-patrikeeva)
- Use pip install . instead of calling setup.py ([dask#5139](https://github.com/dask/dask/pull/5139)) [Matthias Bussonier](https://github.com/Carreau)
- Close user survey ([dask#5147](https://github.com/dask/dask/pull/5147)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix Google Calendar meeting link ([dask#5155](https://github.com/dask/dask/pull/5155)) [Loïc Estève](https://github.com/lesteve)
- Add docker image customization example ([dask#5171](https://github.com/dask/dask/pull/5171)) [James Bourbeau](https://github.com/jrbourbeau)
- Update remote-data-services after fsspec ([dask#5170](https://github.com/dask/dask/pull/5170)) [Martin Durant](https://github.com/martindurant)
- Fix typo in spark.rst ([dask#5164](https://github.com/dask/dask/pull/5164)) [Xavier Holt](https://github.com/xavi-ai)
- Update setup/python docs for async/await API ([dask#5163](https://github.com/dask/dask/pull/5163)) [Matthew Rocklin](https://github.com/mrocklin)
- Update Local Storage HPC documentation ([dask#5165](https://github.com/dask/dask/pull/5165)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v2-1-0-2019-07-08"></a>

## 2.1.0 / 2019-07-08

### Array

- Add `recompute=` keyword to `svd_compressed` for lower-memory use ([dask#5041](https://github.com/dask/dask/pull/5041)) [Matthew Rocklin](https://github.com/mrocklin)
- Change `__array_function__` implementation for backwards compatibility ([dask#5043](https://github.com/dask/dask/pull/5043)) [Ralf Gommers](https://github.com/rgommers)
- Added `dtype` and `shape` kwargs to `apply_along_axis` ([dask#3742](https://github.com/dask/dask/pull/3742)) [Davis Bennett](https://github.com/d-v-b)
- Fix reduction with empty tuple axis ([dask#5025](https://github.com/dask/dask/pull/5025)) [Peter Andreas Entschev](https://github.com/pentschev)
- Drop size 0 arrays in `stack` ([dask#4978](https://github.com/dask/dask/pull/4978)) [John A Kirkham](https://github.com/jakirkham)

### Core

- Removes index keyword from pandas `to_parquet` call ([dask#5075](https://github.com/dask/dask/pull/5075)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixes upstream dev CI build installation ([dask#5072](https://github.com/dask/dask/pull/5072)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure scalar arrays are not rendered to SVG ([dask#5058](https://github.com/dask/dask/pull/5058)) [Willi Rath](https://github.com/willirath)
- Environment creation overhaul ([dask#5038](https://github.com/dask/dask/pull/5038)) [Tom Augspurger](https://github.com/tomaugspurger)
- s3fs, moto compatibility ([dask#5033](https://github.com/dask/dask/pull/5033)) [Tom Augspurger](https://github.com/tomaugspurger)
- pytest 5.0 compat ([dask#5027](https://github.com/dask/dask/pull/5027)) [Tom Augspurger](https://github.com/tomaugspurger)

### DataFrame

- Fix `compute_meta` recursion in blockwise ([dask#5048](https://github.com/dask/dask/pull/5048)) [Peter Andreas Entschev](https://github.com/pentschev)
- Remove hard dependency on pandas in `get_dummies` ([dask#5057](https://github.com/dask/dask/pull/5057)) [GALI PREM SAGAR](https://github.com/galipremsagar)
- Check dtypes unchanged when using `DataFrame.assign` ([dask#5047](https://github.com/dask/dask/pull/5047)) [asmith26](https://github.com/asmith26)
- Fix cumulative functions on tables with more than 1 partition ([dask#5034](https://github.com/dask/dask/pull/5034)) [tshatrov](https://github.com/tshatrov)
- Handle non-divisible sizes in repartition ([dask#5013](https://github.com/dask/dask/pull/5013)) [George Sakkis](https://github.com/gsakkis)
- Handles timestamp and `preserve_index` changes in pyarrow ([dask#5018](https://github.com/dask/dask/pull/5018)) [Richard J Zamora](https://github.com/rjzamora)
- Fix undefined `meta` for `str.split(expand=False)` ([dask#5022](https://github.com/dask/dask/pull/5022)) [Brett Naul](https://github.com/bnaul)
- Removed checks used for debugging `merge_asof` ([dask#5011](https://github.com/dask/dask/pull/5011)) [Cody Johnson](https://github.com/codercody)
- Don’t use type when getting accessor in dataframes ([dask#4992](https://github.com/dask/dask/pull/4992)) [Matthew Rocklin](https://github.com/mrocklin)
- Add `melt` as a method of Dask DataFrame ([dask#4984](https://github.com/dask/dask/pull/4984)) [Dustin Tindall](https://github.com/dustindall)
- Adds path-like support to `to_hdf` ([dask#5003](https://github.com/dask/dask/pull/5003)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Point to latest K8s setup article in JupyterHub docs ([dask#5065](https://github.com/dask/dask/pull/5065)) [Sean McKenna](https://github.com/seanmck)
- Changes vizualize to visualize ([dask#5061](https://github.com/dask/dask/pull/5061)) [David Brochart](https://github.com/davidbrochart)
- Fix `from_sequence` typo in delayed best practices ([dask#5045](https://github.com/dask/dask/pull/5045)) [James Bourbeau](https://github.com/jrbourbeau)
- Add user survey link to docs ([dask#5026](https://github.com/dask/dask/pull/5026)) [James Bourbeau](https://github.com/jrbourbeau)
- Fixes typo in optimization docs ([dask#5015](https://github.com/dask/dask/pull/5015)) [James Bourbeau](https://github.com/jrbourbeau)
- Update community meeting information ([dask#5006](https://github.com/dask/dask/pull/5006)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v2-0-0-2019-06-25"></a>

## 2.0.0 / 2019-06-25

### Array

- Support automatic chunking in da.indices ([dask#4981](https://github.com/dask/dask/pull/4981)) [James Bourbeau](https://github.com/jrbourbeau)
- Err if there are no arrays to stack ([dask#4975](https://github.com/dask/dask/pull/4975)) [John A Kirkham](https://github.com/jakirkham)
- Asymmetrical Array Overlap ([dask#4863](https://github.com/dask/dask/pull/4863)) [Michael Eaton](https://github.com/mpeaton)
- Dispatch concatenate where possible within dask array ([dask#4669](https://github.com/dask/dask/pull/4669)) [Hameer Abbasi](https://github.com/hameerabbasi)
- Fix tokenization of memmapped numpy arrays on different part of same file ([dask#4931](https://github.com/dask/dask/pull/4931)) [Henry Pinkard](https://github.com/)
- Preserve NumPy condition in da.asarray to preserve output shape ([dask#4945](https://github.com/dask/dask/pull/4945)) [Alistair Miles](https://github.com/alimanfoo)
- Expand foo_like_safe usage ([dask#4946](https://github.com/dask/dask/pull/4946)) [Peter Andreas Entschev](https://github.com/pentschev)
- Defer order/casting einsum parameters to NumPy implementation ([dask#4914](https://github.com/dask/dask/pull/4914)) [Peter Andreas Entschev](https://github.com/pentschev)
- Remove numpy warning in moment calculation ([dask#4921](https://github.com/dask/dask/pull/4921)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix meta_from_array to support Xarray test suite ([dask#4938](https://github.com/dask/dask/pull/4938)) [Matthew Rocklin](https://github.com/mrocklin)
- Cache chunk boundaries for integer slicing ([dask#4923](https://github.com/dask/dask/pull/4923)) [Bruce Merry](https://github.com/bmerry)
- Drop size 0 arrays in concatenate ([dask#4167](https://github.com/dask/dask/pull/4167)) [John A Kirkham](https://github.com/jakirkham)
- Raise ValueError if concatenate is given no arrays ([dask#4927](https://github.com/dask/dask/pull/4927)) [John A Kirkham](https://github.com/jakirkham)
- Promote types in concatenate using \_meta ([dask#4925](https://github.com/dask/dask/pull/4925)) [John A Kirkham](https://github.com/jakirkham)
- Add chunk type to html repr in Dask array ([dask#4895](https://github.com/dask/dask/pull/4895)) [Matthew Rocklin](https://github.com/mrocklin)
- Add Dask Array._meta attribute ([dask#4543](https://github.com/dask/dask/pull/4543)) [Peter Andreas Entschev](https://github.com/pentschev)
  : - Fix \_meta slicing of flexible types ([dask#4912](https://github.com/dask/dask/pull/4912)) [Peter Andreas Entschev](https://github.com/pentschev)
    - Minor meta construction cleanup in concatenate ([dask#4937](https://github.com/dask/dask/pull/4937)) [Peter Andreas Entschev](https://github.com/pentschev)
    - Further relax Array meta checks for Xarray ([dask#4944](https://github.com/dask/dask/pull/4944)) [Matthew Rocklin](https://github.com/mrocklin)
    - Support meta= keyword in da.from_delayed ([dask#4972](https://github.com/dask/dask/pull/4972)) [Matthew Rocklin](https://github.com/mrocklin)
    - Concatenate meta along axis ([dask#4977](https://github.com/dask/dask/pull/4977)) [John A Kirkham](https://github.com/jakirkham)
    - Use meta in stack ([dask#4976](https://github.com/dask/dask/pull/4976)) [John A Kirkham](https://github.com/jakirkham)
    - Move blockwise_meta to more general compute_meta function ([dask#4954](https://github.com/dask/dask/pull/4954)) [Matthew Rocklin](https://github.com/mrocklin)
- Alias .partitions to .blocks attribute of dask arrays ([dask#4853](https://github.com/dask/dask/pull/4853)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Drop outdated numpy_compat functions ([dask#4850](https://github.com/dask/dask/pull/4850)) [John A Kirkham](https://github.com/jakirkham)
- Allow da.eye to support arbitrary chunking sizes with chunks=’auto’  ([dask#4834](https://github.com/dask/dask/pull/4834)) [Anderson Banihirwe](https://github.com/andersy005)
- Fix CI warnings in dask.array tests ([dask#4805](https://github.com/dask/dask/pull/4805)) [Tom Augspurger](https://github.com/tomaugspurger)
- Make map_blocks work with drop_axis + block_info ([dask#4831](https://github.com/dask/dask/pull/4831)) [Bruce Merry](https://github.com/bmerry)
- Add SVG image and table in Array._repr_html_ ([dask#4794](https://github.com/dask/dask/pull/4794)) [Matthew Rocklin](https://github.com/mrocklin)
- ufunc: avoid \_\_array_wrap_\_ in favor of \_\_array_function_\_ ([dask#4708](https://github.com/dask/dask/pull/4708)) [Peter Andreas Entschev](https://github.com/pentschev)
- Ensure trivial padding returns the original array ([dask#4990](https://github.com/dask/dask/pull/4990)) [John A Kirkham](https://github.com/jakirkham)
- Test `da.block` with 0-size arrays ([dask#4991](https://github.com/dask/dask/pull/4991)) [John A Kirkham](https://github.com/jakirkham)

### Core

- **Drop Python 2.7** ([dask#4919](https://github.com/dask/dask/pull/4919)) [Jim Crist](https://github.com/jcrist)
- Quiet dependency installs in CI ([dask#4960](https://github.com/dask/dask/pull/4960)) [Tom Augspurger](https://github.com/tomaugspurger)
- Raise on warnings in tests ([dask#4916](https://github.com/dask/dask/pull/4916)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add a diagnostics extra to setup.py (includes bokeh) ([dask#4924](https://github.com/dask/dask/pull/4924)) [John A Kirkham](https://github.com/jakirkham)
- Add newline delimter keyword to OpenFile ([dask#4935](https://github.com/dask/dask/pull/4935)) [btw08](https://github.com/btw08)
- Overload HighLevelGraphs values method ([dask#4918](https://github.com/dask/dask/pull/4918)) [James Bourbeau](https://github.com/jrbourbeau)
- Add \_\_await_\_ method to Dask collections ([dask#4901](https://github.com/dask/dask/pull/4901)) [Matthew Rocklin](https://github.com/mrocklin)
- Also ignore AttributeErrors which may occur if snappy (not python-snappy) is installed ([dask#4908](https://github.com/dask/dask/pull/4908)) [Mark Bell](https://github.com/MarkCBell)
- Canonicalize key names in config.rename ([dask#4903](https://github.com/dask/dask/pull/4903)) [Ian Bolliger](https://github.com/bolliger32)
- Bump minimum partd to 0.3.10 ([dask#4890](https://github.com/dask/dask/pull/4890)) [Tom Augspurger](https://github.com/tomaugspurger)
- Catch async def SyntaxError ([dask#4836](https://github.com/dask/dask/pull/4836)) [James Bourbeau](https://github.com/jrbourbeau)
- catch IOError in ensure_file ([dask#4806](https://github.com/dask/dask/pull/4806)) [Justin Poehnelt](https://github.com/jpoehnelt)
- Cleanup CI warnings ([dask#4798](https://github.com/dask/dask/pull/4798)) [Tom Augspurger](https://github.com/tomaugspurger)
- Move distributed’s parse and format functions to dask.utils ([dask#4793](https://github.com/dask/dask/pull/4793)) [Matthew Rocklin](https://github.com/mrocklin)
- Apply black formatting ([dask#4983](https://github.com/dask/dask/pull/4983)) [James Bourbeau](https://github.com/jrbourbeau)
- Package license file in wheels ([dask#4988](https://github.com/dask/dask/pull/4988)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Add an optional partition_size parameter to repartition ([dask#4416](https://github.com/dask/dask/pull/4416)) [George Sakkis](https://github.com/gsakkis)
- merge_asof and prefix_reduction ([dask#4877](https://github.com/dask/dask/pull/4877)) [Cody Johnson](https://github.com/codercody)
- Allow dataframes to be indexed by dask arrays ([dask#4882](https://github.com/dask/dask/pull/4882)) [Endre Mark Borza](https://github.com/endremborza)
- Avoid deprecated message parameter in pytest.raises ([dask#4962](https://github.com/dask/dask/pull/4962)) [James Bourbeau](https://github.com/jrbourbeau)
- Update test_to_records to test with lengths argument([dask#4515](https://github.com/dask/dask/pull/4515)) [asmith26](https://github.com/asmith26)
- Remove pandas pinning in Dataframe accessors ([dask#4955](https://github.com/dask/dask/pull/4955)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix correlation of series with same names ([dask#4934](https://github.com/dask/dask/pull/4934)) [Philipp S. Sommer](https://github.com/Chilipp)
- Map Dask Series to Dask Series ([dask#4872](https://github.com/dask/dask/pull/4872)) [Justin Waugh](https://github.com/bluecoconut)
- Warn in dd.merge on dtype warning ([dask#4917](https://github.com/dask/dask/pull/4917)) [mcsoini](https://github.com/mcsoini)
- Add groupby Covariance/Correlation ([dask#4889](https://github.com/dask/dask/pull/4889)) [Benjamin Zaitlen](https://github.com/quasiben)
- keep index name with to_datetime ([dask#4905](https://github.com/dask/dask/pull/4905)) [Ian Bolliger](https://github.com/bolliger32)
- Add Parallel variance computation for dataframes ([dask#4865](https://github.com/dask/dask/pull/4865)) [Ksenia Bobrova](https://github.com/almaleksia)
- Add divmod implementation to arrays and dataframes ([dask#4884](https://github.com/dask/dask/pull/4884)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Add documentation for dataframe reshape methods ([dask#4896](https://github.com/dask/dask/pull/4896)) [tpanza](https://github.com/tpanza)
- Avoid use of pandas.compat ([dask#4881](https://github.com/dask/dask/pull/4881)) [Tom Augspurger](https://github.com/tomaugspurger)
- Added accessor registration for Series, DataFrame, and Index ([dask#4829](https://github.com/dask/dask/pull/4829)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add read_function keyword to read_json ([dask#4810](https://github.com/dask/dask/pull/4810)) [Richard J Zamora](https://github.com/rjzamora)
- Provide full type name in check_meta ([dask#4819](https://github.com/dask/dask/pull/4819)) [Matthew Rocklin](https://github.com/mrocklin)
- Correctly estimate bytes per row in read_sql_table ([dask#4807](https://github.com/dask/dask/pull/4807)) [Lijo Jose](https://github.com/lijose)
- Adding support of non-numeric data to describe() ([dask#4791](https://github.com/dask/dask/pull/4791)) [Ksenia Bobrova](https://github.com/almaleksia)
- Scalars for extension dtypes. ([dask#4459](https://github.com/dask/dask/pull/4459)) [Tom Augspurger](https://github.com/tomaugspurger)
- Call head before compute in dd.from_delayed ([dask#4802](https://github.com/dask/dask/pull/4802)) [Matthew Rocklin](https://github.com/mrocklin)
- Add support for rolling operations with larger window that partition size in DataFrames with Time-based index ([dask#4796](https://github.com/dask/dask/pull/4796)) [Jorge Pessoa](https://github.com/jorge-pessoa)
- Update groupby-apply doc with warning ([dask#4800](https://github.com/dask/dask/pull/4800)) [Tom Augspurger](https://github.com/tomaugspurger)
- Change groupby-ness tests in \_maybe_slice ([dask#4786](https://github.com/dask/dask/pull/4786)) [Benjamin Zaitlen](https://github.com/quasiben)
- Add master best practices document ([dask#4745](https://github.com/dask/dask/pull/4745)) [Matthew Rocklin](https://github.com/mrocklin)
- Add document for how Dask works with GPUs ([dask#4792](https://github.com/dask/dask/pull/4792)) [Matthew Rocklin](https://github.com/mrocklin)
- Add cli API docs ([dask#4788](https://github.com/dask/dask/pull/4788)) [James Bourbeau](https://github.com/jrbourbeau)
- Ensure concat output has coherent dtypes ([dask#4692](https://github.com/dask/dask/pull/4692)) [Guillaume Lemaitre](https://github.com/glemaitre)
- Fixes pandas_datareader dependencies installation ([dask#4989](https://github.com/dask/dask/pull/4989)) [James Bourbeau](https://github.com/jrbourbeau)
- Accept pathlib.Path as pattern in read_hdf ([dask#3335](https://github.com/dask/dask/pull/3335)) [Jörg Dietrich](https://github.com/joergdietrich)

### Documentation

- Move CLI API docs to relavant pages ([dask#4980](https://github.com/dask/dask/pull/4980)) [James Bourbeau](https://github.com/jrbourbeau)
- Add to_datetime function to dataframe API docs [Matthew Rocklin](https://github.com/mrocklin)
- Add documentation entry for dask.array.ma.average ([dask#4970](https://github.com/dask/dask/pull/4970)) [Bouwe Andela](https://github.com/bouweandela)
- Add bag.read_avro to bag API docs ([dask#4969](https://github.com/dask/dask/pull/4969)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix typo ([dask#4968](https://github.com/dask/dask/pull/4968)) [mbarkhau](https://github.com/mbarkhau)
- Docs: Drop support for Python 2.7 ([dask#4932](https://github.com/dask/dask/pull/4932)) [Hugo](https://github.com/hugovk)
- Remove requirement to modify changelog ([dask#4915](https://github.com/dask/dask/pull/4915)) [Matthew Rocklin](https://github.com/mrocklin)
- Add documentation about meta column order ([dask#4887](https://github.com/dask/dask/pull/4887)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add documentation note in DataFrame.shift ([dask#4886](https://github.com/dask/dask/pull/4886)) [Tom Augspurger](https://github.com/tomaugspurger)
- Docs: Fix typo ([dask#4868](https://github.com/dask/dask/pull/4868)) [Paweł Kordek](https://github.com/kordek)
- Put do/don’t into boxes for delayed best practice docs ([dask#3821](https://github.com/dask/dask/pull/3821)) [Martin Durant](https://github.com/martindurant)
- Doc fixups ([dask#2528](https://github.com/dask/dask/pull/2528)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add quansight to paid support doc section ([dask#4838](https://github.com/dask/dask/pull/4838)) [Martin Durant](https://github.com/martindurant)
- Add document for custom startup ([dask#4833](https://github.com/dask/dask/pull/4833)) [Matthew Rocklin](https://github.com/mrocklin)
- Allow utils.derive_from to accept functions, apply across array ([dask#4804](https://github.com/dask/dask/pull/4804)) [Martin Durant](https://github.com/martindurant)
- Add “Avoid Large Partitions” section to best practices ([dask#4808](https://github.com/dask/dask/pull/4808)) [Matthew Rocklin](https://github.com/mrocklin)
- Update URL for joblib to new website hosting their doc ([dask#4816](https://github.com/dask/dask/pull/4816)) [Christian Hudon](https://github.com/chrish42)

<a id="v1-2-2-2019-05-08"></a>

## 1.2.2 / 2019-05-08

### Array

- Clarify regions kwarg to array.store ([dask#4759](https://github.com/dask/dask/pull/4759)) [Martin Durant](https://github.com/martindurant)
- Add dtype= parameter to da.random.randint ([dask#4753](https://github.com/dask/dask/pull/4753)) [Matthew Rocklin](https://github.com/mrocklin)
- Use “row major” rather than “C order” in docstring ([dask#4452](https://github.com/dask/dask/pull/4452)) [@asmith26](https://github.com/asmith26)
- Normalize Xarray datasets to Dask arrays ([dask#4756](https://github.com/dask/dask/pull/4756)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove normed keyword in da.histogram ([dask#4755](https://github.com/dask/dask/pull/4755)) [Matthew Rocklin](https://github.com/mrocklin)

### Bag

- Add key argument to Bag.distinct ([dask#4423](https://github.com/dask/dask/pull/4423)) [Daniel Severo](https://github.com/dsevero)

### Core

- Add core dask config file ([dask#4774](https://github.com/dask/dask/pull/4774)) [Matthew Rocklin](https://github.com/mrocklin)
- Add core dask config file to MANIFEST.in ([dask#4780](https://github.com/dask/dask/pull/4780)) [James Bourbeau](https://github.com/jrbourbeau)
- Enabling glob with HTTP file-system ([dask#3926](https://github.com/dask/dask/pull/3926)) [Martin Durant](https://github.com/martindurant)
- HTTPFile.seek with whence=1 ([dask#4751](https://github.com/dask/dask/pull/4751)) [Martin Durant](https://github.com/martindurant)
- Remove config key normalization ([dask#4742](https://github.com/dask/dask/pull/4742)) [Jim Crist](https://github.com/jcrist)

### DataFrame

- Remove explicit references to Pandas in dask.dataframe.groupby ([dask#4778](https://github.com/dask/dask/pull/4778)) [Matthew Rocklin](https://github.com/mrocklin)
- Add support for group_keys kwarg in DataFrame.groupby() ([dask#4771](https://github.com/dask/dask/pull/4771)) [Brian Chu](https://github.com/bchu)
- Describe doc ([dask#4762](https://github.com/dask/dask/pull/4762)) [Martin Durant](https://github.com/martindurant)
- Remove explicit pandas check in cumulative aggregations ([dask#4765](https://github.com/dask/dask/pull/4765)) [Nick Becker](https://github.com/beckernick)
- Added meta for read_json and test ([dask#4588](https://github.com/dask/dask/pull/4588)) [Abhinav Ralhan](https://github.com/abhinavralhan)
- Add test for dtype casting ([dask#4760](https://github.com/dask/dask/pull/4760)) [Martin Durant](https://github.com/martindurant)
- Document alignment in map_partitions ([dask#4757](https://github.com/dask/dask/pull/4757)) [Jim Crist](https://github.com/jcrist)
- Implement Series.str.split(expand=True) ([dask#4744](https://github.com/dask/dask/pull/4744)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Tweaks to develop.rst from trying to run tests ([dask#4772](https://github.com/dask/dask/pull/4772)) [Christian Hudon](https://github.com/chrish42)
- Add document describing phases of computation ([dask#4766](https://github.com/dask/dask/pull/4766)) [Matthew Rocklin](https://github.com/mrocklin)
- Point users to Dask-Yarn from spark documentation ([dask#4770](https://github.com/dask/dask/pull/4770)) [Matthew Rocklin](https://github.com/mrocklin)
- Update images in delayed doc to remove labels ([dask#4768](https://github.com/dask/dask/pull/4768)) [Martin Durant](https://github.com/martindurant)
- Explain intermediate storage for dask arrays ([dask#4025](https://github.com/dask/dask/pull/4025)) [John A Kirkham](https://github.com/jakirkham)
- Specify bash code-block in array best practices ([dask#4764](https://github.com/dask/dask/pull/4764)) [James Bourbeau](https://github.com/jrbourbeau)
- Add array best practices doc ([dask#4705](https://github.com/dask/dask/pull/4705)) [Matthew Rocklin](https://github.com/mrocklin)
- Update optimization docs now that cull is not automatic ([dask#4752](https://github.com/dask/dask/pull/4752)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v1-2-1-2019-04-29"></a>

## 1.2.1 / 2019-04-29

### Array

- Fix map_blocks with block_info and broadcasting ([dask#4737](https://github.com/dask/dask/pull/4737)) [Bruce Merry](https://github.com/bmerry)
- Make ‘minlength’ keyword argument optional in da.bincount ([dask#4684](https://github.com/dask/dask/pull/4684)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add support for map_blocks with no array arguments ([dask#4713](https://github.com/dask/dask/pull/4713)) [Bruce Merry](https://github.com/bmerry)
- Add dask.array.trace ([dask#4717](https://github.com/dask/dask/pull/4717)) [Danilo Horta](https://github.com/horta)
- Add sizeof support for cupy.ndarray ([dask#4715](https://github.com/dask/dask/pull/4715)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add name kwarg to from_zarr ([dask#4663](https://github.com/dask/dask/pull/4663)) [Michael Eaton](https://github.com/mpeaton)
- Add chunks=’auto’ to from_array ([dask#4704](https://github.com/dask/dask/pull/4704)) [Matthew Rocklin](https://github.com/mrocklin)
- Raise TypeError if dask array is given as shape for da.ones, zeros, empty or full ([dask#4707](https://github.com/dask/dask/pull/4707)) [Genevieve Buckley](https://github.com/GenevieveBuckley)
- Add TileDB backend ([dask#4679](https://github.com/dask/dask/pull/4679)) [Isaiah Norton](https://github.com/hnorton)

### Core

- Delay long list arguments ([dask#4735](https://github.com/dask/dask/pull/4735)) [Matthew Rocklin](https://github.com/mrocklin)
- Bump to numpy >= 1.13, pandas >= 0.21.0 ([dask#4720](https://github.com/dask/dask/pull/4720)) [Jim Crist](https://github.com/jcrist)
- Remove file “test” ([dask#4710](https://github.com/dask/dask/pull/4710)) [James Bourbeau](https://github.com/jrbourbeau)
- Reenable development build, uses upstream libraries ([dask#4696](https://github.com/dask/dask/pull/4696)) [Peter Andreas Entschev](https://github.com/pentschev)
- Remove assertion in HighLevelGraph constructor ([dask#4699](https://github.com/dask/dask/pull/4699)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Change cum-aggregation last-nonnull-value algorithm ([dask#4736](https://github.com/dask/dask/pull/4736)) [Nick Becker](https://github.com/beckernick)
- Fixup series-groupby-apply ([dask#4738](https://github.com/dask/dask/pull/4738)) [Jim Crist](https://github.com/jcrist)
- Refactor array.percentile and dataframe.quantile to use t-digest ([dask#4677](https://github.com/dask/dask/pull/4677)) [Janne Vuorela](https://github.com/Dimplexion)
- Allow naive concatenation of sorted dataframes ([dask#4725](https://github.com/dask/dask/pull/4725)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix perf issue in dd.Series.isin ([dask#4727](https://github.com/dask/dask/pull/4727)) [Jim Crist](https://github.com/jcrist)
- Remove hard pandas dependency for melt by using methodcaller ([dask#4719](https://github.com/dask/dask/pull/4719)) [Nick Becker](https://github.com/beckernick)
- A few dataframe metadata fixes ([dask#4695](https://github.com/dask/dask/pull/4695)) [Jim Crist](https://github.com/jcrist)
- Add Dataframe.replace ([dask#4714](https://github.com/dask/dask/pull/4714)) [Matthew Rocklin](https://github.com/mrocklin)
- Add ‘threshold’ parameter to pd.DataFrame.dropna ([dask#4625](https://github.com/dask/dask/pull/4625)) [Nathan Matare](https://github.com/nmatare)

### Documentation

- Add warning about derived docstrings early in the docstring ([dask#4716](https://github.com/dask/dask/pull/4716)) [Matthew Rocklin](https://github.com/mrocklin)
- Create dataframe best practices doc ([dask#4703](https://github.com/dask/dask/pull/4703)) [Matthew Rocklin](https://github.com/mrocklin)
- Uncomment dask_sphinx_theme ([dask#4728](https://github.com/dask/dask/pull/4728)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix minor typo fix in a Queue/fire_and_forget example ([dask#4709](https://github.com/dask/dask/pull/4709)) [Matthew Rocklin](https://github.com/mrocklin)
- Update from_pandas docstring to match signature ([dask#4698](https://github.com/dask/dask/pull/4698)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v1-2-0-2019-04-12"></a>

## 1.2.0 / 2019-04-12

### Array

- Fixed mean() and moment() on sparse arrays ([dask#4525](https://github.com/dask/dask/pull/4525)) [Peter Andreas Entschev](https://github.com/pentschev)
- Add test for NEP-18. ([dask#4675](https://github.com/dask/dask/pull/4675)) [Hameer Abbasi](https://github.com/hameerabbasi)
- Allow None to say “no chunking” in normalize_chunks ([dask#4656](https://github.com/dask/dask/pull/4656)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix limit value in auto_chunks ([dask#4645](https://github.com/dask/dask/pull/4645)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Updated diagnostic bokeh test for compatibility with bokeh>=1.1.0 ([dask#4680](https://github.com/dask/dask/pull/4680)) [Philipp Rudiger](https://github.com/philippjfr)
- Adjusts codecov’s target/threshold, disable patch ([dask#4671](https://github.com/dask/dask/pull/4671)) [Peter Andreas Entschev](https://github.com/pentschev)
- Always start with empty http buffer, not None ([dask#4673](https://github.com/dask/dask/pull/4673)) [Martin Durant](https://github.com/martindurant)

### DataFrame

- Propagate index dtype and name when create dask dataframe from array ([dask#4686](https://github.com/dask/dask/pull/4686)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Fix ordering of quantiles in describe ([dask#4647](https://github.com/dask/dask/pull/4647)) [gregrf](https://github.com/gregrf)
- Clean up and document rearrange_column_by_tasks ([dask#4674](https://github.com/dask/dask/pull/4674)) [Matthew Rocklin](https://github.com/mrocklin)
- Mark some parquet tests xfail ([dask#4667](https://github.com/dask/dask/pull/4667)) [Peter Andreas Entschev](https://github.com/pentschev)
- Fix parquet breakages with arrow 0.13.0 ([dask#4668](https://github.com/dask/dask/pull/4668)) [Martin Durant](https://github.com/martindurant)
- Allow sample to be False when reading CSV from a remote URL ([dask#4634](https://github.com/dask/dask/pull/4634)) [Ian Rose](https://github.com/ian-r-rose)
- Fix timezone metadata inference on parquet load ([dask#4655](https://github.com/dask/dask/pull/4655)) [Martin Durant](https://github.com/martindurant)
- Use is_dataframe/index_like in dd.utils ([dask#4657](https://github.com/dask/dask/pull/4657)) [Matthew Rocklin](https://github.com/mrocklin)
- Add min_count parameter to groupby sum method ([dask#4648](https://github.com/dask/dask/pull/4648)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Correct quantile to handle unsorted quantiles ([dask#4650](https://github.com/dask/dask/pull/4650)) [gregrf](https://github.com/gregrf)

### Documentation

- Add delayed extra dependencies to install docs ([dask#4660](https://github.com/dask/dask/pull/4660)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v1-1-5-2019-03-29"></a>

## 1.1.5 / 2019-03-29

### Array

- Ensure that we use the dtype keyword in normalize_chunks ([dask#4646](https://github.com/dask/dask/pull/4646)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Use recursive glob in LocalFileSystem ([dask#4186](https://github.com/dask/dask/pull/4186)) [Brett Naul](https://github.com/bnaul)
- Avoid YAML deprecation ([dask#4603](https://github.com/dask/dask/pull/4603))
- Fix CI and add set -e ([dask#4605](https://github.com/dask/dask/pull/4605)) [James Bourbeau](https://github.com/jrbourbeau)
- Support builtin sequence types in dask.visualize ([dask#4602](https://github.com/dask/dask/pull/4602))
- unpack/repack orderedDict ([dask#4623](https://github.com/dask/dask/pull/4623)) [Justin Poehnelt](https://github.com/jpoehnelt)
- Add da.random.randint to API docs ([dask#4628](https://github.com/dask/dask/pull/4628)) [James Bourbeau](https://github.com/jrbourbeau)
- Add zarr to CI environment ([dask#4604](https://github.com/dask/dask/pull/4604)) [James Bourbeau](https://github.com/jrbourbeau)
- Enable codecov ([dask#4631](https://github.com/dask/dask/pull/4631)) [Peter Andreas Entschev](https://github.com/pentschev)

### DataFrame

- Support setting the index ([dask#4565](https://github.com/dask/dask/pull/4565))
- DataFrame.itertuples accepts index, name kwargs ([dask#4593](https://github.com/dask/dask/pull/4593)) [Dan O’Donovan](https://github.com/danodonovan)
- Support non-Pandas series in dd.Series.unique ([dask#4599](https://github.com/dask/dask/pull/4599)) [Benjamin Zaitlen](https://github.com/quasiben)
- Replace use of explicit type check with ._is_partition_type predicate ([dask#4533](https://github.com/dask/dask/pull/4533))
- Remove additional pandas warnings in tests ([dask#4576](https://github.com/dask/dask/pull/4576))
- Check object for name/dtype attributes rather than type ([dask#4606](https://github.com/dask/dask/pull/4606))
- Fix comparison against pd.Series ([dask#4613](https://github.com/dask/dask/pull/4613)) [amerkel2](https://github.com/amerkel2)
- Fixing warning from setting categorical codes to floats ([dask#4624](https://github.com/dask/dask/pull/4624)) [Julia Signell](https://github.com/jsignell)
- Fix renaming on index to_frame method ([dask#4498](https://github.com/dask/dask/pull/4498)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Fix divisions when joining two single-partition dataframes ([dask#4636](https://github.com/dask/dask/pull/4636)) [Justin Waugh](https://github.com/bluecoconut)
- Warn if partitions overlap in compute_divisions ([dask#4600](https://github.com/dask/dask/pull/4600)) [Brian Chu](https://github.com/bchu)
- Give informative meta= warning ([dask#4637](https://github.com/dask/dask/pull/4637)) [Matthew Rocklin](https://github.com/mrocklin)
- Add informative error message to Series._\_getitem_\_ ([dask#4638](https://github.com/dask/dask/pull/4638)) [Matthew Rocklin](https://github.com/mrocklin)
- Add clear exception message when using index or index_col in read_csv ([dask#4651](https://github.com/dask/dask/pull/4651)) [Álvaro Abella Bascarán](https://github.com/alvaroabascar)

### Documentation

- Add documentation for custom groupby aggregations ([dask#4571](https://github.com/dask/dask/pull/4571))
- Docs dataframe joins ([dask#4569](https://github.com/dask/dask/pull/4569))
- Specify fork-based contributions  ([dask#4619](https://github.com/dask/dask/pull/4619)) [James Bourbeau](https://github.com/jrbourbeau)
- correct to_parquet example in docs ([dask#4641](https://github.com/dask/dask/pull/4641)) [Aaron Fowles](https://github.com/aaronfowles)
- Update and secure several references ([dask#4649](https://github.com/dask/dask/pull/4649)) [Søren Fuglede Jørgensen](https://github.com/fuglede)

<a id="v1-1-4-2019-03-08"></a>

## 1.1.4 / 2019-03-08

### Array

- Use mask selection in compress ([dask#4548](https://github.com/dask/dask/pull/4548)) [John A Kirkham](https://github.com/jakirkham)
- Use asarray in extract ([dask#4549](https://github.com/dask/dask/pull/4549)) [John A Kirkham](https://github.com/jakirkham)
- Use correct dtype when test concatenation. ([dask#4539](https://github.com/dask/dask/pull/4539)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Fix CuPy tests or properly marks as xfail ([dask#4564](https://github.com/dask/dask/pull/4564)) [Peter Andreas Entschev](https://github.com/pentschev)

### Core

- Fix local scheduler callback to deal with custom caching ([dask#4542](https://github.com/dask/dask/pull/4542)) [Yu Feng](https://github.com/rainwoodman)
- Use parse_bytes in read_bytes(sample=…) ([dask#4554](https://github.com/dask/dask/pull/4554)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Fix up groupby-standard deviation again on object dtype keys ([dask#4541](https://github.com/dask/dask/pull/4541)) [Matthew Rocklin](https://github.com/mrocklin)
- TST/CI: Updates for pandas 0.24.1 ([dask#4551](https://github.com/dask/dask/pull/4551)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add ability to control number of unique elements in timeseries ([dask#4557](https://github.com/dask/dask/pull/4557)) [Matthew Rocklin](https://github.com/mrocklin)
- Add support in read_csv for parameter skiprows for other iterables ([dask#4560](https://github.com/dask/dask/pull/4560)) [@JulianWgs](https://github.com/JulianWgs)

### Documentation

- DataFrame to Array conversion and unknown chunks ([dask#4516](https://github.com/dask/dask/pull/4516)) [Scott Sievert](https://github.com/stsievert)
- Add docs for random array creation ([dask#4566](https://github.com/dask/dask/pull/4566)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix typo in docstring ([dask#4572](https://github.com/dask/dask/pull/4572)) [Shyam Saladi](https://github.com/smsaladi)

<a id="v1-1-3-2019-03-01"></a>

## 1.1.3 / 2019-03-01

### Array

- Modify mean chunk functions to return dicts rather than arrays ([dask#4513](https://github.com/dask/dask/pull/4513)) [Matthew Rocklin](https://github.com/mrocklin)
- Change sparse installation in CI for NumPy/Python2 compatibility ([dask#4537](https://github.com/dask/dask/pull/4537)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Make merge dispatchable on pandas/other dataframe types ([dask#4522](https://github.com/dask/dask/pull/4522)) [Matthew Rocklin](https://github.com/mrocklin)
- read_sql_table - datetime index fix and  index type checking ([dask#4474](https://github.com/dask/dask/pull/4474)) [Joe Corbett](https://github.com/jcorb)
- Use generalized form of index checking (is_index_like) ([dask#4531](https://github.com/dask/dask/pull/4531)) [Benjamin Zaitlen](https://github.com/quasiben)
- Add tests for groupby reductions with object dtypes ([dask#4535](https://github.com/dask/dask/pull/4535)) [Matthew Rocklin](https://github.com/mrocklin)
- Fixes #4467 : Updates time_series for pandas deprecation ([dask#4530](https://github.com/dask/dask/pull/4530)) [@HSR05](https://github.com/HSR05)

### Documentation

- Add missing method to documentation index ([dask#4528](https://github.com/dask/dask/pull/4528)) [Bart Broere](https://github.com/bartbroere)

<a id="v1-1-2-2019-02-25"></a>

## 1.1.2 / 2019-02-25

### Array

- Fix another unicode/mixed-type edge case in normalize_array ([dask#4489](https://github.com/dask/dask/pull/4489)) [Marco Neumann](https://github.com/crepererum)
- Add dask.array.diagonal ([dask#4431](https://github.com/dask/dask/pull/4431)) [Danilo Horta](https://github.com/horta)
- Call asanyarray in unify_chunks ([dask#4506](https://github.com/dask/dask/pull/4506)) [Jim Crist](https://github.com/jcrist)
- Modify moment chunk functions to return dicts ([dask#4519](https://github.com/dask/dask/pull/4519)) [Peter Andreas Entschev](https://github.com/pentschev)

### Bag

- Don’t inline output keys in dask.bag ([dask#4464](https://github.com/dask/dask/pull/4464)) [Jim Crist](https://github.com/jcrist)
- Ensure that bag.from_sequence always includes at least one partition ([dask#4475](https://github.com/dask/dask/pull/4475)) [Anderson Banihirwe](https://github.com/andersy005)
- Implement out_type for bag.fold ([dask#4502](https://github.com/dask/dask/pull/4502)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove map from bag keynames ([dask#4500](https://github.com/dask/dask/pull/4500)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid itertools.repeat in map_partitions ([dask#4507](https://github.com/dask/dask/pull/4507)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Fix relative path parsing on windows when using fastparquet ([dask#4445](https://github.com/dask/dask/pull/4445)) [Janne Vuorela](https://github.com/Dimplexion)
- Fix bug in pyarrow and hdfs ([dask#4453](https://github.com/dask/dask/pull/4453)) ([dask#4455](https://github.com/dask/dask/pull/4455)) [Michał Jastrzębski](https://github.com/inc0)
- df getitem with integer slices is not implemented ([dask#4466](https://github.com/dask/dask/pull/4466)) [Jim Crist](https://github.com/jcrist)
- Replace cudf-specific code with dask-cudf import ([dask#4470](https://github.com/dask/dask/pull/4470)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid groupby.agg(callable) in groupby-var ([dask#4482](https://github.com/dask/dask/pull/4482)) [Matthew Rocklin](https://github.com/mrocklin)
- Consider uint types as numerical in check_meta ([dask#4485](https://github.com/dask/dask/pull/4485)) [Marco Neumann](https://github.com/crepererum)
- Fix some typos in groupby comments ([dask#4494](https://github.com/dask/dask/pull/4494)) [Daniel Saxton](https://github.com/dsaxton)
- Add error message around set_index(inplace=True) ([dask#4501](https://github.com/dask/dask/pull/4501)) [Matthew Rocklin](https://github.com/mrocklin)
- meta_nonempty works with categorical index ([dask#4505](https://github.com/dask/dask/pull/4505)) [Jim Crist](https://github.com/jcrist)
- Add module name to expected meta error message ([dask#4499](https://github.com/dask/dask/pull/4499)) [Matthew Rocklin](https://github.com/mrocklin)
- groupby-nunique works on empty chunk ([dask#4504](https://github.com/dask/dask/pull/4504)) [Jim Crist](https://github.com/jcrist)
- Propagate index metadata if not specified ([dask#4509](https://github.com/dask/dask/pull/4509)) [Jim Crist](https://github.com/jcrist)

### Documentation

- Update docs to use `from_zarr` ([dask#4472](https://github.com/dask/dask/pull/4472)) [John A Kirkham](https://github.com/jakirkham)
- DOC: add section of Using Other S3-Compatible Services for remote-data-services ([dask#4405](https://github.com/dask/dask/pull/4405)) [Aploium](https://github.com/aploium)
- Fix header level of section in changelog ([dask#4483](https://github.com/dask/dask/pull/4483)) [Bruce Merry](https://github.com/bmerry)
- Add quotes to pip install [skip-ci] ([dask#4508](https://github.com/dask/dask/pull/4508)) [James Bourbeau](https://github.com/jrbourbeau)

### Core

- Extend started_cbs AFTER state is initialized ([dask#4460](https://github.com/dask/dask/pull/4460)) [Marco Neumann](https://github.com/crepererum)
- Fix bug in HTTPFile._fetch_range with headers ([dask#4479](https://github.com/dask/dask/pull/4479)) ([dask#4480](https://github.com/dask/dask/pull/4480)) [Ross Petchler](https://github.com/rpetchler)
- Repeat optimize_blockwise for diamond fusion ([dask#4492](https://github.com/dask/dask/pull/4492)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v1-1-1-2019-01-31"></a>

## 1.1.1 / 2019-01-31

### Array

- Add support for cupy.einsum ([dask#4402](https://github.com/dask/dask/pull/4402)) [Johnnie Gray](https://github.com/jcmgray)
- Provide byte size in chunks keyword ([dask#4434](https://github.com/dask/dask/pull/4434)) [Adam Beberg](https://github.com/beberg)
- Raise more informative error for histogram bins and range ([dask#4430](https://github.com/dask/dask/pull/4430)) [James Bourbeau](https://github.com/jrbourbeau)

### DataFrame

- Lazily register more cudf functions and move to backends file ([dask#4396](https://github.com/dask/dask/pull/4396)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix ORC tests for pyarrow 0.12.0 ([dask#4413](https://github.com/dask/dask/pull/4413)) [Jim Crist](https://github.com/jcrist)
- rearrange_by_column: ensure that shuffle arg defaults to ‘disk’ if it’s None in dask.config ([dask#4414](https://github.com/dask/dask/pull/4414)) [George Sakkis](https://github.com/gsakkis)
- Implement filters for \_read_pyarrow ([dask#4415](https://github.com/dask/dask/pull/4415)) [George Sakkis](https://github.com/gsakkis)
- Avoid checking against types in is_dataframe_like ([dask#4418](https://github.com/dask/dask/pull/4418)) [Matthew Rocklin](https://github.com/mrocklin)
- Pass username as ‘user’ when using pyarrow ([dask#4438](https://github.com/dask/dask/pull/4438)) [Roma Sokolov](https://github.com/little-arhat)

### Delayed

- Fix DelayedAttr return value ([dask#4440](https://github.com/dask/dask/pull/4440)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Use SVG for pipeline graphic ([dask#4406](https://github.com/dask/dask/pull/4406)) [John A Kirkham](https://github.com/jakirkham)
- Add doctest-modules to py.test documentation ([dask#4427](https://github.com/dask/dask/pull/4427)) [Daniel Severo](https://github.com/dsevero)

### Core

- Work around psutil 5.5.0 not allowing pickling Process objects [Janne Vuorela](https://github.com/Dimplexion)

<a id="v1-1-0-2019-01-18"></a>

## 1.1.0 / 2019-01-18

### Array

- Fix the average function when there is a masked array ([dask#4236](https://github.com/dask/dask/pull/4236)) [Damien Garaud](https://github.com/geraud)
- Add allow_unknown_chunksizes to hstack and vstack ([dask#4287](https://github.com/dask/dask/pull/4287)) [Paul Vecchio](https://github.com/vecchp)
- Fix tensordot for 27+ dimensions ([dask#4304](https://github.com/dask/dask/pull/4304)) [Johnnie Gray](https://github.com/jcmgray)
- Fixed block_info with axes. ([dask#4301](https://github.com/dask/dask/pull/4301)) [Tom Augspurger](https://github.com/tomaugspurger)
- Use safe_wraps for matmul ([dask#4346](https://github.com/dask/dask/pull/4346)) [Mark Harfouche](https://github.com/hmaarrfk)
- Use chunks=”auto” in array creation routines ([dask#4354](https://github.com/dask/dask/pull/4354)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix np.matmul in dask.array.Array._\_array_ufunc_\_ ([dask#4363](https://github.com/dask/dask/pull/4363)) [Stephan Hoyer](https://github.com/shoyer)
- COMPAT: Re-enable multifield copy->view change ([dask#4357](https://github.com/dask/dask/pull/4357)) [Diane Trout](https://github.com/detrout)
- Calling np.dtype on a delayed object works ([dask#4387](https://github.com/dask/dask/pull/4387)) [Jim Crist](https://github.com/jcrist)
- Rework normalize_array for numpy data ([dask#4312](https://github.com/dask/dask/pull/4312)) [Marco Neumann](https://github.com/crepererum)

### DataFrame

- Add fill_value support for series comparisons ([dask#4250](https://github.com/dask/dask/pull/4250)) [James Bourbeau](https://github.com/jrbourbeau)
- Add schema name in read_sql_table for empty tables ([dask#4268](https://github.com/dask/dask/pull/4268)) [Mina Farid](https://github.com/minafarid)
- Adjust check for bad chunks in map_blocks ([dask#4308](https://github.com/dask/dask/pull/4308)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add dask.dataframe.read_fwf ([dask#4316](https://github.com/dask/dask/pull/4316)) [@slnguyen](https://github.com/slnguyen)
- Use atop fusion in dask dataframe ([dask#4229](https://github.com/dask/dask/pull/4229)) [Matthew Rocklin](https://github.com/mrocklin)
- Use parallel_types() in from_pandas ([dask#4331](https://github.com/dask/dask/pull/4331)) [Matthew Rocklin](https://github.com/mrocklin)
- Change DataFrame._repr_data to method ([dask#4330](https://github.com/dask/dask/pull/4330)) [Matthew Rocklin](https://github.com/mrocklin)
- Install pyarrow fastparquet for Appveyor ([dask#4338](https://github.com/dask/dask/pull/4338)) [Gábor Lipták](https://github.com/gliptak)
- Remove explicit pandas checks and provide cudf lazy registration ([dask#4359](https://github.com/dask/dask/pull/4359)) [Matthew Rocklin](https://github.com/mrocklin)
- Replace isinstance(…, pandas) with is_dataframe_like ([dask#4375](https://github.com/dask/dask/pull/4375)) [Matthew Rocklin](https://github.com/mrocklin)
- ENH: Support 3rd-party ExtensionArrays ([dask#4379](https://github.com/dask/dask/pull/4379)) [Tom Augspurger](https://github.com/tomaugspurger)
- Pandas 0.24.0 compat ([dask#4374](https://github.com/dask/dask/pull/4374)) [Tom Augspurger](https://github.com/tomaugspurger)

### Documentation

- Fix link to ‘map_blocks’ function in array api docs ([dask#4258](https://github.com/dask/dask/pull/4258)) [David Hoese](https://github.com/djhoese)
- Add a paragraph on Dask-Yarn in the cloud docs ([dask#4260](https://github.com/dask/dask/pull/4260)) [Jim Crist](https://github.com/jcrist)
- Copy edit documentation ([dask#4267](https://github.com/dask/dask/pull/4267)), ([dask#4263](https://github.com/dask/dask/pull/4263)), ([dask#4262](https://github.com/dask/dask/pull/4262)), ([dask#4277](https://github.com/dask/dask/pull/4277)), ([dask#4271](https://github.com/dask/dask/pull/4271)), ([dask#4279](https://github.com/dask/dask/pull/4279)), ([dask#4265](https://github.com/dask/dask/pull/4265)), ([dask#4295](https://github.com/dask/dask/pull/4295)), ([dask#4293](https://github.com/dask/dask/pull/4293)), ([dask#4296](https://github.com/dask/dask/pull/4296)), ([dask#4302](https://github.com/dask/dask/pull/4302)), ([dask#4306](https://github.com/dask/dask/pull/4306)), ([dask#4318](https://github.com/dask/dask/pull/4318)), ([dask#4314](https://github.com/dask/dask/pull/4314)), ([dask#4309](https://github.com/dask/dask/pull/4309)), ([dask#4317](https://github.com/dask/dask/pull/4317)), ([dask#4326](https://github.com/dask/dask/pull/4326)), ([dask#4325](https://github.com/dask/dask/pull/4325)), ([dask#4322](https://github.com/dask/dask/pull/4322)), ([dask#4332](https://github.com/dask/dask/pull/4332)), ([dask#4333](https://github.com/dask/dask/pull/4333)), [Miguel Farrajota](https://github.com/farrajota)
- Fix typo in code example ([dask#4272](https://github.com/dask/dask/pull/4272)) [Daniel Li](https://github.com/li-dan)
- Doc: Update array-api.rst ([dask#4259](https://github.com/dask/dask/pull/4259)) ([dask#4282](https://github.com/dask/dask/pull/4282)) [Prabakaran Kumaresshan](https://github.com/nixphix)
- Update hpc doc ([dask#4266](https://github.com/dask/dask/pull/4266)) [Guillaume Eynard-Bontemps](https://github.com/guillaumeeb)
- Doc: Replace from_avro with read_avro in documents ([dask#4313](https://github.com/dask/dask/pull/4313)) [Prabakaran Kumaresshan](https://github.com/nixphix)
- Remove reference to “get” scheduler functions in docs ([dask#4350](https://github.com/dask/dask/pull/4350)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix typo in docstring ([dask#4376](https://github.com/dask/dask/pull/4376)) [Daniel Saxton](https://github.com/dsaxton)
- Added documentation for dask.dataframe.merge ([dask#4382](https://github.com/dask/dask/pull/4382)) [Jendrik Jördening](https://github.com/jendrikjoe)

### Core

- Avoid recursion in dask.core.get ([dask#4219](https://github.com/dask/dask/pull/4219)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove verbose flag from pytest setup.cfg ([dask#4281](https://github.com/dask/dask/pull/4281)) [Matthew Rocklin](https://github.com/mrocklin)
- Support Pytest 4.0 by specifying marks explicitly ([dask#4280](https://github.com/dask/dask/pull/4280)) [Takahiro Kojima](https://github.com/515hikaru)
- Add High Level Graphs ([dask#4092](https://github.com/dask/dask/pull/4092)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix SerializableLock locked and acquire methods ([dask#4294](https://github.com/dask/dask/pull/4294)) [Stephan Hoyer](https://github.com/shoyer)
- Pin boto3 to earlier version in tests to avoid moto conflict ([dask#4276](https://github.com/dask/dask/pull/4276)) [Martin Durant](https://github.com/martindurant)
- Treat None as missing in config when updating ([dask#4324](https://github.com/dask/dask/pull/4324)) [Matthew Rocklin](https://github.com/mrocklin)
- Update Appveyor to Python 3.6 ([dask#4337](https://github.com/dask/dask/pull/4337)) [Gábor Lipták](https://github.com/gliptak)
- Use parse_bytes more liberally in dask.dataframe/bytes/bag ([dask#4339](https://github.com/dask/dask/pull/4339)) [Matthew Rocklin](https://github.com/mrocklin)
- Add a better error message when cloudpickle is missing ([dask#4342](https://github.com/dask/dask/pull/4342)) [Mark Harfouche](https://github.com/hmaarrfk)
- Support pool= keyword argument in threaded/multiprocessing get functions ([dask#4351](https://github.com/dask/dask/pull/4351)) [Matthew Rocklin](https://github.com/mrocklin)
- Allow updates from arbitrary Mappings in config.update, not only dicts. ([dask#4356](https://github.com/dask/dask/pull/4356)) [Stuart Berg](https://github.com/stuarteberg)
- Move dask/array/top.py code to dask/blockwise.py ([dask#4348](https://github.com/dask/dask/pull/4348)) [Matthew Rocklin](https://github.com/mrocklin)
- Add has_parallel_type ([dask#4395](https://github.com/dask/dask/pull/4395)) [Matthew Rocklin](https://github.com/mrocklin)
- CI: Update Appveyor ([dask#4381](https://github.com/dask/dask/pull/4381)) [Tom Augspurger](https://github.com/tomaugspurger)
- Ignore non-readable config files ([dask#4388](https://github.com/dask/dask/pull/4388)) [Jim Crist](https://github.com/jcrist)

<a id="v1-0-0-2018-11-28"></a>

## 1.0.0 / 2018-11-28

### Array

- Add nancumsum/nancumprod unit tests ([dask#4215](https://github.com/dask/dask/pull/4215)) [crusaderky](https://github.com/crusaderky)

### DataFrame

- Add index to to_dask_dataframe docstring ([dask#4232](https://github.com/dask/dask/pull/4232)) [James Bourbeau](https://github.com/jrbourbeau)
- Text and fix when appending categoricals with fastparquet ([dask#4245](https://github.com/dask/dask/pull/4245)) [Martin Durant](https://github.com/martindurant)
- Don’t reread metadata when passing ParquetFile to read_parquet ([dask#4247](https://github.com/dask/dask/pull/4247)) [Martin Durant](https://github.com/martindurant)

### Documentation

- Copy edit documentation ([dask#4222](https://github.com/dask/dask/pull/4222)) ([dask#4224](https://github.com/dask/dask/pull/4224)) ([dask#4228](https://github.com/dask/dask/pull/4228)) ([dask#4231](https://github.com/dask/dask/pull/4231)) ([dask#4230](https://github.com/dask/dask/pull/4230)) ([dask#4234](https://github.com/dask/dask/pull/4234)) ([dask#4235](https://github.com/dask/dask/pull/4235)) ([dask#4254](https://github.com/dask/dask/pull/4254)) [Miguel Farrajota](https://github.com/farrajota)
- Updated doc for the new scheduler keyword ([dask#4251](https://github.com/dask/dask/pull/4251)) [@milesial](https://github.com/milesial)

### Core

- Avoid a few warnings ([dask#4223](https://github.com/dask/dask/pull/4223)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove dask.store module ([dask#4221](https://github.com/dask/dask/pull/4221)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove AUTHORS.md [Jim Crist](https://github.com/jcrist)

<a id="v0-20-2-2018-11-15"></a>

## 0.20.2 / 2018-11-15

### Array

- Avoid fusing dependencies of atop reductions ([dask#4207](https://github.com/dask/dask/pull/4207)) [Matthew Rocklin](https://github.com/mrocklin)

### Dataframe

- Improve memory footprint for dataframe correlation ([dask#4193](https://github.com/dask/dask/pull/4193)) [Damien Garaud](https://github.com/geraud)
- Add empty DataFrame check to boundary_slice ([dask#4212](https://github.com/dask/dask/pull/4212)) [James Bourbeau](https://github.com/jrbourbeau)

### Documentation

- Copy edit documentation ([dask#4197](https://github.com/dask/dask/pull/4197)) ([dask#4204](https://github.com/dask/dask/pull/4204)) ([dask#4198](https://github.com/dask/dask/pull/4198)) ([dask#4199](https://github.com/dask/dask/pull/4199)) ([dask#4200](https://github.com/dask/dask/pull/4200)) ([dask#4202](https://github.com/dask/dask/pull/4202)) ([dask#4209](https://github.com/dask/dask/pull/4209)) [Miguel Farrajota](https://github.com/farrajota)
- Add stats module namespace ([dask#4206](https://github.com/dask/dask/pull/4206)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix link in dataframe documentation ([dask#4208](https://github.com/dask/dask/pull/4208)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v0-20-1-2018-11-09"></a>

## 0.20.1 / 2018-11-09

### Array

- Only allocate the result space in wrapped_pad_func ([dask#4153](https://github.com/dask/dask/pull/4153)) [John A Kirkham](https://github.com/jakirkham)
- Generalize expand_pad_width to expand_pad_value ([dask#4150](https://github.com/dask/dask/pull/4150)) [John A Kirkham](https://github.com/jakirkham)
- Test da.pad with 2D linear_ramp case ([dask#4162](https://github.com/dask/dask/pull/4162)) [John A Kirkham](https://github.com/jakirkham)
- Fix import for broadcast_to. ([dask#4168](https://github.com/dask/dask/pull/4168)) [samc0de](https://github.com/samc0de)
- Rewrite Dask Array’s pad to add only new chunks ([dask#4152](https://github.com/dask/dask/pull/4152)) [John A Kirkham](https://github.com/jakirkham)
- Validate index inputs to atop ([dask#4182](https://github.com/dask/dask/pull/4182)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Dask.config set and get normalize underscores and hyphens ([dask#4143](https://github.com/dask/dask/pull/4143)) [James Bourbeau](https://github.com/jrbourbeau)
- Only subs on core collections, not subclasses ([dask#4159](https://github.com/dask/dask/pull/4159)) [Matthew Rocklin](https://github.com/mrocklin)
- Add block_size=0 option to HTTPFileSystem. ([dask#4171](https://github.com/dask/dask/pull/4171)) [Martin Durant](https://github.com/martindurant)
- Add traverse support for dataclasses ([dask#4165](https://github.com/dask/dask/pull/4165)) [Armin Berres](https://github.com/aberres)
- Avoid optimization on sharedicts without dependencies ([dask#4181](https://github.com/dask/dask/pull/4181)) [Matthew Rocklin](https://github.com/mrocklin)
- Update the pytest version for TravisCI ([dask#4189](https://github.com/dask/dask/pull/4189)) [Damien Garaud](https://github.com/geraud)
- Use key_split rather than funcname in visualize names ([dask#4160](https://github.com/dask/dask/pull/4160)) [Matthew Rocklin](https://github.com/mrocklin)

### Dataframe

- Add fix for  DataFrame._\_setitem_\_ for index ([dask#4151](https://github.com/dask/dask/pull/4151)) [Anderson Banihirwe](https://github.com/andersy005)
- Fix column choice when passing list of files to fastparquet ([dask#4174](https://github.com/dask/dask/pull/4174)) [Martin Durant](https://github.com/martindurant)
- Pass engine_kwargs from read_sql_table to sqlalchemy ([dask#4187](https://github.com/dask/dask/pull/4187)) [Damien Garaud](https://github.com/geraud)

### Documentation

- Fix documentation in Delayed best practices example that returned an empty list ([dask#4147](https://github.com/dask/dask/pull/4147)) [Jonathan Fraine](https://github.com/exowanderer)
- Copy edit documentation ([dask#4164](https://github.com/dask/dask/pull/4164)) ([dask#4175](https://github.com/dask/dask/pull/4175)) ([dask#4185](https://github.com/dask/dask/pull/4185)) ([dask#4192](https://github.com/dask/dask/pull/4192)) ([dask#4191](https://github.com/dask/dask/pull/4191)) ([dask#4190](https://github.com/dask/dask/pull/4190)) ([dask#4180](https://github.com/dask/dask/pull/4180)) [Miguel Farrajota](https://github.com/farrajota)
- Fix typo in docstring ([dask#4183](https://github.com/dask/dask/pull/4183)) [Carlos Valiente](https://github.com/carletes)

<a id="v0-20-0-2018-10-26"></a>

## 0.20.0 / 2018-10-26

### Array

- Fuse Atop operations ([dask#3998](https://github.com/dask/dask/pull/3998)), ([dask#4081](https://github.com/dask/dask/pull/4081)) [Matthew Rocklin](https://github.com/mrocklin)
- Support da.asanyarray on dask dataframes ([dask#4080](https://github.com/dask/dask/pull/4080)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove unnecessary endianness check in datetime test ([dask#4113](https://github.com/dask/dask/pull/4113)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Set name=False in array foo_like functions ([dask#4116](https://github.com/dask/dask/pull/4116)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove dask.array.ghost module ([dask#4121](https://github.com/dask/dask/pull/4121)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix use of getargspec in dask array ([dask#4125](https://github.com/dask/dask/pull/4125)) [Stephan Hoyer](https://github.com/shoyer)
- Adds dask.array.invert ([dask#4127](https://github.com/dask/dask/pull/4127)), ([dask#4131](https://github.com/dask/dask/pull/4131)) [Anderson Banihirwe](https://github.com/andersy005)
- Raise informative error on arg-reduction on unknown chunksize ([dask#4128](https://github.com/dask/dask/pull/4128)), ([dask#4135](https://github.com/dask/dask/pull/4135)) [Matthew Rocklin](https://github.com/mrocklin)
- Normalize reversed slices in dask array ([dask#4126](https://github.com/dask/dask/pull/4126)) [Matthew Rocklin](https://github.com/mrocklin)

### Bag

- Add bag.to_avro ([dask#4076](https://github.com/dask/dask/pull/4076)) [Martin Durant](https://github.com/martindurant)

### Core

- Pull num_workers from config.get ([dask#4086](https://github.com/dask/dask/pull/4086)), ([dask#4093](https://github.com/dask/dask/pull/4093)) [James Bourbeau](https://github.com/jrbourbeau)
- Fix invalid escape sequences with raw strings ([dask#4112](https://github.com/dask/dask/pull/4112)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Raise an error on the use of the get= keyword and set_options ([dask#4077](https://github.com/dask/dask/pull/4077)) [Matthew Rocklin](https://github.com/mrocklin)
- Add import for Azure DataLake storage, and add docs ([dask#4132](https://github.com/dask/dask/pull/4132)) [Martin Durant](https://github.com/martindurant)
- Avoid collections.Mapping/Sequence ([dask#4138](https://github.com/dask/dask/pull/4138))  [Matthew Rocklin](https://github.com/mrocklin)

### Dataframe

- Include index keyword in to_dask_dataframe ([dask#4071](https://github.com/dask/dask/pull/4071)) [Matthew Rocklin](https://github.com/mrocklin)
- add support for duplicate column names ([dask#4087](https://github.com/dask/dask/pull/4087)) [Jan Koch](https://github.com/datajanko)
- Implement min_count for the DataFrame methods sum and prod ([dask#4090](https://github.com/dask/dask/pull/4090)) [Bart Broere](https://github.com/bartbroere)
- Remove pandas warnings in concat ([dask#4095](https://github.com/dask/dask/pull/4095)) [Matthew Rocklin](https://github.com/mrocklin)
- DataFrame.to_csv header option to only output headers in the first chunk ([dask#3909](https://github.com/dask/dask/pull/3909)) [Rahul Vaidya](https://github.com/rvaidya)
- Remove Series.to_parquet ([dask#4104](https://github.com/dask/dask/pull/4104)) [Justin Dennison](https://github.com/justin1dennison)
- Avoid warnings and deprecated pandas methods ([dask#4115](https://github.com/dask/dask/pull/4115)) [Matthew Rocklin](https://github.com/mrocklin)
- Swap ‘old’ and ‘previous’ when reporting append error ([dask#4130](https://github.com/dask/dask/pull/4130)) [Martin Durant](https://github.com/martindurant)

### Documentation

- Copy edit documentation ([dask#4073](https://github.com/dask/dask/pull/4073)), ([dask#4074](https://github.com/dask/dask/pull/4074)), ([dask#4094](https://github.com/dask/dask/pull/4094)), ([dask#4097](https://github.com/dask/dask/pull/4097)), ([dask#4107](https://github.com/dask/dask/pull/4107)), ([dask#4124](https://github.com/dask/dask/pull/4124)), ([dask#4133](https://github.com/dask/dask/pull/4133)), ([dask#4139](https://github.com/dask/dask/pull/4139)) [Miguel Farrajota](https://github.com/farrajota)
- Fix typo in code example ([dask#4089](https://github.com/dask/dask/pull/4089)) [Antonino Ingargiola](https://github.com/tritemio)
- Add pycon 2018 presentation ([dask#4102](https://github.com/dask/dask/pull/4102)) [Javad](https://github.com/javad94)
- Quick description for gcsfs ([dask#4109](https://github.com/dask/dask/pull/4109)) [Martin Durant](https://github.com/martindurant)
- Fixed typo in docstrings of read_sql_table method ([dask#4114](https://github.com/dask/dask/pull/4114)) [TakaakiFuruse](https://github.com/TakaakiFuruse)
- Make target directories in redirects if they don’t exist ([dask#4136](https://github.com/dask/dask/pull/4136)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-19-4-2018-10-09"></a>

## 0.19.4 / 2018-10-09

### Array

- Implement `apply_gufunc(..., axes=..., keepdims=...)` ([dask#3985](https://github.com/dask/dask/pull/3985)) [Markus Gonser](https://github.com/magonser)

### Bag

- Fix typo in datasets.make_people ([dask#4069](https://github.com/dask/dask/pull/4069)) [Matthew Rocklin](https://github.com/mrocklin)

### Dataframe

- Added percentiles options for dask.dataframe.describe method ([dask#4067](https://github.com/dask/dask/pull/4067)) [Zhenqing Li](https://github.com/DigitalPig)
- Add DataFrame.partitions accessor similar to Array.blocks ([dask#4066](https://github.com/dask/dask/pull/4066)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Pass get functions and Clients through scheduler keyword ([dask#4062](https://github.com/dask/dask/pull/4062)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Fix Typo on hpc example. (missing = in kwarg). ([dask#4068](https://github.com/dask/dask/pull/4068)) [Matthias Bussonier](https://github.com/Carreau)
- Extensive copy-editing: ([dask#4065](https://github.com/dask/dask/pull/4065)), ([dask#4064](https://github.com/dask/dask/pull/4064)), ([dask#4063](https://github.com/dask/dask/pull/4063)) [Miguel Farrajota](https://github.com/farrajota)

<a id="v0-19-3-2018-10-05"></a>

## 0.19.3 / 2018-10-05

### Array

- Make da.RandomState extensible to other modules ([dask#4041](https://github.com/dask/dask/pull/4041)) [Matthew Rocklin](https://github.com/mrocklin)
- Support unknown dims in ravel no-op case ([dask#4055](https://github.com/dask/dask/pull/4055)) [Jim Crist](https://github.com/jcrist)
- Add basic infrastructure for cupy ([dask#4019](https://github.com/dask/dask/pull/4019)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid asarray and lock arguments for from_array(getitem) ([dask#4044](https://github.com/dask/dask/pull/4044)) [Matthew Rocklin](https://github.com/mrocklin)
- Move local imports in corrcoef to global imports ([dask#4030](https://github.com/dask/dask/pull/4030)) [John A Kirkham](https://github.com/jakirkham)
- Move local indices import to global import ([dask#4029](https://github.com/dask/dask/pull/4029)) [John A Kirkham](https://github.com/jakirkham)
- Fix-up Dask Array’s fromfunction w.r.t. dtype and kwargs ([dask#4028](https://github.com/dask/dask/pull/4028)) [John A Kirkham](https://github.com/jakirkham)
- Don’t use dummy expansion for trim_internal in overlapped ([dask#3964](https://github.com/dask/dask/pull/3964)) [Mark Harfouche](https://github.com/hmaarrfk)
- Add unravel_index ([dask#3958](https://github.com/dask/dask/pull/3958)) [John A Kirkham](https://github.com/jakirkham)

### Bag

- Sort result in Bag.frequencies ([dask#4033](https://github.com/dask/dask/pull/4033)) [Matthew Rocklin](https://github.com/mrocklin)
- Add support for npartitions=1 edge case in groupby ([dask#4050](https://github.com/dask/dask/pull/4050)) [James Bourbeau](https://github.com/jrbourbeau)
- Add new random dataset for people ([dask#4018](https://github.com/dask/dask/pull/4018)) [Matthew Rocklin](https://github.com/mrocklin)
- Improve performance of bag.read_text on small files ([dask#4013](https://github.com/dask/dask/pull/4013)) [Eric Wolak](https://github.com/epall)
- Add bag.read_avro ([dask#4000](https://github.com/dask/dask/pull/4000)) ([dask#4007](https://github.com/dask/dask/pull/4007)) [Martin Durant](https://github.com/martindurant)

### Dataframe

- Added an `index` parameter to [`dask.dataframe.from_dask_array()`](generated/dask.dataframe.from_dask_array.md#dask.dataframe.from_dask_array) for creating a dask DataFrame from a dask Array with a given index. ([dask#3991](https://github.com/dask/dask/pull/3991)) [Tom Augspurger](https://github.com/tomaugspurger)
- Improve sub-classability of dask dataframe ([dask#4015](https://github.com/dask/dask/pull/4015)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix failing hdfs test [test-hdfs] ([dask#4046](https://github.com/dask/dask/pull/4046)) [Jim Crist](https://github.com/jcrist)
- fuse_subgraphs works without normal fuse ([dask#4042](https://github.com/dask/dask/pull/4042)) [Jim Crist](https://github.com/jcrist)
- Make path for reading many parquet files without prescan ([dask#3978](https://github.com/dask/dask/pull/3978)) [Martin Durant](https://github.com/martindurant)
- Index in dd.from_dask_array ([dask#3991](https://github.com/dask/dask/pull/3991)) [Tom Augspurger](https://github.com/tomaugspurger)
- Making skiprows accept lists ([dask#3975](https://github.com/dask/dask/pull/3975)) [Julia Signell](https://github.com/jsignell)
- Fail early in fastparquet read for nonexistent column ([dask#3989](https://github.com/dask/dask/pull/3989)) [Martin Durant](https://github.com/martindurant)

### Core

- Add support for npartitions=1 edge case in groupby ([dask#4050](https://github.com/dask/dask/pull/4050)) [James Bourbeau](https://github.com/jrbourbeau)
- Automatically wrap large arguments with dask.delayed in map_blocks/partitions ([dask#4002](https://github.com/dask/dask/pull/4002)) [Matthew Rocklin](https://github.com/mrocklin)
- Fuse linear chains of subgraphs ([dask#3979](https://github.com/dask/dask/pull/3979)) [Jim Crist](https://github.com/jcrist)
- Make multiprocessing context configurable ([dask#3763](https://github.com/dask/dask/pull/3763)) [Itamar Turner-Trauring](https://github.com/itamarst)

### Documentation

- Extensive copy-editing  ([dask#4049](https://github.com/dask/dask/pull/4049)), ([dask#4034](https://github.com/dask/dask/pull/4034)),  ([dask#4031](https://github.com/dask/dask/pull/4031)), ([dask#4020](https://github.com/dask/dask/pull/4020)), ([dask#4021](https://github.com/dask/dask/pull/4021)), ([dask#4022](https://github.com/dask/dask/pull/4022)), ([dask#4023](https://github.com/dask/dask/pull/4023)), ([dask#4016](https://github.com/dask/dask/pull/4016)), ([dask#4017](https://github.com/dask/dask/pull/4017)), ([dask#4010](https://github.com/dask/dask/pull/4010)), ([dask#3997](https://github.com/dask/dask/pull/3997)), ([dask#3996](https://github.com/dask/dask/pull/3996)), [Miguel Farrajota](https://github.com/farrajota)
- Update shuffle method selection docs ([dask#4048](https://github.com/dask/dask/pull/4048)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove docs/source/examples, point to examples.dask.org ([dask#4014](https://github.com/dask/dask/pull/4014)) [Matthew Rocklin](https://github.com/mrocklin)
- Replace readthedocs links with dask.org ([dask#4008](https://github.com/dask/dask/pull/4008)) [Matthew Rocklin](https://github.com/mrocklin)
- Updates DataFrame.to_hdf docstring for returned values ([dask#3992](https://github.com/dask/dask/pull/3992)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v0-19-2-2018-09-17"></a>

## 0.19.2 / 2018-09-17

### Array

- `apply_gufunc` implements automatic infer of functions output dtypes ([dask#3936](https://github.com/dask/dask/pull/3936)) [Markus Gonser](https://github.com/magonser)
- Fix array histogram range error when array has nans ([dask#3980](https://github.com/dask/dask/pull/3980)) [James Bourbeau](https://github.com/jrbourbeau)
- Issue 3937 follow up, int type checks. ([dask#3956](https://github.com/dask/dask/pull/3956)) [Yu Feng](https://github.com/rainwoodman)
- from_array: add @martindurant’s explaining of how hashing is done for an array. ([dask#3965](https://github.com/dask/dask/pull/3965)) [Mark Harfouche](https://github.com/hmaarrfk)
- Support gradient with coordinate ([dask#3949](https://github.com/dask/dask/pull/3949)) [Keisuke Fujii](https://github.com/fujiisoup)

### Core

- Fix use of has_keyword with partial in Python 2.7 ([dask#3966](https://github.com/dask/dask/pull/3966)) [Mark Harfouche](https://github.com/hmaarrfk)
- Set pyarrow as default for HDFS ([dask#3957](https://github.com/dask/dask/pull/3957)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- Use dask_sphinx_theme ([dask#3963](https://github.com/dask/dask/pull/3963)) [Matthew Rocklin](https://github.com/mrocklin)
- Use JupyterLab in Binder links from main page [Matthew Rocklin](https://github.com/mrocklin)
- DOC: fixed sphinx syntax ([dask#3960](https://github.com/dask/dask/pull/3960)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v0-19-1-2018-09-06"></a>

## 0.19.1 / 2018-09-06

### Array

- Don’t enforce dtype if result has no dtype ([dask#3928](https://github.com/dask/dask/pull/3928)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix NumPy issubtype deprecation warning ([dask#3939](https://github.com/dask/dask/pull/3939)) [Bruce Merry](https://github.com/bmerry)
- Fix arg reduction tokens to be unique with different arguments ([dask#3955](https://github.com/dask/dask/pull/3955)) [Tobias de Jong](https://github.com/tadejong)
- Coerce numpy integers to ints in slicing code ([dask#3944](https://github.com/dask/dask/pull/3944)) [Yu Feng](https://github.com/rainwoodman)
- Linalg.norm ndim along axis partial fix ([dask#3933](https://github.com/dask/dask/pull/3933)) [Tobias de Jong](https://github.com/tadejong)

### Dataframe

- Deterministic DataFrame.set_index ([dask#3867](https://github.com/dask/dask/pull/3867)) [George Sakkis](https://github.com/gsakkis)
- Fix divisions in read_parquet when dealing with filters #3831 #3930 ([dask#3923](https://github.com/dask/dask/pull/3923)) ([dask#3931](https://github.com/dask/dask/pull/3931))  [@andrethrill](https://github.com/andrethrill)
- Fixing returning type in categorical.as_known  ([dask#3888](https://github.com/dask/dask/pull/3888)) [Sriharsha Hatwar](https://github.com/Sriharsha-hatwar)
- Fix DataFrame.assign for callables ([dask#3919](https://github.com/dask/dask/pull/3919)) [Tom Augspurger](https://github.com/tomaugspurger)
- Include partitions with no width in repartition ([dask#3941](https://github.com/dask/dask/pull/3941)) [Matthew Rocklin](https://github.com/mrocklin)
- Don’t constrict stage/k dtype in dataframe shuffle ([dask#3942](https://github.com/dask/dask/pull/3942)) [Matthew Rocklin](https://github.com/mrocklin)

### Documentation

- DOC: Add hint on how to render task graphs horizontally ([dask#3922](https://github.com/dask/dask/pull/3922)) [Uwe Korn](https://github.com/xhochy)
- Add try-now button to main landing page ([dask#3924](https://github.com/dask/dask/pull/3924)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-19-0-2018-08-29"></a>

## 0.19.0 / 2018-08-29

### Array

- Support coordinate in gradient ([dask#3949](https://github.com/dask/dask/pull/3949)) [Keisuke Fujii](https://github.com/fujiisoup)
- Fix argtopk split_every bug ([dask#3810](https://github.com/dask/dask/pull/3810)) [crusaderky](https://github.com/crusaderky)
- Ensure result computing dask.array.isnull() always gives a numpy array ([dask#3825](https://github.com/dask/dask/pull/3825)) [Stephan Hoyer](https://github.com/shoyer)
- Support concatenate for scipy.sparse in dask array ([dask#3836](https://github.com/dask/dask/pull/3836)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix argtopk on 32-bit systems. ([dask#3823](https://github.com/dask/dask/pull/3823)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Normalize keys in rechunk ([dask#3820](https://github.com/dask/dask/pull/3820)) [Matthew Rocklin](https://github.com/mrocklin)
- Allow shape of dask.array to be a numpy array ([dask#3844](https://github.com/dask/dask/pull/3844)) [Mark Harfouche](https://github.com/hmaarrfk)
- Fix numpy deprecation warning on tuple indexing ([dask#3851](https://github.com/dask/dask/pull/3851)) [Tobias de Jong](https://github.com/tadejong)
- Rename ghost module to overlap ([dask#3830](https://github.com/dask/dask/pull/3830)) [Robert Sare](https://github.com/rmsare)
- Re-add the ghost import to da \_\_init_\_ ([dask#3861](https://github.com/dask/dask/pull/3861)) [Jim Crist](https://github.com/jcrist)
- Ensure copy preserves masked arrays ([dask#3852](https://github.com/dask/dask/pull/3852)) [Tobias de Jong](https://github.com/tadejong)

### DataFrame

- Added `dtype` and `sparse` keywords to [`dask.dataframe.get_dummies()`](generated/dask.dataframe.get_dummies.md#dask.dataframe.get_dummies) ([dask#3792](https://github.com/dask/dask/pull/3792)) [Tom Augspurger](https://github.com/tomaugspurger)
- Added `dask.dataframe.to_dask_array()` for converting a Dask Series or DataFrame to a
  Dask Array, possibly with known chunk sizes ([dask#3884](https://github.com/dask/dask/pull/3884)) Tom Augspurger
- Changed the behavior for [`dask.array.asarray()`](generated/dask.array.asarray.md#dask.array.asarray) for dask dataframe and series inputs. Previously,
  the series was eagerly converted to an in-memory NumPy array before creating a dask array with known
  chunks sizes. This caused unexpectedly high memory usage. Now, no intermediate NumPy array is created,
  and a Dask array with unknown chunk sizes is returned ([dask#3884](https://github.com/dask/dask/pull/3884)) Tom Augspurger
- DataFrame.iloc ([dask#3805](https://github.com/dask/dask/pull/3805)) [Tom Augspurger](https://github.com/tomaugspurger)
- When reading multiple paths, expand globs. ([dask#3828](https://github.com/dask/dask/pull/3828)) [Irina Truong](https://github.com/j-bennet)
- Added index column name after resample ([dask#3833](https://github.com/dask/dask/pull/3833)) [Eric Bonfadini](https://github.com/eric-bonfadini)
- Add (lazy) shape property to dataframe and series ([dask#3212](https://github.com/dask/dask/pull/3212)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Fix failing hdfs test [test-hdfs] ([dask#3858](https://github.com/dask/dask/pull/3858)) [Jim Crist](https://github.com/jcrist)
- Fixes for pyarrow 0.10.0 release ([dask#3860](https://github.com/dask/dask/pull/3860)) [Jim Crist](https://github.com/jcrist)
- Rename to_csv keys for diagnostics ([dask#3890](https://github.com/dask/dask/pull/3890)) [Matthew Rocklin](https://github.com/mrocklin)
- Match pandas warnings for concat sort ([dask#3897](https://github.com/dask/dask/pull/3897)) [Tom Augspurger](https://github.com/tomaugspurger)
- Include filename in read_csv ([dask#3908](https://github.com/dask/dask/pull/3908)) [Julia Signell](https://github.com/jsignell)

### Core

- Better error message on import when missing common dependencies ([dask#3771](https://github.com/dask/dask/pull/3771)) [Danilo Horta](https://github.com/horta)
- Drop Python 3.4 support ([dask#3840](https://github.com/dask/dask/pull/3840)) [Jim Crist](https://github.com/jcrist)
- Remove expired deprecation warnings ([dask#3841](https://github.com/dask/dask/pull/3841)) [Jim Crist](https://github.com/jcrist)
- Add DASK_ROOT_CONFIG environment variable ([dask#3849](https://github.com/dask/dask/pull/3849)) [Joe Hamman](https://github.com/jhamman)
- Don’t cull in local scheduler, do cull in delayed ([dask#3856](https://github.com/dask/dask/pull/3856)) [Jim Crist](https://github.com/jcrist)
- Increase conda download retries ([dask#3857](https://github.com/dask/dask/pull/3857)) [Jim Crist](https://github.com/jcrist)
- Add python_requires and Trove classifiers ([dask#3855](https://github.com/dask/dask/pull/3855)) [@hugovk](https://github.com/hugovk)
- Fix collections.abc deprecation warnings in Python 3.7.0 ([dask#3876](https://github.com/dask/dask/pull/3876)) [Jan Margeta](https://github.com/jmargeta)
- Allow dot jpeg to xfail in visualize tests ([dask#3896](https://github.com/dask/dask/pull/3896)) [Matthew Rocklin](https://github.com/mrocklin)
- Add Python 3.7 to travis.yml ([dask#3894](https://github.com/dask/dask/pull/3894)) [Matthew Rocklin](https://github.com/mrocklin)
- Add expand_environment_variables to dask.config ([dask#3893](https://github.com/dask/dask/pull/3893)) [Joe Hamman](https://github.com/jhamman)

### Docs

- Fix typo in import statement of diagnostics ([dask#3826](https://github.com/dask/dask/pull/3826)) [John Mrziglod](https://github.com/JohnMrziglod)
- Add link to YARN docs ([dask#3838](https://github.com/dask/dask/pull/3838)) [Jim Crist](https://github.com/jcrist)
- fix of minor typos in landing page index.html ([dask#3746](https://github.com/dask/dask/pull/3746)) [Christoph Moehl](https://github.com/cmohl2013)
- Update delayed-custom.rst ([dask#3850](https://github.com/dask/dask/pull/3850)) [Anderson Banihirwe](https://github.com/andersy005)
- DOC: clarify delayed docstring ([dask#3709](https://github.com/dask/dask/pull/3709)) [Scott Sievert](https://github.com/stsievert)
- Add new presentations ([dask#3880](https://github.com/dask/dask/pull/3880)) [Javad](https://github.com/javad94)
- Add dask array normalize_chunks to documentation ([dask#3878](https://github.com/dask/dask/pull/3878)) [Daniel Rothenberg](https://github.com/darothen)
- Docs: Fix link to snakeviz ([dask#3900](https://github.com/dask/dask/pull/3900)) [Hans Moritz Günther](https://github.com/hamogu)
- Add missing \` to docstring ([dask#3915](https://github.com/dask/dask/pull/3915)) [@rtobar](https://github.com/rtobar)

<a id="v0-18-2-2018-07-23"></a>

## 0.18.2 / 2018-07-23

### Array

- Reimplemented `argtopk` to make it release the GIL ([dask#3610](https://github.com/dask/dask/pull/3610)) [crusaderky](https://github.com/crusaderky)
- Don’t overlap on non-overlapped dimensions in `map_overlap` ([dask#3653](https://github.com/dask/dask/pull/3653)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix `linalg.tsqr` for dimensions of uncertain length ([dask#3662](https://github.com/dask/dask/pull/3662)) [Jeremy Chen](https://github.com/convexset)
- Break apart uneven array-of-int slicing to separate chunks ([dask#3648](https://github.com/dask/dask/pull/3648)) [Matthew Rocklin](https://github.com/mrocklin)
- Align auto chunks to provided chunks, rather than shape ([dask#3679](https://github.com/dask/dask/pull/3679)) [Matthew Rocklin](https://github.com/mrocklin)
- Adds endpoint and retstep support for linspace ([dask#3675](https://github.com/dask/dask/pull/3675)) [James Bourbeau](https://github.com/jrbourbeau)
- Implement `.blocks` accessor ([dask#3689](https://github.com/dask/dask/pull/3689)) [Matthew Rocklin](https://github.com/mrocklin)
- Add `block_info` keyword to `map_blocks` functions ([dask#3686](https://github.com/dask/dask/pull/3686)) [Matthew Rocklin](https://github.com/mrocklin)
- Slice by dask array of ints ([dask#3407](https://github.com/dask/dask/pull/3407)) [crusaderky](https://github.com/crusaderky)
- Support `dtype` in `arange` ([dask#3722](https://github.com/dask/dask/pull/3722)) [crusaderky](https://github.com/crusaderky)
- Fix `argtopk` with uneven chunks ([dask#3720](https://github.com/dask/dask/pull/3720)) [crusaderky](https://github.com/crusaderky)
- Raise error when `replace=False` in `da.choice` ([dask#3765](https://github.com/dask/dask/pull/3765)) [James Bourbeau](https://github.com/jrbourbeau)
- Update chunks in `Array.__setitem__` ([dask#3767](https://github.com/dask/dask/pull/3767)) [Itamar Turner-Trauring](https://github.com/itamarst)
- Add a `chunksize` convenience property ([dask#3777](https://github.com/dask/dask/pull/3777)) [Jacob Tomlinson](https://github.com/jacobtomlinson)
- Fix and simplify array slicing behavior when `step < 0` ([dask#3702](https://github.com/dask/dask/pull/3702)) [Ziyao Wei](https://github.com/ZiyaoWei)
- Ensure `to_zarr` with `return_stored` `True` returns a Dask Array ([dask#3786](https://github.com/dask/dask/pull/3786)) [John A Kirkham](https://github.com/jakirkham)

### Bag

- Add `last_endline` optional parameter in `to_textfiles` ([dask#3745](https://github.com/dask/dask/pull/3745)) [George Sakkis](https://github.com/gsakkis)

### Dataframe

- Add aggregate function for rolling objects ([dask#3772](https://github.com/dask/dask/pull/3772)) [Gerome Pistre](https://github.com/GPistre)
- Properly tokenize cumulative groupby aggregations ([dask#3799](https://github.com/dask/dask/pull/3799)) [Cloves Almeida](https://github.com/cjalmeida)

### Delayed

- Add the `@` operator to the delayed objects ([dask#3691](https://github.com/dask/dask/pull/3691)) [Mark Harfouche](https://github.com/hmaarrfk)
- Add delayed best practices to documentation ([dask#3737](https://github.com/dask/dask/pull/3737)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix `@delayed` decorator for methods and add tests ([dask#3757](https://github.com/dask/dask/pull/3757)) [Ziyao Wei](https://github.com/ZiyaoWei)

### Core

- Fix extra progressbar ([dask#3669](https://github.com/dask/dask/pull/3669)) [Mike Neish](https://github.com/neishm)
- Allow tasks back onto ordering stack if they have one dependency ([dask#3652](https://github.com/dask/dask/pull/3652)) [Matthew Rocklin](https://github.com/mrocklin)
- Prefer end-tasks with low numbers of dependencies when ordering ([dask#3588](https://github.com/dask/dask/pull/3588)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add `assert_eq` to top-level modules ([dask#3726](https://github.com/dask/dask/pull/3726)) [Matthew Rocklin](https://github.com/mrocklin)
- Test that dask collections can hold `scipy.sparse` arrays ([dask#3738](https://github.com/dask/dask/pull/3738)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix setup of lz4 decompression functions ([dask#3782](https://github.com/dask/dask/pull/3782)) [Elliott Sales de Andrade](https://github.com/QuLogic)
- Add datasets module ([dask#3780](https://github.com/dask/dask/pull/3780)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-18-1-2018-06-22"></a>

## 0.18.1 / 2018-06-22

### Array

- `from_array` now supports scalar types and nested lists/tuples in input,
  just like all numpy functions do; it also produces a simpler graph when the
  input is a plain ndarray ([dask#3568](https://github.com/dask/dask/pull/3568)) [crusaderky](https://github.com/crusaderky)
- Fix slicing of big arrays due to cumsum dtype bug ([dask#3620](https://github.com/dask/dask/pull/3620)) [Marco Rossi](https://github.com/m-rossi)
- Add Dask Array implementation of pad ([dask#3578](https://github.com/dask/dask/pull/3578)) [John A Kirkham](https://github.com/jakirkham)
- Fix array random API examples ([dask#3625](https://github.com/dask/dask/pull/3625)) [James Bourbeau](https://github.com/jrbourbeau)
- Add average function to dask array ([dask#3640](https://github.com/dask/dask/pull/3640)) [James Bourbeau](https://github.com/jrbourbeau)
- Tokenize ghost_internal with axes ([dask#3643](https://github.com/dask/dask/pull/3643))  [Matthew Rocklin](https://github.com/mrocklin)
- Add outer for Dask Arrays ([dask#3658](https://github.com/dask/dask/pull/3658)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Add Index.to_series method ([dask#3613](https://github.com/dask/dask/pull/3613)) [Henrique Ribeiro](https://github.com/henriqueribeiro)
- Fix missing partition columns in pyarrow-parquet ([dask#3636](https://github.com/dask/dask/pull/3636)) [Martin Durant](https://github.com/martindurant)

### Core

- Minor tweaks to CI ([dask#3629](https://github.com/dask/dask/pull/3629)) [crusaderky](https://github.com/crusaderky)
- Add back dask.utils.effective_get ([dask#3642](https://github.com/dask/dask/pull/3642)) [Matthew Rocklin](https://github.com/mrocklin)
- DASK_CONFIG dictates config write location ([dask#3621](https://github.com/dask/dask/pull/3621)) [Jim Crist](https://github.com/jcrist)
- Replace ‘collections’ key in unpack_collections with unique key ([dask#3632](https://github.com/dask/dask/pull/3632)) [Yu Feng](https://github.com/rainwoodman)
- Avoid deepcopy in dask.config.set ([dask#3649](https://github.com/dask/dask/pull/3649)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-18-0-2018-06-14"></a>

## 0.18.0 / 2018-06-14

### Array

- Add to/from_zarr for Zarr-format datasets and arrays ([dask#3460](https://github.com/dask/dask/pull/3460)) [Martin Durant](https://github.com/martindurant)
- Experimental addition of generalized ufunc support, `apply_gufunc`, `gufunc`, and
  `as_gufunc` ([dask#3109](https://github.com/dask/dask/pull/3109)) ([dask#3526](https://github.com/dask/dask/pull/3526)) ([dask#3539](https://github.com/dask/dask/pull/3539)) [Markus Gonser](https://github.com/magonser)
- Avoid unnecessary rechunking tasks ([dask#3529](https://github.com/dask/dask/pull/3529)) [Matthew Rocklin](https://github.com/mrocklin)
- Compute dtypes at runtime for fft ([dask#3511](https://github.com/dask/dask/pull/3511)) [Matthew Rocklin](https://github.com/mrocklin)
- Generate UUIDs for all da.store operations ([dask#3540](https://github.com/dask/dask/pull/3540)) [Martin Durant](https://github.com/martindurant)
- Correct internal dimension of Dask’s SVD ([dask#3517](https://github.com/dask/dask/pull/3517)) [John A Kirkham](https://github.com/jakirkham)
- BUG: do not raise IndexError for identity slice in array.vindex ([dask#3559](https://github.com/dask/dask/pull/3559)) [Scott Sievert](https://github.com/stsievert)
- Adds isneginf and isposinf ([dask#3581](https://github.com/dask/dask/pull/3581)) [John A Kirkham](https://github.com/jakirkham)
- Drop Dask Array’s learn module ([dask#3580](https://github.com/dask/dask/pull/3580)) [John A Kirkham](https://github.com/jakirkham)
- added sfqr (short-and-fat) as a counterpart to tsqr… ([dask#3575](https://github.com/dask/dask/pull/3575)) [Jeremy Chen](https://github.com/convexset)
- Allow 0-width chunks in dask.array.rechunk ([dask#3591](https://github.com/dask/dask/pull/3591)) [Marc Pfister](https://github.com/drwelby)
- Document Dask Array’s nan_to_num in public API ([dask#3599](https://github.com/dask/dask/pull/3599)) [John A Kirkham](https://github.com/jakirkham)
- Show block example ([dask#3601](https://github.com/dask/dask/pull/3601)) [John A Kirkham](https://github.com/jakirkham)
- Replace token= keyword with name= in map_blocks ([dask#3597](https://github.com/dask/dask/pull/3597)) [Matthew Rocklin](https://github.com/mrocklin)
- Disable locking in to_zarr (needed for using to_zarr in a distributed context) ([dask#3607](https://github.com/dask/dask/pull/3607)) [John A Kirkham](https://github.com/jakirkham)
- Support Zarr Arrays in to_zarr/from_zarr ([dask#3561](https://github.com/dask/dask/pull/3561)) [John A Kirkham](https://github.com/jakirkham)
- Added recursion to array/linalg/tsqr to better manage the single core bottleneck ([dask#3586](https://github.com/dask/dask/pull/3586)) [Jeremy Chan](https://github.com/convexset)
  ([dask#3396](https://github.com/dask/dask/pull/3396)) [crusaderky](https://github.com/crusaderky)

### Dataframe

- Add to/read_json ([dask#3494](https://github.com/dask/dask/pull/3494)) [Martin Durant](https://github.com/martindurant)
- Adds `index` to unsupported arguments for `DataFrame.rename` method ([dask#3522](https://github.com/dask/dask/pull/3522)) [James Bourbeau](https://github.com/jrbourbeau)
- Adds support to subset Dask DataFrame columns using `numpy.ndarray`, `pandas.Series`, and
  `pandas.Index` objects ([dask#3536](https://github.com/dask/dask/pull/3536)) [James Bourbeau](https://github.com/jrbourbeau)
- Raise error if meta columns do not match dataframe ([dask#3485](https://github.com/dask/dask/pull/3485)) [Christopher Ren](https://github.com/cr458)
- Add index to unsupprted argument for DataFrame.rename ([dask#3522](https://github.com/dask/dask/pull/3522)) [James Bourbeau](https://github.com/jrbourbeau)
- Adds support for subsetting DataFrames with pandas Index/Series and numpy ndarrays ([dask#3536](https://github.com/dask/dask/pull/3536)) [James Bourbeau](https://github.com/jrbourbeau)
- Dataframe sample method docstring fix ([dask#3566](https://github.com/dask/dask/pull/3566)) [James Bourbeau](https://github.com/jrbourbeau)
- fixes dd.read_json to infer file compression ([dask#3594](https://github.com/dask/dask/pull/3594)) [Matt Lee](https://github.com/mathewlee11)
- Adds n to sample method ([dask#3606](https://github.com/dask/dask/pull/3606)) [James Bourbeau](https://github.com/jrbourbeau)
- Add fastparquet ParquetFile object support ([dask#3573](https://github.com/dask/dask/pull/3573)) [@andrethrill](https://github.com/andrethrill)

### Bag

- Rename method= keyword to shuffle= in bag.groupby ([dask#3470](https://github.com/dask/dask/pull/3470)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Replace get= keyword with scheduler= keyword ([dask#3448](https://github.com/dask/dask/pull/3448)) [Matthew Rocklin](https://github.com/mrocklin)
- Add centralized dask.config module to handle configuration for all Dask
  subprojects ([dask#3432](https://github.com/dask/dask/pull/3432)) ([dask#3513](https://github.com/dask/dask/pull/3513)) ([dask#3520](https://github.com/dask/dask/pull/3520)) [Matthew Rocklin](https://github.com/mrocklin)
- Add dask-ssh CLI Options and Description. ([dask#3476](https://github.com/dask/dask/pull/3476)) [@beomi](https://github.com/beomi)
- Read whole files fix regardless of header for HTTP ([dask#3496](https://github.com/dask/dask/pull/3496)) [Martin Durant](https://github.com/martindurant)
- Adds synchronous scheduler syntax to debugging docs ([dask#3509](https://github.com/dask/dask/pull/3509)) [James Bourbeau](https://github.com/jrbourbeau)
- Replace dask.set_options with dask.config.set ([dask#3502](https://github.com/dask/dask/pull/3502)) [Matthew Rocklin](https://github.com/mrocklin)
- Update sphinx readthedocs-theme ([dask#3516](https://github.com/dask/dask/pull/3516)) [Matthew Rocklin](https://github.com/mrocklin)
- Introduce “auto” value for normalize_chunks ([dask#3507](https://github.com/dask/dask/pull/3507)) [Matthew Rocklin](https://github.com/mrocklin)
- Fix check in configuration with env=None ([dask#3562](https://github.com/dask/dask/pull/3562)) [Simon Perkins](https://github.com/sjperkins)
- Update sizeof definitions ([dask#3582](https://github.com/dask/dask/pull/3582)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove –verbose flag from travis-ci ([dask#3477](https://github.com/dask/dask/pull/3477)) [Matthew Rocklin](https://github.com/mrocklin)
- Remove “da.random” from random array keys ([dask#3604](https://github.com/dask/dask/pull/3604)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-17-5-2018-05-16"></a>

## 0.17.5 / 2018-05-16

### Array

- Fix `rechunk` with chunksize of -1 in a dict ([dask#3469](https://github.com/dask/dask/pull/3469)) [Stephan Hoyer](https://github.com/shoyer)
- `einsum` now accepts the `split_every` parameter ([dask#3471](https://github.com/dask/dask/pull/3471)) [crusaderky](https://github.com/crusaderky)
- Improved slicing performance ([dask#3479](https://github.com/dask/dask/pull/3479)) [Yu Feng](https://github.com/rainwoodman)

### DataFrame

- Compatibility with pandas 0.23.0 ([dask#3499](https://github.com/dask/dask/pull/3499)) [Tom Augspurger](https://github.com/tomaugspurger)

<a id="v0-17-4-2018-05-03"></a>

## 0.17.4 / 2018-05-03

### Dataframe

- Add support for indexing Dask DataFrames with string subclasses ([dask#3461](https://github.com/dask/dask/pull/3461)) [James Bourbeau](https://github.com/jrbourbeau)
- Allow using both sorted_index and chunksize in read_hdf ([dask#3463](https://github.com/dask/dask/pull/3463)) [Pierre Bartet](https://github.com/Pierre-Bartet)
- Pass filesystem to arrow piece reader ([dask#3466](https://github.com/dask/dask/pull/3466)) [Martin Durant](https://github.com/martindurant)
- Switches to using dask.compat string_types ([dask#3462](https://github.com/dask/dask/pull/3462)) [James Bourbeau](https://github.com/jrbourbeau)

<a id="v0-17-3-2018-05-02"></a>

## 0.17.3 / 2018-05-02

### Array

- Add `einsum` for Dask Arrays ([dask#3412](https://github.com/dask/dask/pull/3412)) [Simon Perkins](https://github.com/sjperkins)
- Add `piecewise` for Dask Arrays ([dask#3350](https://github.com/dask/dask/pull/3350)) [John A Kirkham](https://github.com/jakirkham)
- Fix handling of `nan` in `broadcast_shapes` ([dask#3356](https://github.com/dask/dask/pull/3356)) [John A Kirkham](https://github.com/jakirkham)
- Add `isin` for dask arrays ([dask#3363](https://github.com/dask/dask/pull/3363)). [Stephan Hoyer](https://github.com/shoyer)
- Overhauled `topk` for Dask Arrays: faster algorithm, particularly for large k’s; added support
  for multiple axes, recursive aggregation, and an option to pick the bottom k elements instead.
  ([dask#3395](https://github.com/dask/dask/pull/3395)) [crusaderky](https://github.com/crusaderky)
- The `topk` API has changed from topk(k, array) to the more conventional topk(array, k).
  The legacy API still works but is now deprecated. ([dask#2965](https://github.com/dask/dask/pull/2965)) [crusaderky](https://github.com/crusaderky)
- New function `argtopk` for Dask Arrays ([dask#3396](https://github.com/dask/dask/pull/3396)) [crusaderky](https://github.com/crusaderky)
- Fix handling partial depth and boundary in `map_overlap` ([dask#3445](https://github.com/dask/dask/pull/3445)) [John A Kirkham](https://github.com/jakirkham)
- Add `gradient` for Dask Arrays ([dask#3434](https://github.com/dask/dask/pull/3434)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Allow t as shorthand for table in to_hdf for pandas compatibility ([dask#3330](https://github.com/dask/dask/pull/3330)) [Jörg Dietrich](https://github.com/joergdietrich)
- Added top level isna method for Dask DataFrames ([dask#3294](https://github.com/dask/dask/pull/3294)) [Christopher Ren](https://github.com/cr458)
- Fix selection on partition column on `read_parquet` for `engine="pyarrow"` ([dask#3207](https://github.com/dask/dask/pull/3207)) [Uwe Korn](https://github.com/xhochy)
- Added DataFrame.squeeze method ([dask#3366](https://github.com/dask/dask/pull/3366)) [Christopher Ren](https://github.com/cr458)
- Added infer_divisions option to `read_parquet` to specify whether read engines should compute divisions ([dask#3387](https://github.com/dask/dask/pull/3387)) [Jon Mease](https://github.com/jonmmease)
- Added support for inferring division for `engine="pyarrow"` ([dask#3387](https://github.com/dask/dask/pull/3387)) [Jon Mease](https://github.com/jonmmease)
- Provide more informative error message for meta= errors ([dask#3343](https://github.com/dask/dask/pull/3343)) [Matthew Rocklin](https://github.com/mrocklin)
- add orc reader ([dask#3284](https://github.com/dask/dask/pull/3284)) [Martin Durant](https://github.com/martindurant)
- Default compression for parquet now always Snappy, in line with pandas ([dask#3373](https://github.com/dask/dask/pull/3373)) [Martin Durant](https://github.com/martindurant)
- Fixed bug in Dask DataFrame and Series comparisons with NumPy scalars ([dask#3436](https://github.com/dask/dask/pull/3436)) [James Bourbeau](https://github.com/jrbourbeau)
- Remove outdated requirement from repartition docstring ([dask#3440](https://github.com/dask/dask/pull/3440)) [Jörg Dietrich](https://github.com/joergdietrich)
- Fixed bug in aggregation when only a Series is selected ([dask#3446](https://github.com/dask/dask/pull/3446)) [Jörg Dietrich](https://github.com/joergdietrich)
- Add default values to make_timeseries ([dask#3421](https://github.com/dask/dask/pull/3421)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Support traversing collections in persist, visualize, and optimize ([dask#3410](https://github.com/dask/dask/pull/3410)) [Jim Crist](https://github.com/jcrist)
- Add schedule= keyword to compute and persist.  This replaces common use of the get= keyword ([dask#3448](https://github.com/dask/dask/pull/3448)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-17-2-2018-03-21"></a>

## 0.17.2 / 2018-03-21

### Array

- Add `broadcast_arrays` for Dask Arrays ([dask#3217](https://github.com/dask/dask/pull/3217)) [John A Kirkham](https://github.com/jakirkham)
- Add `bitwise_*` ufuncs ([dask#3219](https://github.com/dask/dask/pull/3219)) [John A Kirkham](https://github.com/jakirkham)
- Add optional `axis` argument to `squeeze` ([dask#3261](https://github.com/dask/dask/pull/3261)) [John A Kirkham](https://github.com/jakirkham)
- Validate inputs to atop ([dask#3307](https://github.com/dask/dask/pull/3307)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid calls to astype in concatenate if all parts have the same dtype ([dask#3301](https://github.com/dask/dask/pull/3301)) [Martin Durant](https://github.com/martindurant)

### DataFrame

- Fixed bug in shuffle due to aggressive truncation ([dask#3201](https://github.com/dask/dask/pull/3201)) [Matthew Rocklin](https://github.com/mrocklin)
- Support specifying categorical columns on `read_parquet` with `categories=[…]` for `engine="pyarrow"` ([dask#3177](https://github.com/dask/dask/pull/3177)) [Uwe Korn](https://github.com/xhochy)
- Add `dd.tseries.Resampler.agg` ([dask#3202](https://github.com/dask/dask/pull/3202)) [Richard Postelnik](https://github.com/postelrich)
- Support operations that mix dataframes and arrays ([dask#3230](https://github.com/dask/dask/pull/3230)) [Matthew Rocklin](https://github.com/mrocklin)
- Support extra Scalar and Delayed args in `dd.groupby._Groupby.apply` ([dask#3256](https://github.com/dask/dask/pull/3256)) [Gabriele Lanaro](https://github.com/gabrielelanaro)

### Bag

- Support joining against single-partitioned bags and delayed objects ([dask#3254](https://github.com/dask/dask/pull/3254)) [Matthew Rocklin](https://github.com/mrocklin)

### Core

- Fixed bug when using unexpected but hashable types for keys ([dask#3238](https://github.com/dask/dask/pull/3238)) [Daniel Collins](https://github.com/dancollins34)
- Fix bug in task ordering so that we break ties consistently with the key name ([dask#3271](https://github.com/dask/dask/pull/3271)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid sorting tasks in order when the number of tasks is very large ([dask#3298](https://github.com/dask/dask/pull/3298)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-17-1-2018-02-22"></a>

## 0.17.1 / 2018-02-22

### Array

- Corrected dimension chunking in indices ([dask#3166](https://github.com/dask/dask/issues/3166), [dask#3167](https://github.com/dask/dask/pull/3167)) [Simon Perkins](https://github.com/sjperkins)
- Inline `store_chunk` calls for `store`’s `return_stored` option ([dask#3153](https://github.com/dask/dask/pull/3153)) [John A Kirkham](https://github.com/jakirkham)
- Compatibility with struct dtypes for NumPy 1.14.1 release ([dask#3187](https://github.com/dask/dask/pull/3187)) [Matthew Rocklin](https://github.com/mrocklin)

### DataFrame

- Bugfix to allow column assignment of pandas datetimes([dask#3164](https://github.com/dask/dask/pull/3164)) [Max Epstein](https://github.com/MaxPowerWasTaken)

### Core

- New file-system for HTTP(S), allowing direct loading from specific URLs ([dask#3160](https://github.com/dask/dask/pull/3160)) [Martin Durant](https://github.com/martindurant)
- Fix bug when tokenizing partials with no keywords ([dask#3191](https://github.com/dask/dask/pull/3191)) [Matthew Rocklin](https://github.com/mrocklin)
- Use more recent LZ4 API ([dask#3157](https://github.com/dask/dask/pull/3157)) [Thrasibule](https://github.com/thrasibule)
- Introduce output stream parameter for progress bar ([dask#3185](https://github.com/dask/dask/pull/3185)) [Dieter Weber](https://github.com/uellue)

<a id="v0-17-0-2018-02-09"></a>

## 0.17.0 / 2018-02-09

### Array

- Added a support object-type arrays for nansum, nanmin, and nanmax ([dask#3133](https://github.com/dask/dask/issues/3133)) [Keisuke Fujii](https://github.com/fujiisoup)
- Update error handling when len is called with empty chunks ([dask#3058](https://github.com/dask/dask/issues/3058)) [Xander Johnson](https://github.com/metasyn)
- Fixes a metadata bug with `store`’s `return_stored` option ([dask#3064](https://github.com/dask/dask/pull/3064)) [John A Kirkham](https://github.com/jakirkham)
- Fix a bug in `optimization.fuse_slice` to properly handle when first input is `None` ([dask#3076](https://github.com/dask/dask/pull/3076)) [James Bourbeau](https://github.com/jrbourbeau)
- Support arrays with unknown chunk sizes in percentile ([dask#3107](https://github.com/dask/dask/pull/3107)) [Matthew Rocklin](https://github.com/mrocklin)
- Tokenize scipy.sparse arrays and np.matrix ([dask#3060](https://github.com/dask/dask/pull/3060)) [Roman Yurchak](https://github.com/rth)

### DataFrame

- Support month timedeltas in repartition(freq=…) ([dask#3110](https://github.com/dask/dask/pull/3110)) [Matthew Rocklin](https://github.com/mrocklin)
- Avoid mutation in dataframe groupby tests ([dask#3118](https://github.com/dask/dask/pull/3118)) [Matthew Rocklin](https://github.com/mrocklin)
- `read_csv`, `read_table`, and `read_parquet` accept iterables of paths
  ([dask#3124](https://github.com/dask/dask/pull/3124)) [Jim Crist](https://github.com/jcrist)
- Deprecates the `dd.to_delayed` *function* in favor of the existing method
  ([dask#3126](https://github.com/dask/dask/pull/3126)) [Jim Crist](https://github.com/jcrist)
- Return dask.arrays from df.map_partitions calls when the UDF returns a numpy array ([dask#3147](https://github.com/dask/dask/pull/3147)) [Matthew Rocklin](https://github.com/mrocklin)
- Change handling of `columns` and `index` in `dd.read_parquet` to be more
  consistent, especially in handling of multi-indices ([dask#3149](https://github.com/dask/dask/pull/3149)) [Jim Crist](https://github.com/jcrist)
- fastparquet append=True allowed to create new dataset ([dask#3097](https://github.com/dask/dask/pull/3097)) [Martin Durant](https://github.com/martindurant)
- dtype rationalization for sql queries ([dask#3100](https://github.com/dask/dask/pull/3100)) [Martin Durant](https://github.com/martindurant)

### Bag

- Document `bag.map_paritions` function may receive either a list or generator. ([dask#3150](https://github.com/dask/dask/pull/3150)) [Nir](https://github.com/nirizr)

### Core

- Change default task ordering to prefer nodes with few dependents and then
  many downstream dependencies ([dask#3056](https://github.com/dask/dask/pull/3056)) [Matthew Rocklin](https://github.com/mrocklin)
- Add color= option to visualize to color by task order ([dask#3057](https://github.com/dask/dask/pull/3057)) ([dask#3122](https://github.com/dask/dask/pull/3122)) [Matthew Rocklin](https://github.com/mrocklin)
- Deprecate `dask.bytes.open_text_files` ([dask#3077](https://github.com/dask/dask/pull/3077)) [Jim Crist](https://github.com/jcrist)
- Remove short-circuit hdfs reads handling due to maintenance costs. May be
  re-added in a more robust manner later ([dask#3079](https://github.com/dask/dask/pull/3079)) [Jim Crist](https://github.com/jcrist)
- Add `dask.base.optimize` for optimizing multiple collections without
  computing. ([dask#3071](https://github.com/dask/dask/pull/3071)) [Jim Crist](https://github.com/jcrist)
- Rename `dask.optimize` module to `dask.optimization` ([dask#3071](https://github.com/dask/dask/pull/3071)) [Jim Crist](https://github.com/jcrist)
- Change task ordering to do a full traversal ([dask#3066](https://github.com/dask/dask/pull/3066)) [Matthew Rocklin](https://github.com/mrocklin)
- Adds an `optimize_graph` keyword to all `to_delayed` methods to allow
  controlling whether optimizations occur on conversion. ([dask#3126](https://github.com/dask/dask/pull/3126)) [Jim Crist](https://github.com/jcrist)
- Support using `pyarrow` for hdfs integration ([dask#3123](https://github.com/dask/dask/pull/3123)) [Jim Crist](https://github.com/jcrist)
- Move HDFS integration and tests into dask repo ([dask#3083](https://github.com/dask/dask/pull/3083)) [Jim Crist](https://github.com/jcrist)
- Remove write_bytes ([dask#3116](https://github.com/dask/dask/pull/3116)) [Jim Crist](https://github.com/jcrist)

<a id="v0-16-1-2018-01-09"></a>

## 0.16.1 / 2018-01-09

### Array

- Fix handling of scalar percentile values in `percentile` ([dask#3021](https://github.com/dask/dask/pull/3021)) [James Bourbeau](https://github.com/jrbourbeau)
- Prevent `bool()` coercion from calling compute ([dask#2958](https://github.com/dask/dask/pull/2958)) [Albert DeFusco](https://github.com/AlbertDeFusco)
- Add `matmul` ([dask#2904](https://github.com/dask/dask/pull/2904)) [John A Kirkham](https://github.com/jakirkham)
- Support N-D arrays with `matmul` ([dask#2909](https://github.com/dask/dask/pull/2909)) [John A Kirkham](https://github.com/jakirkham)
- Add `vdot` ([dask#2910](https://github.com/dask/dask/pull/2910)) [John A Kirkham](https://github.com/jakirkham)
- Explicit `chunks` argument for `broadcast_to` ([dask#2943](https://github.com/dask/dask/pull/2943)) [Stephan Hoyer](https://github.com/shoyer)
- Add `meshgrid` ([dask#2938](https://github.com/dask/dask/pull/2938)) [John A Kirkham](https://github.com/jakirkham) and ([dask#3001](https://github.com/dask/dask/pull/3001)) [Markus Gonser](https://github.com/magonser)
- Preserve singleton chunks in `fftshift`/`ifftshift` ([dask#2733](https://github.com/dask/dask/pull/2733)) [John A Kirkham](https://github.com/jakirkham)
- Fix handling of negative indexes in `vindex` and raise errors for out of bounds indexes ([dask#2967](https://github.com/dask/dask/pull/2967)) [Stephan Hoyer](https://github.com/shoyer)
- Add `flip`, `flipud`, `fliplr` ([dask#2954](https://github.com/dask/dask/pull/2954)) [John A Kirkham](https://github.com/jakirkham)
- Add `float_power` ufunc ([dask#2962](https://github.com/dask/dask/pull/2962)) ([dask#2969](https://github.com/dask/dask/pull/2969)) [John A Kirkham](https://github.com/jakirkham)
- Compatibility for changes to structured arrays in the upcoming NumPy 1.14 release ([dask#2964](https://github.com/dask/dask/pull/2964)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add `block` ([dask#2650](https://github.com/dask/dask/pull/2650)) [John A Kirkham](https://github.com/jakirkham)
- Add `frompyfunc` ([dask#3030](https://github.com/dask/dask/pull/3030)) [Jim Crist](https://github.com/jcrist)
- Add the `return_stored` option to `store` for chaining stored results ([dask#2980](https://github.com/dask/dask/pull/2980)) [John A Kirkham](https://github.com/jakirkham)

### DataFrame

- Fixed naming bug in cumulative aggregations ([dask#3037](https://github.com/dask/dask/issues/3037)) [Martijn Arts](https://github.com/mfaafm)
- Fixed `dd.read_csv` when `names` is given but `header` is not set to `None` ([dask#2976](https://github.com/dask/dask/issues/2976)) [Martijn Arts](https://github.com/mfaafm)
- Fixed `dd.read_csv` so that passing instances of `CategoricalDtype` in `dtype` will result in known categoricals ([dask#2997](https://github.com/dask/dask/pull/2997)) [Tom Augspurger](https://github.com/tomaugspurger)
- Prevent `bool()` coercion from calling compute ([dask#2958](https://github.com/dask/dask/pull/2958)) [Albert DeFusco](https://github.com/AlbertDeFusco)
- `DataFrame.read_sql()` ([dask#2928](https://github.com/dask/dask/pull/2928)) to an empty database tables returns an empty dask dataframe [Apostolos Vlachopoulos](https://github.com/avlahop)
- Compatibility for reading Parquet files written by PyArrow 0.8.0 ([dask#2973](https://github.com/dask/dask/pull/2973)) [Tom Augspurger](https://github.com/tomaugspurger)
- Correctly handle the column name (df.columns.name) when reading in `dd.read_parquet` ([dask#2973](https://github.com/dask/dask/pull/2973)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed `dd.concat` losing the index dtype when the data contained a categorical ([dask#2932](https://github.com/dask/dask/issues/2932)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add `dd.Series.rename` ([dask#3027](https://github.com/dask/dask/pull/3027)) [Jim Crist](https://github.com/jcrist)
- `DataFrame.merge()` now supports merging on a combination of columns and the index ([dask#2960](https://github.com/dask/dask/pull/2960)) [Jon Mease](https://github.com/jonmmease)
- Removed the deprecated `dd.rolling*` methods, in preparation for their removal in the next pandas release ([dask#2995](https://github.com/dask/dask/pull/2995)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix metadata inference bug in which single-partition series were mistakenly special cased ([dask#3035](https://github.com/dask/dask/pull/3035)) [Jim Crist](https://github.com/jcrist)
- Add support for `Series.str.cat` ([dask#3028](https://github.com/dask/dask/pull/3028)) [Jim Crist](https://github.com/jcrist)

### Core

- Improve 32-bit compatibility ([dask#2937](https://github.com/dask/dask/pull/2937)) [Matthew Rocklin](https://github.com/mrocklin)
- Change task prioritization to avoid upwards branching ([dask#3017](https://github.com/dask/dask/pull/3017)) [Matthew Rocklin](https://github.com/mrocklin)

<a id="v0-16-0-2017-11-17"></a>

## 0.16.0 / 2017-11-17

This is a major release.  It includes breaking changes, new protocols, and a
large number of bug fixes.

### Array

- Add `atleast_1d`, `atleast_2d`, and `atleast_3d` ([dask#2760](https://github.com/dask/dask/pull/2760)) ([dask#2765](https://github.com/dask/dask/pull/2765)) [John A Kirkham](https://github.com/jakirkham)
- Add `allclose` ([dask#2771](https://github.com/dask/dask/pull/2771)) by [John A Kirkham](https://github.com/jakirkham)
- Remove `random.different_seeds` from Dask Array API docs ([dask#2772](https://github.com/dask/dask/pull/2772)) [John A Kirkham](https://github.com/jakirkham)
- Deprecate `vnorm` in favor of `dask.array.linalg.norm` ([dask#2773](https://github.com/dask/dask/pull/2773)) [John A Kirkham](https://github.com/jakirkham)
- Reimplement `unique` to be lazy ([dask#2775](https://github.com/dask/dask/pull/2775)) [John A Kirkham](https://github.com/jakirkham)
- Support broadcasting of Dask Arrays with 0-length dimensions ([dask#2784](https://github.com/dask/dask/pull/2784)) [John A Kirkham](https://github.com/jakirkham)
- Add `asarray` and `asanyarray` to Dask Array API docs ([dask#2787](https://github.com/dask/dask/pull/2787)) [James Bourbeau](https://github.com/jrbourbeau)
- Support `unique`’s `return_*` arguments ([dask#2779](https://github.com/dask/dask/pull/2779)) [John A Kirkham](https://github.com/jakirkham)
- Simplify `_unique_internal` ([dask#2850](https://github.com/dask/dask/pull/2850)) ([dask#2855](https://github.com/dask/dask/pull/2855)) [John A Kirkham](https://github.com/jakirkham)
- Avoid removing some getter calls in array optimizations ([dask#2826](https://github.com/dask/dask/pull/2826)) [Jim Crist](https://github.com/jcrist)

### DataFrame

- Support `pyarrow` in `dd.to_parquet` ([dask#2868](https://github.com/dask/dask/pull/2868)) [Jim Crist](https://github.com/jcrist)
- Fixed `DataFrame.quantile` and `Series.quantile` returning `nan` when missing values are present ([dask#2791](https://github.com/dask/dask/pull/2791)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed `DataFrame.quantile` losing the result `.name` when `q` is a scalar ([dask#2791](https://github.com/dask/dask/pull/2791)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fixed `dd.concat` return a `dask.Dataframe` when concatenating a single series along the columns, matching pandas’ behavior ([dask#2800](https://github.com/dask/dask/pull/2800)) [James Munroe](https://github.com/jmunroe)
- Fixed default inplace parameter for `DataFrame.eval` to match the pandas defualt for pandas >= 0.21.0 ([dask#2838](https://github.com/dask/dask/pull/2838)) [Tom Augspurger](https://github.com/tomaugspurger)
- Fix exception when calling `DataFrame.set_index` on text column where one of the partitions was empty ([dask#2831](https://github.com/dask/dask/pull/2831)) [Jesse Vogt](https://github.com/jessevogt)
- Do not raise exception when calling `DataFrame.set_index` on empty dataframe ([dask#2827](https://github.com/dask/dask/pull/2827)) [Jesse Vogt](https://github.com/jessevogt)
- Fixed bug in `Dataframe.fillna` when filling with a `Series` value ([dask#2810](https://github.com/dask/dask/pull/2810)) [Tom Augspurger](https://github.com/tomaugspurger)
- Deprecate old argument ordering in `dd.to_parquet` to better match convention of putting the dataframe first ([dask#2867](https://github.com/dask/dask/pull/2867)) [Jim Crist](https://github.com/jcrist)
- df.astype(categorical_dtype -> known categoricals ([dask#2835](https://github.com/dask/dask/pull/2835)) [Jim Crist](https://github.com/jcrist)
- Test against Pandas release candidate ([dask#2814](https://github.com/dask/dask/pull/2814)) [Tom Augspurger](https://github.com/tomaugspurger)
- Add more tests for read_parquet(engine=’pyarrow’) ([dask#2822](https://github.com/dask/dask/pull/2822)) [Uwe Korn](https://github.com/xhochy)
- Remove unnecessary map_partitions in aggregate ([dask#2712](https://github.com/dask/dask/pull/2712)) [Christopher Prohm](https://github.com/chmp)
- Fix bug calling sample on empty partitions ([dask#2818](https://github.com/dask/dask/pull/2818)) [@xwang777](https://github.com/xwang777)
- Error nicely when parsing dates in read_csv ([dask#2863](https://github.com/dask/dask/pull/2863)) [Jim Crist](https://github.com/jcrist)
- Cleanup handling of passing filesystem objects to PyArrow readers ([dask#2527](https://github.com/dask/dask/pull/2527)) [@fjetter](https://github.com/fjetter)
- Support repartitioning even if there are no divisions ([dask#2873](https://github.com/dask/dask/pull/2873)) [@Ced4](https://github.com/Ced4)
- Support reading/writing to hdfs using `pyarrow` in `dd.to_parquet` ([dask#2894](https://github.com/dask/dask/pull/2894), [dask#2881](https://github.com/dask/dask/pull/2881)) [Jim Crist](https://github.com/jcrist)

### Core

- Allow tuples as sharedict keys ([dask#2763](https://github.com/dask/dask/pull/2763)) [Matthew Rocklin](https://github.com/mrocklin)
- Calling compute within a dask.distributed task defaults to distributed scheduler ([dask#2762](https://github.com/dask/dask/pull/2762)) [Matthew Rocklin](https://github.com/mrocklin)
- Auto-import gcsfs when gcs:// protocol is used ([dask#2776](https://github.com/dask/dask/pull/2776)) [Matthew Rocklin](https://github.com/mrocklin)
- Fully remove dask.async module, use dask.local instead ([dask#2828](https://github.com/dask/dask/pull/2828)) [Thomas Caswell](https://github.com/tacaswell)
- Compatibility with bokeh 0.12.10 ([dask#2844](https://github.com/dask/dask/pull/2844)) [Tom Augspurger](https://github.com/tomaugspurger)
- Reduce test memory usage ([dask#2782](https://github.com/dask/dask/pull/2782)) [Jim Crist](https://github.com/jcrist)
- Add Dask collection interface ([dask#2748](https://github.com/dask/dask/pull/2748)) [Jim Crist](https://github.com/jcrist)
- Update Dask collection interface during XArray integration ([dask#2847](https://github.com/dask/dask/pull/2847)) [Matthew Rocklin](https://github.com/mrocklin)
- Close resource profiler process on \_\_exit_\_ ([dask#2871](https://github.com/dask/dask/pull/2871)) [Jim Crist](https://github.com/jcrist)
- Fix S3 tests ([dask#2875](https://github.com/dask/dask/pull/2875)) [Jim Crist](https://github.com/jcrist)
- Fix port for bokeh dashboard in docs ([dask#2889](https://github.com/dask/dask/pull/2889)) [Ian Hopkinson](https://github.com/IanHopkinson)
- Wrap Dask filesystems for PyArrow compatibility ([dask#2881](https://github.com/dask/dask/pull/2881)) [Jim Crist](https://github.com/jcrist)

<a id="v0-15-4-2017-10-06"></a>

## 0.15.4 / 2017-10-06

### Array

- `da.random.choice` now works with array arguments ([dask#2781](https://github.com/dask/dask/pull/2781))
- Support indexing in arrays with np.int (fixes regression) ([dask#2719](https://github.com/dask/dask/pull/2719))
- Handle zero dimension with rechunking ([dask#2747](https://github.com/dask/dask/pull/2747))
- Support -1 as an alias for “size of the dimension” in `chunks` ([dask#2749](https://github.com/dask/dask/pull/2749))
- Call mkdir in array.to_npy_stack ([dask#2709](https://github.com/dask/dask/pull/2709))

### DataFrame

- Added the .str accessor to Categoricals with string categories ([dask#2743](https://github.com/dask/dask/pull/2743))
- Support int96 (spark) datetimes in parquet writer ([dask#2711](https://github.com/dask/dask/pull/2711))
- Pass on file scheme to fastparquet ([dask#2714](https://github.com/dask/dask/pull/2714))
- Support Pandas 0.21 ([dask#2737](https://github.com/dask/dask/pull/2737))

### Bag

- Add tree reduction support for foldby ([dask#2710](https://github.com/dask/dask/pull/2710))

### Core

- Drop s3fs from `pip install dask[complete]` ([dask#2750](https://github.com/dask/dask/pull/2750))

<a id="v0-15-3-2017-09-24"></a>

## 0.15.3 / 2017-09-24

### Array

- Add masked arrays ([dask#2301](https://github.com/dask/dask/pull/2301))
- Add `*_like array creation functions` ([dask#2640](https://github.com/dask/dask/pull/2640))
- Indexing with unsigned integer array ([dask#2647](https://github.com/dask/dask/pull/2647))
- Improved slicing with boolean arrays of different dimensions ([dask#2658](https://github.com/dask/dask/pull/2658))
- Support literals in `top` and `atop` ([dask#2661](https://github.com/dask/dask/pull/2661))
- Optional axis argument in cumulative functions ([dask#2664](https://github.com/dask/dask/pull/2664))
- Improve tests on scalars with `assert_eq` ([dask#2681](https://github.com/dask/dask/pull/2681))
- Fix norm keepdims ([dask#2683](https://github.com/dask/dask/pull/2683))
- Add `ptp` ([dask#2691](https://github.com/dask/dask/pull/2691))
- Add apply_along_axis ([dask#2690](https://github.com/dask/dask/pull/2690)) and apply_over_axes ([dask#2702](https://github.com/dask/dask/pull/2702))

### DataFrame

- Added `Series.str[index]` ([dask#2634](https://github.com/dask/dask/pull/2634))
- Allow the groupby by param to handle columns and index levels ([dask#2636](https://github.com/dask/dask/pull/2636))
- `DataFrame.to_csv` and `Bag.to_textfiles` now return the filenames to
  : which they have written ([dask#2655](https://github.com/dask/dask/pull/2655))
- Fix combination of `partition_on` and `append` in `to_parquet`
  ([dask#2645](https://github.com/dask/dask/pull/2645))
- Fix for parquet file schemes ([dask#2667](https://github.com/dask/dask/pull/2667))
- Repartition works with mixed categoricals ([dask#2676](https://github.com/dask/dask/pull/2676))

### Core

- `python setup.py test` now runs tests ([dask#2641](https://github.com/dask/dask/pull/2641))
- Added new cheatsheet ([dask#2649](https://github.com/dask/dask/pull/2649))
- Remove resize tool in Bokeh plots ([dask#2688](https://github.com/dask/dask/pull/2688))

<a id="v0-15-2-2017-08-25"></a>

## 0.15.2 / 2017-08-25

### Array

- Remove spurious keys from map_overlap graph ([dask#2520](https://github.com/dask/dask/pull/2520))
- where works with non-bool condition and scalar values ([dask#2543](https://github.com/dask/dask/pull/2543)) ([dask#2549](https://github.com/dask/dask/pull/2549))
- Improve compress ([dask#2541](https://github.com/dask/dask/pull/2541)) ([dask#2545](https://github.com/dask/dask/pull/2545)) ([dask#2555](https://github.com/dask/dask/pull/2555))
- Add argwhere, \_nonzero, and where(cond) ([dask#2539](https://github.com/dask/dask/pull/2539))
- Generalize vindex in dask.array to handle multi-dimensional indices ([dask#2573](https://github.com/dask/dask/pull/2573))
- Add choose method ([dask#2584](https://github.com/dask/dask/pull/2584))
- Split code into reorganized files ([dask#2595](https://github.com/dask/dask/pull/2595))
- Add linalg.norm ([dask#2597](https://github.com/dask/dask/pull/2597))
- Add diff, ediff1d ([dask#2607](https://github.com/dask/dask/pull/2607)), ([dask#2609](https://github.com/dask/dask/pull/2609))
- Improve dtype inference and reflection ([dask#2571](https://github.com/dask/dask/pull/2571))

### Bag

- Remove deprecated Bag behaviors ([dask#2525](https://github.com/dask/dask/pull/2525))

### DataFrame

- Support callables in assign ([dask#2513](https://github.com/dask/dask/pull/2513))
- better error messages for read_csv ([dask#2522](https://github.com/dask/dask/pull/2522))
- Add dd.to_timedelta ([dask#2523](https://github.com/dask/dask/pull/2523))
- Verify metadata in from_delayed ([dask#2534](https://github.com/dask/dask/pull/2534)) ([dask#2591](https://github.com/dask/dask/pull/2591))
- Add DataFrame.isin ([dask#2558](https://github.com/dask/dask/pull/2558))
- Read_hdf supports iterables of files ([dask#2547](https://github.com/dask/dask/pull/2547))

### Core

- Remove bare `except:` blocks everywhere ([dask#2590](https://github.com/dask/dask/pull/2590))

<a id="v0-15-1-2017-07-08"></a>

## 0.15.1 / 2017-07-08

- Add storage_options to to_textfiles and to_csv ([dask#2466](https://github.com/dask/dask/pull/2466))
- Rechunk and simplify rfftfreq ([dask#2473](https://github.com/dask/dask/pull/2473)), ([dask#2475](https://github.com/dask/dask/pull/2475))
- Better support ndarray subclasses ([dask#2486](https://github.com/dask/dask/pull/2486))
- Import star in dask.distributed ([dask#2503](https://github.com/dask/dask/pull/2503))
- Threadsafe cache handling with tokenization ([dask#2511](https://github.com/dask/dask/pull/2511))

<a id="v0-15-0-2017-06-09"></a>

## 0.15.0 / 2017-06-09

### Array

- Add dask.array.stats submodule ([dask#2269](https://github.com/dask/dask/pull/2269))
- Support `ufunc.outer` ([dask#2345](https://github.com/dask/dask/pull/2345))
- Optimize fancy indexing by reducing graph overhead ([dask#2333](https://github.com/dask/dask/pull/2333)) ([dask#2394](https://github.com/dask/dask/pull/2394))
- Faster array tokenization using alternative hashes ([dask#2377](https://github.com/dask/dask/pull/2377))
- Added the matmul `@` operator ([dask#2349](https://github.com/dask/dask/pull/2349))
- Improved coverage of the `numpy.fft` module ([dask#2320](https://github.com/dask/dask/pull/2320)) ([dask#2322](https://github.com/dask/dask/pull/2322)) ([dask#2327](https://github.com/dask/dask/pull/2327)) ([dask#2323](https://github.com/dask/dask/pull/2323))
- Support NumPy’s `__array_ufunc__` protocol ([dask#2438](https://github.com/dask/dask/pull/2438))

### Bag

- Fix bug where reductions on bags with no partitions would fail ([dask#2324](https://github.com/dask/dask/pull/2324))
- Add broadcasting and variadic `db.map` top-level function.  Also remove
  auto-expansion of tuples as map arguments ([dask#2339](https://github.com/dask/dask/pull/2339))
- Rename `Bag.concat` to `Bag.flatten` ([dask#2402](https://github.com/dask/dask/pull/2402))

### DataFrame

- Parquet improvements ([dask#2277](https://github.com/dask/dask/pull/2277)) ([dask#2422](https://github.com/dask/dask/pull/2422))

### Core

- Move dask.async module to dask.local ([dask#2318](https://github.com/dask/dask/pull/2318))
- Support callbacks with nested scheduler calls ([dask#2397](https://github.com/dask/dask/pull/2397))
- Support pathlib.Path objects as uris  ([dask#2310](https://github.com/dask/dask/pull/2310))

<a id="v0-14-3-2017-05-05"></a>

## 0.14.3 / 2017-05-05

### DataFrame

- Pandas 0.20.0 support

<a id="v0-14-2-2017-05-03"></a>

## 0.14.2 / 2017-05-03

### Array

- Add da.indices ([dask#2268](https://github.com/dask/dask/pull/2268)), da.tile ([dask#2153](https://github.com/dask/dask/pull/2153)), da.roll ([dask#2135](https://github.com/dask/dask/pull/2135))
- Simultaneously support drop_axis and new_axis in da.map_blocks ([dask#2264](https://github.com/dask/dask/pull/2264))
- Rechunk and concatenate work with unknown chunksizes ([dask#2235](https://github.com/dask/dask/pull/2235)) and ([dask#2251](https://github.com/dask/dask/pull/2251))
- Support non-numpy container arrays, notably sparse arrays ([dask#2234](https://github.com/dask/dask/pull/2234))
- Tensordot contracts over multiple axes ([dask#2186](https://github.com/dask/dask/pull/2186))
- Allow delayed targets in da.store ([dask#2181](https://github.com/dask/dask/pull/2181))
- Support interactions against lists and tuples ([dask#2148](https://github.com/dask/dask/pull/2148))
- Constructor plugins for debugging ([dask#2142](https://github.com/dask/dask/pull/2142))
- Multi-dimensional FFTs (single chunk) ([dask#2116](https://github.com/dask/dask/pull/2116))

### Bag

- to_dataframe enforces consistent types ([dask#2199](https://github.com/dask/dask/pull/2199))

### DataFrame

- Set_index always fully sorts the index ([dask#2290](https://github.com/dask/dask/pull/2290))
- Support compatibility with pandas 0.20.0 ([dask#2249](https://github.com/dask/dask/pull/2249)), ([dask#2248](https://github.com/dask/dask/pull/2248)), and ([dask#2246](https://github.com/dask/dask/pull/2246))
- Support Arrow Parquet reader ([dask#2223](https://github.com/dask/dask/pull/2223))
- Time-based rolling windows ([dask#2198](https://github.com/dask/dask/pull/2198))
- Repartition can now create more partitions, not just less ([dask#2168](https://github.com/dask/dask/pull/2168))

### Core

- Always use absolute paths when on POSIX file system ([dask#2263](https://github.com/dask/dask/pull/2263))
- Support user provided graph optimizations ([dask#2219](https://github.com/dask/dask/pull/2219))
- Refactor path handling ([dask#2207](https://github.com/dask/dask/pull/2207))
- Improve fusion performance ([dask#2129](https://github.com/dask/dask/pull/2129)), ([dask#2131](https://github.com/dask/dask/pull/2131)), and ([dask#2112](https://github.com/dask/dask/pull/2112))

<a id="v0-14-1-2017-03-22"></a>

## 0.14.1 / 2017-03-22

### Array

- Micro-optimize optimizations ([dask#2058](https://github.com/dask/dask/pull/2058))
- Change slicing optimizations to avoid fusing raw numpy arrays ([dask#2075](https://github.com/dask/dask/pull/2075)) ([dask#2080](https://github.com/dask/dask/pull/2080))
- Dask.array operations now work on numpy arrays ([dask#2079](https://github.com/dask/dask/pull/2079))
- Reshape now works in a much broader set of cases ([dask#2089](https://github.com/dask/dask/pull/2089))
- Support deepcopy python protocol ([dask#2090](https://github.com/dask/dask/pull/2090))
- Allow user-provided FFT implementations in `da.fft` ([dask#2093](https://github.com/dask/dask/pull/2093))

### DataFrame

- Fix to_parquet with empty partitions ([dask#2020](https://github.com/dask/dask/pull/2020))
- Optional `npartitions='auto'` mode in `set_index` ([dask#2025](https://github.com/dask/dask/pull/2025))
- Optimize shuffle performance ([dask#2032](https://github.com/dask/dask/pull/2032))
- Support efficient repartitioning along time windows like `repartition(freq='12h')` ([dask#2059](https://github.com/dask/dask/pull/2059))
- Improve speed of categorize ([dask#2010](https://github.com/dask/dask/pull/2010))
- Support single-row dataframe arithmetic ([dask#2085](https://github.com/dask/dask/pull/2085))
- Automatically avoid shuffle when setting index with a sorted column ([dask#2091](https://github.com/dask/dask/pull/2091))
- Improve handling of integer-na handling in read_csv ([dask#2098](https://github.com/dask/dask/pull/2098))

### Delayed

- Repeated attribute access on delayed objects uses the same key ([dask#2084](https://github.com/dask/dask/pull/2084))

### Core

- Improve naming of nodes in dot visuals to avoid generic `apply`
  ([dask#2070](https://github.com/dask/dask/pull/2070))
- Ensure that worker processes have different random seeds ([dask#2094](https://github.com/dask/dask/pull/2094))

<a id="v0-14-0-2017-02-24"></a>

## 0.14.0 / 2017-02-24

### Array

- Fix corner cases with zero shape and misaligned values in `arange`
  ([dask#1902](https://github.com/dask/dask/pull/1902)), ([dask#1904](https://github.com/dask/dask/pull/1904)), ([dask#1935](https://github.com/dask/dask/pull/1935)), ([dask#1955](https://github.com/dask/dask/pull/1955)), ([dask#1956](https://github.com/dask/dask/pull/1956))
- Improve concatenation efficiency ([dask#1923](https://github.com/dask/dask/pull/1923))
- Avoid hashing in `from_array` if name is provided ([dask#1972](https://github.com/dask/dask/pull/1972))

### Bag

- Repartition can now increase number of partitions ([dask#1934](https://github.com/dask/dask/pull/1934))
- Fix bugs in some reductions with empty partitions ([dask#1939](https://github.com/dask/dask/pull/1939)), ([dask#1950](https://github.com/dask/dask/pull/1950)),
  ([dask#1953](https://github.com/dask/dask/pull/1953))

### DataFrame

- Support non-uniform categoricals ([dask#1877](https://github.com/dask/dask/pull/1877)), ([dask#1930](https://github.com/dask/dask/pull/1930))
- Groupby cumulative reductions ([dask#1909](https://github.com/dask/dask/pull/1909))
- DataFrame.loc indexing now supports lists ([dask#1913](https://github.com/dask/dask/pull/1913))
- Improve multi-level groupbys ([dask#1914](https://github.com/dask/dask/pull/1914))
- Improved HTML and string repr for DataFrames ([dask#1637](https://github.com/dask/dask/pull/1637))
- Parquet append ([dask#1940](https://github.com/dask/dask/pull/1940))
- Add `dd.demo.daily_stock` function for teaching ([dask#1992](https://github.com/dask/dask/pull/1992))

### Delayed

- Add `traverse=` keyword to delayed to optionally avoid traversing nested
  data structures ([dask#1899](https://github.com/dask/dask/pull/1899))
- Support Futures in from_delayed functions ([dask#1961](https://github.com/dask/dask/pull/1961))
- Improve serialization of decorated delayed functions ([dask#1969](https://github.com/dask/dask/pull/1969))

### Core

- Improve windows path parsing in corner cases ([dask#1910](https://github.com/dask/dask/pull/1910))
- Rename tasks when fusing ([dask#1919](https://github.com/dask/dask/pull/1919))
- Add top level `persist` function ([dask#1927](https://github.com/dask/dask/pull/1927))
- Propagate `errors=` keyword in byte handling ([dask#1954](https://github.com/dask/dask/pull/1954))
- Dask.compute traverses Python collections ([dask#1975](https://github.com/dask/dask/pull/1975))
- Structural sharing between graphs in dask.array and dask.delayed ([dask#1985](https://github.com/dask/dask/pull/1985))

<a id="v0-13-0-2017-01-02"></a>

## 0.13.0 / 2017-01-02

### Array

- Mandatory dtypes on dask.array.  All operations maintain dtype information
  and UDF functions like map_blocks now require a dtype= keyword if it can not
  be inferred.  ([dask#1755](https://github.com/dask/dask/pull/1755))
- Support arrays without known shapes, such as arises when slicing arrays with
  arrays or converting dataframes to arrays ([dask#1838](https://github.com/dask/dask/pull/1838))
- Support mutation by setting one array with another ([dask#1840](https://github.com/dask/dask/pull/1840))
- Tree reductions for covariance and correlations.  ([dask#1758](https://github.com/dask/dask/pull/1758))
- Add SerializableLock for better use with distributed scheduling ([dask#1766](https://github.com/dask/dask/pull/1766))
- Improved atop support ([dask#1800](https://github.com/dask/dask/pull/1800))
- Rechunk optimization ([dask#1737](https://github.com/dask/dask/pull/1737)), ([dask#1827](https://github.com/dask/dask/pull/1827))

### Bag

- Avoid wrong results when recomputing the same groupby twice ([dask#1867](https://github.com/dask/dask/pull/1867))

### DataFrame

- Add `map_overlap` for custom rolling operations ([dask#1769](https://github.com/dask/dask/pull/1769))
- Add `shift` ([dask#1773](https://github.com/dask/dask/pull/1773))
- Add Parquet support ([dask#1782](https://github.com/dask/dask/pull/1782)) ([dask#1792](https://github.com/dask/dask/pull/1792)) ([dask#1810](https://github.com/dask/dask/pull/1810)), ([dask#1843](https://github.com/dask/dask/pull/1843)),
  ([dask#1859](https://github.com/dask/dask/pull/1859)), ([dask#1863](https://github.com/dask/dask/pull/1863))
- Add missing methods combine, abs, autocorr, sem, nsmallest, first, last,
  prod, ([dask#1787](https://github.com/dask/dask/pull/1787))
- Approximate nunique ([dask#1807](https://github.com/dask/dask/pull/1807)), ([dask#1824](https://github.com/dask/dask/pull/1824))
- Reductions with multiple output partitions (for operations like
  drop_duplicates) ([dask#1808](https://github.com/dask/dask/pull/1808)), ([dask#1823](https://github.com/dask/dask/pull/1823)) ([dask#1828](https://github.com/dask/dask/pull/1828))
- Add delitem and copy to DataFrames, increasing mutation support ([dask#1858](https://github.com/dask/dask/pull/1858))

### Delayed

- Changed behaviour for `delayed(nout=0)` and `delayed(nout=1)`:
  `delayed(nout=1)` does not default to `out=None` anymore, and
  `delayed(nout=0)` is also enabled. I.e. functions with return
  tuples of length 1 or 0 can be handled correctly. This is especially
  handy, if functions with a variable amount of outputs are wrapped by
  `delayed`. E.g. a trivial example:
  `delayed(lambda *args: args, nout=len(vals))(*vals)`

### Core

- Refactor core byte ingest ([dask#1768](https://github.com/dask/dask/pull/1768)), ([dask#1774](https://github.com/dask/dask/pull/1774))
- Improve import time ([dask#1833](https://github.com/dask/dask/pull/1833))

<a id="v0-12-0-2016-11-03"></a>

## 0.12.0 / 2016-11-03

### DataFrame

- Return a series when functions given to `dataframe.map_partitions` return
  scalars ([dask#1515](https://github.com/dask/dask/pull/1515))
- Fix type size inference for series ([dask#1513](https://github.com/dask/dask/pull/1513))
- `dataframe.DataFrame.categorize` no longer includes missing values
  in the `categories`. This is for compatibility with a [pandas change](https://github.com/pydata/pandas/pull/10929) ([dask#1565](https://github.com/dask/dask/pull/1565))
- Fix head parser error in `dataframe.read_csv` when some lines have quotes
  ([dask#1495](https://github.com/dask/dask/pull/1495))
- Add `dataframe.reduction` and `series.reduction` methods to apply generic
  row-wise reduction to dataframes and series ([dask#1483](https://github.com/dask/dask/pull/1483))
- Add `dataframe.select_dtypes`, which mirrors the [pandas method](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.select_dtypes.html) ([dask#1556](https://github.com/dask/dask/pull/1556))
- `dataframe.read_hdf` now supports reading `Series` ([dask#1564](https://github.com/dask/dask/pull/1564))
- Support Pandas 0.19.0 ([dask#1540](https://github.com/dask/dask/pull/1540))
- Implement `select_dtypes` ([dask#1556](https://github.com/dask/dask/pull/1556))
- String accessor works with indexes ([dask#1561](https://github.com/dask/dask/pull/1561))
- Add pipe method to dask.dataframe ([dask#1567](https://github.com/dask/dask/pull/1567))
- Add `indicator` keyword to merge ([dask#1575](https://github.com/dask/dask/pull/1575))
- Support Series in `read_hdf` ([dask#1575](https://github.com/dask/dask/pull/1575))
- Support Categories with missing values ([dask#1578](https://github.com/dask/dask/pull/1578))
- Support inplace operators like `df.x += 1` ([dask#1585](https://github.com/dask/dask/pull/1585))
- Str accessor passes through args and kwargs ([dask#1621](https://github.com/dask/dask/pull/1621))
- Improved groupby support for single-machine multiprocessing scheduler
  ([dask#1625](https://github.com/dask/dask/pull/1625))
- Tree reductions ([dask#1663](https://github.com/dask/dask/pull/1663))
- Pivot tables ([dask#1665](https://github.com/dask/dask/pull/1665))
- Add clip ([dask#1667](https://github.com/dask/dask/pull/1667)), align ([dask#1668](https://github.com/dask/dask/pull/1668)), combine_first ([dask#1725](https://github.com/dask/dask/pull/1725)), and
  any/all ([dask#1724](https://github.com/dask/dask/pull/1724))
- Improved handling of divisions on dask-pandas merges ([dask#1666](https://github.com/dask/dask/pull/1666))
- Add `groupby.aggregate` method ([dask#1678](https://github.com/dask/dask/pull/1678))
- Add `dd.read_table` function ([dask#1682](https://github.com/dask/dask/pull/1682))
- Improve support for multi-level columns ([dask#1697](https://github.com/dask/dask/pull/1697)) ([dask#1712](https://github.com/dask/dask/pull/1712))
- Support 2d indexing in `loc` ([dask#1726](https://github.com/dask/dask/pull/1726))
- Extend `resample` to include DataFrames ([dask#1741](https://github.com/dask/dask/pull/1741))
- Support dask.array ufuncs on dask.dataframe objects ([dask#1669](https://github.com/dask/dask/pull/1669))

### Array

- Add information about how `dask.array` `chunks` argument work ([dask#1504](https://github.com/dask/dask/pull/1504))
- Fix field access with non-scalar fields in `dask.array` ([dask#1484](https://github.com/dask/dask/pull/1484))
- Add concatenate= keyword to atop to concatenate chunks of contracted dimensions
- Optimized slicing performance ([dask#1539](https://github.com/dask/dask/pull/1539)) ([dask#1731](https://github.com/dask/dask/pull/1731))
- Extend `atop` with a `concatenate=` ([dask#1609](https://github.com/dask/dask/pull/1609)) `new_axes=`
  ([dask#1612](https://github.com/dask/dask/pull/1612)) and `adjust_chunks=` ([dask#1716](https://github.com/dask/dask/pull/1716)) keywords
- Add clip ([dask#1610](https://github.com/dask/dask/pull/1610)) swapaxes ([dask#1611](https://github.com/dask/dask/pull/1611)) round ([dask#1708](https://github.com/dask/dask/pull/1708)) repeat
- Automatically align chunks in `atop`-backed operations ([dask#1644](https://github.com/dask/dask/pull/1644))
- Cull dask.arrays on slicing ([dask#1709](https://github.com/dask/dask/pull/1709))

### Bag

- Fix issue with callables in `bag.from_sequence` being interpreted as
  tasks ([dask#1491](https://github.com/dask/dask/pull/1491))
- Avoid non-lazy memory use in reductions ([dask#1747](https://github.com/dask/dask/pull/1747))

### Administration

- Added changelog ([dask#1526](https://github.com/dask/dask/pull/1526))
- Create new threadpool when operating from thread ([dask#1487](https://github.com/dask/dask/pull/1487))
- Unify example documentation pages into one ([dask#1520](https://github.com/dask/dask/pull/1520))
- Add versioneer for git-commit based versions ([dask#1569](https://github.com/dask/dask/pull/1569))
- Pass through node_attr and edge_attr keywords in dot visualization
  ([dask#1614](https://github.com/dask/dask/pull/1614))
- Add continuous testing for Windows with Appveyor ([dask#1648](https://github.com/dask/dask/pull/1648))
- Remove use of multiprocessing.Manager ([dask#1653](https://github.com/dask/dask/pull/1653))
- Add global optimizations keyword to compute ([dask#1675](https://github.com/dask/dask/pull/1675))
- Micro-optimize get_dependencies ([dask#1722](https://github.com/dask/dask/pull/1722))

<a id="v0-11-0-2016-08-24"></a>

## 0.11.0 / 2016-08-24

### Major Points

DataFrames now enforce knowing full metadata (columns, dtypes) everywhere.
Previously we would operate in an ambiguous state when functions lost dtype
information (such as `apply`).  Now all dataframes always know their dtypes
and raise errors asking for information if they are unable to infer (which
they usually can).  Some internal attributes like `_pd` and
`_pd_nonempty` have been moved.

The internals of the distributed scheduler have been refactored to
transition tasks between explicit states.  This improves resilience,
reasoning about scheduling, plugin operation, and logging.  It also makes
the scheduler code easier to understand for newcomers.

### Breaking Changes

- The `distributed.s3` and `distributed.hdfs` namespaces are gone.  Use
  protocols in normal methods like `read_text('s3://...'` instead.
- `Dask.array.reshape` now errs in some cases where previously it would have
  create a very large number of tasks

<a id="v0-10-2-2016-07-27"></a>

## 0.10.2 / 2016-07-27

- More Dataframe shuffles now work in distributed settings, ranging from
  setting-index to hash joins, to sorted joins and groupbys.
- Dask passes the full test suite when run when under in Python’s
  optimized-OO mode.
- On-disk shuffles were found to produce wrong results in some
  highly-concurrent situations, especially on Windows.  This has been resolved
  by a fix to the partd library.
- Fixed a growth of open file descriptors that occurred under large data
  communications
- Support ports in the `--bokeh-whitelist` option ot dask-scheduler to better
  routing of web interface messages behind non-trivial network settings
- Some improvements to resilience to worker failure (though other known
  failures persist)
- You can now start an IPython kernel on any worker for improved debugging and
  analysis
- Improvements to `dask.dataframe.read_hdf`, especially when reading from
  multiple files and docs

<a id="v0-10-0-2016-06-13"></a>

## 0.10.0 / 2016-06-13

### Major Changes

- This version drops support for Python 2.6
- Conda packages are built and served from conda-forge
- The `dask.distributed` executables have been renamed from dfoo to dask-foo.
  For example dscheduler is renamed to dask-scheduler
- Both Bag and DataFrame include a preliminary distributed shuffle.

### Bag

- Add task-based shuffle for distributed groupbys
- Add accumulate for cumulative reductions

### DataFrame

- Add a task-based shuffle suitable for distributed joins, groupby-applys, and
  set_index operations.  The single-machine shuffle remains untouched (and
  much more efficient.)
- Add support for new Pandas rolling API with improved communication
  performance on distributed systems.
- Add `groupby.std/var`
- Pass through S3/HDFS storage options in `read_csv`
- Improve categorical partitioning
- Add eval, info, isnull, notnull for dataframes

### Distributed

- Rename executables like dscheduler to dask-scheduler
- Improve scheduler performance in the many-fast-tasks case (important for
  shuffling)
- Improve work stealing to be aware of expected function run-times and data
  sizes.  The drastically increases the breadth of algorithms that can be
  efficiently run on the distributed scheduler without significant user
  expertise.
- Support maximum buffer sizes in streaming queues
- Improve Windows support when using the Bokeh diagnostic web interface
- Support compression of very-large-bytestrings in protocol
- Support clean cancellation of submitted futures in Joblib interface

### Other

- All dask-related projects (dask, distributed, s3fs, hdfs, partd) are now
  building conda packages on conda-forge.
- Change credential handling in s3fs to only pass around delegated credentials
  if explicitly given secret/key.  The default now is to rely on managed
  environments.  This can be changed back by explicitly providing a keyword
  argument.  Anonymous mode must be explicitly declared if desired.

<a id="v0-9-0-2016-05-11"></a>

## 0.9.0 / 2016-05-11

### API Changes

- `dask.do` and `dask.value` have been renamed to `dask.delayed`
- `dask.bag.from_filenames` has been renamed to `dask.bag.read_text`
- All S3/HDFS data ingest functions like `db.from_s3` or
  `distributed.s3.read_csv` have been moved into the plain `read_text`,
  `read_csv functions`, which now support protocols, like
  `dd.read_csv('s3://bucket/keys*.csv')`

### Array

- Add support for `scipy.LinearOperator`
- Improve optional locking to on-disk data structures
- Change rechunk to expose the intermediate chunks

### Bag

- Rename `from_filename`s to `read_text`
- Remove `from_s3` in favor of `read_text('s3://...')`

### DataFrame

- Fixed numerical stability issue for correlation and covariance
- Allow no-hash `from_pandas` for speedy round-trips to and from-pandas
  objects
- Generally reengineered `read_csv` to be more in line with Pandas behavior
- Support fast `set_index` operations for sorted columns

### Delayed

- Rename `do/value` to `delayed`
- Rename `to/from_imperative` to `to/from_delayed`

### Distributed

- Move s3 and hdfs functionality into the dask repository
- Adaptively oversubscribe workers for very fast tasks
- Improve PyPy support
- Improve work stealing for unbalanced workers
- Scatter data efficiently with tree-scatters

### Other

- Add lzma/xz compression support
- Raise a warning when trying to split unsplittable compression types, like
  gzip or bz2
- Improve hashing for single-machine shuffle operations
- Add new callback method for start state
- General performance tuning

<a id="v0-8-1-2016-03-11"></a>

## 0.8.1 / 2016-03-11

### Array

- Bugfix for range slicing that could periodically lead to incorrect results.
- Improved support and resiliency of `arg` reductions (`argmin`, `argmax`, etc.)

### Bag

- Add `zip` function

### DataFrame

- Add `corr` and `cov` functions
- Add `melt` function
- Bugfixes for io to bcolz and hdf5

<a id="v0-8-0-2016-02-20"></a>

## 0.8.0 / 2016-02-20

### Array

- Changed default array reduction split from 32 to 4
- Linear algebra, `tril`, `triu`, `LU`, `inv`, `cholesky`,
  `solve`, `solve_triangular`, `eye`, `lstsq`, `diag`, `corrcoef`.

### Bag

- Add tree reductions
- Add range function
- drop `from_hdfs` function (better functionality now exists in hdfs3 and
  distributed projects)

### DataFrame

- Refactor `dask.dataframe` to include a full empty pandas dataframe as
  metadata.  Drop the `.columns` attribute on Series
- Add Series categorical accessor, series.nunique, drop the `.columns`
  attribute for series.
- `read_csv` fixes (multi-column parse_dates, integer column names, etc. )
- Internal changes to improve graph serialization

### Other

- Documentation updates
- Add from_imperative and to_imperative functions for all collections
- Aesthetic changes to profiler plots
- Moved the dask project to a new dask organization

<a id="v0-7-6-2016-01-05"></a>

## 0.7.6 / 2016-01-05

### Array

- Improve thread safety
- Tree reductions
- Add `view`, `compress`, `hstack`, `dstack`, `vstack` methods
- `map_blocks` can now remove and add dimensions

### DataFrame

- Improve thread safety
- Extend sampling to include replacement options

### Imperative

- Removed optimization passes that fused results.

### Core

- Removed `dask.distributed`
- Improved performance of blocked file reading
- Serialization improvements
- Test Python 3.5

<a id="v0-7-4-2015-10-23"></a>

## 0.7.4 / 2015-10-23

This was mostly a bugfix release. Some notable changes:

- Fix minor bugs associated with the release of numpy 1.10 and pandas 0.17
- Fixed a bug with random number generation that would cause repeated blocks
  due to the birthday paradox
- Use locks in `dask.dataframe.read_hdf` by default to avoid concurrency
  issues
- Change `dask.get` to point to `dask.async.get_sync` by default
- Allow visualization functions to accept general graphviz graph options like
  rankdir=’LR’
- Add reshape and ravel to `dask.array`
- Support the creation of `dask.arrays` from `dask.imperative` objects

### Deprecation

This release also includes a deprecation warning for `dask.distributed`, which
will be removed in the next version.

Future development in distributed computing for dask is happening here:
[https://distributed.dask.org](https://distributed.dask.org) . General feedback on that project is most
welcome from this community.

<a id="v0-7-3-2015-09-25"></a>

## 0.7.3 / 2015-09-25

### Diagnostics

- A utility for profiling memory and cpu usage has been added to the
  `dask.diagnostics` module.

### DataFrame

This release improves coverage of the pandas API. Among other things
it includes `nunique`, `nlargest`, `quantile`. Fixes encoding issues
with reading non-ascii csv files. Performance improvements and  bug fixes
with resample. More flexible read_hdf with globbing. And many more. Various
bug fixes in `dask.imperative` and `dask.bag`.

<a id="v0-7-0-2015-08-15"></a>

## 0.7.0 / 2015-08-15

### DataFrame

This release includes significant bugfixes and alignment with the Pandas API.
This has resulted both from use and from recent involvement by Pandas core
developers.

- New operations: query, rolling operations, drop
- Improved operations: quantiles, arithmetic on full dataframes, dropna,
  constructor logic, merge/join, elemwise operations, groupby aggregations

### Bag

- Fixed a bug in fold where with a null default argument

### Array

- New operations: da.fft module, da.image.imread

### Infrastructure

- The array and dataframe collections create graphs with deterministic keys.
  These tend to be longer (hash strings) but should be consistent between
  computations.  This will be useful for caching in the future.
- All collections (Array, Bag, DataFrame) inherit from common subclass

<a id="v0-6-1-2015-07-23"></a>

## 0.6.1 / 2015-07-23

### Distributed

- Improved (though not yet sufficient) resiliency for `dask.distributed`
  when workers die

### DataFrame

- Improved writing to various formats, including to_hdf, to_castra, and
  to_csv
- Improved creation of dask DataFrames from dask Arrays and Bags
- Improved support for categoricals and various other methods

### Array

- Various bug fixes
- Histogram function

### Scheduling

- Added tie-breaking ordering of tasks within parallel workloads to
  better handle and clear intermediate results

### Other

- Added the dask.do function for explicit construction of graphs with
  normal python code
- Traded pydot for graphviz library for graph printing to support Python3
- There is also a gitter chat room and a stackoverflow tag


# cheatsheet.html.md

# Dask Cheat Sheet

The 300KB PDF [`Dask cheat sheet`](daskcheatsheet.pdf)
is a single page summary about using Dask.
It is commonly distributed at conferences and trade shows.


# cli.html.md

# Command Line Interface

Dask provides a `dask` executable for a command line interface.
Dask’s CLI is [designed to be extensible](#extending-cli) allowing
other projects in the Dask ecosystem (such as `distributed`) to
add subcommands.

## Built-in commands

`dask` comes with the following commands.

### dask info

Information about your dask installation.

### Usage

```shell
dask info [OPTIONS] COMMAND [ARGS]...
```

#### versions

Print versions of Dask related projects.

### Usage

```shell
dask info versions [OPTIONS]
```

### dask docs

Open Dask documentation ([https://docs.dask.org/](https://docs.dask.org/)) in a web browser.

### Usage

```shell
dask docs [OPTIONS]
```

<a id="extending-cli"></a>

## Extending the Dask CLI

#### NOTE
This section is intended for library authors who want to
integrate their library with the `dask` CLI.

Third party packages can extend the `dask` command line tool via
entry points and [Click](https://click.palletsprojects.com/). Dask will discover [`click.Command`](https://click.palletsprojects.com/en/stable/api/#click.Command) and
[`click.Group`](https://click.palletsprojects.com/en/stable/api/#click.Group) objects registered as entry points under the
`dask_cli` namespace. Below you’ll find two examples which augment
the `dask` CLI by adding a `dask_cli` entry point to a project.

Click provides great documentation for writing commands; more
documentation on entry points can be found at:

- [The python packaging documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
- [The setuptools user guide](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
- [The poetry plugins documentation](https://python-poetry.org/docs/pyproject/#plugins).

### Example: PEP-621

Since [PEP-621](https://peps.python.org/pep-0621/), if starting a
new project, the canonical way to add an entry point to your Python
project is to use the `[project.entry-points]` table in the
`pyproject.toml` file. This method should be picked up by any Python
build system that is compatible with `PEP-621`’s `project`
configuration. [Hatch](https://github.com/pypa/hatch), [Flit](https://flit.pypa.io/), and [setuptools](https://setuptools.pypa.io/en/latest/index.html) (version 61.0.0 or
later) are three example build systems which are PEP-621 compatible
and use `[project.entry-points]`.

For example, if your project is called `mypackage`, and it contains
a `cli.py` module under the `mypackage` namespace with the
following contents:

```default
# in the file mypackage/cli.py
import click

@click.command(name="mycommand")
@click.argument("name", type=str)
@click.option("-c", "--count", default=1)
def main(name, count):
    for _ in range(count):
        click.echo(f"hello {name} from mycommand!")
```

You can create an entry point that will be discovered by Dask by
adding to `pyproject.toml`:

```toml
[project.entry-points."dask_cli"]
mycommand = "mypackage.cli:main"
```

After installing `mypackage`, the `mycommand` subcommand should be
available to the `dask` CLI:

```shell
$ dask mycommand world
hello world from mycommand!

$ dask mycommand user -c 3
hello user from mycommand!
hello user from mycommand!
hello user from mycommand!
```

### Example: setup.cfg and setup.py

#### NOTE
If you are starting a new project the recommendation from the
Python Packaging Authority ([PyPA](https://pypa.io/)) is to use PEP-621, these
setuptools instructions are provided for existing projects.

If your project already uses `setuptools` with a `setup.cfg` file
and/or a `setup.py` file, we can create an entry point for the same
`mycommand.cli:main` function introduced in the previous section. If
using `setup.cfg`, the entry point can be registered by adding the
following block to the file:

```ini
[options.entry_points]
dask_cli =
    mycommand = mypackage.cli:main
```

Or the entry point can be registered directly in `setup.py` with:

```python
from setuptools import setup

setup(
    ...
    entry_points="""
        [dask_cli]
        mycommand=mypackage.cli:main
    """,
)
```


# configuration.html.md

# Configuration

Taking full advantage of Dask sometimes requires user configuration.
This might be to control logging verbosity, specify cluster configuration,
provide credentials for security, or any of several other options that arise in
production.

Configuration is specified in one of the following ways:

1. YAML files in `~/.config/dask/` or `/etc/dask/`
2. Environment variables like `DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True`
3. Default settings within sub-libraries

This combination makes it easy to specify configuration in a variety of
settings ranging from personal workstations, to IT-mandated configuration, to
docker images.

## Access Configuration

| [`dask.config.get`](#dask.config.get)(key[, default, config, ...])   | Get elements from global config   |
|----------------------------------------------------------------------|-----------------------------------|

Dask’s configuration system is usually accessed using the `dask.config.get` function.
You can use `.` for nested access, for example:

```python
>>> import dask
>>> import dask.distributed  # populate config with distributed defaults

>>> dask.config.get("distributed.client") # use `.` for nested access
{'heartbeat': '5s', 'scheduler-info-interval': '2s'}

>>> dask.config.get("distributed.scheduler.unknown-task-duration")
'500ms'
```

You may wish to inspect the `dask.config.config` dictionary to get a sense
for what configuration is being used by your current system.

Note that the `get` function treats underscores and hyphens identically.
For example, `dask.config.get("temporary-directory")` is equivalent to
`dask.config.get("temporary_directory")`.

Values like `"128 MiB"` and `"10s"` are parsed using the functions in
[Utilities](api.md#api-utilities).

## Specify Configuration

### YAML files

You can specify configuration values in YAML files. For example:

```yaml
array:
  chunk-size: 128 MiB

distributed:
  worker:
    memory:
      spill: 0.85  # default: 0.7
      target: 0.75  # default: 0.6
      terminate: 0.98  # default: 0.95

  dashboard:
    # Locate the dashboard if working on a Jupyter Hub server
    link: /user/<user>/proxy/8787/status
```

These files can live in any of the following locations:

1. The `~/.config/dask` directory in the user’s home directory
2. The `{sys.prefix}/etc/dask` directory local to Python
3. The `{prefix}/etc/dask` directories with `{prefix}` in [site.PREFIXES](https://docs.python.org/3/library/site.html#site.PREFIXES)
4. The root directory (specified by the `DASK_ROOT_CONFIG` environment
   variable or `/etc/dask/` by default)

Dask searches for *all* YAML files within each of these directories and merges
them together, preferring configuration files closer to the user over system
configuration files (preference follows the order in the list above).
Additionally, users can specify a path with the `DASK_CONFIG` environment
variable, which takes precedence at the top of the list above.

The contents of these YAML files are merged together, allowing different
Dask subprojects like `dask-kubernetes` or `dask-ml` to manage configuration
files separately, but have them merge into the same global configuration.

### Dealing with obsolete configuration files

You may have old configuration files with deprecated keys in your file system. Ideally,
you should update them to match the current configuration schema. If that’s not possible
e.g. because you don’t have enough privileges, you can override obsolete keys with the
updated keys in a higher-priority configuration file (as explained in the previous
paragraph). This will silence warnings about deprecated keys. If a key has been outright
removed from the latest schema, you can suppress the warnings by adding it under a
special `deprecated-keys` section of your higher-priority configuration file.

### Environment Variables

You can also specify configuration values with environment variables like
the following:

```bash
export DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=True
export DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES=5
export DASK_DISTRIBUTED__DASHBOARD__LINK="/user/<user>/proxy/8787/status"
```

resulting in configuration values like the following:

```python
{
    'distributed': {
        'scheduler': {
            'work-stealing': True,
            'allowed-failures': 5
        }
    }
}
```

Dask searches for all environment variables that start with `DASK_`, then
transforms keys by converting to lower case and changing double-underscores to
nested structures.

Dask tries to parse all values with [ast.literal_eval](https://docs.python.org/3/library/ast.html#ast.literal_eval), letting users
pass numeric and boolean values (such as `True` in the example above) as well
as lists, dictionaries, and so on with normal Python syntax.

Environment variables take precedence over configuration values found in YAML
files.

### Defaults

Additionally, individual subprojects may add their own default values when they
are imported.  These are always added with lower priority than the YAML files
or environment variables mentioned above:

```python
>>> import dask.config
>>> dask.config.config  # no configuration by default
{}

>>> import dask.distributed
>>> dask.config.config  # New values have been added
{
    'scheduler': ...,
    'worker': ...,
    'tls': ...
}
```

### Directly within Python

| [`dask.config.set`](#dask.config.set)(arg, config, lock, \*\*kwargs)   | Temporarily set configuration values within a context manager   |
|------------------------------------------------------------------------|-----------------------------------------------------------------|

Configuration is stored within a normal Python dictionary in
`dask.config.config` and can be modified using normal Python operations.

Additionally, you can temporarily set a configuration value using the
`dask.config.set` function.  This function accepts a dictionary as an input
and interprets `"."` as nested access:

```python
>>> dask.config.set({'optimization.fuse.ave-width': 4})
```

This function can also be used as a context manager for consistent cleanup:

```python
>>> with dask.config.set({'optimization.fuse.ave-width': 4}):
...     arr2, = dask.optimize(arr)
```

Note that the `set` function treats underscores and hyphens identically.
For example, `dask.config.set({'optimization.fuse.ave_width': 4})` is
equivalent to `dask.config.set({'optimization.fuse.ave-width': 4})`.

Finally, note that persistent objects may acquire configuration settings when
they are initialized. These settings may also be cached for performance reasons.
This is particularly true for `dask.distributed` objects such as Client, Scheduler,
Worker, and Nanny.

### Directly from CLI

Configuration can also be set and viewed from the CLI.

```default
$ dask config set optimization.fuse.ave-width 4
Updated [optimization.fuse.ave-width] to [4], config saved to ~/dask/dask.yaml

$ dask config get optimization.fuse.ave-width
4
```

### Distributing configuration

It may also be desirable to package up your whole Dask configuration for use on
another machine. This is used in some Dask Distributed libraries to ensure remote components
have the same configuration as your local system.

This is typically handled by the downstream libraries which use base64 encoding to pass
config via the `DASK_INTERNAL_INHERIT_CONFIG` environment variable.

| `dask.config.serialize`(data)   | Serialize config data into a string.               |
|---------------------------------|----------------------------------------------------|
| `dask.config.deserialize`(data) | De-serialize config data into the original object. |

### Conversion Utility

It is possible to configure Dask inline with dot notation, with YAML or via environment variables. You can enter
your own configuration items below to convert back and forth.

#### WARNING
This utility is designed to improve understanding of converting between different notations
and does not claim to be a perfect implementation. Please use for reference only.

**YAML**

<textarea id="configConvertUtilYAML" name="configConvertUtilYAML" rows="10" cols="50" class="configTextArea" wrap="off">
array:
   chunk-size: 128 MiB

distributed:
   workers:
      memory:
         spill: 0.85
         target: 0.75
         terminate: 0.98
</textarea>

**Environment variable**

<textarea id="configConvertUtilEnv" name="configConvertUtilEnv" rows="10" cols="50" class="configTextArea" wrap="off">
export DASK_ARRAY_\_CHUNK_SIZE="128 MiB"
export DASK_DISTRIBUTED_\_WORKERS_\_MEMORY_\_SPILL=0.85
export DASK_DISTRIBUTED_\_WORKERS_\_MEMORY_\_TARGET=0.75
export DASK_DISTRIBUTED_\_WORKERS_\_MEMORY_\_TERMINATE=0.98
</textarea>

**Inline with dot notation**

<textarea id="configConvertUtilCode" name="configConvertUtilCode" rows="10" cols="50" class="configTextArea" wrap="off">
>>> dask.config.set({"array.chunk-size": "128 MiB"})
>>> dask.config.set({"distributed.workers.memory.spill": 0.85})
>>> dask.config.set({"distributed.workers.memory.target": 0.75})
>>> dask.config.set({"distributed.workers.memory.terminate": 0.98})
</textarea>

## Updating Configuration

### Manipulating configuration dictionaries

| [`dask.config.merge`](#dask.config.merge)(\*dicts)                                              | Update a sequence of nested dictionaries                   |
|-------------------------------------------------------------------------------------------------|------------------------------------------------------------|
| [`dask.config.update`](#dask.config.update)(old, new[, priority, ...])                          | Update a nested dictionary with values from another        |
| [`dask.config.expand_environment_variables`](#dask.config.expand_environment_variables)(config) | Expand environment variables in a nested config dictionary |

As described above, configuration can come from many places, including several
YAML files, environment variables, and project defaults.  Each of these
provides a configuration that is possibly nested like the following:

```python
x = {'a': 0, 'c': {'d': 4}}
y = {'a': 1, 'b': 2, 'c': {'e': 5}}
```

Dask will merge these configurations respecting nested data structures, and
respecting order:

```python
>>> dask.config.merge(x, y)
{'a': 1, 'b': 2, 'c': {'d': 4, 'e': 5}}
```

You can also use the `update` function to update the existing configuration
in place with a new configuration.  This can be done with priority being given
to either config.  This is often used to update the global configuration in
`dask.config.config`:

```python
dask.config.update(dask.config, new, priority='new')  # Give priority to new values
dask.config.update(dask.config, new, priority='old')  # Give priority to old values
```

Sometimes it is useful to expand environment variables stored within a
configuration.  This can be done with the `expand_environment_variables`
function:

```python
dask.config.config = dask.config.expand_environment_variables(dask.config.config)
```

### Refreshing Configuration

| [`dask.config.collect`](#dask.config.collect)([paths, env])            | Collect configuration from paths and environment variables      |
|------------------------------------------------------------------------|-----------------------------------------------------------------|
| [`dask.config.refresh`](#dask.config.refresh)([config, defaults, ...]) | Update configuration by re-reading yaml files and env variables |

If you change your environment variables or YAML files, Dask will not
immediately see the changes.  Instead, you can call `refresh` to go through
the configuration collection process and update the default configuration:

```python
>>> dask.config.config
{}

>>> # make some changes to yaml files

>>> dask.config.refresh()
>>> dask.config.config
{...}
```

This function uses `dask.config.collect`, which returns the configuration
without modifying the global configuration.  You might use this to determine
the configuration of particular paths not yet on the config path:

```python
>>> dask.config.collect(paths=[...])
{...}
```

## Downstream Libraries

| [`dask.config.ensure_file`](#dask.config.ensure_file)(source[, ...])   | Copy file to default location if it does not already exist   |
|------------------------------------------------------------------------|--------------------------------------------------------------|
| [`dask.config.update`](#dask.config.update)(old, new[, priority, ...]) | Update a nested dictionary with values from another          |
| `dask.config.update_defaults`(new[, config, ...])                      | Add a new set of defaults to the configuration               |

Downstream Dask libraries often follow a standard convention to use the central
Dask configuration.  This section provides recommendations for integration
using a fictional project, `dask-foo`, as an example.

Downstream projects typically follow the following convention:

1. Maintain default configuration in a YAML file within their source
   directory:
   ```default
   setup.py
   dask_foo/__init__.py
   dask_foo/config.py
   dask_foo/core.py
   dask_foo/foo.yaml  # <---
   ```
2. Place configuration in that file within a namespace for the project:
   ```yaml
   # dask_foo/foo.yaml

   foo:
     color: red
     admin:
       a: 1
       b: 2
   ```
3. Within a config.py file (or anywhere) load that default config file and
   update it into the global configuration:
   ```python
   # dask_foo/config.py
   import os
   import yaml

   import dask.config

   fn = os.path.join(os.path.dirname(__file__), 'foo.yaml')

   with open(fn) as f:
       defaults = yaml.safe_load(f)

   dask.config.update_defaults(defaults)
   ```
4. Ensure that this file is run on import by including it in `__init__.py`:
   ```python
   # dask_foo/__init__.py

   from . import config
   ```
5. Within `dask_foo` code, use the `dask.config.get` function to access
   configuration values:
   ```python
   # dask_foo/core.py

   def process(fn, color=dask.config.get('foo.color')):
       ...
   ```
6. You may also want to ensure that your yaml configuration files are included
   in your package.  This can be accomplished by including the following line
   in your MANIFEST.in:
   ```default
   recursive-include <PACKAGE_NAME> *.yaml
   ```

   and the following in your setup.py `setup` call:
   ```python
   from setuptools import setup

   setup(...,
         include_package_data=True,
         ...)
   ```

This process keeps configuration in a central place, but also keeps it safe
within namespaces.  It places config files in an easy to access location
by default (`~/.config/dask/\*.yaml`), so that users can easily discover what
they can change, but maintains the actual defaults within the source code, so
that they more closely track changes in the library.

However, downstream libraries may choose alternative solutions, such as
isolating their configuration within their library, rather than using the
global dask.config system.  All functions in the `dask.config` module also
work with parameters, and do not need to mutate global state.

## API

### dask.config.get(key: str, default: ~typing.Any = <no_default>, config: dict | None = None, override_with: ~typing.Any = None) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Get elements from global config

If `override_with` is not None this value will be passed straight back.
Useful for getting kwarg defaults from Dask config.

Use ‘.’ for nested access

#### SEE ALSO
[`dask.config.set`](#dask.config.set)

### Examples

```pycon
>>> from dask import config
>>> config.get('foo')
{'x': 1, 'y': 2}
```

```pycon
>>> config.get('foo.x')
1
```

```pycon
>>> config.get('foo.x.y', default=123)
123
```

```pycon
>>> config.get('foo.y', override_with=None)
2
```

```pycon
>>> config.get('foo.y', override_with=3)
3
```

<!-- !! processed by numpydoc !! -->

### dask.config.set(arg: Mapping | None = None, config: dict = None, lock: threading.Lock = <unlocked \_thread.lock object>, \*\*kwargs)

Temporarily set configuration values within a context manager

* **Parameters:**
  **arg**
  : A mapping of configuration key-value pairs to set.

  **\*\*kwargs**
  : Additional key-value pairs to set. If `arg` is provided, values set
    in `arg` will be applied before those in `kwargs`.
    Double-underscores (`__`) in keyword arguments will be replaced with
    `.`, allowing nested values to be easily set.

#### SEE ALSO
[`dask.config.get`](#dask.config.get)

### Examples

```pycon
>>> import dask
```

Set `'foo.bar'` in a context, by providing a mapping.

```pycon
>>> with dask.config.set({'foo.bar': 123}):
...     pass
```

Set `'foo.bar'` in a context, by providing a keyword argument.

```pycon
>>> with dask.config.set(foo__bar=123):
...     pass
```

Set `'foo.bar'` globally.

```pycon
>>> dask.config.set(foo__bar=123)
```

<!-- !! processed by numpydoc !! -->

### dask.config.merge(\*dicts: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Update a sequence of nested dictionaries

This prefers the values in the latter dictionaries to those in the former

#### SEE ALSO
[`dask.config.update`](#dask.config.update)

### Examples

```pycon
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'y': {'b': 3}}
>>> merge(a, b)
{'x': 1, 'y': {'a': 2, 'b': 3}}
```

<!-- !! processed by numpydoc !! -->

### dask.config.update(old: [dict](https://docs.python.org/3/library/stdtypes.html#dict), new: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping), priority: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['old', 'new', 'new-defaults'] = 'new', defaults: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Update a nested dictionary with values from another

This is like dict.update except that it smoothly merges nested values

This operates in-place and modifies old

* **Parameters:**
  **priority: string {‘old’, ‘new’, ‘new-defaults’}**
  : If new (default) then the new dictionary has preference.
    Otherwise the old dictionary does.
    If ‘new-defaults’, a mapping should be given of the current defaults.
    Only if a value in `old` matches the current default, it will be
    updated with `new`.

#### SEE ALSO
[`dask.config.merge`](#dask.config.merge)

### Examples

```pycon
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b)
{'x': 2, 'y': {'a': 2, 'b': 3}}
```

```pycon
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'b': 3}}
>>> update(a, b, priority='old')
{'x': 1, 'y': {'a': 2, 'b': 3}}
```

```pycon
>>> d = {'x': 0, 'y': {'a': 2}}
>>> a = {'x': 1, 'y': {'a': 2}}
>>> b = {'x': 2, 'y': {'a': 3, 'b': 3}}
>>> update(a, b, priority='new-defaults', defaults=d)
{'x': 1, 'y': {'a': 3, 'b': 3}}
```

<!-- !! processed by numpydoc !! -->

### dask.config.collect(paths: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] = ['/etc/dask', '/home/docs/checkouts/readthedocs.org/user_builds/dask/envs/stable/etc/dask', '/home/docs/.config/dask'], env: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Collect configuration from paths and environment variables

* **Parameters:**
  **paths**
  : A list of paths to search for yaml config files

  **env**
  : The system environment variables
* **Returns:**
  config: dict

#### SEE ALSO
[`dask.config.refresh`](#dask.config.refresh)
: collect configuration and update into primary config

<!-- !! processed by numpydoc !! -->

### dask.config.refresh(config: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, defaults: [list](https://docs.python.org/3/library/stdtypes.html#list)[[Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)] = [{'admin': {'async-client-fallback': None, 'traceback': {'shorten': ['concurrent[\\\\\\\\\\\\/]futures[\\\\\\\\\\\\/]', 'dask[\\\\\\\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\\\.py', 'dask[\\\\\\\\\\\\/]array[\\\\\\\\\\\\/]core\\\\.py', 'dask[\\\\\\\\\\\\/]dataframe[\\\\\\\\\\\\/](core|methods)\\\\.py', 'dask[\\\\\\\\\\\\/]_task_spec\\\\.py', 'distributed[\\\\\\\\\\\\/](client|scheduler|utils|worker)\\\\.py', 'tornado[\\\\\\\\\\\\/]gen\\\\.py', 'pandas[\\\\\\\\\\\\/]core[\\\\\\\\\\\\/]']}}, 'array': {'backend': 'numpy', 'chunk-size': '128MiB', 'chunk-size-tolerance': 1.25, 'query-planning': None, 'rechunk': {'method': None, 'threshold': 4}, 'slicing': {'split-large-chunks': None}, 'svg': {'size': 120}}, 'dataframe': {'backend': 'pandas', 'convert-string': None, 'parquet': {'metadata-task-size-local': 512, 'metadata-task-size-remote': 1, 'minimum-partition-size': 75000000}, 'query-planning': None, 'shuffle': {'compression': None, 'method': None}}, 'optimization': {'annotations': {'fuse': True}, 'fuse': {'active': None, 'ave-width': 1, 'delayed': False, 'max-depth-new-edges': None, 'max-height': inf, 'max-width': None, 'rename-keys': True}, 'tune': {'active': True}}, 'temporary-directory': None, 'tokenize': {'ensure-deterministic': False}, 'visualization': {'engine': None}}, {'distributed': {'adaptive': {'interval': '1s', 'maximum': inf, 'minimum': 0, 'target-duration': '5s', 'wait-count': 3}, 'admin': {'event-loop': 'tornado', 'large-graph-warning-threshold': '10MB', 'log-format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s', 'log-length': 10000, 'low-level-log-length': 1000, 'max-error-length': 10000, 'pdb-on-err': False, 'system-monitor': {'disk': True, 'gil': {'enabled': True, 'interval': '1ms'}, 'host-cpu': False, 'interval': '500ms', 'log-length': 7200}, 'tick': {'cycle': '1s', 'interval': '20ms', 'limit': '3s'}}, 'client': {'direct-to-workers': None, 'heartbeat': '5s', 'preload': [], 'preload-argv': [], 'scheduler-info-interval': '2s', 'security-loader': None}, 'comm': {'compression': False, 'default-scheme': 'tcp', 'offload': '10MiB', 'require-encryption': None, 'retry': {'count': 0, 'delay': {'max': '20s', 'min': '1s'}}, 'shard': '64MiB', 'socket-backlog': 2048, 'timeouts': {'connect': '30s', 'tcp': '30s'}, 'tls': {'ca-file': None, 'ciphers': None, 'client': {'cert': None, 'key': None}, 'max-version': None, 'min-version': 1.2, 'scheduler': {'cert': None, 'key': None}, 'worker': {'cert': None, 'key': None}}, 'websockets': {'shard': '8MiB'}, 'zstd': {'level': 3, 'threads': 0}}, 'dashboard': {'export-tool': False, 'graph-max-items': 5000, 'link': '{scheme}://{host}:{port}/status', 'prometheus': {'namespace': 'dask'}}, 'deploy': {'cluster-repr-interval': '500ms', 'lost-worker-timeout': '15s'}, 'diagnostics': {'computations': {'ignore-files': ['runpy', '.\*py\\\\.?test.\*', 'pycharm', 'get_output_via_markers'], 'ignore-modules': ['asyncio', 'functools', 'threading', 'datashader', 'dask', 'debugpy', 'distributed', 'ipykernel', 'coiled', 'cudf', 'cuml', 'matplotlib', 'pluggy', 'prefect', 'rechunker', 'xarray', 'xgboost', 'xdist', '_\_channelexec_\_', 'execnet'], 'max-history': 100, 'nframes': 0}, 'cudf': False, 'erred-tasks': {'max-history': 100}, 'nvml': True}, 'nanny': {'environ': {}, 'pre-spawn-environ': {'MALLOC_TRIM_THRESHOLD_': 65536, 'MKL_NUM_THREADS': 1, 'OMP_NUM_THREADS': 1, 'OPENBLAS_NUM_THREADS': 1}, 'preload': [], 'preload-argv': []}, 'p2p': {'comm': {'buffer': '1 GiB', 'concurrency': 10, 'message-bytes-limit': '2 MiB', 'retry': {'count': 10, 'delay': {'max': '30s', 'min': '1s'}}}, 'storage': {'buffer': '100 MiB', 'disk': True}, 'threads': None}, 'rmm': {'pool-size': None}, 'scheduler': {'active-memory-manager': {'interval': '2s', 'measure': 'optimistic', 'policies': [{'class': 'distributed.active_memory_manager.ReduceReplicas'}], 'start': True}, 'allowed-failures': 3, 'allowed-imports': ['dask', 'distributed'], 'bandwidth': 100000000, 'blocked-handlers': [], 'contact-address': None, 'dashboard': {'bokeh-application': {'allow_websocket_origin': ['\*'], 'check_unused_sessions_milliseconds': 500, 'keep_alive_milliseconds': 500}, 'status': {'task-stream-length': 1000}, 'tasks': {'task-stream-length': 100000}, 'tls': {'ca-file': None, 'cert': None, 'key': None}}, 'default-data-size': '1kiB', 'default-task-durations': {'rechunk-split': '1us', 'split-shuffle': '1us', 'split-stage': '1us', 'split-taskshuffle': '1us'}, 'events-cleanup-delay': '1h', 'http': {'routes': ['distributed.http.scheduler.prometheus', 'distributed.http.scheduler.info', 'distributed.http.scheduler.json', 'distributed.http.health', 'distributed.http.proxy', 'distributed.http.statics']}, 'idle-timeout': None, 'locks': {'lease-timeout': '30s', 'lease-validation-interval': '10s'}, 'no-workers-timeout': None, 'preload': [], 'preload-argv': [], 'reuse-broadcast-comm': True, 'rootish-taskgroup': 5, 'rootish-taskgroup-dependencies': 5, 'unknown-task-duration': '500ms', 'validate': False, 'work-stealing': True, 'work-stealing-interval': '1s', 'worker-saturation': 1.1, 'worker-ttl': '5 minutes'}, 'version': 2, 'worker': {'blocked-handlers': [], 'connections': {'incoming': 10, 'outgoing': 50}, 'daemon': True, 'http': {'routes': ['distributed.http.worker.prometheus', 'distributed.http.health', 'distributed.http.statics']}, 'lifetime': {'duration': None, 'restart': False, 'stagger': '0 seconds'}, 'memory': {'max-spill': False, 'monitor-interval': '100ms', 'pause': 0.8, 'rebalance': {'measure': 'optimistic', 'recipient-max': 0.6, 'sender-min': 0.3, 'sender-recipient-gap': 0.1}, 'recent-to-old-time': '30s', 'spill': 0.7, 'spill-compression': 'auto', 'target': 0.6, 'terminate': 0.95, 'transfer': 0.1}, 'multiprocessing-method': 'spawn', 'preload': [], 'preload-argv': [], 'profile': {'cycle': '1000ms', 'enabled': True, 'interval': '10ms', 'low-level': False}, 'resources': {}, 'transfer': {'message-bytes-limit': '50MB'}, 'use-file-locking': True, 'validate': False}}}], paths: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] = ['/etc/dask', '/home/docs/checkouts/readthedocs.org/user_builds/dask/envs/stable/etc/dask', '/home/docs/.config/dask'], env: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Update configuration by re-reading yaml files and env variables

This mutates the global dask.config.config, or the config parameter if
passed in.

This goes through the following stages:

1. Clearing out all old configuration
2. Updating from the stored defaults from downstream libraries
   (see update_defaults)
3. Updating from yaml files and environment variables
4. Automatically renaming deprecated keys (with a warning)

Note that some functionality only checks configuration once at startup and
may not change behavior, even if configuration changes.  It is recommended
to restart your python process if convenient to ensure that new
configuration changes take place.

#### SEE ALSO
[`dask.config.collect`](#dask.config.collect)
: for parameters

`dask.config.update_defaults`

<!-- !! processed by numpydoc !! -->

### dask.config.ensure_file(source: [str](https://docs.python.org/3/library/stdtypes.html#str), destination: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, comment: [bool](https://docs.python.org/3/library/functions.html#bool) = True) → [None](https://docs.python.org/3/library/constants.html#None)

Copy file to default location if it does not already exist

This tries to move a default configuration file to a default location if
if does not already exist.  It also comments out that file by default.

This is to be used by downstream modules (like dask.distributed) that may
have default configuration files that they wish to include in the default
configuration path.

* **Parameters:**
  **source**
  : Source configuration file, typically within a source directory.

  **destination**
  : Destination directory. Configurable by `DASK_CONFIG` environment
    variable, falling back to ~/.config/dask.

  **comment**
  : Whether or not to comment out the config file when copying.

<!-- !! processed by numpydoc !! -->

### dask.config.expand_environment_variables(config: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Expand environment variables in a nested config dictionary

This function will recursively search through any nested dictionaries
and/or lists.

* **Parameters:**
  **config**
  : Input object to search for environment variables
* **Returns:**
  **config**

### Examples

```pycon
>>> expand_environment_variables({'x': [1, 2, '$USER']})
{'x': [1, 2, 'my-username']}
```

<!-- !! processed by numpydoc !! -->

## Configuration Reference

> * [Dask](#dask)
> * [Distributed Client](#distributed-client)
> * [Distributed Comm](#distributed-comm)
> * [Distributed Dashboard](#distributed-dashboard)
> * [Distributed Deploy](#distributed-deploy)
> * [Distributed Scheduler](#distributed-scheduler)
> * [Distributed Worker](#distributed-worker)
> * [Distributed Nanny](#distributed-nanny)
> * [Distributed Admin](#distributed-admin)
> * [Distributed RMM](#distributed-rmm)

#### NOTE
It is possible to configure Dask inline with dot notation, with YAML or via environment variables.
See the [conversion utility](#conversion-utility) for converting the following dot notation to other forms.

### Dask

<dl class="py data">
                     <dt id="temporary-directory">
                       <code class="sig-prename descclassname">temporary-directory</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#temporary-directory" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Temporary directory for local disk storage /tmp, /scratch,
or /local.  This directory is used during dask spill-to-disk operations.

When the value is "null" (default), dask will create a directory from
where dask was launched: \`cwd/dask-worker-space\`</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="visualization.engine">
                       <code class="sig-prename descclassname">visualization.engine</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#visualization.engine" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Visualization engine to use when calling \`\`.visualize()\`\` on a Dask collection.
Currently supports \`\`'graphviz'\`\`, \`\`'ipycytoscape'\`\`, and \`\`'cytoscape'\`\`
(alias for \`\`'ipycytoscape'\`\`)</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="tokenize.ensure-deterministic">
                       <code class="sig-prename descclassname">tokenize.ensure-deterministic</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#tokenize.ensure-deterministic" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>If \`\`true\`\`, tokenize will error instead of falling back to uuids
when a deterministic token cannot be generated. Defaults to
\`\`false\`\`.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.backend">
                       <code class="sig-prename descclassname">dataframe.backend</code>
                       <em class="property">&nbsp;&nbsp;pandas</p></em>
                       <a class="headerlink" href="#dataframe.backend" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Backend to use for supported dataframe-creation functions.
Default is "pandas".</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.shuffle.method">
                       <code class="sig-prename descclassname">dataframe.shuffle.method</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#dataframe.shuffle.method" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The default shuffle method to choose. Possible values are disk,
tasks, p2p. If null, pick best method depending on application.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.shuffle.compression">
                       <code class="sig-prename descclassname">dataframe.shuffle.compression</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#dataframe.shuffle.compression" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Compression algorithm used for on disk-shuffling. Partd, the library used
for compression supports ZLib, BZ2, and SNAPPY</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.parquet.metadata-task-size-local">
                       <code class="sig-prename descclassname">dataframe.parquet.metadata-task-size-local</code>
                       <em class="property">&nbsp;&nbsp;512</p></em>
                       <a class="headerlink" href="#dataframe.parquet.metadata-task-size-local" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The number of files to handle within each metadata-processing
task when reading a parquet dataset from a LOCAL file system.
Specifying 0 will result in serial execution on the client.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.parquet.metadata-task-size-remote">
                       <code class="sig-prename descclassname">dataframe.parquet.metadata-task-size-remote</code>
                       <em class="property">&nbsp;&nbsp;1</p></em>
                       <a class="headerlink" href="#dataframe.parquet.metadata-task-size-remote" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The number of files to handle within each metadata-processing
task when reading a parquet dataset from a REMOTE file system.
Specifying 0 will result in serial execution on the client.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.parquet.minimum-partition-size">
                       <code class="sig-prename descclassname">dataframe.parquet.minimum-partition-size</code>
                       <em class="property">&nbsp;&nbsp;75000000</p></em>
                       <a class="headerlink" href="#dataframe.parquet.minimum-partition-size" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The minimum in-memory size of a single partition after reading
from parquet. Smaller parquet files will be combined into
a single partitions to reach this threshold.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.convert-string">
                       <code class="sig-prename descclassname">dataframe.convert-string</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#dataframe.convert-string" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to convert string-like data to pyarrow strings.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="dataframe.query-planning">
                       <code class="sig-prename descclassname">dataframe.query-planning</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#dataframe.query-planning" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to use query planning.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.backend">
                       <code class="sig-prename descclassname">array.backend</code>
                       <em class="property">&nbsp;&nbsp;numpy</p></em>
                       <a class="headerlink" href="#array.backend" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Backend to use for supported array-creation functions.
Default is "numpy".</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.chunk-size">
                       <code class="sig-prename descclassname">array.chunk-size</code>
                       <em class="property">&nbsp;&nbsp;128MiB</p></em>
                       <a class="headerlink" href="#array.chunk-size" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The default chunk size to target. Default is "128MiB".</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.chunk-size-tolerance">
                       <code class="sig-prename descclassname">array.chunk-size-tolerance</code>
                       <em class="property">&nbsp;&nbsp;1.25</p></em>
                       <a class="headerlink" href="#array.chunk-size-tolerance" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Upper tolerance for different algorithms when creating output chunks.
Default is 1.25. This means that the algorithms can exceed
the average input chunk size along this dimension by 25%.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.rechunk.method">
                       <code class="sig-prename descclassname">array.rechunk.method</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#array.rechunk.method" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The method to use for rechunking. Possible values are tasks or p2p.
If null, pick best method depending on application.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.rechunk.threshold">
                       <code class="sig-prename descclassname">array.rechunk.threshold</code>
                       <em class="property">&nbsp;&nbsp;4</p></em>
                       <a class="headerlink" href="#array.rechunk.threshold" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The graph growth factor above which task-based shuffling introduces
an intermediate step.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.svg.size">
                       <code class="sig-prename descclassname">array.svg.size</code>
                       <em class="property">&nbsp;&nbsp;120</p></em>
                       <a class="headerlink" href="#array.svg.size" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The size of pixels used when displaying a dask array as an SVG image.
This is used, for example, for nice rendering in a Jupyter notebook</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.slicing.split-large-chunks">
                       <code class="sig-prename descclassname">array.slicing.split-large-chunks</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#array.slicing.split-large-chunks" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>How to handle large chunks created when slicing Arrays. By default a
warning is produced. Set to \`\`False\`\` to silence the warning
and allow large output chunks. Set to \`\`True\`\` to silence the
warning and avoid large output chunks.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="array.query-planning">
                       <code class="sig-prename descclassname">array.query-planning</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#array.query-planning" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to use query planning for arrays.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.annotations.fuse">
                       <code class="sig-prename descclassname">optimization.annotations.fuse</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#optimization.annotations.fuse" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>If adjacent blockwise layers have different annotations (e.g., one has
retries=3 and another has retries=4), Dask can make an attempt to merge
those annotations according to some simple rules. \`\`retries\`\` is set to
the max of the layers, \`\`priority\`\` is set to the max of the layers,
\`\`resources\`\` are set to the max of all the resources, \`\`workers\`\` is
set to the intersection of the requested workers. If this setting is
disabled, then adjacent blockwise layers with different annotations
will \*not\* be fused.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.tune.active">
                       <code class="sig-prename descclassname">optimization.tune.active</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#optimization.tune.active" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Enable/disable the performance-tuning optimization stage. When active,
the optimizer may adjust the partition count based on heuristics. Set
this configuration to False to prevent the partition count from changing
at optimization time. Default is True.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.active">
                       <code class="sig-prename descclassname">optimization.fuse.active</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#optimization.fuse.active" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Turn task fusion on/off. This option refers to the fusion of a
fully-materialized task graph (not a high-Level graph). By default
(None), the active task-fusion option will be treated as \`\`False\`\`
for Dask-Dataframe collections, and as \`\`True\`\` for all other graphs
(including Dask-Array collections).</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.ave-width">
                       <code class="sig-prename descclassname">optimization.fuse.ave-width</code>
                       <em class="property">&nbsp;&nbsp;1</p></em>
                       <a class="headerlink" href="#optimization.fuse.ave-width" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Upper limit for width, where width = num_nodes / height, a good measure of parallelizability</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.max-width">
                       <code class="sig-prename descclassname">optimization.fuse.max-width</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#optimization.fuse.max-width" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Don't fuse if total width is greater than this. Set to null to dynamically adjust to 1.5 + ave_width \* log(ave_width + 1)</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.max-height">
                       <code class="sig-prename descclassname">optimization.fuse.max-height</code>
                       <em class="property">&nbsp;&nbsp;inf</p></em>
                       <a class="headerlink" href="#optimization.fuse.max-height" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Don't fuse more than this many levels</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.max-depth-new-edges">
                       <code class="sig-prename descclassname">optimization.fuse.max-depth-new-edges</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#optimization.fuse.max-depth-new-edges" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Don't fuse if new dependencies are added after this many levels. Set to null to dynamically adjust to ave_width \* 1.5.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.rename-keys">
                       <code class="sig-prename descclassname">optimization.fuse.rename-keys</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#optimization.fuse.rename-keys" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Set to true to rename the fused keys with \`default_fused_keys_renamer\`. Renaming fused keys can keep the graph more understandable and comprehensible, but it comes at the cost of additional processing. If False, then the top-most key will be used. For advanced usage, a function to create the new name is also accepted.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="optimization.fuse.delayed">
                       <code class="sig-prename descclassname">optimization.fuse.delayed</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#optimization.fuse.delayed" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to fuse adjacent delayed calls together. This can be useful to ensure
that an independent chain of delayed calls is executed in a single task to
reduce the memory footprint on a single worker.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="admin.async-client-fallback">
                       <code class="sig-prename descclassname">admin.async-client-fallback</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#admin.async-client-fallback" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>If not null, replace an asynchronous Client in get_scheduler/compute
with the provided scheduler type instead.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="admin.traceback.shorten">
                       <code class="sig-prename descclassname">admin.traceback.shorten</code>
                       <em class="property">&nbsp;&nbsp;['concurrent[\\\\\\\\\\\\/]futures[\\\\\\\\\\\\/]', 'dask[\\\\\\\\\\\\/](base|core|local|multiprocessing|optimization|threaded|utils)\\\\.py', 'dask[\\\\\\\\\\\\/]array[\\\\\\\\\\\\/]core\\\\.py', 'dask[\\\\\\\\\\\\/]dataframe[\\\\\\\\\\\\/](core|methods)\\\\.py', 'dask[\\\\\\\\\\\\/]_task_spec\\\\.py', 'distributed[\\\\\\\\\\\\/](client|scheduler|utils|worker)\\\\.py', 'tornado[\\\\\\\\\\\\/]gen\\\\.py', 'pandas[\\\\\\\\\\\\/]core[\\\\\\\\\\\\/]']</p></em>
                       <a class="headerlink" href="#admin.traceback.shorten" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Clean up Dask tracebacks for readability. Remove all modules that match one of the listed regular expressions. Always preserve the first and last frame.</p></dd>
                    </dl>

                ### Distributed Client

<dl class="py data">
                     <dt id="distributed.client.direct-to-workers">
                       <code class="sig-prename descclassname">distributed.client.direct-to-workers</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.client.direct-to-workers" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to connect directly to workers for gather / scatter.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.client.heartbeat">
                       <code class="sig-prename descclassname">distributed.client.heartbeat</code>
                       <em class="property">&nbsp;&nbsp;5s</p></em>
                       <a class="headerlink" href="#distributed.client.heartbeat" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>This value is the time between heartbeats
The client sends a periodic heartbeat message to the scheduler. If it misses enough of these then the scheduler assumes that it has gone.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.client.scheduler-info-interval">
                       <code class="sig-prename descclassname">distributed.client.scheduler-info-interval</code>
                       <em class="property">&nbsp;&nbsp;2s</p></em>
                       <a class="headerlink" href="#distributed.client.scheduler-info-interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Interval between scheduler-info updates</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.client.security-loader">
                       <code class="sig-prename descclassname">distributed.client.security-loader</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.client.security-loader" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A fully qualified name (e.g. \`\`module.submodule.function\`\`) of
a callback to use for loading security credentials for the
client. If no security object is explicitly passed when creating
a \`\`Client\`\`, this callback is called with a dict containing
client information (currently just \`\`address\`\`), and should
return a \`\`Security\`\` object to use for this client, or \`\`None\`\`
to fallback to the default security configuration.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.client.preload">
                       <code class="sig-prename descclassname">distributed.client.preload</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.client.preload" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Run custom modules during the lifetime of the client

You can run custom modules when the client starts up and closes down.
See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.client.preload-argv">
                       <code class="sig-prename descclassname">distributed.client.preload-argv</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.client.preload-argv" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Arguments to pass into the preload scripts described above

See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                ### Distributed Comm

<dl class="py data">
                     <dt id="distributed.comm.retry.count">
                       <code class="sig-prename descclassname">distributed.comm.retry.count</code>
                       <em class="property">&nbsp;&nbsp;0</p></em>
                       <a class="headerlink" href="#distributed.comm.retry.count" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The number of times to retry a connection</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.retry.delay.min">
                       <code class="sig-prename descclassname">distributed.comm.retry.delay.min</code>
                       <em class="property">&nbsp;&nbsp;1s</p></em>
                       <a class="headerlink" href="#distributed.comm.retry.delay.min" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The first non-zero delay between retry attempts</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.retry.delay.max">
                       <code class="sig-prename descclassname">distributed.comm.retry.delay.max</code>
                       <em class="property">&nbsp;&nbsp;20s</p></em>
                       <a class="headerlink" href="#distributed.comm.retry.delay.max" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum delay between retries</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.compression">
                       <code class="sig-prename descclassname">distributed.comm.compression</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.comm.compression" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The compression algorithm to use. 'auto' defaults to lz4 if installed, otherwise to snappy if installed, otherwise to false. zlib and zstd are only used if explicitly requested here. Uncompressible data and transfers on localhost are always uncompressed, regardless of this setting. See also distributed.worker.memory.spill-compression.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.shard">
                       <code class="sig-prename descclassname">distributed.comm.shard</code>
                       <em class="property">&nbsp;&nbsp;64MiB</p></em>
                       <a class="headerlink" href="#distributed.comm.shard" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum size of a frame to send through a comm

Some network infrastructure doesn't like sending through very large messages.
Dask comms will cut up these large messages into many small ones.
This attribute determines the maximum size of such a shard.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.offload">
                       <code class="sig-prename descclassname">distributed.comm.offload</code>
                       <em class="property">&nbsp;&nbsp;10MiB</p></em>
                       <a class="headerlink" href="#distributed.comm.offload" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The size of message after which we choose to offload serialization to another thread

In some cases, you may also choose to disable this altogether with the value false
This is useful if you want to include serialization in profiling data,
or if you have data types that are particularly sensitive to deserialization</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.default-scheme">
                       <code class="sig-prename descclassname">distributed.comm.default-scheme</code>
                       <em class="property">&nbsp;&nbsp;tcp</p></em>
                       <a class="headerlink" href="#distributed.comm.default-scheme" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The default protocol to use, like tcp or tls</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.socket-backlog">
                       <code class="sig-prename descclassname">distributed.comm.socket-backlog</code>
                       <em class="property">&nbsp;&nbsp;2048</p></em>
                       <a class="headerlink" href="#distributed.comm.socket-backlog" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When shuffling data between workers, there can
really be O(cluster size) connection requests
on a single worker socket, make sure the backlog
is large enough not to lose any.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.zstd.level">
                       <code class="sig-prename descclassname">distributed.comm.zstd.level</code>
                       <em class="property">&nbsp;&nbsp;3</p></em>
                       <a class="headerlink" href="#distributed.comm.zstd.level" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Compression level, between 1 and 22.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.zstd.threads">
                       <code class="sig-prename descclassname">distributed.comm.zstd.threads</code>
                       <em class="property">&nbsp;&nbsp;0</p></em>
                       <a class="headerlink" href="#distributed.comm.zstd.threads" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Number of threads to use.

0 for single-threaded, -1 to infer from cpu count.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.timeouts.connect">
                       <code class="sig-prename descclassname">distributed.comm.timeouts.connect</code>
                       <em class="property">&nbsp;&nbsp;30s</p></em>
                       <a class="headerlink" href="#distributed.comm.timeouts.connect" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.timeouts.tcp">
                       <code class="sig-prename descclassname">distributed.comm.timeouts.tcp</code>
                       <em class="property">&nbsp;&nbsp;30s</p></em>
                       <a class="headerlink" href="#distributed.comm.timeouts.tcp" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.require-encryption">
                       <code class="sig-prename descclassname">distributed.comm.require-encryption</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.require-encryption" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to require encryption on non-local comms</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.ciphers">
                       <code class="sig-prename descclassname">distributed.comm.tls.ciphers</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.ciphers" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Allowed ciphers, specified as an OpenSSL cipher string.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.min-version">
                       <code class="sig-prename descclassname">distributed.comm.tls.min-version</code>
                       <em class="property">&nbsp;&nbsp;1.2</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.min-version" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The minimum TLS version to support. Defaults to TLS 1.2.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.max-version">
                       <code class="sig-prename descclassname">distributed.comm.tls.max-version</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.max-version" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum TLS version to support. Defaults to the maximum
version supported by the platform.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.ca-file">
                       <code class="sig-prename descclassname">distributed.comm.tls.ca-file</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.ca-file" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to a CA file, in pem format</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.scheduler.cert">
                       <code class="sig-prename descclassname">distributed.comm.tls.scheduler.cert</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.scheduler.cert" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to certificate file</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.scheduler.key">
                       <code class="sig-prename descclassname">distributed.comm.tls.scheduler.key</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.scheduler.key" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to key file.

Alternatively, the key can be appended to the cert file
above, and this field left blank</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.worker.key">
                       <code class="sig-prename descclassname">distributed.comm.tls.worker.key</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.worker.key" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to key file.

Alternatively, the key can be appended to the cert file
above, and this field left blank</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.worker.cert">
                       <code class="sig-prename descclassname">distributed.comm.tls.worker.cert</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.worker.cert" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to certificate file</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.client.key">
                       <code class="sig-prename descclassname">distributed.comm.tls.client.key</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.client.key" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to key file.

Alternatively, the key can be appended to the cert file
above, and this field left blank</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.tls.client.cert">
                       <code class="sig-prename descclassname">distributed.comm.tls.client.cert</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.comm.tls.client.cert" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Path to certificate file</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.comm.websockets.shard">
                       <code class="sig-prename descclassname">distributed.comm.websockets.shard</code>
                       <em class="property">&nbsp;&nbsp;8MiB</p></em>
                       <a class="headerlink" href="#distributed.comm.websockets.shard" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum size of a websocket frame to send through a comm.

This is somewhat duplicative of distributed.comm.shard, but websockets
often have much smaller maximum message sizes than other protocols, so
this attribute is used to set a smaller default shard size and to
allow separate control of websocket message sharding.</p></dd>
                    </dl>

                ### Distributed Dashboard

<dl class="py data">
                     <dt id="distributed.dashboard.link">
                       <code class="sig-prename descclassname">distributed.dashboard.link</code>
                       <em class="property">&nbsp;&nbsp;{scheme}://{host}:{port}/status</p></em>
                       <a class="headerlink" href="#distributed.dashboard.link" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The form for the dashboard links

This is used wherever we print out the link for the dashboard
It is filled in with relevant information like the schema, host, and port number</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.dashboard.export-tool">
                       <code class="sig-prename descclassname">distributed.dashboard.export-tool</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.dashboard.export-tool" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.dashboard.graph-max-items">
                       <code class="sig-prename descclassname">distributed.dashboard.graph-max-items</code>
                       <em class="property">&nbsp;&nbsp;5000</p></em>
                       <a class="headerlink" href="#distributed.dashboard.graph-max-items" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>maximum number of tasks to try to plot in "graph" view</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.dashboard.prometheus.namespace">
                       <code class="sig-prename descclassname">distributed.dashboard.prometheus.namespace</code>
                       <em class="property">&nbsp;&nbsp;dask</p></em>
                       <a class="headerlink" href="#distributed.dashboard.prometheus.namespace" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Namespace prefix to use for all prometheus metrics.</p></dd>
                    </dl>

                ### Distributed Deploy

<dl class="py data">
                     <dt id="distributed.deploy.lost-worker-timeout">
                       <code class="sig-prename descclassname">distributed.deploy.lost-worker-timeout</code>
                       <em class="property">&nbsp;&nbsp;15s</p></em>
                       <a class="headerlink" href="#distributed.deploy.lost-worker-timeout" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Interval after which to hard-close a lost worker job

Otherwise we wait for a while to see if a worker will reappear</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.deploy.cluster-repr-interval">
                       <code class="sig-prename descclassname">distributed.deploy.cluster-repr-interval</code>
                       <em class="property">&nbsp;&nbsp;500ms</p></em>
                       <a class="headerlink" href="#distributed.deploy.cluster-repr-interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Interval between calls to update cluster-repr for the widget</p></dd>
                    </dl>

                ### Distributed Scheduler

<dl class="py data">
                     <dt id="distributed.scheduler.allowed-failures">
                       <code class="sig-prename descclassname">distributed.scheduler.allowed-failures</code>
                       <em class="property">&nbsp;&nbsp;3</p></em>
                       <a class="headerlink" href="#distributed.scheduler.allowed-failures" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The number of retries before a task is considered bad

When a worker dies when a task is running that task is rerun elsewhere.
If many workers die while running this same task then we call the task bad, and raise a KilledWorker exception.
This is the number of workers that are allowed to die before this task is marked as bad.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.bandwidth">
                       <code class="sig-prename descclassname">distributed.scheduler.bandwidth</code>
                       <em class="property">&nbsp;&nbsp;100000000</p></em>
                       <a class="headerlink" href="#distributed.scheduler.bandwidth" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The expected bandwidth between any pair of workers

This is used when making scheduling decisions.
The scheduler will use this value as a baseline, but also learn it over time.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.blocked-handlers">
                       <code class="sig-prename descclassname">distributed.scheduler.blocked-handlers</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.scheduler.blocked-handlers" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A list of handlers to exclude

The scheduler operates by receiving messages from various workers and clients
and then performing operations based on those messages.
Each message has an operation like "close-worker" or "task-finished".
In some high security situations administrators may choose to block certain handlers
from running.  Those handlers can be listed here.

For a list of handlers see the \`dask.distributed.Scheduler.handlers\` attribute.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.contact-address">
                       <code class="sig-prename descclassname">distributed.scheduler.contact-address</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.contact-address" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The address that the scheduler advertises to workers for communication with it.

To be specified when the address to which the scheduler binds cannot be the same
as the address that workers use to contact the scheduler (e.g. because the former
is private and the scheduler is in a different network than the workers).</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.default-data-size">
                       <code class="sig-prename descclassname">distributed.scheduler.default-data-size</code>
                       <em class="property">&nbsp;&nbsp;1kiB</p></em>
                       <a class="headerlink" href="#distributed.scheduler.default-data-size" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The default size of a piece of data if we don't know anything about it.

This is used by the scheduler in some scheduling decisions</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.reuse-broadcast-comm">
                       <code class="sig-prename descclassname">distributed.scheduler.reuse-broadcast-comm</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.scheduler.reuse-broadcast-comm" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether to reuse the Scheduler to Worker Comm for repeated broadcasts.

This can be useful to avoid the overhead of creating and destroying Comms when sending multiple
small messages in methods like \`\`Client.run\`\`. The scheduler will persist an open
Comm object for each worker. Set this to False if you want to close the Comm after each broadcast.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.events-cleanup-delay">
                       <code class="sig-prename descclassname">distributed.scheduler.events-cleanup-delay</code>
                       <em class="property">&nbsp;&nbsp;1h</p></em>
                       <a class="headerlink" href="#distributed.scheduler.events-cleanup-delay" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The amount of time to wait until workers or clients are removed from the event log
after they have been removed from the scheduler</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.idle-timeout">
                       <code class="sig-prename descclassname">distributed.scheduler.idle-timeout</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.idle-timeout" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Shut down the scheduler after this duration if no activity has occurred</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.no-workers-timeout">
                       <code class="sig-prename descclassname">distributed.scheduler.no-workers-timeout</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.no-workers-timeout" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Timeout for tasks in an unrunnable state.

If task remains unrunnable for longer than this, it fails. A task is considered unrunnable IFF
it has no pending dependencies, and the task has restrictions that are not satisfied by 
any available worker or no workers are running at all.

In adaptive clusters, this timeout must be set to be safely higher than
the time it takes for workers to spin up.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.work-stealing">
                       <code class="sig-prename descclassname">distributed.scheduler.work-stealing</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.scheduler.work-stealing" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to balance work between workers dynamically

Some times one worker has more work than we expected.
The scheduler will move these tasks around as necessary by default.
Set this to false to disable this behavior</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.work-stealing-interval">
                       <code class="sig-prename descclassname">distributed.scheduler.work-stealing-interval</code>
                       <em class="property">&nbsp;&nbsp;1s</p></em>
                       <a class="headerlink" href="#distributed.scheduler.work-stealing-interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>How frequently to balance worker loads</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.worker-saturation">
                       <code class="sig-prename descclassname">distributed.scheduler.worker-saturation</code>
                       <em class="property">&nbsp;&nbsp;1.1</p></em>
                       <a class="headerlink" href="#distributed.scheduler.worker-saturation" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Controls how many root tasks are sent to workers (like a \`readahead\`).

Up to worker-saturation \* nthreads root tasks are sent to a
worker at a time. If \`.inf\`, all runnable tasks are immediately sent to workers.
The target number is rounded up, so any \`worker-saturation\` value > 1.0 guarantees
at least one extra task will be sent to workers.

Allowing oversaturation (> 1.0) means a worker may start running a new root task as
soon as it completes the previous, even if there is a higher-priority downstream task
to run. This reduces worker idleness, by letting workers do something while waiting for
further instructions from the scheduler, even if it's not the most efficient
thing to do.

This generally comes  at the expense of increased memory usage. It leads to "wider"
(more breadth-first) execution of the graph.

Compute-bound workloads may benefit from oversaturation. Memory-bound workloads should
generally leave \`worker-saturation\` at 1.0, though 1.25-1.5 could slightly improve
performance if ample memory is available.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.rootish-taskgroup">
                       <code class="sig-prename descclassname">distributed.scheduler.rootish-taskgroup</code>
                       <em class="property">&nbsp;&nbsp;5</p></em>
                       <a class="headerlink" href="#distributed.scheduler.rootish-taskgroup" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Controls when a specific task group is identified as rootish when
worker saturation is set.

A task group is identified as rootish if it has only up to a certain number
of dependencies (5 by default). This can be faulty for very large datasets
where the number of data tasks from xarray can be higher than 5.

Increasing this limit will capture these root tasks successfully but increase
the risk of misidentifying task groups as rootish, which can have
performance implications.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.rootish-taskgroup-dependencies">
                       <code class="sig-prename descclassname">distributed.scheduler.rootish-taskgroup-dependencies</code>
                       <em class="property">&nbsp;&nbsp;5</p></em>
                       <a class="headerlink" href="#distributed.scheduler.rootish-taskgroup-dependencies" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Controls the number of transitive dependencies a task group can have to be considered rootish.
It checks the number of dependencies each dependency of a rootish task group has.

The same caveats as for \`rootish-taskgroup\` apply.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.worker-ttl">
                       <code class="sig-prename descclassname">distributed.scheduler.worker-ttl</code>
                       <em class="property">&nbsp;&nbsp;5 minutes</p></em>
                       <a class="headerlink" href="#distributed.scheduler.worker-ttl" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Time to live for workers.

If we don't receive a heartbeat faster than this then we assume that the worker has died.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.preload">
                       <code class="sig-prename descclassname">distributed.scheduler.preload</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.scheduler.preload" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Run custom modules during the lifetime of the scheduler

You can run custom modules when the scheduler starts up and closes down.
See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.preload-argv">
                       <code class="sig-prename descclassname">distributed.scheduler.preload-argv</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.scheduler.preload-argv" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Arguments to pass into the preload scripts described above

See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.unknown-task-duration">
                       <code class="sig-prename descclassname">distributed.scheduler.unknown-task-duration</code>
                       <em class="property">&nbsp;&nbsp;500ms</p></em>
                       <a class="headerlink" href="#distributed.scheduler.unknown-task-duration" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Default duration for all tasks with unknown durations

Over time the scheduler learns a duration for tasks.
However when it sees a new type of task for the first time it has to make a guess
as to how long it will take.  This value is that guess.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.default-task-durations.rechunk-split">
                       <code class="sig-prename descclassname">distributed.scheduler.default-task-durations.rechunk-split</code>
                       <em class="property">&nbsp;&nbsp;1us</p></em>
                       <a class="headerlink" href="#distributed.scheduler.default-task-durations.rechunk-split" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.default-task-durations.split-shuffle">
                       <code class="sig-prename descclassname">distributed.scheduler.default-task-durations.split-shuffle</code>
                       <em class="property">&nbsp;&nbsp;1us</p></em>
                       <a class="headerlink" href="#distributed.scheduler.default-task-durations.split-shuffle" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.default-task-durations.split-taskshuffle">
                       <code class="sig-prename descclassname">distributed.scheduler.default-task-durations.split-taskshuffle</code>
                       <em class="property">&nbsp;&nbsp;1us</p></em>
                       <a class="headerlink" href="#distributed.scheduler.default-task-durations.split-taskshuffle" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.default-task-durations.split-stage">
                       <code class="sig-prename descclassname">distributed.scheduler.default-task-durations.split-stage</code>
                       <em class="property">&nbsp;&nbsp;1us</p></em>
                       <a class="headerlink" href="#distributed.scheduler.default-task-durations.split-stage" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.validate">
                       <code class="sig-prename descclassname">distributed.scheduler.validate</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.scheduler.validate" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to run consistency checks during execution.
This is typically only used for debugging.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.status.task-stream-length">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.status.task-stream-length</code>
                       <em class="property">&nbsp;&nbsp;1000</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.status.task-stream-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum number of tasks to include in the task stream plot</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.tasks.task-stream-length">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.tasks.task-stream-length</code>
                       <em class="property">&nbsp;&nbsp;100000</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.tasks.task-stream-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum number of tasks to include in the task stream plot</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.tls.ca-file">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.tls.ca-file</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.tls.ca-file" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.tls.key">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.tls.key</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.tls.key" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.tls.cert">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.tls.cert</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.tls.cert" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.bokeh-application.allow_websocket_origin">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.bokeh-application.allow_websocket_origin</code>
                       <em class="property">&nbsp;&nbsp;['\*']</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.bokeh-application.allow_websocket_origin" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.bokeh-application.keep_alive_milliseconds">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.bokeh-application.keep_alive_milliseconds</code>
                       <em class="property">&nbsp;&nbsp;500</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.bokeh-application.keep_alive_milliseconds" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.dashboard.bokeh-application.check_unused_sessions_milliseconds">
                       <code class="sig-prename descclassname">distributed.scheduler.dashboard.bokeh-application.check_unused_sessions_milliseconds</code>
                       <em class="property">&nbsp;&nbsp;500</p></em>
                       <a class="headerlink" href="#distributed.scheduler.dashboard.bokeh-application.check_unused_sessions_milliseconds" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.locks.lease-validation-interval">
                       <code class="sig-prename descclassname">distributed.scheduler.locks.lease-validation-interval</code>
                       <em class="property">&nbsp;&nbsp;10s</p></em>
                       <a class="headerlink" href="#distributed.scheduler.locks.lease-validation-interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The interval in which the scheduler validates staleness of all acquired leases. Must always be smaller than the lease-timeout itself.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.locks.lease-timeout">
                       <code class="sig-prename descclassname">distributed.scheduler.locks.lease-timeout</code>
                       <em class="property">&nbsp;&nbsp;30s</p></em>
                       <a class="headerlink" href="#distributed.scheduler.locks.lease-timeout" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Maximum interval to wait for a Client refresh before a lease is invalidated and released.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.http.routes">
                       <code class="sig-prename descclassname">distributed.scheduler.http.routes</code>
                       <em class="property">&nbsp;&nbsp;['distributed.http.scheduler.prometheus', 'distributed.http.scheduler.info', 'distributed.http.scheduler.json', 'distributed.http.health', 'distributed.http.proxy', 'distributed.http.statics']</p></em>
                       <a class="headerlink" href="#distributed.scheduler.http.routes" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A list of modules like "prometheus" and "health" that can be included or excluded as desired

These modules will have a \`\`routes\`\` keyword that gets added to the main HTTP Server.
This is also a list that can be extended with user defined modules.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.allowed-imports">
                       <code class="sig-prename descclassname">distributed.scheduler.allowed-imports</code>
                       <em class="property">&nbsp;&nbsp;['dask', 'distributed']</p></em>
                       <a class="headerlink" href="#distributed.scheduler.allowed-imports" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A list of trusted root modules the scheduler is allowed to import (incl. submodules). For security reasons, the
scheduler does not import arbitrary Python modules.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.active-memory-manager.start">
                       <code class="sig-prename descclassname">distributed.scheduler.active-memory-manager.start</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.scheduler.active-memory-manager.start" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>set to true to auto-start the AMM on Scheduler init</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.active-memory-manager.interval">
                       <code class="sig-prename descclassname">distributed.scheduler.active-memory-manager.interval</code>
                       <em class="property">&nbsp;&nbsp;2s</p></em>
                       <a class="headerlink" href="#distributed.scheduler.active-memory-manager.interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Time expression, e.g. "2s". Run the AMM cycle every <interval>.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.active-memory-manager.measure">
                       <code class="sig-prename descclassname">distributed.scheduler.active-memory-manager.measure</code>
                       <em class="property">&nbsp;&nbsp;optimistic</p></em>
                       <a class="headerlink" href="#distributed.scheduler.active-memory-manager.measure" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>One of the attributes of distributed.scheduler.MemoryState</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.scheduler.active-memory-manager.policies">
                       <code class="sig-prename descclassname">distributed.scheduler.active-memory-manager.policies</code>
                       <em class="property">&nbsp;&nbsp;[{'class': 'distributed.active_memory_manager.ReduceReplicas'}]</p></em>
                       <a class="headerlink" href="#distributed.scheduler.active-memory-manager.policies" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                ### Distributed Worker

<dl class="py data">
                     <dt id="distributed.worker.blocked-handlers">
                       <code class="sig-prename descclassname">distributed.worker.blocked-handlers</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.worker.blocked-handlers" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A list of handlers to exclude

The scheduler operates by receiving messages from various workers and clients
and then performing operations based on those messages.
Each message has an operation like "close-worker" or "task-finished".
In some high security situations administrators may choose to block certain handlers
from running.  Those handlers can be listed here.

For a list of handlers see the \`dask.distributed.Scheduler.handlers\` attribute.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.multiprocessing-method">
                       <code class="sig-prename descclassname">distributed.worker.multiprocessing-method</code>
                       <em class="property">&nbsp;&nbsp;spawn</p></em>
                       <a class="headerlink" href="#distributed.worker.multiprocessing-method" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>How we create new workers, one of "spawn", "forkserver", or "fork"

This is passed to the \`\`multiprocessing.get_context\`\` function.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.use-file-locking">
                       <code class="sig-prename descclassname">distributed.worker.use-file-locking</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.worker.use-file-locking" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to use lock files when creating workers

Workers create a local directory in which to place temporary files.
When many workers are created on the same process at once
these workers can conflict with each other by trying to create this directory all at the same time.

To avoid this, Dask usually used a file-based lock.
However, on some systems file-based locks don't work.
This is particularly common on HPC NFS systems, where users may want to set this to false.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.transfer.message-bytes-limit">
                       <code class="sig-prename descclassname">distributed.worker.transfer.message-bytes-limit</code>
                       <em class="property">&nbsp;&nbsp;50MB</p></em>
                       <a class="headerlink" href="#distributed.worker.transfer.message-bytes-limit" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The maximum amount of data for a worker to request from another in a single gather operation

Tasks are gathered in batches, and if the first task in a batch is larger than this value,
the task will still be gathered to ensure progress. Hence, this limit is not absolute.
Note that this limit applies to a single gather operation and a worker may gather data from
multiple workers in parallel.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.connections.outgoing">
                       <code class="sig-prename descclassname">distributed.worker.connections.outgoing</code>
                       <em class="property">&nbsp;&nbsp;50</p></em>
                       <a class="headerlink" href="#distributed.worker.connections.outgoing" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.connections.incoming">
                       <code class="sig-prename descclassname">distributed.worker.connections.incoming</code>
                       <em class="property">&nbsp;&nbsp;10</p></em>
                       <a class="headerlink" href="#distributed.worker.connections.incoming" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.preload">
                       <code class="sig-prename descclassname">distributed.worker.preload</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.worker.preload" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Run custom modules during the lifetime of the worker

You can run custom modules when the worker starts up and closes down.
See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.preload-argv">
                       <code class="sig-prename descclassname">distributed.worker.preload-argv</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.worker.preload-argv" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Arguments to pass into the preload scripts described above

See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.daemon">
                       <code class="sig-prename descclassname">distributed.worker.daemon</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.worker.daemon" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to run our process as a daemon process</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.validate">
                       <code class="sig-prename descclassname">distributed.worker.validate</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.worker.validate" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to run consistency checks during execution.
This is typically only used for debugging.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.lifetime.duration">
                       <code class="sig-prename descclassname">distributed.worker.lifetime.duration</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.worker.lifetime.duration" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time after creation to close the worker, like "1 hour"</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.lifetime.stagger">
                       <code class="sig-prename descclassname">distributed.worker.lifetime.stagger</code>
                       <em class="property">&nbsp;&nbsp;0 seconds</p></em>
                       <a class="headerlink" href="#distributed.worker.lifetime.stagger" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Random amount by which to stagger lifetimes

If you create many workers at the same time,
you may want to avoid having them kill themselves all at the same time.
To avoid this you might want to set a stagger time,
so that they close themselves with some random variation, like "5 minutes"

That way some workers can die, new ones can be brought up,
and data can be transferred over smoothly.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.lifetime.restart">
                       <code class="sig-prename descclassname">distributed.worker.lifetime.restart</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.worker.lifetime.restart" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Do we try to resurrect the worker after the lifetime deadline?</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.profile.enabled">
                       <code class="sig-prename descclassname">distributed.worker.profile.enabled</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.worker.profile.enabled" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to enable profiling</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.profile.interval">
                       <code class="sig-prename descclassname">distributed.worker.profile.interval</code>
                       <em class="property">&nbsp;&nbsp;10ms</p></em>
                       <a class="headerlink" href="#distributed.worker.profile.interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time between polling the worker threads, typically short like 10ms</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.profile.cycle">
                       <code class="sig-prename descclassname">distributed.worker.profile.cycle</code>
                       <em class="property">&nbsp;&nbsp;1000ms</p></em>
                       <a class="headerlink" href="#distributed.worker.profile.cycle" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time between bundling together this data and sending it to the scheduler

This controls the granularity at which people can query the profile information
on the time axis.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.profile.low-level">
                       <code class="sig-prename descclassname">distributed.worker.profile.low-level</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.worker.profile.low-level" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Whether or not to use the libunwind and stacktrace libraries
to gather profiling information at the lower level (beneath Python)

To get this to work you will need to install the experimental stacktrace library at

conda install -c numba stacktrace

See https://github.com/numba/stacktrace</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.recent-to-old-time">
                       <code class="sig-prename descclassname">distributed.worker.memory.recent-to-old-time</code>
                       <em class="property">&nbsp;&nbsp;30s</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.recent-to-old-time" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When there is an increase in process memory (as observed by the operating system) that is not accounted for by the dask keys stored on the worker, ignore it for this long before considering it in non-time-sensitive heuristics. This should be set to be longer than the duration of most dask tasks.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.rebalance.measure">
                       <code class="sig-prename descclassname">distributed.worker.memory.rebalance.measure</code>
                       <em class="property">&nbsp;&nbsp;optimistic</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.rebalance.measure" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Which of the properties of distributed.scheduler.MemoryState should be used for measuring worker memory usage</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.rebalance.sender-min">
                       <code class="sig-prename descclassname">distributed.worker.memory.rebalance.sender-min</code>
                       <em class="property">&nbsp;&nbsp;0.3</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.rebalance.sender-min" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Fraction of worker process memory at which we start potentially transferring data to other workers.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.rebalance.recipient-max">
                       <code class="sig-prename descclassname">distributed.worker.memory.rebalance.recipient-max</code>
                       <em class="property">&nbsp;&nbsp;0.6</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.rebalance.recipient-max" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Fraction of worker process memory at which we stop potentially receiving data from other workers. Ignored when max_memory is not set.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.rebalance.sender-recipient-gap">
                       <code class="sig-prename descclassname">distributed.worker.memory.rebalance.sender-recipient-gap</code>
                       <em class="property">&nbsp;&nbsp;0.1</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.rebalance.sender-recipient-gap" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Fraction of worker process memory, around the cluster mean, where a worker is neither a sender nor a recipient of data during a rebalance operation. E.g. if the mean cluster occupation is 50%, sender-recipient-gap=0.1 means that only nodes above 55% will donate data and only nodes below 45% will receive them. This helps avoid data from bouncing around the cluster repeatedly.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.transfer">
                       <code class="sig-prename descclassname">distributed.worker.memory.transfer</code>
                       <em class="property">&nbsp;&nbsp;0.1</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.transfer" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When the total size of incoming data transfers gets above this amount, we start throttling incoming data transfers</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.target">
                       <code class="sig-prename descclassname">distributed.worker.memory.target</code>
                       <em class="property">&nbsp;&nbsp;0.6</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.target" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When the process memory (as observed by the operating system) gets above this amount, we start spilling the dask keys holding the oldest chunks of data to disk</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.spill">
                       <code class="sig-prename descclassname">distributed.worker.memory.spill</code>
                       <em class="property">&nbsp;&nbsp;0.7</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.spill" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When the process memory (as observed by the operating system) gets above this amount, we spill data to disk, starting from the dask keys holding the oldest chunks of data, until the process memory falls below the target threshold.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.pause">
                       <code class="sig-prename descclassname">distributed.worker.memory.pause</code>
                       <em class="property">&nbsp;&nbsp;0.8</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.pause" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When the process memory (as observed by the operating system) gets above this amount, we no longer start new tasks or fetch new data on the worker.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.terminate">
                       <code class="sig-prename descclassname">distributed.worker.memory.terminate</code>
                       <em class="property">&nbsp;&nbsp;0.95</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.terminate" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>When the process memory reaches this level the nanny process will kill the worker (if a nanny is present)</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.max-spill">
                       <code class="sig-prename descclassname">distributed.worker.memory.max-spill</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.max-spill" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Limit of number of bytes to be spilled on disk.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.spill-compression">
                       <code class="sig-prename descclassname">distributed.worker.memory.spill-compression</code>
                       <em class="property">&nbsp;&nbsp;auto</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.spill-compression" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The compression algorithm to use. 'auto' defaults to lz4 if installed, otherwise to snappy if installed, otherwise to false. zlib and zstd are only used if explicitly requested here. Uncompressible data is always uncompressed, regardless of this setting. See also distributed.comm.compression.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.memory.monitor-interval">
                       <code class="sig-prename descclassname">distributed.worker.memory.monitor-interval</code>
                       <em class="property">&nbsp;&nbsp;100ms</p></em>
                       <a class="headerlink" href="#distributed.worker.memory.monitor-interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Interval between checks for the spill, pause, and terminate thresholds</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.worker.http.routes">
                       <code class="sig-prename descclassname">distributed.worker.http.routes</code>
                       <em class="property">&nbsp;&nbsp;['distributed.http.worker.prometheus', 'distributed.http.health', 'distributed.http.statics']</p></em>
                       <a class="headerlink" href="#distributed.worker.http.routes" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>A list of modules like "prometheus" and "health" that can be included or excluded as desired

These modules will have a \`\`routes\`\` keyword that gets added to the main HTTP Server.
This is also a list that can be extended with user defined modules.</p></dd>
                    </dl>

                ### Distributed Nanny

<dl class="py data">
                     <dt id="distributed.nanny.preload">
                       <code class="sig-prename descclassname">distributed.nanny.preload</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.nanny.preload" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Run custom modules during the lifetime of the nanny

You can run custom modules when the nanny starts up and closes down.
See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.nanny.preload-argv">
                       <code class="sig-prename descclassname">distributed.nanny.preload-argv</code>
                       <em class="property">&nbsp;&nbsp;[]</p></em>
                       <a class="headerlink" href="#distributed.nanny.preload-argv" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Arguments to pass into the preload scripts described above

See https://docs.dask.org/en/latest/how-to/customize-initialization.html for more information</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.nanny.pre-spawn-environ.MALLOC_TRIM_THRESHOLD_">
                       <code class="sig-prename descclassname">distributed.nanny.pre-spawn-environ.MALLOC_TRIM_THRESHOLD_</code>
                       <em class="property">&nbsp;&nbsp;65536</p></em>
                       <a class="headerlink" href="#distributed.nanny.pre-spawn-environ.MALLOC_TRIM_THRESHOLD_" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.nanny.pre-spawn-environ.OMP_NUM_THREADS">
                       <code class="sig-prename descclassname">distributed.nanny.pre-spawn-environ.OMP_NUM_THREADS</code>
                       <em class="property">&nbsp;&nbsp;1</p></em>
                       <a class="headerlink" href="#distributed.nanny.pre-spawn-environ.OMP_NUM_THREADS" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.nanny.pre-spawn-environ.MKL_NUM_THREADS">
                       <code class="sig-prename descclassname">distributed.nanny.pre-spawn-environ.MKL_NUM_THREADS</code>
                       <em class="property">&nbsp;&nbsp;1</p></em>
                       <a class="headerlink" href="#distributed.nanny.pre-spawn-environ.MKL_NUM_THREADS" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.nanny.pre-spawn-environ.OPENBLAS_NUM_THREADS">
                       <code class="sig-prename descclassname">distributed.nanny.pre-spawn-environ.OPENBLAS_NUM_THREADS</code>
                       <em class="property">&nbsp;&nbsp;1</p></em>
                       <a class="headerlink" href="#distributed.nanny.pre-spawn-environ.OPENBLAS_NUM_THREADS" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>No Comment</p></dd>
                    </dl>

                ### Distributed Admin

<dl class="py data">
                     <dt id="distributed.admin.large-graph-warning-threshold">
                       <code class="sig-prename descclassname">distributed.admin.large-graph-warning-threshold</code>
                       <em class="property">&nbsp;&nbsp;10MB</p></em>
                       <a class="headerlink" href="#distributed.admin.large-graph-warning-threshold" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Threshold in bytes for when a warning is raised about a large
submitted task graph.
Default is 10MB.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.tick.interval">
                       <code class="sig-prename descclassname">distributed.admin.tick.interval</code>
                       <em class="property">&nbsp;&nbsp;20ms</p></em>
                       <a class="headerlink" href="#distributed.admin.tick.interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time between ticks, default 20ms</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.tick.limit">
                       <code class="sig-prename descclassname">distributed.admin.tick.limit</code>
                       <em class="property">&nbsp;&nbsp;3s</p></em>
                       <a class="headerlink" href="#distributed.admin.tick.limit" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time allowed before triggering a warning</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.tick.cycle">
                       <code class="sig-prename descclassname">distributed.admin.tick.cycle</code>
                       <em class="property">&nbsp;&nbsp;1s</p></em>
                       <a class="headerlink" href="#distributed.admin.tick.cycle" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The time in between verifying event loop speed</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.max-error-length">
                       <code class="sig-prename descclassname">distributed.admin.max-error-length</code>
                       <em class="property">&nbsp;&nbsp;10000</p></em>
                       <a class="headerlink" href="#distributed.admin.max-error-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Maximum length of traceback as text

Some Python tracebacks can be very very long
(particularly in stack overflow errors)

If the traceback is larger than this size (in bytes) then we truncate it.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.log-length">
                       <code class="sig-prename descclassname">distributed.admin.log-length</code>
                       <em class="property">&nbsp;&nbsp;10000</p></em>
                       <a class="headerlink" href="#distributed.admin.log-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Maximum length of worker/scheduler logs to keep in memory.
They can be retrieved with get_scheduler_logs() / get_worker_logs().
Set to null for unlimited.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.log-format">
                       <code class="sig-prename descclassname">distributed.admin.log-format</code>
                       <em class="property">&nbsp;&nbsp;%(asctime)s - %(name)s - %(levelname)s - %(message)s</p></em>
                       <a class="headerlink" href="#distributed.admin.log-format" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The log format to emit.

See https://docs.python.org/3/library/logging.html#logrecord-attributes</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.low-level-log-length">
                       <code class="sig-prename descclassname">distributed.admin.low-level-log-length</code>
                       <em class="property">&nbsp;&nbsp;1000</p></em>
                       <a class="headerlink" href="#distributed.admin.low-level-log-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Maximum length of various event logs for developers.
Set to null for unlimited.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.pdb-on-err">
                       <code class="sig-prename descclassname">distributed.admin.pdb-on-err</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.admin.pdb-on-err" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Enter Python Debugger on scheduling error</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.interval">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.interval</code>
                       <em class="property">&nbsp;&nbsp;500ms</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Polling time to query cpu/memory statistics default 500ms</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.log-length">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.log-length</code>
                       <em class="property">&nbsp;&nbsp;7200</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.log-length" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Maximum number of samples to keep in memory.
Multiply by \`interval\` to obtain log duration.
Set to null for unlimited.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.disk">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.disk</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.disk" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Should we include disk metrics?  (they can cause issues in some systems)</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.host-cpu">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.host-cpu</code>
                       <em class="property">&nbsp;&nbsp;False</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.host-cpu" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Should we include host-wide CPU usage, with very granular breakdown?</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.gil.enabled">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.gil.enabled</code>
                       <em class="property">&nbsp;&nbsp;True</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.gil.enabled" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>Enable monitoring of GIL contention</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.system-monitor.gil.interval">
                       <code class="sig-prename descclassname">distributed.admin.system-monitor.gil.interval</code>
                       <em class="property">&nbsp;&nbsp;1ms</p></em>
                       <a class="headerlink" href="#distributed.admin.system-monitor.gil.interval" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>GIL polling interval. More frequent polling will reflect a more accurate GIL
contention metric but will be more likely to impact runtime performance.</p></dd>
                    </dl>

                <dl class="py data">
                     <dt id="distributed.admin.event-loop">
                       <code class="sig-prename descclassname">distributed.admin.event-loop</code>
                       <em class="property">&nbsp;&nbsp;tornado</p></em>
                       <a class="headerlink" href="#distributed.admin.event-loop" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The event loop to use,

Must be one of tornado, asyncio, or uvloop</p></dd>
                    </dl>

                ### Distributed RMM

<dl class="py data">
                     <dt id="distributed.rmm.pool-size">
                       <code class="sig-prename descclassname">distributed.rmm.pool-size</code>
                       <em class="property">&nbsp;&nbsp;None</p></em>
                       <a class="headerlink" href="#distributed.rmm.pool-size" title="Permalink to this definition">¶</a>
                     </dt>
                    <dd><p>The size of the memory pool in bytes.</p></dd>
                    </dl>


# connect-to-remote-data.html.md

# Connect to remote data

Dask can read data from a variety of data stores including local file systems,
network file systems, cloud object stores, and Hadoop.  Typically this is done
by prepending a protocol like `"s3://"` to paths used in common data access
functions like `dd.read_csv`:

```python
import dask.dataframe as dd
df = dd.read_csv('s3://bucket/path/to/data-*.csv')
df = dd.read_parquet('gcs://bucket/path/to/data-*.parq')

import dask.bag as db
b = db.read_text('hdfs://path/to/*.json').map(json.loads)
```

Dask uses [fsspec](https://filesystem-spec.readthedocs.io/) for local, cluster and remote data IO. Other file interaction, such
as loading of configuration, is done using ordinary python method.

The following remote services are well supported and tested against the main
codebase:

- **Local or Network File System**: `file://` - the local file system, default in the absence of any protocol.
- **Hadoop File System**: `hdfs://` - Hadoop Distributed File System, for resilient, replicated
  files within a cluster. This uses [PyArrow](https://arrow.apache.org/docs/python/) as the backend.
- **Amazon S3**: `s3://` - Amazon S3 remote binary store, often used with Amazon EC2,
  using the library [s3fs](https://s3fs.readthedocs.io/).
- **Google Cloud Storage**: `gcs://` or `gs://` - Google Cloud Storage, typically used with Google Compute
  resource using [gcsfs](https://gcsfs.readthedocs.io/en/latest/).
- **Microsoft Azure Storage**: `adl://`, `abfs://` or `az://` - Microsoft Azure Storage using [adlfs](https://github.com/dask/adlfs).
- **Hugging Face**: `hf://` - Hugging Face Hub of datasets for AI, using the [huggingface_hub](https://huggingface.co/docs/huggingface_hub) library.
- **HTTP(s)**: `http://` or `https://` for reading data directly from HTTP web servers.

[fsspec](https://filesystem-spec.readthedocs.io/) also provides other file systems that may be of interest to Dask users, such as
ssh, ftp, webhdfs and dropbox. See the documentation for more information.

When specifying a storage location, a URL should be provided using the general
form `protocol://path/to/data`.  If no protocol is provided, the local
file system is assumed (same as `file://`).

Lower-level details on how Dask handles remote data is described
below in the Internals section

## Optional Parameters

Two methods exist for passing parameters to the backend file system driver:
extending the URL to include username, password, server, port, etc.; and
providing `storage_options`, a dictionary of parameters to pass on. The
second form is more general, as any number of file system-specific options
can be passed.

Examples:

```python
df = dd.read_csv('hdfs://user@server:port/path/*.csv')

df = dd.read_parquet('s3://bucket/path',
                     storage_options={'anon': True, 'use_ssl': False})
```

Details on how to provide configuration for the main back-ends
are listed next, but further details can be found in the documentation pages of the
relevant back-end.

Each back-end has additional installation requirements and may not be available
at runtime. The dictionary `fsspec.registry` contains the
currently imported file systems. To see which backends `fsspec` knows how
to import, you can do

```python
from fsspec.registry import known_implementations
known_implementations
```

Note that some backends appear twice, if they can be referenced with multiple
protocol strings, like “http” and “https”.

<a id="connect-to-remote-data-local"></a>

## Local File System

Local files are always accessible, and all parameters passed as part of the URL
(beyond the path itself) or with the `storage_options`
dictionary will be ignored.

This is the default back-end, and the one used if no protocol is passed at all.

We assume here that each worker has access to the same file system - either
the workers are co-located on the same machine, or a network file system
is mounted and referenced at the same path location for every worker node.

Dask’s data IO methods will typically normalize relative paths to absolute paths
so that data can be read by a worker even if it has a different working
directory than your client. But other functions, including ones you’ve written,
might not normalize the paths. If the current working directory of your Client
differs from the working directory of the Worker, you might run issues where a
file can’t be found since the relative path will be incorrect.

In general, things will be simplest if your Client and Workers (and perhaps
Scheduler) all share the same working directory. You can check this with a
snippet like:

```python
>>> import os
>>> from dask.distributed import Client, LocalCluster
>>> client = Client(LocalCluster())
>>> client.run(os.getcwd)
{'tcp://127.0.0.1:64597': '/home/coder',
 'tcp://127.0.0.1:64598': '/home/coder'}
```

## Hadoop File System

The Hadoop File System (HDFS) is a widely deployed, distributed, data-local file
system written in Java. This file system backs many clusters running Hadoop and
Spark. HDFS support can be provided by [PyArrow](https://arrow.apache.org/docs/python/).

By default, the back-end attempts to read the default server and port from
local Hadoop configuration files on each node, so it may be that no
configuration is required. However, the server, port, and user can be passed as
part of the url: `hdfs://user:pass@server:port/path/to/data`, or using the
`storage_options=` kwarg.

### Extra Configuration for PyArrow

The following additional options may be passed to the `PyArrow` driver via
`storage_options`:

> - `host`, `port`, `user`: Basic authentication
> - `kerb_ticket`: Path to kerberos ticket cache

PyArrow’s `libhdfs` driver can also be affected by a few environment
variables. For more information on these, see the [PyArrow documentation](https://arrow.apache.org/docs/python/filesystems_deprecated.html#hadoop-file-system-hdfs).

<a id="connect-to-remote-data-s3"></a>

## Amazon S3

Amazon S3 (Simple Storage Service) is a web service offered by Amazon Web
Services.

The S3 back-end available to Dask is [s3fs](https://s3fs.readthedocs.io/), and is importable when Dask is
imported.

Authentication for S3 is provided by the underlying library boto3. As described
in the [auth docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html), this could be achieved by placing credentials files in one
of several locations on each node: `~/.aws/credentials`, `~/.aws/config`,
`/etc/boto.cfg`, and `~/.boto`. Alternatively, for nodes located
within Amazon EC2, IAM roles can be set up for each node, and then no further
configuration is required. The final authentication option for user
credentials can be passed directly in the URL
(`s3://keyID:keySecret/bucket/key/name`) or using `storage_options`. In
this case, however, the key/secret will be passed to all workers in-the-clear,
so this method is only recommended on well-secured networks.

The following parameters may be passed to s3fs using `storage_options`:

> - anon: Whether access should be anonymous (default False)
> - key, secret: For user authentication
> - token: If authentication has been done with some other S3 client
> - use_ssl: Whether connections are encrypted and secure (default True)
> - client_kwargs: Dict passed to the [boto3 client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client), with keys such
>   as region_name or endpoint_url. Notice: do not pass the config
>   option here, please pass it’s content to config_kwargs instead.
> - config_kwargs: Dict passed to the [s3fs.S3FileSystem](https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem), which passes it to
>   the [boto3 client’s config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) option.
> - requester_pays: Set True if the authenticated user will assume transfer
>   costs, which is required by some providers of bulk data
> - default_block_size, default_fill_cache: These are not of particular
>   interest to Dask users, as they concern the behaviour of the buffer
>   between successive reads
> - kwargs: Other parameters are passed to the [boto3 Session](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) object,
>   such as profile_name, to pick one of the authentication sections from
>   the configuration files referred to above (see [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#shared-credentials-file))

### Using Other S3-Compatible Services

By using the endpoint_url option, you may use other s3-compatible services,
for example, using AlibabaCloud OSS:

```python
dask_function(...,
    storage_options={
        "key": ...,
        "secret": ...,
        "client_kwargs": {
            "endpoint_url": "http://some-region.some-s3-compatible.com",
        },
        # this dict goes to boto3 client's `config`
        #   `addressing_style` is required by AlibabaCloud, other services may not
        "config_kwargs": {"s3": {"addressing_style": "virtual"}},
    })
```

<a id="connect-to-remote-data-gc"></a>

## Google Cloud Storage

Google Cloud Storage is a RESTful online file storage web service for storing
and accessing data on Google’s infrastructure.

The GCS back-end is identified by the
protocol identifiers `gcs` and `gs`, which are identical in their effect.

Multiple modes of authentication are supported. These options should be
included in the `storage_options` dictionary as `{'token': ..}` submitted with your call
to a storage-based Dask function/method. See the [gcsfs](https://gcsfs.readthedocs.io/en/latest/) documentation for further
details.

General recommendations for distributed clusters, in order:

- use `anon` for public data
- use `cloud` if this is available
- use [gcloud](https://cloud.google.com/sdk/docs/) to generate a JSON file, and distribute this to all workers, and
  supply the path to the file
- use gcsfs directly with the `browser` method to generate a token cache file
  (`~/.gcs_tokens`) and distribute this to all workers, thereafter using method `cache`

A final suggestion is shown below, which may be the fastest and simplest for authenticated access (as
opposed to anonymous), since it will not require re-authentication. However, this method
is not secure since credentials will be passed directly around the cluster. This is fine if
you are certain that the cluster is itself secured. You need to create a `GCSFileSystem` object
using any method that works for you and then pass its credentials directly:

```python
gcs = GCSFileSystem(...)
dask_function(..., storage_options={'token': gcs.session.credentials})
```

## Microsoft Azure Storage

Microsoft Azure Storage is comprised of Data Lake Storage (Gen1) and Blob Storage (Gen2).
These are identified by the protocol identifiers `adl` and `abfs`, respectively,
provided by the [adlfs](https://github.com/dask/adlfs) back-end.

Authentication for `adl` requires `tenant_id`, `client_id` and `client_secret`
in the `storage_options` dictionary.

Authentication for `abfs` requires `storage_options` to contain `account_name`,
`tenant_id`, `client_id` and `client_secret` for the [RBAC and ACL](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model/) access models,
or `account_name` and `account_key` for the [shared key](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model#shared-key-and-shared-access-signature-sas-authorization) access model.

## HTTP(S)

Direct file-like access to arbitrary URLs is available over HTTP and HTTPS. However,
there is no such thing as `glob` functionality over HTTP, so only explicit lists
of files can be used.

Server implementations differ in the information they provide - they may or may
not specify the size of a file via a HEAD request or at the start of a download -
and some servers may not respect byte range requests. The HTTPFileSystem therefore
offers best-effort behaviour: the download is streamed but, if more data is seen
than the configured block-size, an error will be raised. To be able to access such
data you must read the whole file in one shot (and it must fit in memory).

Using a block size of 0 will return normal `requests` streaming file-like objects,
which are stable, but provide no random access.

## Developer API

The prototype for any file system back-end can be found in
`fsspec.spec.AbstractFileSystem`. Any new implementation should provide the
same API, or directly subclass, and make itself available as a protocol to Dask. For example, the
following would register the protocol “myproto”, described by the implementation
class `MyProtoFileSystem`. URLs of the form `myproto://` would thereafter
be dispatched to the methods of this class:

```python
fsspec.registry['myproto'] = MyProtoFileSystem
```

However, it would be better to submit a PR to `fsspec` to include the class in
the `known_implementations`.

## Internals

Dask contains internal tools for extensible data ingestion in the
`dask.bytes` package and uses external tools like `open_files` from [fsspec](https://filesystem-spec.readthedocs.io/).
.  *These functions are developer-focused rather than for
direct consumption by users.  These functions power user-facing functions like*
`dd.read_csv` *and* `db.read_text` *which are probably more useful for most
users.*

| [`read_bytes`](#dask.bytes.read_bytes)(urlpath[, delimiter, not_zero, ...])   | Given a path or paths, return delayed objects that read from those paths.   |
|-------------------------------------------------------------------------------|-----------------------------------------------------------------------------|

This function is extensible in its output format (bytes),
its input locations (file system, S3, HDFS), line delimiters, and compression
formats.

This function is *lazy*, returning pointers to blocks of bytes (`read_bytes`).
It handles different storage backends by prepending
protocols like `s3://` or `hdfs://` (see below). It handles compression formats
listed in `fsspec.compression`, some of which may require additional packages
to be installed.

This function is not used for all data sources.  Some data sources like HDF5
are quite particular and receive custom treatment.

### Delimiters

The `read_bytes` function takes a path (or globstring of paths) and produces
a sample of the first file and a list of delayed objects for each of the other
files.  If passed a delimiter such as `delimiter=b'\n'`, it will ensure that
the blocks of bytes start directly after a delimiter and end directly before a
delimiter.  This allows other functions, like `pd.read_csv`, to operate on
these delayed values with expected behavior.

These delimiters are useful both for typical line-based formats (log files,
CSV, JSON) as well as other delimited formats like Avro, which may separate
logical chunks by a complex sentinel string. Note that the delimiter finding
algorithm is simple, and will not account for characters that are escaped,
part of a UTF-8 code sequence or within the quote marks of a string.

### Compression

These functions support widely available compression technologies like `gzip`,
`bz2`, `xz`, `snappy`, and `lz4`.  More compressions can be easily
added by inserting functions into dictionaries available in the
`fsspec.compression` module.  This can be done at runtime and need not be
added directly to the codebase.

However, most compression technologies like `gzip` do not support efficient
random access, and so are useful for streaming `fsspec.open_files` but not useful for
`read_bytes` which splits files at various points.

### API

### dask.bytes.read_bytes(urlpath, delimiter=None, not_zero=False, blocksize='128 MiB', sample='10 kiB', compression=None, include_path=False, \*\*kwargs)

Given a path or paths, return delayed objects that read from those paths.

The path may be a filename like `'2015-01-01.csv'` or a globstring
like `'2015-*-*.csv'`.

The path may be preceded by a protocol, like `s3://` or `hdfs://` if
those libraries are installed.

This cleanly breaks data by a delimiter if given, so that block boundaries
start directly after a delimiter and end on the delimiter.

* **Parameters:**
  **urlpath**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to read from alternative filesystems. To read from multiple files you
    can pass a globstring or a list of paths, with the caveat that they
    must all have the same protocol.

  **delimiter**
  : An optional delimiter, like `b'\n'` on which to split blocks of
    bytes.

  **not_zero**
  : Force seek of start-of-file delimiter, discarding header.

  **blocksize**
  : Chunk size in bytes, defaults to “128 MiB”

  **compression**
  : String like ‘gzip’ or ‘xz’.  Must support efficient random access.

  **sample**
  : Whether or not to return a header sample.
    Values can be `False` for “no sample requested”
    Or an integer or string value like `2**20` or `"1 MiB"`

  **include_path**
  : Whether or not to include the path with the bytes representing a particular file.
    Default is False.

  **\*\*kwargs**
  : Extra options that make sense to a particular storage connection, e.g.
    host, port, username, password, etc.
* **Returns:**
  **sample**
  : The sample header

  **blocks**
  : Each list corresponds to a file, and each delayed object computes to a
    block of bytes from that file.

  **paths**
  : List of same length as blocks, where each item is the path to the file
    represented in the corresponding block.

### Examples

```pycon
>>> sample, blocks = read_bytes('2015-*-*.csv', delimiter=b'\n')
>>> sample, blocks = read_bytes('s3://bucket/2015-*-*.csv', delimiter=b'\n')
>>> sample, paths, blocks = read_bytes('2015-*-*.csv', include_path=True)
```

<!-- !! processed by numpydoc !! -->


# custom-collections.html.md

# Custom Collections

For many problems, the built-in Dask collections (`dask.array`,
`dask.dataframe`, `dask.bag`, and `dask.delayed`) are sufficient. For
cases where they aren’t, it’s possible to create your own Dask collections. Here
we describe the required methods to fulfill the Dask collection interface.

#### NOTE
This is considered an advanced feature. For most cases the built-in
collections are probably sufficient.

Before reading this you should read and understand:

- [overview](graphs.md)
- [graph specification](spec.md)
- [custom graphs](custom-graphs.md)

**Contents**

- [Description of the Dask collection interface](#collection-interface)
- [How this interface is used to implement the core Dask
  methods](#core-method-internals)
- [How to add the core methods to your class](#adding-methods-to-class)
- [Example Dask Collection](#example-dask-collection)
- [How to check if something is a Dask collection](#is-dask-collection)
- [How to make tokenize work with your collection](#deterministic-hashing)

<a id="collection-interface"></a>

## The Dask Collection Interface

To create your own Dask collection, you need to fulfill the interface
defined by the [`dask.typing.DaskCollection`](#dask.typing.DaskCollection) protocol. Note
that there is no required base class.

It is recommended to also read [Internals of the Core Dask Methods](#core-method-internals) to see how this
interface is used inside Dask.

### Collection Protocol

### *class* dask.typing.DaskCollection(\*args, \*\*kwargs)

Protocol defining the interface of a Dask collection.

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_graph_\_() → [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

The Dask task graph.

The core Dask collections (Array, DataFrame, Bag, and Delayed)
use a [`HighLevelGraph`](high-level-graphs.md#dask.highlevelgraph.HighLevelGraph) to
represent the collection task graph. It is also possible to
represent the task graph as a low level graph using a Python
dictionary.

* **Returns:**
  Mapping
  : The Dask task graph. If the instance returns a
    [`dask.highlevelgraph.HighLevelGraph`](high-level-graphs.md#dask.highlevelgraph.HighLevelGraph) then the
    `__dask_layers__()` method must be implemented, as
    defined by the [`HLGDaskCollection`](#dask.typing.HLGDaskCollection)
    protocol.

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_keys_\_() → [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...] | NestedKeys]

The output keys of the task graph.

Note that there are additional constraints on keys for a Dask
collection than those described in the [task graph
specification documentation](spec.md). These additional
constraints are described below.

All keys must either be non-empty strings or tuples where the first element is a
non-empty string, followed by zero or more arbitrary str, bytes, int, float, or
tuples thereof. The non-empty string is commonly known as the *collection name*.
All collections embedded in the dask package have exactly one name, but this is
not a requirement.

These are all valid outputs:

- `[]`
- `["x", "y"]`
- `[[("y", "a", 0), ("y", "a", 1)], [("y", "b", 0), ("y", "b", 1)]`

* **Returns:**
  list
  : A possibly nested list of keys that represent the outputs
    of the graph. After computation, the results will be
    returned in the same layout, with the keys replaced with
    their corresponding outputs.

<!-- !! processed by numpydoc !! -->

#### \_\_dask_optimize_\_ *: [Any](https://docs.python.org/3/library/typing.html#typing.Any)*

Given a graph and keys, return a new optimized graph.

This method can be either a `staticmethod` or a `classmethod`,
but not an `instancemethod`. For example implementations see the
definitions of `__dask_optimize__` in the core Dask collections:
`dask.array.Array`, `dask.dataframe.DataFrame`, etc.

Note that graphs and keys are merged before calling
`__dask_optimize__`; as such, the graph and keys passed to
this method may represent more than one collection sharing the
same optimize method.

* **Parameters:**
  **dsk**
  : The merged graphs from all collections sharing the same
    `__dask_optimize__` method.

  **keys**
  : A list of the outputs from `__dask_keys__` from all
    collections sharing the same `__dask_optimize__` method.

  **\*\*kwargs**
  : Extra keyword arguments forwarded from the call to
    `compute` or `persist`. Can be used or ignored as
    needed.
* **Returns:**
  MutableMapping
  : The optimized Dask graph.

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_postcompute_\_() → [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)]

Finalizer function and optional arguments to construct final result.

Upon computation each key in the collection will have an in
memory result, the postcompute function combines each key’s
result into a final in memory representation. For example,
dask.array.Array concatenates the arrays at each chunk into a
final in-memory array.

* **Returns:**
  PostComputeCallable
  : Callable that receives the sequence of the results of each
    final key along with optional arguments. An example signature
    would be `finalize(results: Sequence[Any], *args)`.

  tuple[Any, …]
  : Optional arguments passed to the function following the
    key results (the \*args part of the
    `PostComputeCallable`. If no additional arguments are to
    be passed then this must be an empty tuple.

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_postpersist_\_() → [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[PostPersistCallable](#dask.typing.PostPersistCallable), [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)]

Rebuilder function and optional arguments to construct a persisted collection.

See also the documentation for [`dask.typing.PostPersistCallable`](#dask.typing.PostPersistCallable).

* **Returns:**
  PostPersistCallable
  : Callable that rebuilds the collection. The signature
    should be
    `rebuild(dsk: Mapping, *args: Any, rename: Mapping[str, str] | None)`
    (as defined by the
    [`PostPersistCallable`](#dask.typing.PostPersistCallable) protocol).
    The callable should return an equivalent Dask collection
    with the same keys as self, but with results that are
    computed through a different graph. In the case of
    [`dask.persist()`](api.md#dask.persist), the new graph will have just the
    output keys and the values already computed.

  tuple[Any, …]
  : Optional arguments passed to the rebuild callable. If no
    additional arguments are to be passed then this must be an
    empty tuple.

<!-- !! processed by numpydoc !! -->

#### \_\_dask_scheduler_\_ *: staticmethod*

The default scheduler `get` to use for this object.

Usually attached to the class as a staticmethod, e.g.:

```pycon
>>> import dask.threaded
>>> class MyCollection:
...     # Use the threaded scheduler by default
...     __dask_scheduler__ = staticmethod(dask.threaded.get)
```

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_tokenize_\_() → [Hashable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)

Value that must fully represent the object.

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* compute(\*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Compute this dask collection.

This turns a lazy Dask collection into its in-memory
equivalent. For example a Dask array turns into a NumPy array
and a Dask dataframe turns into a Pandas dataframe. The entire
dataset must fit into memory before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or
    “processes”. If not provided, the default is to check the
    global settings first, and then fall back to the
    collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before
    computation. Otherwise the graph is run as is. This can be
    useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  The collection’s computed result.

#### SEE ALSO
[`dask.compute`](api.md#dask.compute)

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* persist(\*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → CollType

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with
the same metadata, but now with the results fully computed or
actively computing in the background.

The action of function differs significantly depending on the
active task scheduler. If the task scheduler supports
asynchronous computing, such as is the case of the
dask.distributed scheduler, then persist will return
*immediately* and the return value’s task graph will contain
Dask Future objects. However if the task scheduler only
supports blocking computation then the call to persist will
*block* and the return value’s task graph will contain
concrete Python results.

This function is particularly useful when using distributed
systems, because the results will be kept in distributed
memory, rather than returned to the local process as with
compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or
    “processes”. If not provided, the default is to check the
    global settings first, and then fall back to the
    collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before
    computation. Otherwise the graph is run as is. This can be
    useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](api.md#dask.persist)

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* visualize(filename: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'mydask', format: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, optimize_graph: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs: Any) → DisplayObject | [None](https://docs.python.org/3/library/constants.html#None)

Render the computation of this object’s task graph using graphviz.

Requires `graphviz` to be installed.

* **Parameters:**
  **filename**
  : The name of the file to write to disk. If the provided
    filename doesn’t include an extension, ‘.png’ will be
    used by default. If filename is None, no file will be
    written, and we communicate with dot using only pipes.

  **format**
  : Format in which to write output file. Default is ‘png’.

  **optimize_graph**
  : If True, the graph is optimized before rendering.
    Otherwise, the graph is displayed as is. Default is False.

  **color: {None, ‘order’}, optional**
  : Options to color nodes. Provide `cmap=` keyword for
    additional colormap

  **\*\*kwargs**
  : Additional keyword arguments to forward to `to_graphviz`.
* **Returns:**
  **result**
  : See dask.dot.dot_graph for more information.

#### SEE ALSO
[`dask.visualize`](api.md#dask.visualize)

`dask.dot.dot_graph`

### Notes

For more information on optimization see here:

[https://docs.dask.org/en/latest/optimize.html](https://docs.dask.org/en/latest/optimize.html)

### Examples

```pycon
>>> x.visualize(filename='dask.pdf')
>>> x.visualize(filename='dask.pdf', color='order')
```

<!-- !! processed by numpydoc !! -->

### HLG Collection Protocol

#### NOTE
HighLevelGraphs are being deprecated in favor of expressions. New projects are encouraged to not implement their own HLG Layers.

Collections backed by Dask’s [High Level Graphs](high-level-graphs.md#high-level-graphs) must implement
an additional method, defined by this protocol:

### *class* dask.typing.HLGDaskCollection(\*args, \*\*kwargs)

Protocol defining a Dask collection that uses HighLevelGraphs.

This protocol is nearly identical to
[`DaskCollection`](#dask.typing.DaskCollection), with the addition of the
`__dask_layers__` method (required for collections backed by
high level graphs).

<!-- !! processed by numpydoc !! -->

#### *abstractmethod* \_\_dask_layers_\_() → [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str)]

Names of the HighLevelGraph layers.

<!-- !! processed by numpydoc !! -->

### Scheduler `get` Protocol

The `SchedulerGetProtocol` defines the signature that a Dask
collection’s `__dask_scheduler__` definition must adhere to.

### *class* dask.typing.SchedulerGetCallable(\*args, \*\*kwargs)

Protocol defining the signature of a `__dask_scheduler__` callable.

<!-- !! processed by numpydoc !! -->

#### \_\_call_\_(dsk: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)], keys: [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]] | [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Method called as the default scheduler for a collection.

* **Parameters:**
  **dsk**
  : The task graph.

  **keys**
  : Key(s) corresponding to the desired data.

  **\*\*kwargs**
  : Additional arguments.
* **Returns:**
  Any
  : Result(s) associated with keys

<!-- !! processed by numpydoc !! -->

### Post-persist Callable Protocol

Collections must define a `__dask_postpersist__` method which
returns a callable that adheres to the `PostPersistCallable`
interface.

### *class* dask.typing.PostPersistCallable(\*args, \*\*kwargs)

Protocol defining the signature of a `__dask_postpersist__` callable.

<!-- !! processed by numpydoc !! -->

#### \_\_call_\_(dsk: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)], \*args: [Any](https://docs.python.org/3/library/typing.html#typing.Any), rename: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None) → CollType_co

Method called to rebuild a persisted collection.

* **Parameters:**
  **dsk: Mapping**
  : A mapping which contains at least the output keys returned
    by \_\_dask_keys_\_().

  **\*args**
  : Additional optional arguments If no extra arguments are
    necessary, it must be an empty tuple.

  **rename**
  : If defined, it indicates that output keys may be changing
    too; e.g. if the previous output of `__dask_keys__()`
    was `[("a", 0), ("a", 1)]`, after calling
    `rebuild(dsk, *extra_args, rename={"a": "b"})`
    it must become `[("b", 0), ("b", 1)]`.
    The `rename` mapping may not contain the collection
    name(s); in such case the associated keys do not change.
    It may contain replacements for unexpected names, which
    must be ignored.
* **Returns:**
  Collection
  : An equivalent Dask collection with the same keys as
    computed through a different graph.

<!-- !! processed by numpydoc !! -->

<a id="core-method-internals"></a>

## Internals of the Core Dask Methods

Dask has a few *core* functions (and corresponding methods) that implement
common operations:

- `compute`: Convert one or more Dask collections into their in-memory
  counterparts
- `persist`: Convert one or more Dask collections into equivalent Dask
  collections with their results already computed and cached in memory
- `optimize`: Convert one or more Dask collections into equivalent Dask
  collections sharing one large optimized graph
- `visualize`: Given one or more Dask collections, draw out the graph that
  would be passed to the scheduler during a call to `compute` or `persist`

Here we briefly describe the internals of these functions to illustrate how
they relate to the above interface.

### Compute

The operation of `compute` can be broken into three stages:

1. **Graph Merging, finalization**

   First, the individual collections are converted to a single large expression
   and nested list of keys. This is done by
   `collections_to_expr()` and ensures that all collections are
   optimized together to eliminate common sub-expressions.

   #### NOTE
   At this stage, legacy HLG graphs are wrapped into a `HLGExpr` that
   encodes \_\_dask_postcompute_\_ and the low level optimizer as determined by \_\_dask_optimize_\_ into the expression.

   The `optimize_graph` argument is only relevant for HLG graphs and
   controls whether low level optimizations are considered.
   - If `optimize_graph` is `True` (default), then the collections are
     first grouped by their `__dask_optimize__` methods.  All collections with the same `__dask_optimize__` method have their graphs merged and keys concatenated, and then a single call to each respective `__dask_optimize__` is made with the merged graphs and keys.  The resulting graphs are then merged.
   - If `optimize_graph` is `False`, then all the graphs are merged and
     all the keys concatenated.

   The combined graph is \_finalized_ with a `FinalizeCompute` expression
   which instructs the expression / graph to reduce to a single partition,
   suitable to be returned to the user after compute. This is either done by
   implementing the `__dask_postcompute__` method of the collection or by
   implementing an optimization path of the expression.

   For the example of a DataFrame, the `FinalizeCompute` simplifies to a `RepartitionToFewer(..., npartition=1)` which simply concatenates all results to one ordinary DataFrame.
2. **(Expression) Optimization**

   The merged expression is optimized. This step should not be confused with
   the low level optimization that is defined by \_\_dask_optimize_\_ for legacy
   graphs. This is a step that is always performed and is a required step to
   simplify and lower expressions to their final form that can be used to
   actually generate the executable task graph. See also, [Optimizer](dataframe-optimizer.md).

   For legacy HLG graphs, the low level optimization step is embedded in the
   graph materialization which typically only happens after the graph has been
   passed to the scheduler (see below).
3. **Computation**

   After the graphs are merged and any optimizations performed, the resulting
   large graph and nested list of keys are passed on to the scheduler.  The
   scheduler to use is chosen as follows:
   - If a `get` function is specified directly as a keyword, use that
   - Otherwise, if a global scheduler is set, use that
   - Otherwise fall back to the default scheduler for the given collections.
     Note that if all collections don’t share the same `__dask_scheduler__`
     then an error will be raised.

   Once the appropriate scheduler `get` function is determined, it is called
   with the merged graph, keys, and extra keyword arguments.  After this stage,
   `results` is a nested list of values. The structure of this list mirrors
   that of `keys`, with each key substituted with its corresponding result.

### Persist

Persist is very similar to `compute`, except for how the return values are
created. It too has three stages:

1. **Graph Merging, \*no\* finalization**

   Same as in `compute` but without a finalization. In the case of persist we
   do not want to concatenate all output partitions but instead want to return a
   future for every partition.
2. **(Expression) Optimization**

   Same as in `compute`.
3. **Computation**

   Same as in `compute` with the difference that the returned results are a list of Futures.
4. **Postpersist**

   The futures returned by the scheduler are used with `__dask_postpersist__` to rebuild a collection that is pointing to the remote data.

   `__dask_postpersist__` returns two things:
   - A `rebuild` function, which takes in a persisted graph.  The keys of
     this graph are the same as `__dask_keys__` for the corresponding
     collection, and the values are computed results (for the single-machine
     scheduler) or futures (for the distributed scheduler).
   - A tuple of extra arguments to pass to `rebuild` after the graph

   To build the outputs of `persist`, the list of collections and results is
   iterated over, and the rebuilder for each collection is called on the graph
   for its respective results.

### Optimize

The operation of `optimize` can be broken into two stages:

1. **Graph Merging, \*no\* finalization**

   Same as in `persist`.
2. **(Expression) Optimization**

   Same as in `compute` and `persist`.
3. **Materialization and Rebuilding**

   The entire graph is materialized (which also performs low level optimization).
   Similar to `persist`, the `rebuild` function and arguments from
   `__dask_postpersist__` are used to reconstruct equivalent collections from
   the optimized graph.

### Visualize

Visualize is the simplest of the 4 core functions. It only has two stages:

1. **Graph Merging & Optimization**

   Same as in `compute`.
2. **Graph Drawing**

   The resulting merged graph is drawn using `graphviz` and outputs to the
   specified file.

<a id="adding-methods-to-class"></a>

## Adding the Core Dask Methods to Your Class

Defining the above interface will allow your object to used by the core Dask
functions (`dask.compute`, `dask.persist`, `dask.visualize`, etc.). To
add corresponding method versions of these, you can subclass from
`dask.base.DaskMethodsMixin` which adds implementations of `compute`,
`persist`, and `visualize` based on the interface above.

## Expressions to define computation

It is recommended to define dask graphs using the `dask.expr.Expr` class.
To get started, a minimal set of methods have to be implemented.

### *class* dask.Expr(\*args, \_determ_token=None, \*\*kwargs)

<!-- !! processed by numpydoc !! -->

#### \_\_dask_graph_\_()

Traverse expression tree, collect layers

Subclasses generally do not want to override this method unless custom
logic is required to treat (e.g. ignore) specific operands during graph
generation.

#### SEE ALSO
[`Expr._layer`](#dask.Expr._layer)

[`Expr._task`](#dask.Expr._task)

<!-- !! processed by numpydoc !! -->

#### \_\_dask_keys_\_()

The keys for this expression

This is used to determine the keys of the output collection
when this expression is computed.

* **Returns:**
  keys: list
  : The keys for this expression

<!-- !! processed by numpydoc !! -->

#### \_layer() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

The graph layer added by this expression.

Simple expressions that apply one task per partition can choose to only
implement Expr._task instead.

* **Returns:**
  layer: dict
  : The Dask task graph added by this expression

#### SEE ALSO
[`Expr._task`](#dask.Expr._task)

[`Expr.__dask_graph__`](#dask.Expr.__dask_graph__)

### Examples

```pycon
>>> class Add(Expr):
...     def _layer(self):
...         return {
...            name: Task(
...                name,
...                operator.add,
...                TaskRef((self.left._name, i)),
...                TaskRef((self.right._name, i))
...            )
...            for i, name in enumerate(self.__dask_keys__())
...         }
```

<!-- !! processed by numpydoc !! -->

#### \_task(key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], index: [int](https://docs.python.org/3/library/functions.html#int)) → Task

The task for the i’th partition

* **Parameters:**
  **index:**
  : The index of the partition of this dataframe
* **Returns:**
  task:
  : The Dask task to compute this partition

#### SEE ALSO
[`Expr._layer`](#dask.Expr._layer)

### Examples

```pycon
>>> class Add(Expr):
...     def _task(self, i):
...         return Task(
...            self.__dask_keys__()[i],
...            operator.add,
...            TaskRef((self.left._name, i)),
...            TaskRef((self.right._name, i))
...        )
```

<!-- !! processed by numpydoc !! -->

<a id="example-dask-collection"></a>

## Example Dask Collection

Here we create a Dask collection representing a tuple.  Every element in the
tuple is represented as a task in the graph.  Note that this is for illustration
purposes only - the same user experience could be done using normal tuples with
elements of `dask.delayed`:

```python
import dask
from dask.base import DaskMethodsMixin, replace_name_in_key
from dask.expr import Expr, LLGExpr
from dask.typing import Key
from dask.task_spec import Task, DataNode, Alias


# We subclass from DaskMethodsMixin to add common dask methods to
# our class (compute, persist, and visualize). This is nice but not
# necessary for creating a Dask collection (you can define them
# yourself).
class Tuple(DaskMethodsMixin):
    def __init__(self, expr):
        self._expr = expr

    def __dask_graph__(self):
        return self._expr.__dask_graph__()

    def __dask_keys__(self):
        return self._expr.__dask_keys__()

    # Use the threaded scheduler by default.
    __dask_scheduler__ = staticmethod(dask.threaded.get)

    def __dask_postcompute__(self):
        # We want to return the results as a tuple, so our finalize
        # function is `tuple`. There are no extra arguments, so we also
        # return an empty tuple.
        return tuple, ()

    def __dask_postpersist__(self):
        return Tuple._rebuild, ("mysuffix",)

    @staticmethod
    def _rebuild(futures: dict, name: str):
        expr = LLGExpr({
            (name, i): DataNode((name, i), val)
            for i, val in  enumerate(futures.values())
        })
        return Tuple(expr)

    def __dask_tokenize__(self):
        # For tokenize to work we want to return a value that fully
        # represents this object. In this case this is done by a type
        identifier plus the (also tokenized) name of the expression
        return (type(self), self._expr._name)

class RemoteTuple(Expr):
    @property
    def npartitions(self):
        return len(self.operands)

    def __dask_keys__(self):
        return [(self._name, i) for i in range(self.npartitions)]

    def _task(self, name: Key, index: int) -> Task:
        return DataNode(name, self.operands[index])
```

Demonstrating this class:

```python
>>> from dask_tuple import Tuple

def from_pytuple(pytup: tuple) -> Tuple:
    return Tuple(RemoteTuple(*pytup))

>>> dask_tup = from_pytuple(tuple(range(5)))

>>> dask_tup.__dask_keys__()
[('remotetuple-b7ea9a26c3ab8287c78d11fd45f26793', 0),
('remotetuple-b7ea9a26c3ab8287c78d11fd45f26793', 1),
('remotetuple-b7ea9a26c3ab8287c78d11fd45f26793', 2)]

# Compute turns Tuple into a tuple
>>> dask_tup.compute()
(0, 1, 2)

# Persist turns Tuple into a Tuple, with each task already computed
>>> dask_tup2 = dask_tup.persist()
>>> isinstance(dask_tup2, Tuple)
True

>>> dask_tup2.__dask_graph__()
{('newname', 0): DataNode(0),
('newname', 1): DataNode(1),
('newname', 2): DataNode(2)}

>>> x2.compute()
(0, 1, 2)

# Run-time typechecking
>>> from dask.typing import DaskCollection
>>> isinstance(x, DaskCollection)
True
```

<a id="is-dask-collection"></a>

## Checking if an object is a Dask collection

To check if an object is a Dask collection, use
`dask.base.is_dask_collection`:

```python
>>> from dask.base import is_dask_collection
>>> from dask import delayed

>>> x = delayed(sum)([1, 2, 3])
>>> is_dask_collection(x)
True
>>> is_dask_collection(1)
False
```

<a id="deterministic-hashing"></a>

## Implementing Deterministic Hashing

Dask implements its own deterministic hash function to generate keys based on
the value of arguments.  This function is available as `dask.base.tokenize`.
Many common types already have implementations of `tokenize`, which can be
found in `dask/base.py`.

When creating your own custom classes, you may need to register a `tokenize`
implementation. There are two ways to do this:

1. The `__dask_tokenize__` method

   Where possible, it is recommended to define the `__dask_tokenize__` method.
   This method takes no arguments and should return a value fully
   representative of the object. It is a good idea to call `dask.base.normalize_token`
   from it before returning any non-trivial objects.
2. Register a function with `dask.base.normalize_token`

   If defining a method on the class isn’t possible or you need to
   customize the tokenize function for a class whose super-class is
   already registered (for example if you need to sub-class built-ins),
   you can register a tokenize function with the `normalize_token`
   dispatch.  The function should have the same signature as described
   above.

In both cases the implementation should be the same, where only the location of the
definition is different.

#### NOTE
Both Dask collections and normal Python objects can have
implementations of `tokenize` using either of the methods
described above.

### Example

```python
>>> from dask.base import tokenize, normalize_token

# Define a tokenize implementation using a method.
>>> class Point:
...     def __init__(self, x, y):
...         self.x = x
...         self.y = y
...
...     def __dask_tokenize__(self):
...         # This tuple fully represents self
...         # Wrap non-trivial objects with normalize_token before returning them
...         return normalize_token(Point), self.x, self.y

>>> x = Point(1, 2)
>>> tokenize(x)
'5988362b6e07087db2bc8e7c1c8cc560'
>>> tokenize(x) == tokenize(x)  # token is idempotent
True
>>> tokenize(Point(1, 2)) == tokenize(Point(1, 2))  # token is deterministic
True
>>> tokenize(Point(1, 2)) == tokenize(Point(2, 1))  # tokens are unique
False


# Register an implementation with normalize_token
>>> class Point3D:
...     def __init__(self, x, y, z):
...         self.x = x
...         self.y = y
...         self.z = z

>>> @normalize_token.register(Point3D)
... def normalize_point3d(x):
...     return normalize_token(Point3D), x.x, x.y, x.z

>>> y = Point3D(1, 2, 3)
>>> tokenize(y)
'5a7e9c3645aa44cf13d021c14452152e'
```

For more examples, see `dask/base.py` or any of the built-in Dask collections.


# custom-graphs.html.md

# Custom Graphs

There may be times when you want to do parallel computing but your application
doesn’t fit neatly into something like Dask Array or Dask Bag.  In these
cases, you can interact directly with the Dask schedulers.  These schedulers
operate well as standalone modules.

This separation provides a release valve for complex situations and allows
advanced projects to have additional opportunities for parallel execution, even if
those projects have an internal representation for their computations.  As Dask
schedulers improve or expand to distributed memory, code written to use Dask
schedulers will advance as well.

<a id="custom-graph-example"></a>

## Example

!["Dask graph for data pipeline"](images/pipeline.svg)

As discussed in the [motivation](graphs.md) and [specification](spec.md)
sections, the schedulers take a task graph (which is a dict of tuples of
functions) and a list of desired keys from that graph.

Here is a mocked out example building a graph for a traditional clean and
analyze pipeline:

```python
def load(filename):
    ...

def clean(data):
    ...

def analyze(sequence_of_data):
    ...

def store(result):
    with open(..., 'w') as f:
        f.write(result)

dsk = {'load-1': (load, 'myfile.a.data'),
       'load-2': (load, 'myfile.b.data'),
       'load-3': (load, 'myfile.c.data'),
       'clean-1': (clean, 'load-1'),
       'clean-2': (clean, 'load-2'),
       'clean-3': (clean, 'load-3'),
       'analyze': (analyze, ['clean-%d' % i for i in [1, 2, 3]]),
       'store': (store, 'analyze')}

from dask.threaded import get
get(dsk, 'store')  # executes in parallel
```

## Keyword arguments in custom Dask graphs

Sometimes, you may want to pass keyword arguments to a function
in a custom Dask graph.
You can do that using the [`dask.utils.apply()`](api.md#dask.utils.apply) function, like this:

```python
from dask.utils import apply

task = (apply, func, args, kwargs)  # equivalent to func(*args, **kwargs)

dsk = {'task-name': task,
        ...
       }
```

In the example above:

- `args` should be a tuple (eg: `(arg_1, arg_2, arg_3)`), and
- `kwargs` should be a dictionary (eg: `{"kwarg_1": value, "kwarg_2": value}`

## Related Projects

The following excellent projects also provide parallel execution:

* [Joblib](https://joblib.readthedocs.io/en/latest/)
* [Multiprocessing](https://docs.python.org/3/library/multiprocessing.html)
* [IPython Parallel](https://ipyparallel.readthedocs.io/en/latest/)
* [Concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html)
* [Luigi](https://luigi.readthedocs.io)

Each library lets you dictate how your tasks relate to each other with various
levels of sophistication.  Each library executes those tasks with some internal
logic.

Dask schedulers differ in the following ways:

1. You specify the entire graph as a Python dict rather than using a
   specialized API.
2. You get a variety of schedulers, ranging from a single-machine, single-core
   scheduler to threaded, multi-process, and distributed options.
3. You benefit from logic to execute the graph in a way that minimizes memory
   footprint with the Dask single-machine schedulers.

But the other projects offer different advantages and different programming
paradigms.  One should inspect all such projects before selecting one.


# customize-initialization.html.md

# Customize Initialization

Often we want to run custom code when we start up or tear down a scheduler or
worker.  We might do this manually with functions like `Client.run` or
`Client.run_on_scheduler`, but this is error-prone and difficult to automate.

To resolve this, Dask includes a few mechanisms to run arbitrary code around
the lifecycle of a Scheduler, Worker, Nanny, or Client.

## Preload Scripts

Both `dask-scheduler` and `dask-worker` support a `--preload` option that
allows custom initialization of each scheduler/worker respectively. A module or
Python file passed as a `--preload` value is guaranteed to be imported before
establishing any connection. A `dask_setup(service)` function is called if
found, with a `Scheduler`, `Worker`, `Nanny`, or `Client` instance as
the argument. As the service stops, `dask_teardown(service)` is called if
present.

To support additional configuration, a single `--preload` module may register
additional command-line arguments by exposing `dask_setup` as a  [Click](http://click.pocoo.org/)
command.  This command will be used to parse additional arguments provided to
`dask-worker` or `dask-scheduler` and will be called before service
initialization.

### Example

As an example, consider the following file that creates a
[scheduler plugin](https://distributed.dask.org/en/latest/plugins.html)
and registers it with the scheduler

```python
# scheduler-setup.py
import click

from distributed.diagnostics.plugin import SchedulerPlugin

class MyPlugin(SchedulerPlugin):
    def __init__(self, print_count):
      self.print_count = print_count
      super().__init__()

    def add_worker(self, scheduler=None, worker=None, **kwargs):
        print("Added a new worker at:", worker)
        if self.print_count and scheduler is not None:
            print("Total workers:", len(scheduler.workers))

@click.command()
@click.option("--print-count/--no-print-count", default=False)
def dask_setup(scheduler, print_count):
    plugin = MyPlugin(print_count)
    scheduler.add_plugin(plugin)
```

We can then run this preload script by referring to its filename (or module name
if it is on the path) when we start the scheduler:

```default
dask-scheduler --preload scheduler-setup.py --print-count
```

### Types

Preloads can be specified as any of the following forms:

- A path to a script, like `/path/to/myfile.py`
- A module name that is on the path, like `my_module.initialize`
- The text of a Python script, like `import os; os.environ["A"] = "value"`

### Configuration

Preloads can also be registered with configuration at the following values:

```yaml
distributed:
  scheduler:
    preload:
    - "import os; os.environ['A'] = 'b'"  # use Python text
    - /path/to/myfile.py                  # or a filename
    - my_module                           # or a module name
    preload-argv:
    - []                                  # Pass optional keywords
    - ["--option", "value"]
    - []
  worker:
    preload: []
    preload-argv: []
  nanny:
    preload: []
    preload-argv: []
  client:
    preload: []
    preload-argv: []
```

#### NOTE
Because the `dask-worker` command needs to accept keywords for both the
Worker and the Nanny (if a nanny is used) it has both a `--preload` and
`--preload-nanny` keyword.  All extra keywords (like `--print-count`
above) will be sent to the workers rather than the nanny.  There is no way
to specify extra keywords to the nanny preload scripts on the command line.
We recommend the use of the more flexible configuration if this is
necessary.

## Worker Lifecycle Plugins

You can also create a class with `setup`, `teardown`, and `transition` methods,
and register that class with the scheduler to give to every worker using the
`Client.register_worker_plugin` method.


# dashboard.html.md

# Dashboard Diagnostics

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/N_GqzcuGLCY"
        frameborder="0"
        allow="autoplay; encrypted-media"
        style="margin: 0 auto 20px auto; display: block;"
        allowfullscreen>
</iframe>

Profiling parallel code can be challenging, but the interactive dashboard provided with
Dask’s [distributed scheduler](scheduling.md) makes this easier with live monitoring
of your Dask computations. The dashboard is built with [Bokeh](https://docs.bokeh.org)
and will start up automatically, returning a link to the dashboard whenever the scheduler is created.

Locally, this is when you create a [`Client`](futures.md#distributed.Client) and connect the scheduler:

```python
from dask.distributed import Client
client = Client()  # start distributed scheduler locally.
```

In a Jupyter Notebook or JupyterLab session displaying the client object will show the dashboard address:

![Client html repr displaying the dashboard link in a JupyterLab session.](images/dashboard_link.png)

You can also query the address from `client.dashboard_link` (or for older versions of distributed, `client.scheduler_info()['services']`).

By default, when starting a scheduler on your local machine the dashboard will be served at `http://localhost:8787/status`. You can type this address into your browser to access the dashboard, but may be directed
elsewhere if port 8787 is taken. You can also configure the address using the `dashboard_address`
parameter (see [`LocalCluster`](deploying-python.md#distributed.deploy.local.LocalCluster)).

There are numerous diagnostic plots available. In this guide you’ll learn about some
of the most commonly used plots shown on the entry point for the dashboard:

- [Bytes Stored and Bytes per Worker](#dashboard-memory): Cluster memory and Memory per worker
- [Task Processing/CPU Utilization/Occupancy/Data Transfer](#dashboard-proc-cpu-occ):  Tasks being processed by each worker/ CPU Utilization per worker/ Expected runtime for all tasks currently on a worker.
- [Task Stream](#dashboard-task-stream): Individual task across threads.
- [Progress](#dashboard-progress): Progress of a set of tasks.

![Main dashboard with five panes arranged into two columns. In the left column there are three bar charts. The top two show total bytes stored and bytes per worker. The bottom has three tabs to toggle between task processing, CPU utilization, and occupancy. In the right column, there are two bar charts with corresponding colors showing task activity over time, referred to as task stream and progress.](images/dashboard_status.png)

<a id="dashboard-memory"></a>

## Bytes Stored and Bytes per Worker

These two plots show a summary of the overall memory usage on the cluster (Bytes Stored),
as well as the individual usage on each worker (Bytes per Worker). The colors on these plots
indicate the following.

<table>
    <tr>
        <td>
            <div role="img" aria-label="blue square" style="color:rgba(0, 0, 255, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Memory under target (default 60% of memory available) </td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="orange square" style="color:rgba(255, 165, 0, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td> Memory is close to the spilling to disk target (default 70% of memory available)</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="red square" style="color:rgba(255, 0, 0, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>When the worker (or at least one worker) is paused (default 80% of memory available) or retiring</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="grey square" style="color:rgba(128, 128, 128, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Memory spilled to disk</td>
    </tr>
</table>![Two bar charts on memory usage. The top chart shows the total cluster memory in a single bar with mostly under target memory - changing colors according to memory usage, (blue - under target, orange - Memory is about to be spilled, red - paused or retiring, and a small part of spilled to disk in grey. The bottom chart displays the memory usage per worker, with a separate bar for each of the four workers. The four bars can be seen in various colours as in blue when under target, orange as their worker's memory are close to the spilling to disk target, with the second and fourth worker standing out with a portion in grey that correspond to the amount spilled to disk, also fourth worker in red is paused or about to retire.](images/dashboard_memory_new.gif)

The different levels of transparency on these plots are related to the type of memory
(Managed, Unmanaged and Unmanaged recent), and you can find a detailed explanation of them in the
Worker Memory management documentation

<a id="dashboard-proc-cpu-occ"></a>

## Task Processing/CPU Utilization/Occupancy/Data Transfer

**Task Processing**

The *Processing* tab in the figure shows the number of tasks that have been assigned to each worker. Not all of these
tasks are necessarily *executing* at the moment: a worker only executes as many tasks at once as it has threads. Any
extra tasks assigned to the worker will wait to run, depending on their priority and whether their
dependencies are in memory on the worker.

The scheduler will try to ensure that the workers are processing about the same number of tasks. If one of the bars is
completely white it means that worker has no tasks and is waiting for them. This usually happens when the computations
are close to finished (nothing to worry about), but it can also mean that the distribution of the task across workers is
not optimized.

There are three different colors that can appear in this plot:

<table>
    <tr>
        <td>
            <div role="img" aria-label="blue square" style="color:rgba(0, 0, 255, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Processing tasks.</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="green square" style="color:rgba(0, 128, 0, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Saturated: It has enough work to stay busy.</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="red square" style="color:rgba(255, 0, 0, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Idle: Does not have enough work to stay busy.</td>
    </tr>
</table>![Task Processing bar chart, showing a relatively even number of tasks on each worker.](images/dashboard_task_processing.png)

In this plot on the dashboard we have two extra tabs with the following information:

**CPU Utilization**

The *CPU* tab shows the cpu usage per-worker as reported by `psutil` metrics.

**Occupancy**

The *Occupancy* tab shows the occupancy, in time, per worker. The total occupancy for a worker is the amount of time Dask expects it would take
to run all the tasks, and transfer any of their dependencies from other workers, *if the execution and transfers happened one-by-one*.
For example, if a worker has an occupancy of 10s, and it has 2 threads, you can expect it to take about 5s of wall-clock time for the worker
to complete all its tasks.

**Data Transfer**

The *Data Transfer* tab shows the size of open data transfers from/to other workers, per worker.

<a id="dashboard-task-stream"></a>

## Task Stream

The task stream is a view of which tasks have been running on each thread of each worker. Each row represents a thread, and each rectangle represents
an individual task. The color for each rectangle corresponds to the task-prefix of the task being performed, and matches the color
of the [Progress plot](#dashboard-progress). This means that all the individual tasks which are part of the `inc` task-prefix, for example, will have
the same color (which is chosen randomly from the viridis color map).

Note that when a new worker joins, it will get a new row, even if it’s replacing a worker that recently left. So it’s possible to temporarily
see more rows on the task stream than there are currently threads in the cluster, because both the history of the old worker and the new worker
will be displayed.

There are certain colors that are reserved for a specific kinds of operations:

<table>
    <tr>
        <td>
            <div role="img" aria-label="light red square" style="color:rgba(255, 0, 0, 0.4); font-size: 25px ">&#9632;</div>
        </td>
        <td>Transferring data between workers.</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="light orange square" style="color: rgba(255,165,0, 0.4); font-size: 25px ">&#9632;</div>
        </td>
        <td>Reading from or writing to disk.</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="light grey square" style="color:rgba(128,128,128, 0.4); font-size: 25px ">&#9632;</div>
        </td>
        <td>Serializing/deserializing data.</td>
    </tr>
    <tr>
        <td>
            <div role="img" aria-label="black square" style="color:rgba(0, 0, 0, 1); font-size: 25px ">&#9632;</div>
        </td>
        <td>Erred tasks.</td>
    </tr>
</table>

In some scenarios, the dashboard will have white spaces between each rectangle. During that time, the worker thread was idle.
Having too much white space is an indication of sub-optimal use of resources. Additionally, a lot of long red bars (transfers) can indicate
a performance problem, due to anything from too large of chunk sizes, too complex of a graph, or even poor scheduling choices.

![The stacked bar chart, with one bar per worker-thread, has different shades of blue and green for different tasks, with occasional, very narrow red bars overlapping them.](images/dashboard_taskstream_healthy.png)![The stacked bar chart, with one bar per worker-thread, is mostly empty (white). Each row only has a few occasional spurts of activity. There are also red and orange bars, some of which are long and don't overlap with the other colors.](images/dashboard_task_stream_unhealthy.png)

<a id="dashboard-progress"></a>

## Progress

The progress bars plot shows the progress of each individual task-prefix. The color of each bar matches the color of the
individual tasks on the task stream from the same task-prefix. Each horizontal bar has four different components, from left to right:

<ul style="list-style-type: none">
    <li>
        <span role="img" aria-label="light teal square" style="background:rgba(30,151,138, 0.6); width: 0.6em; height: 0.6em; border: 1px solid rgba(30,151,138, 0.6); display: inline-block"></span>
        <span>Tasks that have completed, are not needed anymore, and now have been released from memory.</span>
    </li>
    <li>
        <span role="img" aria-label="teal square" style="background:rgba(30,151,138, 1); width: 0.6em; height: 0.6em; border: 1px solid rgba(30,151,138, 1); display: inline-block"></span>
        <span> Tasks that have completed and are in memory.</span>
    </li>
    <li>
        <span role="img" aria-label="light grey square" style="background:rgba(128,128,128, 0.4); width: 0.6em; height: 0.6em; border: 1px solid rgba(128,128,128, 0.4); display: inline-block"></span>
        <span>Tasks that are ready to run.</span>
    </li>
    <li>
        <span role="img" aria-label="hashed light grey square" style="background-image: linear-gradient(135deg, rgba(128,128,128, 0.4) 25%, #ffffff 25%, #ffffff 50%, rgba(128,128,128, 0.4) 50%, rgba(128,128,128, 0.4) 75%, #ffffff 75%, #ffffff 100%); width: 0.6em; height: 0.6em; border: 1px solid rgba(128,128,128, 0.4); display: inline-block"></span>
        <span>Tasks that are <a href="https://distributed.dask.org/en/stable/scheduling-policies.html#queuing">queued</a>. They are ready to run, but not assigned to workers yet, so higher-priority tasks can run first.</span>
    </li>
    <li>
        <span role="img" aria-label="hashed red square" style="background-image: linear-gradient(135deg, rgba(255,0,0, 0.35) 20%, rgba(0,0,0, 0.35) 25%, rgba(0,0,0, 0.35) 50%, rgba(255,0,0, 0.35) 50%, rgba(255,0,0, 0.35) 75%, rgba(0,0,0, 0.35) 75%, rgba(0,0,0, 0.35) 100%); width: 0.6em; height: 0.6em; border: 1px solid rgba(128,128,128, 0.4); display: inline-block"></span>
        <span>Tasks that do not have a worker to run on due to <a href="https://distributed.dask.org/en/stable/locality.html#user-control">restrictions</a> or limited <a href="https://distributed.dask.org/en/stable/resources.html">resources</a>.</span>
    </li>
</ul>![Progress bar chart with one bar for each task-prefix matching with the names "add", "double", "inc", and "sum". The "double", "inc" and "add" bars have a progress of approximately one third of the total tasks, displayed in their individual color with different transparency levels. The "double" and "inc" bars have a striped grey background, and the "sum" bar is empty.](images/dashboard_progress.png)

## Dask JupyterLab Extension

The [JupyterLab Dask extension](https://github.com/dask/dask-labextension#dask-jupyterlab-extension)
allows you to embed Dask’s dashboard plots directly into JupyterLab panes.

Once the JupyterLab Dask extension is installed you can choose any of the individual plots available and
integrated as a pane in your JupyterLab session. For example, in the figure below we selected the *Task Stream*,
*Progress*, *Workers Memory*, and *Graph* plots.

![Dask JupyterLab extension showing an arrangement of four panes selected from a display of plot options. The panes displayed are the Task stream, Bytes per worker, Progress and the Task Graph.](images/dashboard_jupyterlab.png)


# dask.array.Array.all.html.md

# dask.array.Array.all

#### Array.all(axis=None, keepdims=False, split_every=None, out=None)

Returns True if all elements evaluate to True.

Refer to [`dask.array.all()`](dask.array.all.md#dask.array.all) for full documentation.

#### SEE ALSO
[`dask.array.all`](dask.array.all.md#dask.array.all)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.any.html.md

# dask.array.Array.any

#### Array.any(axis=None, keepdims=False, split_every=None, out=None)

Returns True if any of the elements evaluate to True.

Refer to [`dask.array.any()`](dask.array.any.md#dask.array.any) for full documentation.

#### SEE ALSO
[`dask.array.any`](dask.array.any.md#dask.array.any)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.argmax.html.md

# dask.array.Array.argmax

#### Array.argmax(axis=None, , keepdims=False, split_every=None, out=None)

Return indices of the maximum values along the given axis.

Refer to [`dask.array.argmax()`](dask.array.argmax.md#dask.array.argmax) for full documentation.

#### SEE ALSO
[`dask.array.argmax`](dask.array.argmax.md#dask.array.argmax)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.argmin.html.md

# dask.array.Array.argmin

#### Array.argmin(axis=None, , keepdims=False, split_every=None, out=None)

Return indices of the minimum values along the given axis.

Refer to [`dask.array.argmin()`](dask.array.argmin.md#dask.array.argmin) for full documentation.

#### SEE ALSO
[`dask.array.argmin`](dask.array.argmin.md#dask.array.argmin)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.argtopk.html.md

# dask.array.Array.argtopk

#### Array.argtopk(k, axis=-1, split_every=None)

The indices of the top k elements of an array.

Refer to [`dask.array.argtopk()`](dask.array.argtopk.md#dask.array.argtopk) for full documentation.

#### SEE ALSO
[`dask.array.argtopk`](dask.array.argtopk.md#dask.array.argtopk)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.astype.html.md

# dask.array.Array.astype

#### Array.astype(dtype, \*\*kwargs)

Copy of the array, cast to a specified type.

* **Parameters:**
  **dtype**
  : Typecode or data-type to which the array is cast.

  **casting**
  : Controls what kind of data casting may occur. Defaults to ‘unsafe’
    for backwards compatibility.
    * ‘no’ means the data types should not be cast at all.
    * ‘equiv’ means only byte-order changes are allowed.
    * ‘safe’ means only casts which can preserve values are allowed.
    * ‘same_kind’ means only safe casts or casts within a kind,
      : like float64 to float32, are allowed.
    * ‘unsafe’ means any data conversions may be done.

  **copy**
  : By default, astype always returns a newly allocated array. If this
    is set to False and the dtype requirement is satisfied, the input
    array is returned instead of a copy.
    <br/>
    #### NOTE
    Dask does not respect the contiguous memory layout of the array,
    and will ignore the `order` keyword argument.
    The default order is ‘C’ contiguous.

<!-- !! processed by numpydoc !! -->


# dask.array.Array.blocks.html.md

# dask.array.Array.blocks

#### *property* Array.blocks

An array-like interface to the blocks of an array.

This returns a `Blockview` object that provides an array-like interface
to the blocks of a dask array.  Numpy-style indexing of a `Blockview` object
returns a selection of blocks as a new dask array.

You can index `array.blocks` like a numpy array of shape
equal to the number of blocks in each dimension, (available as
array.blocks.size).  The dimensionality of the output array matches
the dimension of this array, even if integer indices are passed.
Slicing with `np.newaxis` or multiple lists is not supported.

* **Returns:**
  An instance of `dask.array.Blockview`

### Examples

```pycon
>>> import dask.array as da
>>> x = da.arange(8, chunks=2)
>>> x.blocks.shape # aliases x.numblocks
(4,)
>>> x.blocks[0].compute()
array([0, 1])
>>> x.blocks[:3].compute()
array([0, 1, 2, 3, 4, 5])
>>> x.blocks[::2].compute()
array([0, 1, 4, 5])
>>> x.blocks[[-1, 0]].compute()
array([6, 7, 0, 1])
>>> x.blocks.ravel()
[dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>]
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.choose.html.md

# dask.array.Array.choose

#### Array.choose(choices)

Use an index array to construct a new array from a set of choices.

Refer to [`dask.array.choose()`](dask.array.choose.md#dask.array.choose) for full documentation.

#### SEE ALSO
[`dask.array.choose`](dask.array.choose.md#dask.array.choose)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.chunks.html.md

# dask.array.Array.chunks

#### *property* Array.chunks

Chunks property.

<!-- !! processed by numpydoc !! -->


# dask.array.Array.chunksize.html.md

# dask.array.Array.chunksize

#### *property* Array.chunksize *: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float), ...]*

<!-- !! processed by numpydoc !! -->


# dask.array.Array.clip.html.md

# dask.array.Array.clip

#### Array.clip(min=None, max=None)

Return an array whose values are limited to `[min, max]`.
One of max or min must be given.

Refer to [`dask.array.clip()`](dask.array.clip.md#dask.array.clip) for full documentation.

#### SEE ALSO
[`dask.array.clip`](dask.array.clip.md#dask.array.clip)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.compute.html.md

# dask.array.Array.compute

#### Array.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.array.Array.compute_chunk_sizes.html.md

# dask.array.Array.compute_chunk_sizes

#### Array.compute_chunk_sizes()

Compute the chunk sizes for a Dask array. This is especially useful
when the chunk sizes are unknown (e.g., when indexing one Dask array
with another).

### Notes

This function modifies the Dask array in-place.

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> x = da.from_array([-2, -1, 0, 1, 2], chunks=2)
>>> x.chunks
((2, 2, 1),)
>>> y = x[x <= 0]
>>> y.chunks
((nan, nan, nan),)
>>> y.compute_chunk_sizes()  # in-place computation
dask.array<getitem, shape=(3,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>
>>> y.chunks
((2, 1, 0),)
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.conj.html.md

# dask.array.Array.conj

#### Array.conj()

Complex-conjugate all elements.

Refer to [`dask.array.conj()`](dask.array.conj.md#dask.array.conj) for full documentation.

#### SEE ALSO
[`dask.array.conj`](dask.array.conj.md#dask.array.conj)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.copy.html.md

# dask.array.Array.copy

#### Array.copy()

Copy array.  This is a no-op for dask.arrays, which are immutable

<!-- !! processed by numpydoc !! -->


# dask.array.Array.cumprod.html.md

# dask.array.Array.cumprod

#### Array.cumprod(axis, dtype=None, out=None, , method='sequential')

Return the cumulative product of the elements along the given axis.

Refer to [`dask.array.cumprod()`](dask.array.cumprod.md#dask.array.cumprod) for full documentation.

#### SEE ALSO
[`dask.array.cumprod`](dask.array.cumprod.md#dask.array.cumprod)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.cumsum.html.md

# dask.array.Array.cumsum

#### Array.cumsum(axis, dtype=None, out=None, , method='sequential')

Return the cumulative sum of the elements along the given axis.

Refer to [`dask.array.cumsum()`](dask.array.cumsum.md#dask.array.cumsum) for full documentation.

#### SEE ALSO
[`dask.array.cumsum`](dask.array.cumsum.md#dask.array.cumsum)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.dask.html.md

# dask.array.Array.dask

#### Array.dask

<!-- !! processed by numpydoc !! -->


# dask.array.Array.dot.html.md

# dask.array.Array.dot

#### Array.dot(other)

Dot product of self and other.

Refer to [`dask.array.tensordot()`](dask.array.tensordot.md#dask.array.tensordot) for full documentation.

#### SEE ALSO
[`dask.array.dot`](dask.array.dot.md#dask.array.dot)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.dtype.html.md

# dask.array.Array.dtype

#### *property* Array.dtype

<!-- !! processed by numpydoc !! -->


# dask.array.Array.flatten.html.md

# dask.array.Array.flatten

#### Array.flatten()

Return a flattened array.

Refer to [`dask.array.ravel()`](dask.array.ravel.md#dask.array.ravel) for full documentation.

#### SEE ALSO
[`dask.array.ravel`](dask.array.ravel.md#dask.array.ravel)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.html.md

# dask.array.Array

### *class* dask.array.Array(dask, name, chunks, dtype=None, meta=None, shape=None)

Parallel Dask Array

A parallel nd-array comprised of many numpy arrays arranged in a grid.

This constructor is for advanced uses only.  For normal use see the
[`dask.array.from_array()`](dask.array.from_array.md#dask.array.from_array) function.

* **Parameters:**
  **dask**
  : Task dependency graph

  **name**
  : Name of array in dask

  **chunks: iterable of tuples**
  : block sizes along each dimension

  **dtype**
  : Typecode or data-type for the new Dask Array

  **meta**
  : empty ndarray created with same NumPy backend, ndim and dtype as the
    Dask Array being created (overrides dtype)

  **shape**
  : Shape of the entire array

#### SEE ALSO
[`dask.array.from_array`](dask.array.from_array.md#dask.array.from_array)

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(\*args, \*\*kwargs)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.array.Array.__init__)(\*args, \*\*kwargs)                                                            |                                                                                                                   |
|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|
| [`all`](dask.array.Array.all.md#dask.array.Array.all)([axis, keepdims, split_every, out])                               | Returns True if all elements evaluate to True.                                                                    |
| [`any`](dask.array.Array.any.md#dask.array.Array.any)([axis, keepdims, split_every, out])                               | Returns True if any of the elements evaluate to True.                                                             |
| [`argmax`](dask.array.Array.argmax.md#dask.array.Array.argmax)([axis, keepdims, split_every, out])                      | Return indices of the maximum values along the given axis.                                                        |
| [`argmin`](dask.array.Array.argmin.md#dask.array.Array.argmin)([axis, keepdims, split_every, out])                      | Return indices of the minimum values along the given axis.                                                        |
| [`argtopk`](dask.array.Array.argtopk.md#dask.array.Array.argtopk)(k[, axis, split_every])                               | The indices of the top k elements of an array.                                                                    |
| [`astype`](dask.array.Array.astype.md#dask.array.Array.astype)(dtype, \*\*kwargs)                                       | Copy of the array, cast to a specified type.                                                                      |
| [`choose`](dask.array.Array.choose.md#dask.array.Array.choose)(choices)                                                 | Use an index array to construct a new array from a set of choices.                                                |
| [`clip`](dask.array.Array.clip.md#dask.array.Array.clip)([min, max])                                                    | Return an array whose values are limited to `[min, max]`.                                                         |
| [`compute`](dask.array.Array.compute.md#dask.array.Array.compute)(\*\*kwargs)                                           | Compute this dask collection                                                                                      |
| [`compute_chunk_sizes`](dask.array.Array.compute_chunk_sizes.md#dask.array.Array.compute_chunk_sizes)()                 | Compute the chunk sizes for a Dask array.                                                                         |
| [`conj`](dask.array.Array.conj.md#dask.array.Array.conj)()                                                              | Complex-conjugate all elements.                                                                                   |
| [`copy`](dask.array.Array.copy.md#dask.array.Array.copy)()                                                              | Copy array.                                                                                                       |
| [`cumprod`](dask.array.Array.cumprod.md#dask.array.Array.cumprod)(axis[, dtype, out, method])                           | Return the cumulative product of the elements along the given axis.                                               |
| [`cumsum`](dask.array.Array.cumsum.md#dask.array.Array.cumsum)(axis[, dtype, out, method])                              | Return the cumulative sum of the elements along the given axis.                                                   |
| [`dot`](dask.array.Array.dot.md#dask.array.Array.dot)(other)                                                            | Dot product of self and other.                                                                                    |
| [`flatten`](dask.array.Array.flatten.md#dask.array.Array.flatten)()                                                     | Return a flattened array.                                                                                         |
| [`map_blocks`](dask.array.Array.map_blocks.md#dask.array.Array.map_blocks)(\*args[, name, token, dtype, ...])           | Map a function across all blocks of a dask array.                                                                 |
| [`map_overlap`](dask.array.Array.map_overlap.md#dask.array.Array.map_overlap)(func, depth[, boundary, trim])            | Map a function over blocks of the array with some overlap                                                         |
| [`max`](dask.array.Array.max.md#dask.array.Array.max)([axis, keepdims, split_every, out])                               | Return the maximum along a given axis.                                                                            |
| [`mean`](dask.array.Array.mean.md#dask.array.Array.mean)([axis, dtype, keepdims, split_every, out])                     | Returns the average of the array elements along given axis.                                                       |
| [`min`](dask.array.Array.min.md#dask.array.Array.min)([axis, keepdims, split_every, out])                               | Return the minimum along a given axis.                                                                            |
| [`moment`](dask.array.Array.moment.md#dask.array.Array.moment)(order[, axis, dtype, keepdims, ddof, ...])               | Calculate the nth centralized moment.                                                                             |
| [`nonzero`](dask.array.Array.nonzero.md#dask.array.Array.nonzero)()                                                     | Return the indices of the elements that are non-zero.                                                             |
| [`persist`](dask.array.Array.persist.md#dask.array.Array.persist)(\*\*kwargs)                                           | Persist this dask collection into memory                                                                          |
| [`prod`](dask.array.Array.prod.md#dask.array.Array.prod)([axis, dtype, keepdims, split_every, out])                     | Return the product of the array elements over the given axis                                                      |
| [`ravel`](dask.array.Array.ravel.md#dask.array.Array.ravel)()                                                           | Return a flattened array.                                                                                         |
| [`rechunk`](dask.array.Array.rechunk.md#dask.array.Array.rechunk)([chunks, threshold, ...])                             | Convert blocks in dask array x for new chunks.                                                                    |
| [`repeat`](dask.array.Array.repeat.md#dask.array.Array.repeat)(repeats[, axis])                                         | Repeat elements of an array.                                                                                      |
| [`reshape`](dask.array.Array.reshape.md#dask.array.Array.reshape)(\*shape[, merge_chunks, limit])                       | Reshape array to new shape                                                                                        |
| [`round`](dask.array.Array.round.md#dask.array.Array.round)([decimals])                                                 | Return array with each element rounded to the given number of decimals.                                           |
| [`shuffle`](dask.array.Array.shuffle.md#dask.array.Array.shuffle)(indexer, axis[, chunks])                              | Reorders one dimensions of a Dask Array based on an indexer.                                                      |
| [`squeeze`](dask.array.Array.squeeze.md#dask.array.Array.squeeze)([axis])                                               | Remove axes of length one from array.                                                                             |
| [`std`](dask.array.Array.std.md#dask.array.Array.std)([axis, dtype, keepdims, ddof, ...])                               | Returns the standard deviation of the array elements along given axis.                                            |
| [`store`](dask.array.Array.store.md#dask.array.Array.store)(targets[, lock, regions, compute, ...])                     | Store dask arrays in array-like objects, overwrite data in target                                                 |
| [`sum`](dask.array.Array.sum.md#dask.array.Array.sum)([axis, dtype, keepdims, split_every, out])                        | Return the sum of the array elements over the given axis.                                                         |
| [`swapaxes`](dask.array.Array.swapaxes.md#dask.array.Array.swapaxes)(axis1, axis2)                                      | Return a view of the array with `axis1` and `axis2` interchanged.                                                 |
| [`to_backend`](dask.array.Array.to_backend.md#dask.array.Array.to_backend)([backend])                                   | Move to a new Array backend                                                                                       |
| [`to_dask_dataframe`](dask.array.Array.to_dask_dataframe.md#dask.array.Array.to_dask_dataframe)([columns, index, meta]) | Convert dask Array to dask Dataframe                                                                              |
| [`to_delayed`](dask.array.Array.to_delayed.md#dask.array.Array.to_delayed)([optimize_graph])                            | Convert into an array of [`dask.delayed.Delayed`](../delayed-api.md#dask.delayed.Delayed) objects, one per chunk. |
| [`to_hdf5`](dask.array.Array.to_hdf5.md#dask.array.Array.to_hdf5)(filename, datapath, \*\*kwargs)                       | Store array in HDF5 file                                                                                          |
| [`to_svg`](dask.array.Array.to_svg.md#dask.array.Array.to_svg)([size])                                                  | Convert chunks from Dask Array into an SVG Image                                                                  |
| [`to_tiledb`](dask.array.Array.to_tiledb.md#dask.array.Array.to_tiledb)(uri, \*args, \*\*kwargs)                        | Save array to the TileDB storage manager                                                                          |
| [`to_zarr`](dask.array.Array.to_zarr.md#dask.array.Array.to_zarr)(\*args, \*\*kwargs)                                   | Save array to the zarr storage format                                                                             |
| [`topk`](dask.array.Array.topk.md#dask.array.Array.topk)(k[, axis, split_every])                                        | The top k elements of an array.                                                                                   |
| [`trace`](dask.array.Array.trace.md#dask.array.Array.trace)([offset, axis1, axis2, dtype])                              | Return the sum along diagonals of the array.                                                                      |
| [`transpose`](dask.array.Array.transpose.md#dask.array.Array.transpose)(\*axes)                                         | Reverse or permute the axes of an array.                                                                          |
| [`var`](dask.array.Array.var.md#dask.array.Array.var)([axis, dtype, keepdims, ddof, ...])                               | Returns the variance of the array elements, along given axis.                                                     |
| [`view`](dask.array.Array.view.md#dask.array.Array.view)([dtype, order])                                                | Get a view of the array as a new data type                                                                        |
| [`visualize`](dask.array.Array.visualize.md#dask.array.Array.visualize)([filename, format, optimize_graph])             | Render the computation of this object's task graph using graphviz.                                                |

### Attributes

| [`dask`](dask.array.Array.dask.md#dask.array.Array.dask)                      |                                                    |
|-------------------------------------------------------------------------------|----------------------------------------------------|
| `A`                                                                           |                                                    |
| `T`                                                                           |                                                    |
| [`blocks`](dask.array.Array.blocks.md#dask.array.Array.blocks)                | An array-like interface to the blocks of an array. |
| [`chunks`](dask.array.Array.chunks.md#dask.array.Array.chunks)                | Chunks property.                                   |
| [`chunksize`](dask.array.Array.chunksize.md#dask.array.Array.chunksize)       |                                                    |
| [`dtype`](dask.array.Array.dtype.md#dask.array.Array.dtype)                   |                                                    |
| [`imag`](dask.array.Array.imag.md#dask.array.Array.imag)                      |                                                    |
| [`itemsize`](dask.array.Array.itemsize.md#dask.array.Array.itemsize)          | Length of one array element in bytes               |
| [`name`](dask.array.Array.name.md#dask.array.Array.name)                      |                                                    |
| [`nbytes`](dask.array.Array.nbytes.md#dask.array.Array.nbytes)                | Number of bytes in array                           |
| [`ndim`](dask.array.Array.ndim.md#dask.array.Array.ndim)                      |                                                    |
| [`npartitions`](dask.array.Array.npartitions.md#dask.array.Array.npartitions) |                                                    |
| [`numblocks`](dask.array.Array.numblocks.md#dask.array.Array.numblocks)       |                                                    |
| [`partitions`](dask.array.Array.partitions.md#dask.array.Array.partitions)    | Slice an array by partitions.                      |
| [`real`](dask.array.Array.real.md#dask.array.Array.real)                      |                                                    |
| [`shape`](dask.array.Array.shape.md#dask.array.Array.shape)                   |                                                    |
| [`size`](dask.array.Array.size.md#dask.array.Array.size)                      | Number of elements in array                        |
| [`vindex`](dask.array.Array.vindex.md#dask.array.Array.vindex)                | Vectorized indexing with broadcasting.             |


# dask.array.Array.imag.html.md

# dask.array.Array.imag

#### *property* Array.imag

<!-- !! processed by numpydoc !! -->


# dask.array.Array.itemsize.html.md

# dask.array.Array.itemsize

#### *property* Array.itemsize *: [int](https://docs.python.org/3/library/functions.html#int)*

Length of one array element in bytes

<!-- !! processed by numpydoc !! -->


# dask.array.Array.map_blocks.html.md

# dask.array.Array.map_blocks

#### Array.map_blocks(\*args, name=None, token=None, dtype=None, chunks=None, drop_axis=None, new_axis=None, enforce_ndim=False, meta=None, \*\*kwargs)

Map a function across all blocks of a dask array.

Note that `map_blocks` will attempt to automatically determine the output
array type by calling `func` on 0-d versions of the inputs. Please refer to
the `meta` keyword argument below if you expect that the function will not
succeed when operating on 0-d arrays.

* **Parameters:**
  **func**
  : Function to apply to every block in the array.
    If `func` accepts `block_info=` or `block_id=`
    as keyword arguments, these will be passed dictionaries
    containing information about input and output chunks/arrays
    during computation. See examples for details.

  **args**

  **dtype**
  : The `dtype` of the output array. It is recommended to provide this.
    If not provided, will be inferred by applying the function to a small
    set of fake data.

  **chunks**
  : Chunk shape of resulting blocks if the function does not preserve
    shape. If not provided, the resulting array is assumed to have the same
    block structure as the first input array.

  **drop_axis**
  : Dimensions lost by the function.

  **new_axis**
  : New dimensions created by the function. Note that these are applied
    after `drop_axis` (if present). The size of each chunk along this
    dimension will be set to 1. Please specify `chunks` if the individual
    chunks have a different size.

  **enforce_ndim**
  : Whether to enforce at runtime that the dimensionality of the array
    produced by `func` actually matches that of the array returned by
    `map_blocks`.
    If True, this will raise an error when there is a mismatch.

  **token**
  : The key prefix to use for the output array. If not provided, will be
    determined from the function name.

  **name**
  : The key name to use for the output array. Note that this fully
    specifies the output key name, and must be unique. If not provided,
    will be determined by a hash of the arguments.

  **meta**
  : The `meta` of the output array, when specified is expected to be an
    array of the same type and dtype of that returned when calling `.compute()`
    on the array returned by this function. When not provided, `meta` will be
    inferred by applying the function to a small set of fake data, usually a
    0-d array. It’s important to ensure that `func` can successfully complete
    computation without raising exceptions when 0-d is passed to it, providing
    `meta` will be required otherwise. If the output type is known beforehand
    (e.g., `np.ndarray`, `cupy.ndarray`), an empty array of such type dtype
    can be passed, for example: `meta=np.array((), dtype=np.int32)`.

  **\*\*kwargs**
  : Other keyword arguments to pass to function. Values must be constants
    (not dask.arrays)

#### SEE ALSO
[`dask.array.map_overlap`](dask.array.map_overlap.md#dask.array.map_overlap)
: Generalized operation with overlap between neighbors.

[`dask.array.blockwise`](dask.array.blockwise.md#dask.array.blockwise)
: Generalized operation with control over block alignment.

### Examples

```pycon
>>> import dask.array as da
>>> x = da.arange(6, chunks=3)
```

```pycon
>>> x.map_blocks(lambda x: x * 2).compute()
array([ 0,  2,  4,  6,  8, 10])
```

The `da.map_blocks` function can also accept multiple arrays.

```pycon
>>> d = da.arange(5, chunks=2)
>>> e = da.arange(5, chunks=2)
```

```pycon
>>> f = da.map_blocks(lambda a, b: a + b**2, d, e)
>>> f.compute()
array([ 0,  2,  6, 12, 20])
```

If the function changes shape of the blocks then you must provide chunks
explicitly.

```pycon
>>> y = x.map_blocks(lambda x: x[::2], chunks=((2, 2),))
```

You have a bit of freedom in specifying chunks.  If all of the output chunk
sizes are the same, you can provide just that chunk size as a single tuple.

```pycon
>>> a = da.arange(18, chunks=(6,))
>>> b = a.map_blocks(lambda x: x[:3], chunks=(3,))
```

If the function changes the dimension of the blocks you must specify the
created or destroyed dimensions.

```pycon
>>> b = a.map_blocks(lambda x: x[None, :, None], chunks=(1, 6, 1),
...                  new_axis=[0, 2])
```

If `chunks` is specified but `new_axis` is not, then it is inferred to
add the necessary number of axes on the left.

Note that `map_blocks()` will concatenate chunks along axes specified by
the keyword parameter `drop_axis` prior to applying the function.
This is illustrated in the figure below:

![image](images/map_blocks_drop_axis.png)

Due to memory-size-constraints, it is often not advisable to use `drop_axis`
on an axis that is chunked.  In that case, it is better not to use
`map_blocks` but rather
`dask.array.reduction(..., axis=dropped_axes, concatenate=False)` which
maintains a leaner memory footprint while it drops any axis.

Map_blocks aligns blocks by block positions without regard to shape. In the
following example we have two arrays with the same number of blocks but
with different shape and chunk sizes.

```pycon
>>> x = da.arange(1000, chunks=(100,))
>>> y = da.arange(100, chunks=(10,))
```

The relevant attribute to match is numblocks.

```pycon
>>> x.numblocks
(10,)
>>> y.numblocks
(10,)
```

If these match (up to broadcasting rules) then we can map arbitrary
functions across blocks

```pycon
>>> def func(a, b):
...     return np.array([a.max(), b.max()])
```

```pycon
>>> da.map_blocks(func, x, y, chunks=(2,), dtype='i8')
dask.array<func, shape=(20,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>
```

```pycon
>>> _.compute()
array([ 99,   9, 199,  19, 299,  29, 399,  39, 499,  49, 599,  59, 699,
        69, 799,  79, 899,  89, 999,  99])
```

Your block function can get information about where it is in the array by
accepting a special `block_info` or `block_id` keyword argument.
During computation, they will contain information about each of the input
and output chunks (and dask arrays) relevant to each call of `func`.

```pycon
>>> def func(block_info=None):
...     pass
```

This will receive the following information:

```pycon
>>> block_info
{0: {'shape': (1000,),
     'num-chunks': (10,),
     'chunk-location': (4,),
     'array-location': [(400, 500)]},
 None: {'shape': (1000,),
        'num-chunks': (10,),
        'chunk-location': (4,),
        'array-location': [(400, 500)],
        'chunk-shape': (100,),
        'dtype': dtype('float64')}}
```

The keys to the `block_info` dictionary indicate which is the input and
output Dask array:

- **Input Dask array(s):** `block_info[0]` refers to the first input Dask array.
  The dictionary key is `0` because that is the argument index corresponding
  to the first input Dask array.
  In cases where multiple Dask arrays have been passed as input to the function,
  you can access them with the number corresponding to the input argument,
  eg: `block_info[1]`, `block_info[2]`, etc.
  (Note that if you pass multiple Dask arrays as input to map_blocks,
  the arrays must match each other by having matching numbers of chunks,
  along corresponding dimensions up to broadcasting rules.)
- **Output Dask array:** `block_info[None]` refers to the output Dask array,
  and contains information about the output chunks.
  The output chunk shape and dtype may may be different than the input chunks.

For each dask array, `block_info` describes:

- `shape`: the shape of the full Dask array,
- `num-chunks`: the number of chunks of the full array in each dimension,
- `chunk-location`: the chunk location (for example the fourth chunk over
  in the first dimension), and
- `array-location`: the array location within the full Dask array
  (for example the slice corresponding to `40:50`).

In addition to these, there are two extra parameters described by
`block_info` for the output array (in `block_info[None]`):

- `chunk-shape`: the output chunk shape, and
- `dtype`: the output dtype.

These features can be combined to synthesize an array from scratch, for
example:

```pycon
>>> def func(block_info=None):
...     loc = block_info[None]['array-location'][0]
...     return np.arange(loc[0], loc[1])
```

```pycon
>>> da.map_blocks(func, chunks=((4, 4),), dtype=np.float64)
dask.array<func, shape=(8,), dtype=float64, chunksize=(4,), chunktype=numpy.ndarray>
```

```pycon
>>> _.compute()
array([0, 1, 2, 3, 4, 5, 6, 7])
```

`block_id` is similar to `block_info` but contains only the `chunk_location`:

```pycon
>>> def func(block_id=None):
...     pass
```

This will receive the following information:

```pycon
>>> block_id
(4, 3)
```

You may specify the key name prefix of the resulting task in the graph with
the optional `token` keyword argument.

```pycon
>>> x.map_blocks(lambda x: x + 1, name='increment')
dask.array<increment, shape=(1000,), dtype=int64, chunksize=(100,), chunktype=numpy.ndarray>
```

For functions that may not handle 0-d arrays, it’s also possible to specify
`meta` with an empty array matching the type of the expected result. In
the example below, `func` will result in an `IndexError` when computing
`meta`:

```pycon
>>> rng = da.random.default_rng()
>>> da.map_blocks(lambda x: x[2], rng.random(5), meta=np.array(()))
dask.array<lambda, shape=(5,), dtype=float64, chunksize=(5,), chunktype=numpy.ndarray>
```

Similarly, it’s possible to specify a non-NumPy array to `meta`, and provide
a `dtype`:

```pycon
>>> import cupy
>>> rng = da.random.default_rng(cupy.random.default_rng())
>>> dt = np.float32
>>> da.map_blocks(lambda x: x[2], rng.random(5, dtype=dt), meta=cupy.array((), dtype=dt))
dask.array<lambda, shape=(5,), dtype=float32, chunksize=(5,), chunktype=cupy.ndarray>
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.map_overlap.html.md

# dask.array.Array.map_overlap

#### Array.map_overlap(func, depth, boundary=None, trim=True, \*\*kwargs)

Map a function over blocks of the array with some overlap

Refer to [`dask.array.map_overlap()`](dask.array.map_overlap.md#dask.array.map_overlap) for full documentation.

#### SEE ALSO
[`dask.array.map_overlap`](dask.array.map_overlap.md#dask.array.map_overlap)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.max.html.md

# dask.array.Array.max

#### Array.max(axis=None, keepdims=False, split_every=None, out=None)

Return the maximum along a given axis.

Refer to [`dask.array.max()`](dask.array.max.md#dask.array.max) for full documentation.

#### SEE ALSO
[`dask.array.max`](dask.array.max.md#dask.array.max)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.mean.html.md

# dask.array.Array.mean

#### Array.mean(axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Returns the average of the array elements along given axis.

Refer to [`dask.array.mean()`](dask.array.mean.md#dask.array.mean) for full documentation.

#### SEE ALSO
[`dask.array.mean`](dask.array.mean.md#dask.array.mean)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.min.html.md

# dask.array.Array.min

#### Array.min(axis=None, keepdims=False, split_every=None, out=None)

Return the minimum along a given axis.

Refer to [`dask.array.min()`](dask.array.min.md#dask.array.min) for full documentation.

#### SEE ALSO
[`dask.array.min`](dask.array.min.md#dask.array.min)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.moment.html.md

# dask.array.Array.moment

#### Array.moment(order, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Calculate the nth centralized moment.

Refer to [`dask.array.moment()`](dask.array.moment.md#dask.array.moment) for the full documentation.

#### SEE ALSO
[`dask.array.moment`](dask.array.moment.md#dask.array.moment)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.name.html.md

# dask.array.Array.name

#### *property* Array.name

<!-- !! processed by numpydoc !! -->


# dask.array.Array.nbytes.html.md

# dask.array.Array.nbytes

#### *property* Array.nbytes *: [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float)*

Number of bytes in array

<!-- !! processed by numpydoc !! -->


# dask.array.Array.ndim.html.md

# dask.array.Array.ndim

#### *property* Array.ndim *: [int](https://docs.python.org/3/library/functions.html#int)*

<!-- !! processed by numpydoc !! -->


# dask.array.Array.nonzero.html.md

# dask.array.Array.nonzero

#### Array.nonzero()

Return the indices of the elements that are non-zero.

Refer to [`dask.array.nonzero()`](dask.array.nonzero.md#dask.array.nonzero) for full documentation.

#### SEE ALSO
[`dask.array.nonzero`](dask.array.nonzero.md#dask.array.nonzero)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.npartitions.html.md

# dask.array.Array.npartitions

#### *property* Array.npartitions

<!-- !! processed by numpydoc !! -->


# dask.array.Array.numblocks.html.md

# dask.array.Array.numblocks

#### *property* Array.numblocks

<!-- !! processed by numpydoc !! -->


# dask.array.Array.partitions.html.md

# dask.array.Array.partitions

#### *property* Array.partitions

Slice an array by partitions. Alias of dask array .blocks attribute.

This alias allows you to write agnostic code that works with both
dask arrays and dask dataframes.

This returns a `Blockview` object that provides an array-like interface
to the blocks of a dask array.  Numpy-style indexing of a `Blockview` object
returns a selection of blocks as a new dask array.

You can index `array.blocks` like a numpy array of shape
equal to the number of blocks in each dimension, (available as
array.blocks.size).  The dimensionality of the output array matches
the dimension of this array, even if integer indices are passed.
Slicing with `np.newaxis` or multiple lists is not supported.

* **Returns:**
  An instance of `da.array.Blockview`

### Examples

```pycon
>>> import dask.array as da
>>> x = da.arange(8, chunks=2)
>>> x.partitions.shape # aliases x.numblocks
(4,)
>>> x.partitions[0].compute()
array([0, 1])
>>> x.partitions[:3].compute()
array([0, 1, 2, 3, 4, 5])
>>> x.partitions[::2].compute()
array([0, 1, 4, 5])
>>> x.partitions[[-1, 0]].compute()
array([6, 7, 0, 1])
>>> x.partitions.ravel()
[dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>,
 dask.array<blocks, shape=(2,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>]
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.persist.html.md

# dask.array.Array.persist

#### Array.persist(\*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.array.Array.prod.html.md

# dask.array.Array.prod

#### Array.prod(axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Return the product of the array elements over the given axis

Refer to [`dask.array.prod()`](dask.array.prod.md#dask.array.prod) for full documentation.

#### SEE ALSO
[`dask.array.prod`](dask.array.prod.md#dask.array.prod)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.ravel.html.md

# dask.array.Array.ravel

#### Array.ravel()

Return a flattened array.

Refer to [`dask.array.ravel()`](dask.array.ravel.md#dask.array.ravel) for full documentation.

#### SEE ALSO
[`dask.array.ravel`](dask.array.ravel.md#dask.array.ravel)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.real.html.md

# dask.array.Array.real

#### *property* Array.real

<!-- !! processed by numpydoc !! -->


# dask.array.Array.rechunk.html.md

# dask.array.Array.rechunk

#### Array.rechunk(chunks='auto', threshold=None, block_size_limit=None, balance=False, method=None)

Convert blocks in dask array x for new chunks.

Refer to [`dask.array.rechunk()`](dask.array.rechunk.md#dask.array.rechunk) for full documentation.

#### SEE ALSO
[`dask.array.rechunk`](dask.array.rechunk.md#dask.array.rechunk)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.repeat.html.md

# dask.array.Array.repeat

#### Array.repeat(repeats, axis=None)

Repeat elements of an array.

Refer to [`dask.array.repeat()`](dask.array.repeat.md#dask.array.repeat) for full documentation.

#### SEE ALSO
[`dask.array.repeat`](dask.array.repeat.md#dask.array.repeat)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.reshape.html.md

# dask.array.Array.reshape

#### Array.reshape(\*shape, merge_chunks=True, limit=None)

Reshape array to new shape

Refer to [`dask.array.reshape()`](dask.array.reshape.md#dask.array.reshape) for full documentation.

#### SEE ALSO
[`dask.array.reshape`](dask.array.reshape.md#dask.array.reshape)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.round.html.md

# dask.array.Array.round

#### Array.round(decimals=0)

Return array with each element rounded to the given number of decimals.

Refer to [`dask.array.round()`](dask.array.round.md#dask.array.round) for full documentation.

#### SEE ALSO
[`dask.array.round`](dask.array.round.md#dask.array.round)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.shape.html.md

# dask.array.Array.shape

#### *property* Array.shape *: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float), ...]*

<!-- !! processed by numpydoc !! -->


# dask.array.Array.shuffle.html.md

# dask.array.Array.shuffle

#### Array.shuffle(indexer: [list](https://docs.python.org/3/library/stdtypes.html#list)[[list](https://docs.python.org/3/library/stdtypes.html#list)[[int](https://docs.python.org/3/library/functions.html#int)]], axis: [int](https://docs.python.org/3/library/functions.html#int), chunks: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['auto'] = 'auto')

Reorders one dimensions of a Dask Array based on an indexer.

Refer to [`dask.array.shuffle()`](dask.array.shuffle.md#dask.array.shuffle) for full documentation.

#### SEE ALSO
[`dask.array.shuffle`](dask.array.shuffle.md#dask.array.shuffle)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.size.html.md

# dask.array.Array.size

#### *property* Array.size *: [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float)*

Number of elements in array

<!-- !! processed by numpydoc !! -->


# dask.array.Array.squeeze.html.md

# dask.array.Array.squeeze

#### Array.squeeze(axis=None)

Remove axes of length one from array.

Refer to [`dask.array.squeeze()`](dask.array.squeeze.md#dask.array.squeeze) for full documentation.

#### SEE ALSO
[`dask.array.squeeze`](dask.array.squeeze.md#dask.array.squeeze)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.std.html.md

# dask.array.Array.std

#### Array.std(axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Returns the standard deviation of the array elements along given axis.

Refer to [`dask.array.std()`](dask.array.std.md#dask.array.std) for full documentation.

#### SEE ALSO
[`dask.array.std`](dask.array.std.md#dask.array.std)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.store.html.md

# dask.array.Array.store

#### Array.store(targets: ArrayLike | [Delayed](../delayed-api.md#dask.delayed.Delayed) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[ArrayLike | [Delayed](../delayed-api.md#dask.delayed.Delayed)], lock: [bool](https://docs.python.org/3/library/functions.html#bool) | lock = True, regions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[slice](https://docs.python.org/3/library/functions.html#slice), ...] | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[slice](https://docs.python.org/3/library/functions.html#slice), ...]] | [None](https://docs.python.org/3/library/constants.html#None) = None, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True, return_stored: [bool](https://docs.python.org/3/library/functions.html#bool) = False, load_stored: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Store dask arrays in array-like objects, overwrite data in target

This stores dask arrays into object that supports numpy-style setitem
indexing.  It stores values chunk by chunk so that it does not have to
fill up memory.  For best performance you can align the block size of
the storage target with the block size of your array.

If your data fits in memory then you may prefer calling
`np.array(myarray)` instead.

* **Parameters:**
  **sources: Array or collection of Arrays**

  **targets: array-like or Delayed or collection of array-likes and/or Delayeds**
  : These should support setitem syntax `target[10:20] = ...`.
    If sources is a single item, targets must be a single item; if sources is a
    collection of arrays, targets must be a matching collection.

  **lock: boolean or threading.Lock, optional**
  : Whether or not to lock the data stores while storing.
    Pass True (lock each file individually), False (don’t lock) or a
    particular [`threading.Lock`](https://docs.python.org/3/library/threading.html#threading.Lock) object to be shared among all writes.

  **regions: tuple of slices or collection of tuples of slices, optional**
  : Each `region` tuple in `regions` should be such that
    `target[region].shape = source.shape`
    for the corresponding source and target in sources and targets,
    respectively. If this is a tuple, the contents will be assumed to be
    slices, so do not provide a tuple of tuples.

  **compute: boolean, optional**
  : If true compute immediately; return [`dask.delayed.Delayed`](../delayed-api.md#dask.delayed.Delayed) otherwise.

  **return_stored: boolean, optional**
  : Optionally return the stored result (default False).

  **load_stored: boolean, optional**
  : Optionally return the stored result, loaded in to memory (default None).
    If None, `load_stored` is True if `return_stored` is True and
    `compute` is False. *This is an advanced option.*
    When False, store will return the appropriate `target` for each chunk that is stored.
    Directly computing this result is not what you want.
    Instead, you can use the returned `target` to execute followup operations to the store.

  **kwargs:**
  : Parameters passed to compute/persist (only used if compute=True)
* **Returns:**
  If return_stored=True
  : tuple of Arrays

  If return_stored=False and compute=True
  : None

  If return_stored=False and compute=False
  : Delayed

### Examples

```pycon
>>> import h5py
>>> f = h5py.File('myfile.hdf5', mode='a')
>>> dset = f.create_dataset('/data', shape=x.shape,
...                                  chunks=x.chunks,
...                                  dtype='f8')
```

```pycon
>>> store(x, dset)
```

Alternatively store many arrays at the same time

```pycon
>>> store([x, y, z], [dset1, dset2, dset3])
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.sum.html.md

# dask.array.Array.sum

#### Array.sum(axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Return the sum of the array elements over the given axis.

Refer to [`dask.array.sum()`](dask.array.sum.md#dask.array.sum) for full documentation.

#### SEE ALSO
[`dask.array.sum`](dask.array.sum.md#dask.array.sum)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.swapaxes.html.md

# dask.array.Array.swapaxes

#### Array.swapaxes(axis1, axis2)

Return a view of the array with `axis1` and `axis2` interchanged.

Refer to [`dask.array.swapaxes()`](dask.array.swapaxes.md#dask.array.swapaxes) for full documentation.

#### SEE ALSO
[`dask.array.swapaxes`](dask.array.swapaxes.md#dask.array.swapaxes)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_backend.html.md

# dask.array.Array.to_backend

#### Array.to_backend(backend: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Move to a new Array backend

* **Parameters:**
  **backend**
  : The name of the new backend to move to. The default
    is the current “array.backend” configuration.
* **Returns:**
  Array

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_dask_dataframe.html.md

# dask.array.Array.to_dask_dataframe

#### Array.to_dask_dataframe(columns=None, index=None, meta=None)

Convert dask Array to dask Dataframe

* **Parameters:**
  **columns: list or string**
  : list of column names if DataFrame, single string if Series

  **index**
  : An optional *dask* Index to use for the output Series or DataFrame.
    <br/>
    The default output index depends on whether the array has any unknown
    chunks. If there are any unknown chunks, the output has `None`
    for all the divisions (one per chunk). If all the chunks are known,
    a default index with known divisions is created.
    <br/>
    Specifying `index` can be useful if you’re conforming a Dask Array
    to an existing dask Series or DataFrame, and you would like the
    indices to match.

  **meta**
  : An optional meta parameter can be passed for dask
    to specify the concrete dataframe type to use for partitions of
    the Dask dataframe. By default, pandas DataFrame is used.

#### SEE ALSO
[`dask.dataframe.from_dask_array`](dask.dataframe.from_dask_array.md#dask.dataframe.from_dask_array)

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_delayed.html.md

# dask.array.Array.to_delayed

#### Array.to_delayed(optimize_graph=True)

Convert into an array of [`dask.delayed.Delayed`](../delayed-api.md#dask.delayed.Delayed) objects, one per chunk.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    [`dask.delayed.Delayed`](../delayed-api.md#dask.delayed.Delayed) objects.

#### SEE ALSO
[`dask.array.from_delayed`](dask.array.from_delayed.md#dask.array.from_delayed)

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_hdf5.html.md

# dask.array.Array.to_hdf5

#### Array.to_hdf5(filename, datapath, \*\*kwargs)

Store array in HDF5 file

```pycon
>>> x.to_hdf5('myfile.hdf5', '/x')
```

Optionally provide arguments as though to `h5py.File.create_dataset`

```pycon
>>> x.to_hdf5('myfile.hdf5', '/x', compression='lzf', shuffle=True)
```

#### SEE ALSO
[`dask.array.store`](dask.array.store.md#dask.array.store)

`h5py.File.create_dataset`

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_svg.html.md

# dask.array.Array.to_svg

#### Array.to_svg(size=500)

Convert chunks from Dask Array into an SVG Image

* **Parameters:**
  **chunks: tuple**

  **size: int**
  : Rough size of the image
* **Returns:**
  text: An svg string depicting the array as a grid of chunks

### Examples

```pycon
>>> x.to_svg(size=500)
```

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_tiledb.html.md

# dask.array.Array.to_tiledb

#### Array.to_tiledb(uri, \*args, \*\*kwargs)

Save array to the TileDB storage manager

See [https://docs.tiledb.io](https://docs.tiledb.io) for details about the format and engine.

See function [`dask.array.to_tiledb()`](dask.array.to_tiledb.md#dask.array.to_tiledb) for argument documentation.

#### SEE ALSO
[`dask.array.to_tiledb`](dask.array.to_tiledb.md#dask.array.to_tiledb)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.to_zarr.html.md

# dask.array.Array.to_zarr

#### Array.to_zarr(\*args, \*\*kwargs)

Save array to the zarr storage format

See [https://zarr.readthedocs.io](https://zarr.readthedocs.io) for details about the format.

Refer to [`dask.array.to_zarr()`](dask.array.to_zarr.md#dask.array.to_zarr) for full documentation.

#### SEE ALSO
[`dask.array.to_zarr`](dask.array.to_zarr.md#dask.array.to_zarr)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.topk.html.md

# dask.array.Array.topk

#### Array.topk(k, axis=-1, split_every=None)

The top k elements of an array.

Refer to [`dask.array.topk()`](dask.array.topk.md#dask.array.topk) for full documentation.

#### SEE ALSO
[`dask.array.topk`](dask.array.topk.md#dask.array.topk)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.trace.html.md

# dask.array.Array.trace

#### Array.trace(offset=0, axis1=0, axis2=1, dtype=None)

Return the sum along diagonals of the array.

Refer to [`dask.array.trace()`](dask.array.trace.md#dask.array.trace) for full documentation.

#### SEE ALSO
[`dask.array.trace`](dask.array.trace.md#dask.array.trace)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.transpose.html.md

# dask.array.Array.transpose

#### Array.transpose(\*axes)

Reverse or permute the axes of an array. Return the modified array.

Refer to [`dask.array.transpose()`](dask.array.transpose.md#dask.array.transpose) for full documentation.

#### SEE ALSO
[`dask.array.transpose`](dask.array.transpose.md#dask.array.transpose)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.var.html.md

# dask.array.Array.var

#### Array.var(axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Returns the variance of the array elements, along given axis.

Refer to [`dask.array.var()`](dask.array.var.md#dask.array.var) for full documentation.

#### SEE ALSO
[`dask.array.var`](dask.array.var.md#dask.array.var)
: equivalent function

<!-- !! processed by numpydoc !! -->


# dask.array.Array.view.html.md

# dask.array.Array.view

#### Array.view(dtype=None, order='C')

Get a view of the array as a new data type

* **Parameters:**
  **dtype:**
  : The dtype by which to view the array.
    The default, None, results in the view having the same data-type
    as the original array.

  **order: string**
  : ‘C’ or ‘F’ (Fortran) ordering

  **This reinterprets the bytes of the array under a new dtype.  If that**

  **dtype does not have the same size as the original array then the shape**

  **will change.**

  **Beware that both numpy and dask.array can behave oddly when taking**

  **shape-changing views of arrays under Fortran ordering.  Under some**

  **versions of NumPy this function will fail when taking shape-changing**

  **views of Fortran ordered arrays if the first dimension has chunks of**

  **size one.**

<!-- !! processed by numpydoc !! -->


# dask.array.Array.vindex.html.md

# dask.array.Array.vindex

#### *property* Array.vindex

Vectorized indexing with broadcasting.

This is equivalent to numpy’s advanced indexing, using arrays that are
broadcast against each other. This allows for pointwise indexing:

```pycon
>>> import dask.array as da
>>> x = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>> x = da.from_array(x, chunks=2)
>>> x.vindex[[0, 1, 2], [0, 1, 2]].compute()
array([1, 5, 9])
```

Mixed basic/advanced indexing with slices/arrays is also supported. The
order of dimensions in the result follows those proposed for
[ndarray.vindex](https://github.com/numpy/numpy/pull/6256):
the subspace spanned by arrays is followed by all slices.

Note: `vindex` provides more general functionality than standard
indexing, but it also has fewer optimizations and can be significantly
slower.

<!-- !! processed by numpydoc !! -->


# dask.array.Array.visualize.html.md

# dask.array.Array.visualize

#### Array.visualize(filename='mydask', format=None, optimize_graph=False, \*\*kwargs)

Render the computation of this object’s task graph using graphviz.

Requires `graphviz` to be installed.

* **Parameters:**
  **filename**
  : The name of the file to write to disk. If the provided filename
    doesn’t include an extension, ‘.png’ will be used by default.
    If filename is None, no file will be written, and we communicate
    with dot using only pipes.

  **format**
  : Format in which to write output file.  Default is ‘png’.

  **optimize_graph**
  : If True, the graph is optimized before rendering.  Otherwise,
    the graph is displayed as is. Default is False.

  **color: {None, ‘order’}, optional**
  : Options to color nodes.  Provide `cmap=` keyword for additional
    colormap

  **\*\*kwargs**
  : Additional keyword arguments to forward to `to_graphviz`.
* **Returns:**
  **result**
  : See dask.dot.dot_graph for more information.

#### SEE ALSO
[`dask.visualize`](../api.md#dask.visualize)

`dask.dot.dot_graph`

### Notes

For more information on optimization see here:

[https://docs.dask.org/en/latest/optimize.html](https://docs.dask.org/en/latest/optimize.html)

### Examples

```pycon
>>> x.visualize(filename='dask.pdf')
>>> x.visualize(filename='dask.pdf', color='order')
```

<!-- !! processed by numpydoc !! -->


# dask.array.abs.html.md

# dask.array.abs

### dask.array.abs(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'absolute'>*

This docstring was copied from numpy.absolute.

Some inconsistencies with the Dask version may exist.

Calculate the absolute value element-wise.

`np.abs` is a shorthand for this function.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **absolute**
  : An ndarray containing the absolute value of
    each element in x.  For complex input, `a + ib`, the
    absolute value is $\sqrt{ a^2 + b^2 }$.
    This is a scalar if x is a scalar.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([-1.2, 1.2])
>>> np.absolute(x)
array([ 1.2,  1.2])
>>> np.absolute(1.2 + 1j)
1.5620499351813308
```

Plot the function over `[-10, 10]`:

```pycon
>>> import matplotlib.pyplot as plt
```

```pycon
>>> x = np.linspace(start=-10, stop=10, num=101)
>>> plt.plot(x, np.absolute(x))
>>> plt.show()
```

Plot the function over the complex plane:

```pycon
>>> xx = x + 1j * x[:, np.newaxis]
>>> plt.imshow(np.abs(xx), extent=[-10, 10, -10, 10], cmap='gray')
>>> plt.show()
```

The abs function can be used as a shorthand for `np.absolute` on
ndarrays.

```pycon
>>> x = np.array([-1.2, 1.2])
>>> abs(x)
array([1.2, 1.2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.absolute.html.md

# dask.array.absolute

### dask.array.absolute(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'absolute'>*

This docstring was copied from numpy.absolute.

Some inconsistencies with the Dask version may exist.

Calculate the absolute value element-wise.

`np.abs` is a shorthand for this function.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **absolute**
  : An ndarray containing the absolute value of
    each element in x.  For complex input, `a + ib`, the
    absolute value is $\sqrt{ a^2 + b^2 }$.
    This is a scalar if x is a scalar.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([-1.2, 1.2])
>>> np.absolute(x)
array([ 1.2,  1.2])
>>> np.absolute(1.2 + 1j)
1.5620499351813308
```

Plot the function over `[-10, 10]`:

```pycon
>>> import matplotlib.pyplot as plt
```

```pycon
>>> x = np.linspace(start=-10, stop=10, num=101)
>>> plt.plot(x, np.absolute(x))
>>> plt.show()
```

Plot the function over the complex plane:

```pycon
>>> xx = x + 1j * x[:, np.newaxis]
>>> plt.imshow(np.abs(xx), extent=[-10, 10, -10, 10], cmap='gray')
>>> plt.show()
```

The abs function can be used as a shorthand for `np.absolute` on
ndarrays.

```pycon
>>> x = np.array([-1.2, 1.2])
>>> abs(x)
array([1.2, 1.2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.add.html.md

# dask.array.add

### dask.array.add(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'add'>*

This docstring was copied from numpy.add.

Some inconsistencies with the Dask version may exist.

Add arguments element-wise.

* **Parameters:**
  **x1, x2**
  : The arrays to be added.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **add**
  : The sum of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

### Notes

Equivalent to x1 + x2 in terms of array broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.add(1.0, 4.0)
5.0
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> np.add(x1, x2)
array([[  0.,   2.,   4.],
       [  3.,   5.,   7.],
       [  6.,   8.,  10.]])
```

The `+` operator can be used as a shorthand for `np.add` on ndarrays.

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> x1 + x2
array([[ 0.,  2.,  4.],
       [ 3.,  5.,  7.],
       [ 6.,  8., 10.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.all.html.md

# dask.array.all

### dask.array.all(a, axis=None, keepdims=False, split_every=None, out=None)

Test whether all array elements along a given axis evaluate to True.

This docstring was copied from numpy.all.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array.

  **axis**
  : Axis or axes along which a logical AND reduction is performed.
    The default (`axis=None`) is to perform a logical AND over all
    the dimensions of the input array. axis may be negative, in
    which case it counts from the last to the first axis. If this
    is a tuple of ints, a reduction is performed on multiple
    axes, instead of a single axis or all the axes as before.

  **out**
  : Alternate output array in which to place the result.
    It must have the same shape as the expected output and its
    type is preserved (e.g., if `dtype(out)` is float, the result
    will consist of 0.0’s and 1.0’s). See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)
    for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the all method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **where**
  : Elements to include in checking for all True values.
    See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **all**
  : A new boolean or array is returned unless out is specified,
    in which case a reference to out is returned.

#### SEE ALSO
`ndarray.all`
: equivalent method

[`any`](dask.array.any.md#dask.array.any)
: Test whether any element along a given axis evaluates to True.

### Notes

Not a Number (NaN), positive infinity and negative infinity
evaluate to True because these are not equal to zero.

#### Versionchanged
Changed in version 2.0: Before NumPy 2.0, `all` did not return booleans for object dtype
input arrays.
This behavior is still available via `np.logical_and.reduce`.

### Examples

```pycon
>>> import numpy as np
>>> np.all([[True,False],[True,True]])
False
```

```pycon
>>> np.all([[True,False],[True,True]], axis=0)
array([ True, False])
```

```pycon
>>> np.all([-1, 4, 5])
True
```

```pycon
>>> np.all([1.0, np.nan])
True
```

```pycon
>>> np.all([[True, True], [False, True]], where=[[True], [False]])
True
```

```pycon
>>> o=np.array(False)
>>> z=np.all([-1, 4, 5], out=o)
>>> id(z), id(o), z
(28293632, 28293632, array(True)) # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.allclose.html.md

# dask.array.allclose

### dask.array.allclose(arr1, arr2, rtol=1e-05, atol=1e-08, equal_nan=False)

Returns True if two arrays are element-wise equal within a tolerance.

This docstring was copied from numpy.allclose.

Some inconsistencies with the Dask version may exist.

The tolerance values are positive, typically very small numbers.  The
relative difference (rtol \* abs(b)) and the absolute difference
atol are added together to compare against the absolute difference
between a and b.

#### WARNING
The default atol is not appropriate for comparing numbers
with magnitudes much smaller than one (see Notes).

NaNs are treated as equal if they are in the same place and if
`equal_nan=True`.  Infs are treated as equal if they are in the same
place and of the same sign in both arrays.

* **Parameters:**
  **a, b**
  : Input arrays to compare.

  **rtol**
  : The relative tolerance parameter (see Notes).

  **atol**
  : The absolute tolerance parameter (see Notes).

  **equal_nan**
  : Whether to compare NaN’s as equal.  If True, NaN’s in a will be
    considered equal to NaN’s in b in the output array.
* **Returns:**
  **allclose**
  : Returns True if the two arrays are equal within the given
    tolerance; False otherwise.

#### SEE ALSO
[`isclose`](dask.array.isclose.md#dask.array.isclose), [`all`](dask.array.all.md#dask.array.all), [`any`](dask.array.any.md#dask.array.any), [`equal`](dask.array.equal.md#dask.array.equal)

### Notes

If the following equation is element-wise True, then allclose returns
True.:

```default
absolute(a - b) <= (atol + rtol * absolute(b))
```

The above equation is not symmetric in a and b, so that
`allclose(a, b)` might be different from `allclose(b, a)` in
some rare cases.

The default value of atol is not appropriate when the reference value
b has magnitude smaller than one. For example, it is unlikely that
`a = 1e-9` and `b = 2e-9` should be considered “close”, yet
`allclose(1e-9, 2e-9)` is `True` with default settings. Be sure
to select atol for the use case at hand, especially for defining the
threshold below which a non-zero value in a will be considered “close”
to a very small or zero value in b.

The comparison of a and b uses standard broadcasting, which
means that a and b need not have the same shape in order for
`allclose(a, b)` to evaluate to True.  The same is true for
equal but not array_equal.

allclose is not defined for non-numeric data types.
bool is considered a numeric data-type for this purpose.

### Examples

```pycon
>>> import numpy as np
>>> np.allclose([1e10,1e-7], [1.00001e10,1e-8])
False
```

```pycon
>>> np.allclose([1e10,1e-8], [1.00001e10,1e-9])
True
```

```pycon
>>> np.allclose([1e10,1e-8], [1.0001e10,1e-9])
False
```

```pycon
>>> np.allclose([1.0, np.nan], [1.0, np.nan])
False
```

```pycon
>>> np.allclose([1.0, np.nan], [1.0, np.nan], equal_nan=True)
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.angle.html.md

# dask.array.angle

### dask.array.angle(x, deg=0)

Return the angle of the complex argument.

This docstring was copied from numpy.angle.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **z**
  : A complex number or sequence of complex numbers.

  **deg**
  : Return angle in degrees if True, radians if False (default).
* **Returns:**
  **angle**
  : The counterclockwise angle from the positive real axis on the complex
    plane in the range `(-pi, pi]`, with dtype as numpy.float64.

#### SEE ALSO
[`arctan2`](dask.array.arctan2.md#dask.array.arctan2)

[`absolute`](dask.array.absolute.md#dask.array.absolute)

### Notes

This function passes the imaginary and real parts of the argument to
arctan2 to compute the result; consequently, it follows the convention
of arctan2 when the magnitude of the argument is zero. See example.

### Examples

```pycon
>>> import numpy as np
>>> np.angle([1.0, 1.0j, 1+1j])               # in radians
array([ 0.        ,  1.57079633,  0.78539816]) # may vary
>>> np.angle(1+1j, deg=True)                  # in degrees
45.0
>>> np.angle([0., -0., complex(0., -0.), complex(-0., -0.)])  # convention
array([ 0.        ,  3.14159265, -0.        , -3.14159265])
```

<!-- !! processed by numpydoc !! -->


# dask.array.any.html.md

# dask.array.any

### dask.array.any(a, axis=None, keepdims=False, split_every=None, out=None)

Test whether any array element along a given axis evaluates to True.

This docstring was copied from numpy.any.

Some inconsistencies with the Dask version may exist.

Returns single boolean if axis is `None`

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array.

  **axis**
  : Axis or axes along which a logical OR reduction is performed.
    The default (`axis=None`) is to perform a logical OR over all
    the dimensions of the input array. axis may be negative, in
    which case it counts from the last to the first axis. If this
    is a tuple of ints, a reduction is performed on multiple
    axes, instead of a single axis or all the axes as before.

  **out**
  : Alternate output array in which to place the result.  It must have
    the same shape as the expected output and its type is preserved
    (e.g., if it is of type float, then it will remain so, returning
    1.0 for True and 0.0 for False, regardless of the type of a).
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the any method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **where**
  : Elements to include in checking for any True values.
    See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **any**
  : A new boolean or ndarray is returned unless out is specified,
    in which case a reference to out is returned.

#### SEE ALSO
`ndarray.any`
: equivalent method

[`all`](dask.array.all.md#dask.array.all)
: Test whether all elements along a given axis evaluate to True.

### Notes

Not a Number (NaN), positive infinity and negative infinity evaluate
to True because these are not equal to zero.

#### Versionchanged
Changed in version 2.0: Before NumPy 2.0, `any` did not return booleans for object dtype
input arrays.
This behavior is still available via `np.logical_or.reduce`.

### Examples

```pycon
>>> import numpy as np
>>> np.any([[True, False], [True, True]])
True
```

```pycon
>>> np.any([[True,  False, True ],
...         [False, False, False]], axis=0)
array([ True, False, True])
```

```pycon
>>> np.any([-1, 0, 5])
True
```

```pycon
>>> np.any([[np.nan], [np.inf]], axis=1, keepdims=True)
array([[ True],
       [ True]])
```

```pycon
>>> np.any([[True, False], [False, False]], where=[[False], [True]])
False
```

```pycon
>>> a = np.array([[1, 0, 0],
...               [0, 0, 1],
...               [0, 0, 0]])
>>> np.any(a, axis=0)
array([ True, False,  True])
>>> np.any(a, axis=1)
array([ True,  True, False])
```

```pycon
>>> o=np.array(False)
>>> z=np.any([-1, 4, 5], out=o)
>>> z, o
(array(True), array(True))
>>> # Check now that z is a reference to o
>>> z is o
True
>>> id(z), id(o) # identity of z and o
(191614240, 191614240)
```

<!-- !! processed by numpydoc !! -->


# dask.array.api.normalize_chunks.html.md

# dask.array.api.normalize_chunks

### dask.array.api.normalize_chunks(chunks, shape=None, limit=None, dtype=None, previous_chunks=None)

Normalize chunks to tuple of tuples

This takes in a variety of input types and information and produces a full
tuple-of-tuples result for chunks, suitable to be passed to Array or
rechunk or any other operation that creates a Dask array.

* **Parameters:**
  **chunks: tuple, int, dict, or string**
  : The chunks to be normalized.  See examples below for more details

  **shape: Tuple[int]**
  : The shape of the array

  **limit: int (optional)**
  : The maximum block size to target in bytes,
    if freedom is given to choose

  **dtype: np.dtype**

  **previous_chunks: Tuple[Tuple[int]] optional**
  : Chunks from a previous array that we should use for inspiration when
    rechunking auto dimensions.  If not provided but auto-chunking exists
    then auto-dimensions will prefer square-like chunk shapes.

### Examples

Fully explicit tuple-of-tuples

```pycon
>>> from dask.array.core import normalize_chunks
>>> normalize_chunks(((2, 2, 1), (2, 2, 2)), shape=(5, 6))
((2, 2, 1), (2, 2, 2))
```

Specify uniform chunk sizes

```pycon
>>> normalize_chunks((2, 2), shape=(5, 6))
((2, 2, 1), (2, 2, 2))
```

Cleans up missing outer tuple

```pycon
>>> normalize_chunks((3, 2), (5,))
((3, 2),)
```

Cleans up lists to tuples

```pycon
>>> normalize_chunks([[2, 2], [3, 3]])
((2, 2), (3, 3))
```

Expands integer inputs 10 -> (10, 10)

```pycon
>>> normalize_chunks(10, shape=(30, 5))
((10, 10, 10), (5,))
```

Expands dict inputs

```pycon
>>> normalize_chunks({0: 2, 1: 3}, shape=(6, 6))
((2, 2, 2), (3, 3))
```

The values -1 and None get mapped to full size

```pycon
>>> normalize_chunks((5, -1), shape=(10, 10))
((5, 5), (10,))
>>> normalize_chunks((5, None), shape=(10, 10))
((5, 5), (10,))
```

Use the value “auto” to automatically determine chunk sizes along certain
dimensions.  This uses the `limit=` and `dtype=` keywords to
determine how large to make the chunks.  The term “auto” can be used
anywhere an integer can be used.  See array chunking documentation for more
information.

```pycon
>>> normalize_chunks(("auto",), shape=(20,), limit=5, dtype='uint8')
((5, 5, 5, 5),)
>>> normalize_chunks("auto", (2, 3), dtype=np.int32)
((2,), (3,))
```

You can also use byte sizes (see [`dask.utils.parse_bytes()`](../api.md#dask.utils.parse_bytes)) in place of
“auto” to ask for a particular size

```pycon
>>> normalize_chunks("1kiB", shape=(2000,), dtype='float32')
((256, 256, 256, 256, 256, 256, 256, 208),)
```

Respects null dimensions

```pycon
>>> normalize_chunks(())
()
>>> normalize_chunks((), ())
()
>>> normalize_chunks((1,), ())
()
>>> normalize_chunks((), shape=(0, 0))
((0,), (0,))
```

Handles NaNs

```pycon
>>> normalize_chunks((1, (np.nan,)), (1, np.nan))
((1,), (nan,))
```

<!-- !! processed by numpydoc !! -->


# dask.array.api.normalize_chunks_cached.html.md

# dask.array.api.normalize_chunks_cached

### dask.array.api.normalize_chunks_cached(chunks, shape=None, limit=None, dtype=None, previous_chunks=None)

Cached version of normalize_chunks.

#### NOTE
chunks and previous_chunks are expected to be hashable. Dicts and lists aren’t
allowed for this function.

See [`normalize_chunks()`](dask.array.api.normalize_chunks.md#dask.array.api.normalize_chunks) for further documentation.

<!-- !! processed by numpydoc !! -->


# dask.array.append.html.md

# dask.array.append

### dask.array.append(arr, values, axis=None)

Append values to the end of an array.

This docstring was copied from numpy.append.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **arr**
  : Values are appended to a copy of this array.

  **values**
  : These values are appended to a copy of arr.  It must be of the
    correct shape (the same shape as arr, excluding axis).  If
    axis is not specified, values can be any shape and will be
    flattened before use.

  **axis**
  : The axis along which values are appended.  If axis is not
    given, both arr and values are flattened before use.
* **Returns:**
  **append**
  : A copy of arr with values appended to axis.  Note that
    append does not occur in-place: a new array is allocated and
    filled.  If axis is None, out is a flattened array.

#### SEE ALSO
[`insert`](dask.array.insert.md#dask.array.insert)
: Insert elements into an array.

[`delete`](dask.array.delete.md#dask.array.delete)
: Delete elements from an array.

### Examples

```pycon
>>> import numpy as np
>>> np.append([1, 2, 3], [[4, 5, 6], [7, 8, 9]])
array([1, 2, 3, ..., 7, 8, 9])
```

When axis is specified, values must have the correct shape.

```pycon
>>> np.append([[1, 2, 3], [4, 5, 6]], [[7, 8, 9]], axis=0)
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])
```

```pycon
>>> np.append([[1, 2, 3], [4, 5, 6]], [7, 8, 9], axis=0)
Traceback (most recent call last):
    ...
ValueError: all the input arrays must have same number of dimensions, but
the array at index 0 has 2 dimension(s) and the array at index 1 has 1
dimension(s)
```

```pycon
>>> a = np.array([1, 2], dtype=np.int_)
>>> c = np.append(a, [])
>>> c
array([1., 2.])
>>> c.dtype
float64
```

Default dtype for empty ndarrays is float64 thus making the output of dtype
float64 when appended with dtype int64

<!-- !! processed by numpydoc !! -->


# dask.array.apply_along_axis.html.md

# dask.array.apply_along_axis

### dask.array.apply_along_axis(func1d, axis, arr, \*args, dtype=None, shape=None, \*\*kwargs)

Apply a function to 1-D slices along the given axis.

This docstring was copied from numpy.apply_along_axis.

Some inconsistencies with the Dask version may exist.

This is a blocked variant of [`numpy.apply_along_axis()`](https://numpy.org/doc/stable/reference/generated/numpy.apply_along_axis.html#numpy.apply_along_axis) implemented via
[`dask.array.map_blocks()`](dask.array.map_blocks.md#dask.array.map_blocks)

* **Parameters:**
  **func1d**
  : This function should accept 1-D arrays. It is applied to 1-D
    slices of arr along the specified axis.

  **axis**
  : Axis along which arr is sliced.

  **arr**
  : Input array.

  **args**
  : Additional arguments to func1d.

  **kwargs**
  : Additional named arguments to func1d.
* **Returns:**
  **out**
  : The output array. The shape of out is identical to the shape of
    arr, except along the axis dimension. This axis is removed, and
    replaced with new dimensions equal to the shape of the return value
    of func1d. So if func1d returns a scalar out will have one
    fewer dimensions than arr.

#### SEE ALSO
[`apply_over_axes`](dask.array.apply_over_axes.md#dask.array.apply_over_axes)
: Apply a function repeatedly over multiple axes.

### Notes

If either of dtype or shape are not provided, Dask attempts to
determine them by calling func1d on a dummy array. This may produce
incorrect values for dtype or shape, so we recommend providing them.

Execute func1d(a, \*args, \*\*kwargs) where func1d operates on 1-D arrays
and a is a 1-D slice of arr along axis.

This is equivalent to (but faster than) the following use of ndindex and
s_, which sets each of `ii`, `jj`, and `kk` to a tuple of indices:

```default
Ni, Nk = a.shape[:axis], a.shape[axis+1:]
for ii in ndindex(Ni):
    for kk in ndindex(Nk):
        f = func1d(arr[ii + s_[:,] + kk])
        Nj = f.shape
        for jj in ndindex(Nj):
            out[ii + jj + kk] = f[jj]
```

Equivalently, eliminating the inner loop, this can be expressed as:

```default
Ni, Nk = a.shape[:axis], a.shape[axis+1:]
for ii in ndindex(Ni):
    for kk in ndindex(Nk):
        out[ii + s_[...,] + kk] = func1d(arr[ii + s_[:,] + kk])
```

### Examples

```pycon
>>> import numpy as np
>>> def my_func(a):
...     """Average first and last element of a 1-D array"""
...     return (a[0] + a[-1]) * 0.5
>>> b = np.array([[1,2,3], [4,5,6], [7,8,9]])
>>> np.apply_along_axis(my_func, 0, b)
array([4., 5., 6.])
>>> np.apply_along_axis(my_func, 1, b)
array([2.,  5.,  8.])
```

For a function that returns a 1D array, the number of dimensions in
outarr is the same as arr.

```pycon
>>> b = np.array([[8,1,7], [4,3,9], [5,2,6]])
>>> np.apply_along_axis(sorted, 1, b)
array([[1, 7, 8],
       [3, 4, 9],
       [2, 5, 6]])
```

For a function that returns a higher dimensional array, those dimensions
are inserted in place of the axis dimension.

```pycon
>>> b = np.array([[1,2,3], [4,5,6], [7,8,9]])
>>> np.apply_along_axis(np.diag, -1, b)
array([[[1, 0, 0],
        [0, 2, 0],
        [0, 0, 3]],
       [[4, 0, 0],
        [0, 5, 0],
        [0, 0, 6]],
       [[7, 0, 0],
        [0, 8, 0],
        [0, 0, 9]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.apply_over_axes.html.md

# dask.array.apply_over_axes

### dask.array.apply_over_axes(func, a, axes)

Apply a function repeatedly over multiple axes.

This docstring was copied from numpy.apply_over_axes.

Some inconsistencies with the Dask version may exist.

func is called as res = func(a, axis), where axis is the first
element of axes.  The result res of the function call must have
either the same dimensions as a or one less dimension.  If res
has one less dimension than a, a dimension is inserted before
axis.  The call to func is then repeated for each axis in axes,
with res as the first argument.

* **Parameters:**
  **func**
  : This function must take two arguments, func(a, axis).

  **a**
  : Input array.

  **axes**
  : Axes over which func is applied; the elements must be integers.
* **Returns:**
  **apply_over_axis**
  : The output array.  The number of dimensions is the same as a,
    but the shape can be different.  This depends on whether func
    changes the shape of its output with respect to its input.

#### SEE ALSO
[`apply_along_axis`](dask.array.apply_along_axis.md#dask.array.apply_along_axis)
: Apply a function to 1-D slices of an array along the given axis.

### Notes

This function is equivalent to tuple axis arguments to reorderable ufuncs
with keepdims=True. Tuple axis arguments to ufuncs have been available since
version 1.7.0.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(24).reshape(2,3,4)
>>> a
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]],
       [[12, 13, 14, 15],
        [16, 17, 18, 19],
        [20, 21, 22, 23]]])
```

Sum over axes 0 and 2. The result has same number of dimensions
as the original array:

```pycon
>>> np.apply_over_axes(np.sum, a, [0,2])
array([[[ 60],
        [ 92],
        [124]]])
```

Tuple axis arguments to ufuncs are equivalent:

```pycon
>>> np.sum(a, axis=(0,2), keepdims=True)
array([[[ 60],
        [ 92],
        [124]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.arange.html.md

# dask.array.arange

### dask.array.arange(start=None, , stop=None, step=1, , chunks='auto', like=None, dtype=None)

Return evenly spaced values from start to stop with step size step.

The values are half-open [start, stop), so including start and excluding
stop. This is basically the same as python’s range function but for dask
arrays.

When using a non-integer step, such as 0.1, the results will often not be
consistent. It is better to use linspace for these cases.

* **Parameters:**
  **start**
  : If `stop` is specified, the start of interval (inclusive); otherwise, the end
    of the interval (exclusive). Default: 0 when `stop` is specified.

  **stop**
  : The end of the interval (exclusive).

  **step**
  : The distance between two adjacent elements `(out[i+1] - out[i])`.
    Must not be 0; may be negative, this results in an empty array if stop >= start.
    Default: 1.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if `len(array) % chunks != 0`.
    Defaults to “auto” which will automatically determine chunk sizes.

  **dtype**
  : Output dtype. Omit to infer it from start, stop, step
    Defaults to `None`.

  **like**
  : Array to extract meta from. Defaults to `None`.
* **Returns:**
  **samples**

#### SEE ALSO
[`dask.array.linspace`](dask.array.linspace.md#dask.array.linspace)

<!-- !! processed by numpydoc !! -->


# dask.array.arccos.html.md

# dask.array.arccos

### dask.array.arccos(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arccos'>*

This docstring was copied from numpy.arccos.

Some inconsistencies with the Dask version may exist.

Trigonometric inverse cosine, element-wise.

The inverse of cos so that, if `y = cos(x)`, then `x = arccos(y)`.

* **Parameters:**
  **x**
  : x-coordinate on the unit circle.
    For real arguments, the domain is [-1, 1].

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **angle**
  : The angle of the ray intersecting the unit circle at the given
    x-coordinate in radians [0, pi].
    This is a scalar if x is a scalar.

#### SEE ALSO
[`cos`](dask.array.cos.md#dask.array.cos), [`arctan`](dask.array.arctan.md#dask.array.arctan), [`arcsin`](dask.array.arcsin.md#dask.array.arcsin), `emath.arccos`

### Notes

arccos is a multivalued function: for each x there are infinitely
many numbers z such that `cos(z) = x`. The convention is to return
the angle z whose real part lies in [0, pi].

For real-valued input data types, arccos always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, arccos is a complex analytic function that
has branch cuts `[-inf, -1]` and [1, inf] and is continuous from
above on the former and from below on the latter.

The inverse cos is also known as acos or cos^-1.

### References

M. Abramowitz and I.A. Stegun, “Handbook of Mathematical Functions”,
10th printing, 1964, pp. 79.
[https://personal.math.ubc.ca/~cbm/aands/page_79.htm](https://personal.math.ubc.ca/~cbm/aands/page_79.htm)

### Examples

```pycon
>>> import numpy as np
```

We expect the arccos of 1 to be 0, and of -1 to be pi:

```pycon
>>> np.arccos([1, -1])
array([ 0.        ,  3.14159265])
```

Plot arccos:

```pycon
>>> import matplotlib.pyplot as plt
>>> x = np.linspace(-1, 1, num=100)
>>> plt.plot(x, np.arccos(x))
>>> plt.axis('tight')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.arccosh.html.md

# dask.array.arccosh

### dask.array.arccosh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arccosh'>*

This docstring was copied from numpy.arccosh.

Some inconsistencies with the Dask version may exist.

Inverse hyperbolic cosine, element-wise.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **arccosh**
  : Array of the same shape as x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`cosh`](dask.array.cosh.md#dask.array.cosh), [`arcsinh`](dask.array.arcsinh.md#dask.array.arcsinh), [`sinh`](dask.array.sinh.md#dask.array.sinh), [`arctanh`](dask.array.arctanh.md#dask.array.arctanh), [`tanh`](dask.array.tanh.md#dask.array.tanh)

### Notes

arccosh is a multivalued function: for each x there are infinitely
many numbers z such that cosh(z) = x. The convention is to return the
z whose imaginary part lies in `[-pi, pi]` and the real part in
`[0, inf]`.

For real-valued input data types, arccosh always returns real output.
For each value that cannot be expressed as a real number or infinity, it
yields `nan` and sets the invalid floating point error flag.

For complex-valued input, arccosh is a complex analytical function that
has a branch cut [-inf, 1] and is continuous from above on it.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.arccosh([np.e, 10.0])
array([ 1.65745445,  2.99322285])
>>> np.arccosh(1)
0.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.arcsin.html.md

# dask.array.arcsin

### dask.array.arcsin(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arcsin'>*

This docstring was copied from numpy.arcsin.

Some inconsistencies with the Dask version may exist.

Inverse sine, element-wise.

* **Parameters:**
  **x**
  : y-coordinate on the unit circle.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **angle**
  : The inverse sine of each element in x, in radians and in the
    closed interval `[-pi/2, pi/2]`.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`sin`](dask.array.sin.md#dask.array.sin), [`cos`](dask.array.cos.md#dask.array.cos), [`arccos`](dask.array.arccos.md#dask.array.arccos), [`tan`](dask.array.tan.md#dask.array.tan), [`arctan`](dask.array.arctan.md#dask.array.arctan), [`arctan2`](dask.array.arctan2.md#dask.array.arctan2), `emath.arcsin`

### Notes

arcsin is a multivalued function: for each x there are infinitely
many numbers z such that $sin(z) = x$.  The convention is to
return the angle z whose real part lies in [-pi/2, pi/2].

For real-valued input data types, *arcsin* always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, arcsin is a complex analytic function that
has, by convention, the branch cuts [-inf, -1] and [1, inf]  and is
continuous from above on the former and from below on the latter.

The inverse sine is also known as asin or sin^{-1}.

### References

Abramowitz, M. and Stegun, I. A., *Handbook of Mathematical Functions*,
10th printing, New York: Dover, 1964, pp. 79ff.
[https://personal.math.ubc.ca/~cbm/aands/page_79.htm](https://personal.math.ubc.ca/~cbm/aands/page_79.htm)

### Examples

```pycon
>>> import numpy as np
>>> np.arcsin(1)     # pi/2
1.5707963267948966
>>> np.arcsin(-1)    # -pi/2
-1.5707963267948966
>>> np.arcsin(0)
0.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.arcsinh.html.md

# dask.array.arcsinh

### dask.array.arcsinh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arcsinh'>*

This docstring was copied from numpy.arcsinh.

Some inconsistencies with the Dask version may exist.

Inverse hyperbolic sine, element-wise.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Array of the same shape as x.
    This is a scalar if x is a scalar.

### Notes

arcsinh is a multivalued function: for each x there are infinitely
many numbers z such that sinh(z) = x. The convention is to return the
z whose imaginary part lies in [-pi/2, pi/2].

For real-valued input data types, arcsinh always returns real output.
For each value that cannot be expressed as a real number or infinity, it
returns `nan` and sets the invalid floating point error flag.

For complex-valued input, arcsinh is a complex analytical function that
has branch cuts [1j, infj] and [-1j, -infj] and is continuous from
the right on the former and from the left on the latter.

The inverse hyperbolic sine is also known as asinh or `sinh^-1`.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.arcsinh(np.array([np.e, 10.0]))
array([ 1.72538256,  2.99822295])
```

<!-- !! processed by numpydoc !! -->


# dask.array.arctan.html.md

# dask.array.arctan

### dask.array.arctan(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arctan'>*

This docstring was copied from numpy.arctan.

Some inconsistencies with the Dask version may exist.

Trigonometric inverse tangent, element-wise.

The inverse of tan, so that if `y = tan(x)` then `x = arctan(y)`.

* **Parameters:**
  **x**

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Out has the same shape as x.  Its real part is in
    `[-pi/2, pi/2]` (`arctan(+/-inf)` returns `+/-pi/2`).
    This is a scalar if x is a scalar.

#### SEE ALSO
[`arctan2`](dask.array.arctan2.md#dask.array.arctan2)
: The “four quadrant” arctan of the angle formed by (x, y) and the positive x-axis.

[`angle`](dask.array.angle.md#dask.array.angle)
: Argument of complex values.

### Notes

arctan is a multi-valued function: for each x there are infinitely
many numbers z such that tan(z) = x.  The convention is to return
the angle z whose real part lies in [-pi/2, pi/2].

For real-valued input data types, arctan always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, arctan is a complex analytic function that
has [`1j, infj`] and [`-1j, -infj`] as branch cuts, and is continuous
from the left on the former and from the right on the latter.

The inverse tangent is also known as atan or tan^{-1}.

### References

Abramowitz, M. and Stegun, I. A., *Handbook of Mathematical Functions*,
10th printing, New York: Dover, 1964, pp. 79.
[https://personal.math.ubc.ca/~cbm/aands/page_79.htm](https://personal.math.ubc.ca/~cbm/aands/page_79.htm)

### Examples

We expect the arctan of 0 to be 0, and of 1 to be pi/4:

```pycon
>>> import numpy as np
>>> np.arctan([0, 1])
array([ 0.        ,  0.78539816])
```

```pycon
>>> np.pi/4
0.78539816339744828
```

Plot arctan:

```pycon
>>> import matplotlib.pyplot as plt
>>> x = np.linspace(-10, 10)
>>> plt.plot(x, np.arctan(x))
>>> plt.axis('tight')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.arctan2.html.md

# dask.array.arctan2

### dask.array.arctan2(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arctan2'>*

This docstring was copied from numpy.arctan2.

Some inconsistencies with the Dask version may exist.

Element-wise arc tangent of `x1/x2` choosing the quadrant correctly.

The quadrant (i.e., branch) is chosen so that `arctan2(x1, x2)` is
the signed angle in radians between the ray ending at the origin and
passing through the point (1,0), and the ray ending at the origin and
passing through the point (x2, x1).  (Note the role reversal: the
“y-coordinate” is the first function parameter, the “x-coordinate”
is the second.)  By IEEE convention, this function is defined for
x2 = +/-0 and for either or both of x1 and x2 = +/-inf (see
Notes for specific values).

This function is not defined for complex-valued arguments; for the
so-called argument of complex values, use angle.

* **Parameters:**
  **x1**
  : y-coordinates.

  **x2**
  : x-coordinates.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **angle**
  : Array of angles in radians, in the range `[-pi, pi]`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`arctan`](dask.array.arctan.md#dask.array.arctan), [`tan`](dask.array.tan.md#dask.array.tan), [`angle`](dask.array.angle.md#dask.array.angle)

### Notes

*arctan2* is identical to the atan2 function of the underlying
C library.  The following special values are defined in the C
standard: [[1]](#r347210aad557-1)

| x1     | x2     | arctan2(x1,x2)   |
|--------|--------|------------------|
| +/- 0  | +0     | +/- 0            |
| +/- 0  | -0     | +/- pi           |
| > 0    | +/-inf | +0 / +pi         |
| < 0    | +/-inf | -0 / -pi         |
| +/-inf | +inf   | +/- (pi/4)       |
| +/-inf | -inf   | +/- (3\*pi/4)    |

Note that +0 and -0 are distinct floating point numbers, as are +inf
and -inf.

### References

### Examples

Consider four points in different quadrants:

```pycon
>>> import numpy as np
>>> x = np.array([-1, +1, +1, -1])
>>> y = np.array([-1, -1, +1, +1])
>>> np.arctan2(y, x) * 180 / np.pi
array([-135.,  -45.,   45.,  135.])
```

Note the order of the parameters. arctan2 is defined also when x2 = 0
and at several other special points, obtaining values in
the range `[-pi, pi]`:

```pycon
>>> np.arctan2([1., -1.], [0., 0.])
array([ 1.57079633, -1.57079633])
>>> np.arctan2([0., 0., np.inf], [+0., -0., np.inf])
array([0.        , 3.14159265, 0.78539816])
```

<!-- !! processed by numpydoc !! -->


# dask.array.arctanh.html.md

# dask.array.arctanh

### dask.array.arctanh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'arctanh'>*

This docstring was copied from numpy.arctanh.

Some inconsistencies with the Dask version may exist.

Inverse hyperbolic tangent, element-wise.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Array of the same shape as x.
    This is a scalar if x is a scalar.

#### SEE ALSO
`emath.arctanh`

### Notes

arctanh is a multivalued function: for each x there are infinitely
many numbers z such that `tanh(z) = x`. The convention is to return
the z whose imaginary part lies in [-pi/2, pi/2].

For real-valued input data types, arctanh always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, arctanh is a complex analytical function
that has branch cuts [-1, -inf] and [1, inf] and is continuous from
above on the former and from below on the latter.

The inverse hyperbolic tangent is also known as atanh or `tanh^-1`.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.arctanh([0, -0.5])
array([ 0.        , -0.54930614])
```

<!-- !! processed by numpydoc !! -->


# dask.array.argmax.html.md

# dask.array.argmax

### dask.array.argmax(a, axis=None, keepdims=False, split_every=None, out=None)

Returns the indices of the maximum values along an axis.

This docstring was copied from numpy.argmax.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : By default, the index is into the flattened array, otherwise
    along the specified axis.

  **out**
  : If provided, the result will be inserted into this array. It should
    be of the appropriate shape and dtype.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the array.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **index_array**
  : Array of indices into the array. It has the same shape as `a.shape`
    with the dimension along axis removed. If keepdims is set to True,
    then the size of axis will be 1 with the resulting array having same
    shape as `a.shape`.

#### SEE ALSO
`ndarray.argmax`, [`argmin`](dask.array.argmin.md#dask.array.argmin)

`amax`
: The maximum value along a given axis.

[`unravel_index`](dask.array.unravel_index.md#dask.array.unravel_index)
: Convert a flat index into an index tuple.

`take_along_axis`
: Apply `np.expand_dims(index_array, axis)` from argmax to an array as if by calling max.

### Notes

In case of multiple occurrences of the maximum values, the indices
corresponding to the first occurrence are returned.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(6).reshape(2,3) + 10
>>> a
array([[10, 11, 12],
       [13, 14, 15]])
>>> np.argmax(a)
5
>>> np.argmax(a, axis=0)
array([1, 1, 1])
>>> np.argmax(a, axis=1)
array([2, 2])
```

Indexes of the maximal elements of an N-dimensional array:

```pycon
>>> a.flat[np.argmax(a)]
15
>>> ind = np.unravel_index(np.argmax(a, axis=None), a.shape)
>>> ind
(1, 2)
>>> a[ind]
15
```

```pycon
>>> b = np.arange(6)
>>> b[1] = 5
>>> b
array([0, 5, 2, 3, 4, 5])
>>> np.argmax(b)  # Only the first occurrence is returned.
1
```

```pycon
>>> x = np.array([[4,2,3], [1,0,3]])
>>> index_array = np.argmax(x, axis=-1)
>>> # Same as np.amax(x, axis=-1, keepdims=True)
>>> np.take_along_axis(x, np.expand_dims(index_array, axis=-1), axis=-1)
array([[4],
       [3]])
>>> # Same as np.amax(x, axis=-1)
>>> np.take_along_axis(x, np.expand_dims(index_array, axis=-1),
...     axis=-1).squeeze(axis=-1)
array([4, 3])
```

Setting keepdims to True,

```pycon
>>> x = np.arange(24).reshape((2, 3, 4))
>>> res = np.argmax(x, axis=1, keepdims=True)
>>> res.shape
(2, 1, 4)
```

<!-- !! processed by numpydoc !! -->


# dask.array.argmin.html.md

# dask.array.argmin

### dask.array.argmin(a, axis=None, keepdims=False, split_every=None, out=None)

Returns the indices of the minimum values along an axis.

This docstring was copied from numpy.argmin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : By default, the index is into the flattened array, otherwise
    along the specified axis.

  **out**
  : If provided, the result will be inserted into this array. It should
    be of the appropriate shape and dtype.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the array.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **index_array**
  : Array of indices into the array. It has the same shape as a.shape
    with the dimension along axis removed. If keepdims is set to True,
    then the size of axis will be 1 with the resulting array having same
    shape as a.shape.

#### SEE ALSO
`ndarray.argmin`, [`argmax`](dask.array.argmax.md#dask.array.argmax)

`amin`
: The minimum value along a given axis.

[`unravel_index`](dask.array.unravel_index.md#dask.array.unravel_index)
: Convert a flat index into an index tuple.

`take_along_axis`
: Apply `np.expand_dims(index_array, axis)` from argmin to an array as if by calling min.

### Notes

In case of multiple occurrences of the minimum values, the indices
corresponding to the first occurrence are returned.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(6).reshape(2,3) + 10
>>> a
array([[10, 11, 12],
       [13, 14, 15]])
>>> np.argmin(a)
0
>>> np.argmin(a, axis=0)
array([0, 0, 0])
>>> np.argmin(a, axis=1)
array([0, 0])
```

Indices of the minimum elements of an N-dimensional array:

```pycon
>>> a.flat[np.argmin(a)]
10
>>> ind = np.unravel_index(np.argmin(a, axis=None), a.shape)
>>> ind
(0, 0)
>>> a[ind]
10
```

```pycon
>>> b = np.arange(6) + 10
>>> b[4] = 10
>>> b
array([10, 11, 12, 13, 10, 15])
>>> np.argmin(b)  # Only the first occurrence is returned.
0
```

```pycon
>>> x = np.array([[4,2,3], [1,0,3]])
>>> index_array = np.argmin(x, axis=-1)
>>> # Same as np.amin(x, axis=-1, keepdims=True)
>>> np.take_along_axis(x, np.expand_dims(index_array, axis=-1), axis=-1)
array([[2],
       [0]])
>>> # Same as np.amax(x, axis=-1)
>>> np.take_along_axis(x, np.expand_dims(index_array, axis=-1),
...     axis=-1).squeeze(axis=-1)
array([2, 0])
```

Setting keepdims to True,

```pycon
>>> x = np.arange(24).reshape((2, 3, 4))
>>> res = np.argmin(x, axis=1, keepdims=True)
>>> res.shape
(2, 1, 4)
```

<!-- !! processed by numpydoc !! -->


# dask.array.argtopk.html.md

# dask.array.argtopk

### dask.array.argtopk(a, k, axis=-1, split_every=None)

Extract the indices of the k largest elements from a on the given axis,
and return them sorted from largest to smallest. If k is negative, extract
the indices of the -k smallest elements instead, and return them sorted
from smallest to largest.

This performs best when `k` is much smaller than the chunk size. All
results will be returned in a single chunk along the given axis.

* **Parameters:**
  **x: Array**
  : Data being sorted

  **k: int**

  **axis: int, optional**

  **split_every: int >=2, optional**
  : See [`topk()`](dask.array.topk.md#dask.array.topk). The performance considerations for topk also apply
    here.
* **Returns:**
  Selection of np.intp indices of x with size abs(k) along the given axis.

### Examples

```pycon
>>> import dask.array as da
>>> x = np.array([5, 1, 3, 6])
>>> d = da.from_array(x, chunks=2)
>>> d.argtopk(2).compute()
array([3, 0])
>>> d.argtopk(-2).compute()
array([1, 2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.argwhere.html.md

# dask.array.argwhere

### dask.array.argwhere(a)

Find the indices of array elements that are non-zero, grouped by element.

This docstring was copied from numpy.argwhere.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.
* **Returns:**
  **index_array**
  : Indices of elements that are non-zero. Indices are grouped by element.
    This array will have shape `(N, a.ndim)` where `N` is the number of
    non-zero items.

#### SEE ALSO
[`where`](dask.array.where.md#dask.array.where), [`nonzero`](dask.array.nonzero.md#dask.array.nonzero)

### Notes

`np.argwhere(a)` is almost the same as `np.transpose(np.nonzero(a))`,
but produces a result of the correct shape for a 0D array.

The output of `argwhere` is not suitable for indexing arrays.
For this purpose use `nonzero(a)` instead.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6).reshape(2,3)
>>> x
array([[0, 1, 2],
       [3, 4, 5]])
>>> np.argwhere(x>1)
array([[0, 2],
       [1, 0],
       [1, 1],
       [1, 2]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.around.html.md

# dask.array.around

### dask.array.around(x, decimals=0)

Round an array to the given number of decimals.

This docstring was copied from numpy.around.

Some inconsistencies with the Dask version may exist.

around is an alias of ~numpy.round.

#### SEE ALSO
`ndarray.round`
: equivalent method

[`round`](dask.array.round.md#dask.array.round)
: alias for this function

[`ceil`](dask.array.ceil.md#dask.array.ceil), [`fix`](dask.array.fix.md#dask.array.fix), [`floor`](dask.array.floor.md#dask.array.floor), [`rint`](dask.array.rint.md#dask.array.rint), [`trunc`](dask.array.trunc.md#dask.array.trunc)

<!-- !! processed by numpydoc !! -->


# dask.array.array.html.md

# dask.array.array

### dask.array.array(object, dtype=None, , copy=True, order='K', subok=False, ndmin=0, ndmax=0, like=None)

This docstring was copied from numpy.array.

Some inconsistencies with the Dask version may exist.

Create an array.

* **Parameters:**
  **object**
  : An array, any object exposing the array interface, an object whose
    `__array__` method returns an array, or any (nested) sequence.
    If object is a scalar, a 0-dimensional array containing object is
    returned.

  **dtype**
  : The desired data-type for the array. If not given, NumPy will try to use
    a default `dtype` that can represent the values (by applying promotion
    rules when necessary.)

  **copy**
  : If `True` (default), then the array data is copied. If `None`,
    a copy will only be made if `__array__` returns a copy, if obj is
    a nested sequence, or if a copy is needed to satisfy any of the other
    requirements (`dtype`, `order`, etc.). Note that any copy of
    the data is shallow, i.e., for arrays with object dtype, the new
    array will point to the same objects. See Examples for ndarray.copy.
    For `False` it raises a `ValueError` if a copy cannot be avoided.
    Default: `True`.

  **order**
  : Specify the memory layout of the array. If object is not an array, the
    newly created array will be in C order (row major) unless ‘F’ is
    specified, in which case it will be in Fortran order (column major).
    If object is an array the following holds.
    <br/>
    | order   | no copy   | copy=True                                           |
    |---------|-----------|-----------------------------------------------------|
    | ‘K’     | unchanged | F & C order preserved, otherwise most similar order |
    | ‘A’     | unchanged | F order if input is F and not C, otherwise C order  |
    | ‘C’     | C order   | C order                                             |
    | ‘F’     | F order   | F order                                             |
    <br/>
    When `copy=None` and a copy is made for other reasons, the result is
    the same as if `copy=True`, with some exceptions for ‘A’, see the
    Notes section. The default order is ‘K’.

  **subok**
  : If True, then sub-classes will be passed-through, otherwise
    the returned array will be forced to be a base-class array (default).

  **ndmin**
  : Specifies the minimum number of dimensions that the resulting
    array should have.  Ones will be prepended to the shape as
    needed to meet this requirement.

  **ndmax**
  : Specifies the maximum number of dimensions to create when inferring
    shape from nested sequences. By default (ndmax=0), NumPy recurses
    through all nesting levels (up to the compile-time constant
    `NPY_MAXDIMS`).
    Setting `ndmax` stops recursion at the specified depth, preserving
    deeper nested structures as objects instead of promoting them to
    higher-dimensional arrays. In this case, `dtype=np.object_` is required.
    <br/>
    #### Versionadded
    Added in version 2.4.0.

  **like**
  : Reference object to allow the creation of arrays which are not
    NumPy arrays. If an array-like passed in as `like` supports
    the `__array_function__` protocol, the result will be defined
    by it. In this case, it ensures the creation of an array object
    compatible with that passed in via this argument.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **out**
  : An array object satisfying the specified requirements.

#### SEE ALSO
[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`full_like`](dask.array.full_like.md#dask.array.full_like)
: Return a new array with shape of input filled with value.

[`empty`](dask.array.empty.md#dask.array.empty)
: Return a new uninitialized array.

[`ones`](dask.array.ones.md#dask.array.ones)
: Return a new array setting values to one.

[`zeros`](dask.array.zeros.md#dask.array.zeros)
: Return a new array setting values to zero.

[`full`](dask.array.full.md#dask.array.full)
: Return a new array of given shape filled with value.

[`copy`](https://docs.python.org/3/library/copy.html#module-copy)
: Return an array copy of the given object.

### Notes

When order is ‘A’ and `object` is an array in neither ‘C’ nor ‘F’ order,
and a copy is forced by a change in dtype, then the order of the result is
not necessarily ‘C’ as expected. This is likely a bug.

### Examples

```pycon
>>> import numpy as np
>>> np.array([1, 2, 3])
array([1, 2, 3])
```

Upcasting:

```pycon
>>> np.array([1, 2, 3.0])
array([ 1.,  2.,  3.])
```

More than one dimension:

```pycon
>>> np.array([[1, 2], [3, 4]])
array([[1, 2],
       [3, 4]])
```

Minimum dimensions 2:

```pycon
>>> np.array([1, 2, 3], ndmin=2)
array([[1, 2, 3]])
```

Type provided:

```pycon
>>> np.array([1, 2, 3], dtype=np.complex128)
array([ 1.+0.j,  2.+0.j,  3.+0.j])
```

Data-type consisting of more than one element:

```pycon
>>> x = np.array([(1,2),(3,4)],dtype=[('a','<i4'),('b','<i4')])
>>> x['a']
array([1, 3], dtype=int32)
```

Creating an array from sub-classes:

```pycon
>>> np.array(np.asmatrix('1 2; 3 4'))
array([[1, 2],
       [3, 4]])
```

```pycon
>>> np.array(np.asmatrix('1 2; 3 4'), subok=True)
matrix([[1, 2],
        [3, 4]])
```

Limiting the maximum dimensions with `ndmax`:

```pycon
>>> a = np.array([[1, 2], [3, 4]], dtype=np.object_, ndmax=2)
>>> a
array([[1, 2],
       [3, 4]], dtype=object)
>>> a.shape
(2, 2)
```

```pycon
>>> b = np.array([[1, 2], [3, 4]], dtype=np.object_, ndmax=1)
>>> b
array([list([1, 2]), list([3, 4])], dtype=object)
>>> b.shape
(2,)
```

<!-- !! processed by numpydoc !! -->


# dask.array.asanyarray.html.md

# dask.array.asanyarray

### dask.array.asanyarray(a, dtype=None, order=None, , like=None, inline_array=False)

Convert the input to a dask array.

Subclasses of `np.ndarray` will be passed through as chunks unchanged.

* **Parameters:**
  **a**
  : Input data, in any form that can be converted to a dask array. This
    includes lists, lists of tuples, tuples, tuples of tuples, tuples of
    lists and ndarrays.

  **dtype**
  : By default, the data-type is inferred from the input data.

  **order**
  : Memory layout. ‘A’ and ‘K’ depend on the order of input array a.
    ‘C’ row-major (C-style), ‘F’ column-major (Fortran-style) memory
    representation. ‘A’ (any) means ‘F’ if a is Fortran contiguous, ‘C’
    otherwise ‘K’ (keep) preserve input order. Defaults to ‘C’.

  **like: array-like**
  : Reference object to allow the creation of Dask arrays with chunks
    that are not NumPy arrays. If an array-like passed in as `like`
    supports the `__array_function__` protocol, the chunk type of the
    resulting array will be defined by it. In this case, it ensures the
    creation of a Dask array compatible with that passed in via this
    argument. If `like` is a Dask array, the chunk type of the
    resulting array will be defined by the chunk type of `like`.
    Requires NumPy 1.20.0 or higher.

  **inline_array:**
  : Whether to inline the array in the resulting dask graph. For more information,
    see the documentation for `dask.array.from_array()`.
* **Returns:**
  **out**
  : Dask array interpretation of a.

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> x = np.arange(3)
>>> da.asanyarray(x)
dask.array<array, shape=(3,), dtype=int64, chunksize=(3,), chunktype=numpy.ndarray>
```

```pycon
>>> y = [[1, 2, 3], [4, 5, 6]]
>>> da.asanyarray(y)
dask.array<array, shape=(2, 3), dtype=int64, chunksize=(2, 3), chunktype=numpy.ndarray>
```

#### WARNING
order is ignored if a is an Array, has the attribute `to_dask_array`,
or is a list or tuple of Array’s.

<!-- !! processed by numpydoc !! -->


# dask.array.asarray.html.md

# dask.array.asarray

### dask.array.asarray(a, allow_unknown_chunksizes=False, dtype=None, order=None, , like=None, \*\*kwargs)

Convert the input to a dask array.

* **Parameters:**
  **a**
  : Input data, in any form that can be converted to a dask array. This
    includes lists, lists of tuples, tuples, tuples of tuples, tuples of
    lists and ndarrays.

  **allow_unknown_chunksizes: bool**
  : Allow unknown chunksizes, such as come from converting from dask
    dataframes.  Dask.array is unable to verify that chunks line up.  If
    data comes from differently aligned sources then this can cause
    unexpected results.

  **dtype**
  : By default, the data-type is inferred from the input data.

  **order**
  : Memory layout. ‘A’ and ‘K’ depend on the order of input array a.
    ‘C’ row-major (C-style), ‘F’ column-major (Fortran-style) memory
    representation. ‘A’ (any) means ‘F’ if a is Fortran contiguous, ‘C’
    otherwise ‘K’ (keep) preserve input order. Defaults to ‘C’.

  **like: array-like**
  : Reference object to allow the creation of Dask arrays with chunks
    that are not NumPy arrays. If an array-like passed in as `like`
    supports the `__array_function__` protocol, the chunk type of the
    resulting array will be defined by it. In this case, it ensures the
    creation of a Dask array compatible with that passed in via this
    argument. If `like` is a Dask array, the chunk type of the
    resulting array will be defined by the chunk type of `like`.
    Requires NumPy 1.20.0 or higher.
* **Returns:**
  **out**
  : Dask array interpretation of a.

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> x = np.arange(3)
>>> da.asarray(x)
dask.array<array, shape=(3,), dtype=int64, chunksize=(3,), chunktype=numpy.ndarray>
```

```pycon
>>> y = [[1, 2, 3], [4, 5, 6]]
>>> da.asarray(y)
dask.array<array, shape=(2, 3), dtype=int64, chunksize=(2, 3), chunktype=numpy.ndarray>
```

#### WARNING
order is ignored if a is an Array, has the attribute `to_dask_array`,
or is a list or tuple of Array’s.

<!-- !! processed by numpydoc !! -->


# dask.array.atleast_1d.html.md

# dask.array.atleast_1d

### dask.array.atleast_1d(\*arys)

Convert inputs to arrays with at least one dimension.

This docstring was copied from numpy.atleast_1d.

Some inconsistencies with the Dask version may exist.

Scalar inputs are converted to 1-dimensional arrays, whilst
higher-dimensional inputs are preserved.

* **Parameters:**
  **arys1, arys2, …**
  : One or more input arrays.
* **Returns:**
  **ret**
  : An array, or tuple of arrays, each with `a.ndim >= 1`.
    Copies are made only if necessary.

#### SEE ALSO
[`atleast_2d`](dask.array.atleast_2d.md#dask.array.atleast_2d), [`atleast_3d`](dask.array.atleast_3d.md#dask.array.atleast_3d)

### Examples

```pycon
>>> import numpy as np
>>> np.atleast_1d(1.0)
array([1.])
```

```pycon
>>> x = np.arange(9.0).reshape(3,3)
>>> np.atleast_1d(x)
array([[0., 1., 2.],
       [3., 4., 5.],
       [6., 7., 8.]])
>>> np.atleast_1d(x) is x
True
```

```pycon
>>> np.atleast_1d(1, [3, 4])
(array([1]), array([3, 4]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.atleast_2d.html.md

# dask.array.atleast_2d

### dask.array.atleast_2d(\*arys)

View inputs as arrays with at least two dimensions.

This docstring was copied from numpy.atleast_2d.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **arys1, arys2, …**
  : One or more array-like sequences.  Non-array inputs are converted
    to arrays.  Arrays that already have two or more dimensions are
    preserved.
* **Returns:**
  **res, res2, …**
  : An array, or tuple of arrays, each with `a.ndim >= 2`.
    Copies are avoided where possible, and views with two or more
    dimensions are returned.

#### SEE ALSO
[`atleast_1d`](dask.array.atleast_1d.md#dask.array.atleast_1d), [`atleast_3d`](dask.array.atleast_3d.md#dask.array.atleast_3d)

### Examples

```pycon
>>> import numpy as np
>>> np.atleast_2d(3.0)
array([[3.]])
```

```pycon
>>> x = np.arange(3.0)
>>> np.atleast_2d(x)
array([[0., 1., 2.]])
>>> np.atleast_2d(x).base is x
True
```

```pycon
>>> np.atleast_2d(1, [1, 2], [[1, 2]])
(array([[1]]), array([[1, 2]]), array([[1, 2]]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.atleast_3d.html.md

# dask.array.atleast_3d

### dask.array.atleast_3d(\*arys)

View inputs as arrays with at least three dimensions.

This docstring was copied from numpy.atleast_3d.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **arys1, arys2, …**
  : One or more array-like sequences.  Non-array inputs are converted to
    arrays.  Arrays that already have three or more dimensions are
    preserved.
* **Returns:**
  **res1, res2, …**
  : An array, or tuple of arrays, each with `a.ndim >= 3`.  Copies are
    avoided where possible, and views with three or more dimensions are
    returned.  For example, a 1-D array of shape `(N,)` becomes a view
    of shape `(1, N, 1)`, and a 2-D array of shape `(M, N)` becomes a
    view of shape `(M, N, 1)`.

#### SEE ALSO
[`atleast_1d`](dask.array.atleast_1d.md#dask.array.atleast_1d), [`atleast_2d`](dask.array.atleast_2d.md#dask.array.atleast_2d)

### Examples

```pycon
>>> import numpy as np
>>> np.atleast_3d(3.0)
array([[[3.]]])
```

```pycon
>>> x = np.arange(3.0)
>>> np.atleast_3d(x).shape
(1, 3, 1)
```

```pycon
>>> x = np.arange(12.0).reshape(4,3)
>>> np.atleast_3d(x).shape
(4, 3, 1)
>>> np.atleast_3d(x).base is x.base  # x is a reshape, so not base itself
True
```

```pycon
>>> for arr in np.atleast_3d([1, 2], [[1, 2]], [[[1, 2]]]):
...     print(arr, arr.shape)
...
[[[1]
  [2]]] (1, 2, 1)
[[[1]
  [2]]] (1, 2, 1)
[[[1 2]]] (1, 1, 2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.average.html.md

# dask.array.average

### dask.array.average(a, axis=None, weights=None, returned=False, keepdims=False)

Compute the weighted average along the specified axis.

This docstring was copied from numpy.average.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Array containing data to be averaged. If a is not an array, a
    conversion is attempted.

  **axis**
  : Axis or axes along which to average a.  The default,
    axis=None, will average over all of the elements of the input array.
    If axis is negative it counts from the last to the first axis.
    If axis is a tuple of ints, averaging is performed on all of the axes
    specified in the tuple instead of a single axis or all the axes as
    before.

  **weights**
  : An array of weights associated with the values in a. Each value in
    a contributes to the average according to its associated weight.
    The array of weights must be the same shape as a if no axis is
    specified, otherwise the weights must have dimensions and shape
    consistent with a along the specified axis.
    If weights=None, then all data in a are assumed to have a
    weight equal to one.
    The calculation is:
    ```default
    avg = sum(a * weights) / sum(weights)
    ```
    <br/>
    where the sum is over all included elements.
    The only constraint on the values of weights is that sum(weights)
    must not be 0.

  **returned**
  : Default is False. If True, the tuple (average, sum_of_weights)
    is returned, otherwise only the average is returned.
    If weights=None, sum_of_weights is equivalent to the number of
    elements over which the average is taken.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    *Note:* keepdims will not work with instances of numpy.matrix
    or other classes whose methods do not support keepdims.
    <br/>
    #### Versionadded
    Added in version 1.23.0.
* **Returns:**
  **retval, [sum_of_weights]**
  : Return the average along the specified axis. When returned is True,
    return a tuple with the average as the first element and the sum
    of the weights as the second element. sum_of_weights is of the
    same type as retval. The result dtype follows a general pattern.
    If weights is None, the result dtype will be that of a , or `float64`
    if a is integral. Otherwise, if weights is not None and a is non-
    integral, the result type will be the type of lowest precision capable of
    representing values of both a and weights. If a happens to be
    integral, the previous rules still applies but the result dtype will
    at least be `float64`.
* **Raises:**
  ZeroDivisionError
  : When all weights along axis are zero. See numpy.ma.average for a
    version robust to this type of error.

  TypeError
  : When weights does not have the same shape as a, and axis=None.

  ValueError
  : When weights does not have dimensions and shape consistent with a
    along specified axis.

#### SEE ALSO
[`mean`](dask.array.mean.md#dask.array.mean)

[`ma.average`](dask.array.ma.average.md#dask.array.ma.average)
: average for masked arrays – useful if your data contains “missing” values

[`numpy.result_type`](https://numpy.org/doc/stable/reference/generated/numpy.result_type.html#numpy.result_type)
: Returns the type that results from applying the numpy type promotion rules to the arguments.

### Examples

```pycon
>>> import numpy as np
>>> data = np.arange(1, 5)
>>> data
array([1, 2, 3, 4])
>>> np.average(data)
2.5
>>> np.average(np.arange(1, 11), weights=np.arange(10, 0, -1))
4.0
```

```pycon
>>> data = np.arange(6).reshape((3, 2))
>>> data
array([[0, 1],
       [2, 3],
       [4, 5]])
>>> np.average(data, axis=1, weights=[1./4, 3./4])
array([0.75, 2.75, 4.75])
>>> np.average(data, weights=[1./4, 3./4])
Traceback (most recent call last):
    ...
TypeError: Axis must be specified when shapes of a and weights differ.
```

With `keepdims=True`, the following result has shape (3, 1).

```pycon
>>> np.average(data, axis=1, keepdims=True)
array([[0.5],
       [2.5],
       [4.5]])
```

```pycon
>>> data = np.arange(8).reshape((2, 2, 2))
>>> data
array([[[0, 1],
        [2, 3]],
       [[4, 5],
        [6, 7]]])
>>> np.average(data, axis=(0, 1), weights=[[1./4, 3./4], [1., 1./2]])
array([3.4, 4.4])
>>> np.average(data, axis=0, weights=[[1./4, 3./4], [1., 1./2]])
Traceback (most recent call last):
    ...
ValueError: Shape of weights must be consistent
with shape of a along specified axis.
```

<!-- !! processed by numpydoc !! -->


# dask.array.bincount.html.md

# dask.array.bincount

### dask.array.bincount(x, , weights=None, minlength=0)

This docstring was copied from numpy.bincount.

Some inconsistencies with the Dask version may exist.

Count number of occurrences of each value in array of non-negative ints.

The number of bins (of size 1) is one larger than the largest value in
x. If minlength is specified, there will be at least this number
of bins in the output array (though it will be longer if necessary,
depending on the contents of x).
Each bin gives the number of occurrences of its index value in x.
If weights is specified the input array is weighted by it, i.e. if a
value `n` is found at position `i`, `out[n] += weight[i]` instead
of `out[n] += 1`.

* **Parameters:**
  **x**
  : Input array.

  **weights**
  : Weights, array of the same shape as x.

  **minlength**
  : A minimum number of bins for the output array.
* **Returns:**
  **out**
  : The result of binning the input array.
    The length of out is equal to `np.amax(x)+1`.
* **Raises:**
  ValueError
  : If the input is not 1-dimensional, or contains elements with negative
    values, or if minlength is negative.

  TypeError
  : If the type of the input is float or complex.

#### SEE ALSO
[`histogram`](dask.array.histogram.md#dask.array.histogram), [`digitize`](dask.array.digitize.md#dask.array.digitize), [`unique`](dask.array.unique.md#dask.array.unique)

### Examples

```pycon
>>> import numpy as np
>>> np.bincount(np.arange(5))
array([1, 1, 1, 1, 1])
>>> np.bincount(np.array([0, 1, 1, 3, 2, 1, 7]))
array([1, 3, 1, 1, 0, 0, 0, 1])
```

```pycon
>>> x = np.array([0, 1, 1, 3, 2, 1, 7, 23])
>>> np.bincount(x).size == np.amax(x)+1
True
```

The input array needs to be of integer dtype, otherwise a
TypeError is raised:

```pycon
>>> np.bincount(np.arange(5, dtype=np.float64))
Traceback (most recent call last):
  ...
TypeError: Cannot cast array data from dtype('float64') to dtype('int64')
according to the rule 'safe'
```

A possible use of `bincount` is to perform sums over
variable-size chunks of an array, using the `weights` keyword.

```pycon
>>> w = np.array([0.3, 0.5, 0.2, 0.7, 1., -0.6]) # weights
>>> x = np.array([0, 1, 1, 2, 2, 2])
>>> np.bincount(x,  weights=w)
array([ 0.3,  0.7,  1.1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.bitwise_and.html.md

# dask.array.bitwise_and

### dask.array.bitwise_and(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'bitwise_and'>*

This docstring was copied from numpy.bitwise_and.

Some inconsistencies with the Dask version may exist.

Compute the bit-wise AND of two arrays element-wise.

Computes the bit-wise AND of the underlying binary representation of
the integers in the input arrays. This ufunc implements the C/Python
operator `&`.

* **Parameters:**
  **x1, x2**
  : Only integer and boolean types are handled.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Result.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_and`](dask.array.logical_and.md#dask.array.logical_and)

[`bitwise_or`](dask.array.bitwise_or.md#dask.array.bitwise_or)

[`bitwise_xor`](dask.array.bitwise_xor.md#dask.array.bitwise_xor)

`binary_repr`
: Return the binary representation of the input number as a string.

### Examples

```pycon
>>> import numpy as np
```

The number 13 is represented by `00001101`.  Likewise, 17 is
represented by `00010001`.  The bit-wise AND of 13 and 17 is
therefore `000000001`, or 1:

```pycon
>>> np.bitwise_and(13, 17)
1
```

```pycon
>>> np.bitwise_and(14, 13)
12
>>> np.binary_repr(12)
'1100'
>>> np.bitwise_and([14,3], 13)
array([12,  1])
```

```pycon
>>> np.bitwise_and([11,7], [4,25])
array([0, 1])
>>> np.bitwise_and(np.array([2,5,255]), np.array([3,14,16]))
array([ 2,  4, 16])
>>> np.bitwise_and([True, True], [False, True])
array([False,  True])
```

The `&` operator can be used as a shorthand for `np.bitwise_and` on
ndarrays.

```pycon
>>> x1 = np.array([2, 5, 255])
>>> x2 = np.array([3, 14, 16])
>>> x1 & x2
array([ 2,  4, 16])
```

<!-- !! processed by numpydoc !! -->


# dask.array.bitwise_not.html.md

# dask.array.bitwise_not

### dask.array.bitwise_not(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'invert'>*

This docstring was copied from numpy.invert.

Some inconsistencies with the Dask version may exist.

Compute bit-wise inversion, or bit-wise NOT, element-wise.

Computes the bit-wise NOT of the underlying binary representation of
the integers in the input arrays. This ufunc implements the C/Python
operator `~`.

For signed integer inputs, the bit-wise NOT of the absolute value is
returned. In a two’s-complement system, this operation effectively flips
all the bits, resulting in a representation that corresponds to the
negative of the input plus one. This is the most common method of
representing signed integers on computers [[1]](#r4546e276892f-1). An N-bit two’s-complement
system can represent every integer in the range $-2^{N-1}$ to
$+2^{N-1}-1$.

* **Parameters:**
  **x**
  : Only integer and boolean types are handled.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Result.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`bitwise_and`](dask.array.bitwise_and.md#dask.array.bitwise_and), [`bitwise_or`](dask.array.bitwise_or.md#dask.array.bitwise_or), [`bitwise_xor`](dask.array.bitwise_xor.md#dask.array.bitwise_xor)

[`logical_not`](dask.array.logical_not.md#dask.array.logical_not)

`binary_repr`
: Return the binary representation of the input number as a string.

### Notes

`numpy.bitwise_not` is an alias for invert:

```pycon
>>> np.bitwise_not is np.invert
True
```

### References

### Examples

```pycon
>>> import numpy as np
```

We’ve seen that 13 is represented by `00001101`.
The invert or bit-wise NOT of 13 is then:

```pycon
>>> x = np.invert(np.array(13, dtype=np.uint8))
>>> x
np.uint8(242)
>>> np.binary_repr(x, width=8)
'11110010'
```

The result depends on the bit-width:

```pycon
>>> x = np.invert(np.array(13, dtype=np.uint16))
>>> x
np.uint16(65522)
>>> np.binary_repr(x, width=16)
'1111111111110010'
```

When using signed integer types, the result is the bit-wise NOT of
the unsigned type, interpreted as a signed integer:

```pycon
>>> np.invert(np.array([13], dtype=np.int8))
array([-14], dtype=int8)
>>> np.binary_repr(-14, width=8)
'11110010'
```

Booleans are accepted as well:

```pycon
>>> np.invert(np.array([True, False]))
array([False,  True])
```

The `~` operator can be used as a shorthand for `np.invert` on
ndarrays.

```pycon
>>> x1 = np.array([True, False])
>>> ~x1
array([False,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.bitwise_or.html.md

# dask.array.bitwise_or

### dask.array.bitwise_or(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'bitwise_or'>*

This docstring was copied from numpy.bitwise_or.

Some inconsistencies with the Dask version may exist.

Compute the bit-wise OR of two arrays element-wise.

Computes the bit-wise OR of the underlying binary representation of
the integers in the input arrays. This ufunc implements the C/Python
operator `|`.

* **Parameters:**
  **x1, x2**
  : Only integer and boolean types are handled.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Result.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_or`](dask.array.logical_or.md#dask.array.logical_or)

[`bitwise_and`](dask.array.bitwise_and.md#dask.array.bitwise_and)

[`bitwise_xor`](dask.array.bitwise_xor.md#dask.array.bitwise_xor)

`binary_repr`
: Return the binary representation of the input number as a string.

### Examples

```pycon
>>> import numpy as np
```

The number 13 has the binary representation `00001101`. Likewise,
16 is represented by `00010000`.  The bit-wise OR of 13 and 16 is
then `00011101`, or 29:

```pycon
>>> np.bitwise_or(13, 16)
29
>>> np.binary_repr(29)
'11101'
```

```pycon
>>> np.bitwise_or(32, 2)
34
>>> np.bitwise_or([33, 4], 1)
array([33,  5])
>>> np.bitwise_or([33, 4], [1, 2])
array([33,  6])
```

```pycon
>>> np.bitwise_or(np.array([2, 5, 255]), np.array([4, 4, 4]))
array([  6,   5, 255])
>>> np.array([2, 5, 255]) | np.array([4, 4, 4])
array([  6,   5, 255])
>>> np.bitwise_or(np.array([2, 5, 255, 2147483647], dtype=np.int32),
...               np.array([4, 4, 4, 2147483647], dtype=np.int32))
array([         6,          5,        255, 2147483647], dtype=int32)
>>> np.bitwise_or([True, True], [False, True])
array([ True,  True])
```

The `|` operator can be used as a shorthand for `np.bitwise_or` on
ndarrays.

```pycon
>>> x1 = np.array([2, 5, 255])
>>> x2 = np.array([4, 4, 4])
>>> x1 | x2
array([  6,   5, 255])
```

<!-- !! processed by numpydoc !! -->


# dask.array.bitwise_xor.html.md

# dask.array.bitwise_xor

### dask.array.bitwise_xor(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'bitwise_xor'>*

This docstring was copied from numpy.bitwise_xor.

Some inconsistencies with the Dask version may exist.

Compute the bit-wise XOR of two arrays element-wise.

Computes the bit-wise XOR of the underlying binary representation of
the integers in the input arrays. This ufunc implements the C/Python
operator `^`.

* **Parameters:**
  **x1, x2**
  : Only integer and boolean types are handled.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Result.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_xor`](dask.array.logical_xor.md#dask.array.logical_xor)

[`bitwise_and`](dask.array.bitwise_and.md#dask.array.bitwise_and)

[`bitwise_or`](dask.array.bitwise_or.md#dask.array.bitwise_or)

`binary_repr`
: Return the binary representation of the input number as a string.

### Examples

```pycon
>>> import numpy as np
```

The number 13 is represented by `00001101`. Likewise, 17 is
represented by `00010001`.  The bit-wise XOR of 13 and 17 is
therefore `00011100`, or 28:

```pycon
>>> np.bitwise_xor(13, 17)
28
>>> np.binary_repr(28)
'11100'
```

```pycon
>>> np.bitwise_xor(31, 5)
26
>>> np.bitwise_xor([31,3], 5)
array([26,  6])
```

```pycon
>>> np.bitwise_xor([31,3], [5,6])
array([26,  5])
>>> np.bitwise_xor([True, True], [False, True])
array([ True, False])
```

The `^` operator can be used as a shorthand for `np.bitwise_xor` on
ndarrays.

```pycon
>>> x1 = np.array([True, True])
>>> x2 = np.array([False, True])
>>> x1 ^ x2
array([ True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.block.html.md

# dask.array.block

### dask.array.block(arrays, allow_unknown_chunksizes=False)

Assemble an nd-array from nested lists of blocks.

Blocks in the innermost lists are concatenated along the last
dimension (-1), then these are concatenated along the second-last
dimension (-2), and so on until the outermost list is reached

Blocks can be of any dimension, but will not be broadcasted using the normal
rules. Instead, leading axes of size 1 are inserted, to make `block.ndim`
the same for all blocks. This is primarily useful for working with scalars,
and means that code like `block([v, 1])` is valid, where
`v.ndim == 1`.

When the nested list is two levels deep, this allows block matrices to be
constructed from their components.

* **Parameters:**
  **arrays**
  : If passed a single ndarray or scalar (a nested list of depth 0), this
    is returned unmodified (and not copied).
    <br/>
    Elements shapes must match along the appropriate axes (without
    broadcasting), but leading 1s will be prepended to the shape as
    necessary to make the dimensions match.

  **allow_unknown_chunksizes: bool**
  : Allow unknown chunksizes, such as come from converting from dask
    dataframes.  Dask.array is unable to verify that chunks line up.  If
    data comes from differently aligned sources then this can cause
    unexpected results.
* **Returns:**
  **block_array**
  : The array assembled from the given blocks.
    <br/>
    The dimensionality of the output is equal to the greatest of:
    \* the dimensionality of all the inputs
    \* the depth to which the input list is nested
* **Raises:**
  ValueError
  : * If list depths are mismatched - for instance, `[[a, b], c]` is
      illegal, and should be spelt `[[a, b], [c]]`
    * If lists are empty - for instance, `[[a, b], []]`

#### SEE ALSO
[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)
: Join a sequence of arrays together.

[`stack`](dask.array.stack.md#dask.array.stack)
: Stack arrays in sequence along a new dimension.

[`hstack`](dask.array.hstack.md#dask.array.hstack)
: Stack arrays in sequence horizontally (column wise).

[`vstack`](dask.array.vstack.md#dask.array.vstack)
: Stack arrays in sequence vertically (row wise).

[`dstack`](dask.array.dstack.md#dask.array.dstack)
: Stack arrays in sequence depth wise (along third dimension).

`vsplit`
: Split array into a list of multiple sub-arrays vertically.

### Notes

When called with only scalars, `block` is equivalent to an ndarray
call. So `block([[1, 2], [3, 4]])` is equivalent to
`array([[1, 2], [3, 4]])`.

This function does not enforce that the blocks lie on a fixed grid.
`block([[a, b], [c, d]])` is not restricted to arrays of the form:

```default
AAAbb
AAAbb
cccDD
```

But is also allowed to produce, for some `a, b, c, d`:

```default
AAAbb
AAAbb
cDDDD
```

Since concatenation happens along the last axis first, block is \_not_
capable of producing the following directly:

```default
AAAbb
cccbb
cccDD
```

Matlab’s “square bracket stacking”, `[A, B, ...; p, q, ...]`, is
equivalent to `block([[A, B, ...], [p, q, ...]])`.

<!-- !! processed by numpydoc !! -->


# dask.array.blockwise.html.md

# dask.array.blockwise

### dask.array.blockwise(func, out_ind, \*args, name=None, token=None, dtype=None, adjust_chunks=None, new_axes=None, align_arrays=True, concatenate=None, meta=None, \*\*kwargs)

Tensor operation: Generalized inner and outer products

A broad class of blocked algorithms and patterns can be specified with a
concise multi-index notation.  The `blockwise` function applies an in-memory
function across multiple blocks of multiple inputs in a variety of ways.
Many dask.array operations are special cases of blockwise including
elementwise, broadcasting, reductions, tensordot, and transpose.

* **Parameters:**
  **func**
  : Function to apply to individual tuples of blocks

  **out_ind**
  : Block pattern of the output, something like ‘ijk’ or (1, 2, 3)

  **\*args**
  : You may also pass literal arguments, accompanied by None index
    e.g. (x, ‘ij’, y, ‘jk’, z, ‘i’, some_literal, None)

  **\*\*kwargs**
  : Extra keyword arguments to pass to function

  **dtype**
  : Datatype of resulting array.

  **concatenate**
  : If true concatenate arrays along dummy indices, else provide lists

  **adjust_chunks**
  : Dictionary mapping index to function to be applied to chunk sizes

  **new_axes**
  : New indexes and their dimension lengths

  **align_arrays: bool**
  : Whether or not to align chunks along equally sized dimensions when
    multiple arrays are provided.  This allows for larger chunks in some
    arrays to be broken into smaller ones that match chunk sizes in other
    arrays such that they are compatible for block function mapping. If
    this is false, then an error will be thrown if arrays do not already
    have the same number of blocks in each dimension.

### Examples

2D embarrassingly parallel operation from two arrays, x, and y.

```pycon
>>> import operator, numpy as np, dask.array as da
>>> x = da.from_array([[1, 2],
...                    [3, 4]], chunks=(1, 2))
>>> y = da.from_array([[10, 20],
...                    [0, 0]])
>>> z = blockwise(operator.add, 'ij', x, 'ij', y, 'ij', dtype='f8')
>>> z.compute()
array([[11, 22],
       [ 3,  4]])
```

Outer product multiplying a by b, two 1-d vectors

```pycon
>>> a = da.from_array([0, 1, 2], chunks=1)
>>> b = da.from_array([10, 50, 100], chunks=1)
>>> z = blockwise(np.outer, 'ij', a, 'i', b, 'j', dtype='f8')
>>> z.compute()
array([[  0,   0,   0],
       [ 10,  50, 100],
       [ 20, 100, 200]])
```

z = x.T

```pycon
>>> z = blockwise(np.transpose, 'ji', x, 'ij', dtype=x.dtype)
>>> z.compute()
array([[1, 3],
       [2, 4]])
```

The transpose case above is illustrative because it does transposition
both on each in-memory block by calling `np.transpose` and on the order
of the blocks themselves, by switching the order of the index `ij -> ji`.

We can compose these same patterns with more variables and more complex
in-memory functions

z = X + Y.T

```pycon
>>> z = blockwise(lambda x, y: x + y.T, 'ij', x, 'ij', y, 'ji', dtype='f8')
>>> z.compute()
array([[11,  2],
       [23,  4]])
```

Any index, like `i` missing from the output index is interpreted as a
contraction (note that this differs from Einstein convention; repeated
indices do not imply contraction.)  In the case of a contraction the passed
function should expect an iterable of blocks on any array that holds that
index.  To receive arrays concatenated along contracted dimensions instead
pass `concatenate=True`.

Inner product multiplying a by b, two 1-d vectors

```pycon
>>> def sequence_dot(a_blocks, b_blocks):
...     result = 0
...     for a, b in zip(a_blocks, b_blocks):
...         result += a.dot(b)
...     return result
```

```pycon
>>> z = blockwise(sequence_dot, '', a, 'i', b, 'i', dtype='f8')
>>> z.compute()
np.int64(250)
```

Add new single-chunk dimensions with the `new_axes=` keyword, including
the length of the new dimension.  New dimensions will always be in a single
chunk.

```pycon
>>> def f(a):
...     return a[:, None] * np.ones((1, 5))
```

```pycon
>>> z = blockwise(f, 'az', a, 'a', new_axes={'z': 5}, dtype=a.dtype)
```

New dimensions can also be multi-chunk by specifying a tuple of chunk
sizes.  This has limited utility as is (because the chunks are all the
same), but the resulting graph can be modified to achieve more useful
results (see `da.map_blocks`).

```pycon
>>> z = blockwise(f, 'az', a, 'a', new_axes={'z': (5, 5)}, dtype=x.dtype)
>>> z.chunks
((1, 1, 1), (5, 5))
```

If the applied function changes the size of each chunk you can specify this
with a `adjust_chunks={...}` dictionary holding a function for each index
that modifies the dimension size in that index.

```pycon
>>> def double(x):
...     return np.concatenate([x, x])
```

```pycon
>>> y = blockwise(double, 'ij', x, 'ij',
...               adjust_chunks={'i': lambda n: 2 * n}, dtype=x.dtype)
>>> y.chunks
((2, 2), (2,))
```

Include literals by indexing with None

```pycon
>>> z = blockwise(operator.add, 'ij', x, 'ij', 1234, None, dtype=x.dtype)
>>> z.compute()
array([[1235, 1236],
       [1237, 1238]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.broadcast_arrays.html.md

# dask.array.broadcast_arrays

### dask.array.broadcast_arrays(\*args, subok=False)

Broadcast any number of arrays against each other.

This docstring was copied from numpy.broadcast_arrays.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **\*args**
  : The arrays to broadcast.

  **subok**
  : If True, then sub-classes will be passed-through, otherwise
    the returned arrays will be forced to be a base-class array (default).
* **Returns:**
  **broadcasted**
  : These arrays are views on the original arrays.  They are typically
    not contiguous.  Furthermore, more than one element of a
    broadcasted array may refer to a single memory location. If you need
    to write to the arrays, make copies first. While you can set the
    `writable` flag True, writing to a single output value may end up
    changing more than one location in the output array.
    <br/>
    #### Deprecated
    Deprecated since version 1.17: The output is currently marked so that if written to, a deprecation
    warning will be emitted. A future version will set the
    `writable` flag False so writing to it will raise an error.

#### SEE ALSO
`broadcast`

[`broadcast_to`](dask.array.broadcast_to.md#dask.array.broadcast_to)

`broadcast_shapes`

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([[1,2,3]])
>>> y = np.array([[4],[5]])
>>> np.broadcast_arrays(x, y)
(array([[1, 2, 3],
        [1, 2, 3]]),
 array([[4, 4, 4],
        [5, 5, 5]]))
```

Here is a useful idiom for getting contiguous copies instead of
non-contiguous views.

```pycon
>>> [np.array(a) for a in np.broadcast_arrays(x, y)]
[array([[1, 2, 3],
        [1, 2, 3]]),
 array([[4, 4, 4],
        [5, 5, 5]])]
```

<!-- !! processed by numpydoc !! -->


# dask.array.broadcast_to.html.md

# dask.array.broadcast_to

### dask.array.broadcast_to(x, shape, chunks=None, meta=None)

Broadcast an array to a new shape.

* **Parameters:**
  **x**
  : The array to broadcast.

  **shape**
  : The shape of the desired array.

  **chunks**
  : If provided, then the result will use these chunks instead of the same
    chunks as the source array. Setting chunks explicitly as part of
    broadcast_to is more efficient than rechunking afterwards. Chunks are
    only allowed to differ from the original shape along dimensions that
    are new on the result or have size 1 the input array.

  **meta**
  : empty ndarray created with same NumPy backend, ndim and dtype as the
    Dask Array being created (overrides dtype)
* **Returns:**
  **broadcast**

#### SEE ALSO
[`numpy.broadcast_to()`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast_to.html#numpy.broadcast_to)

<!-- !! processed by numpydoc !! -->


# dask.array.cbrt.html.md

# dask.array.cbrt

### dask.array.cbrt(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'cbrt'>*

This docstring was copied from numpy.cbrt.

Some inconsistencies with the Dask version may exist.

Return the cube-root of an array, element-wise.

* **Parameters:**
  **x**
  : The values whose cube-roots are required.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : An array of the same shape as x, containing the
    cube root of each element in x.
    If out was provided, y is a reference to it.
    This is a scalar if x is a scalar.

### Examples

```pycon
>>> import numpy as np
>>> np.cbrt([1,8,27])
array([ 1.,  2.,  3.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ceil.html.md

# dask.array.ceil

### dask.array.ceil(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'ceil'>*

This docstring was copied from numpy.ceil.

Some inconsistencies with the Dask version may exist.

Return the ceiling of the input, element-wise.

The ceil of the scalar x is the smallest integer i, such that
`i >= x`.  It is often denoted as $\lceil x \rceil$.

* **Parameters:**
  **x**
  : Input data.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The ceiling of each element in x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`floor`](dask.array.floor.md#dask.array.floor), [`trunc`](dask.array.trunc.md#dask.array.trunc), [`rint`](dask.array.rint.md#dask.array.rint), [`fix`](dask.array.fix.md#dask.array.fix)

### Examples

```pycon
>>> import numpy as np
```

```pycon
>>> a = np.array([-1.7, -1.5, -0.2, 0.2, 1.5, 1.7, 2.0])
>>> np.ceil(a)
array([-1., -1., -0.,  1.,  2.,  2.,  2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.choose.html.md

# dask.array.choose

### dask.array.choose(a, choices)

Construct an array from an index array and a list of arrays to choose from.

This docstring was copied from numpy.choose.

Some inconsistencies with the Dask version may exist.

First of all, if confused or uncertain, definitely look at the Examples -
in its full generality, this function is less simple than it might
seem from the following code description:

```default
np.choose(a,c) == np.array([c[a[I]][I] for I in np.ndindex(a.shape)])
```

But this omits some subtleties.  Here is a fully general summary:

Given an “index” array (a) of integers and a sequence of `n` arrays
(choices), a and each choice array are first broadcast, as necessary,
to arrays of a common shape; calling these *Ba* and *Bchoices[i], i =
0,…,n-1* we have that, necessarily, `Ba.shape == Bchoices[i].shape`
for each `i`.  Then, a new array with shape `Ba.shape` is created as
follows:

* if `mode='raise'` (the default), then, first of all, each element of
  `a` (and thus `Ba`) must be in the range `[0, n-1]`; now, suppose
  that `i` (in that range) is the value at the `(j0, j1, ..., jm)`
  position in `Ba` - then the value at the same position in the new array
  is the value in `Bchoices[i]` at that same position;
* if `mode='wrap'`, values in a (and thus Ba) may be any (signed)
  integer; modular arithmetic is used to map integers outside the range
  [0, n-1] back into that range; and then the new array is constructed
  as above;
* if `mode='clip'`, values in a (and thus `Ba`) may be any (signed)
  integer; negative integers are mapped to 0; values greater than `n-1`
  are mapped to `n-1`; and then the new array is constructed as above.

* **Parameters:**
  **a**
  : This array must contain integers in `[0, n-1]`, where `n` is the
    number of choices, unless `mode=wrap` or `mode=clip`, in which
    cases any integers are permissible.

  **choices**
  : Choice arrays. a and all of the choices must be broadcastable to the
    same shape.  If choices is itself an array (not recommended), then
    its outermost dimension (i.e., the one corresponding to
    `choices.shape[0]`) is taken as defining the “sequence”.

  **out**
  : If provided, the result will be inserted into this array. It should
    be of the appropriate shape and dtype. Note that out is always
    buffered if `mode='raise'`; use other modes for better performance.

  **mode**
  : Specifies how indices outside `[0, n-1]` will be treated:
    * ‘raise’ : an exception is raised
    * ‘wrap’ : value becomes value mod `n`
    * ‘clip’ : values < 0 are mapped to 0, values > n-1 are mapped to n-1
* **Returns:**
  **merged_array**
  : The merged result.
* **Raises:**
  ValueError: shape mismatch
  : If a and each choice array are not all broadcastable to the same
    shape.

#### SEE ALSO
`ndarray.choose`
: equivalent method

[`numpy.take_along_axis`](https://numpy.org/doc/stable/reference/generated/numpy.take_along_axis.html#numpy.take_along_axis)
: Preferable if choices is an array

### Notes

To reduce the chance of misinterpretation, even though the following
“abuse” is nominally supported, choices should neither be, nor be
thought of as, a single array, i.e., the outermost sequence-like container
should be either a list or a tuple.

### Examples

```pycon
>>> import numpy as np
>>> choices = [[0, 1, 2, 3], [10, 11, 12, 13],
...   [20, 21, 22, 23], [30, 31, 32, 33]]
>>> np.choose([2, 3, 1, 0], choices
... # the first element of the result will be the first element of the
... # third (2+1) "array" in choices, namely, 20; the second element
... # will be the second element of the fourth (3+1) choice array, i.e.,
... # 31, etc.
... )
array([20, 31, 12,  3])
>>> np.choose([2, 4, 1, 0], choices, mode='clip') # 4 goes to 3 (4-1)
array([20, 31, 12,  3])
>>> # because there are 4 choice arrays
>>> np.choose([2, 4, 1, 0], choices, mode='wrap') # 4 goes to (4 mod 4)
array([20,  1, 12,  3])
>>> # i.e., 0
```

A couple examples illustrating how choose broadcasts:

```pycon
>>> a = [[1, 0, 1], [0, 1, 0], [1, 0, 1]]
>>> choices = [-10, 10]
>>> np.choose(a, choices)
array([[ 10, -10,  10],
       [-10,  10, -10],
       [ 10, -10,  10]])
```

```pycon
>>> # With thanks to Anne Archibald
>>> a = np.array([0, 1]).reshape((2,1,1))
>>> c1 = np.array([1, 2, 3]).reshape((1,3,1))
>>> c2 = np.array([-1, -2, -3, -4, -5]).reshape((1,1,5))
>>> np.choose(a, (c1, c2)) # result is 2x3x5, res[0,:,:]=c1, res[1,:,:]=c2
array([[[ 1,  1,  1,  1,  1],
        [ 2,  2,  2,  2,  2],
        [ 3,  3,  3,  3,  3]],
       [[-1, -2, -3, -4, -5],
        [-1, -2, -3, -4, -5],
        [-1, -2, -3, -4, -5]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.clip.html.md

# dask.array.clip

### dask.array.clip(\*args, \*\*kwargs)

Clip (limit) the values in an array.

This docstring was copied from numpy.clip.

Some inconsistencies with the Dask version may exist.

Given an interval, values outside the interval are clipped to
the interval edges.  For example, if an interval of `[0, 1]`
is specified, values smaller than 0 become 0, and values larger
than 1 become 1.

Equivalent to but faster than `np.minimum(a_max, np.maximum(a, a_min))`.

No check is performed to ensure `a_min < a_max`.

* **Parameters:**
  **a**
  : Array containing elements to clip.

  **a_min, a_max**
  : Minimum and maximum value. If `None`, clipping is not performed on
    the corresponding edge. If both `a_min` and `a_max` are `None`,
    the elements of the returned array stay the same. Both are broadcasted
    against `a`.

  **out**
  : The results will be placed in this array. It may be the input
    array for in-place clipping.  out must be of the right shape
    to hold the output.  Its type is preserved.

  **min, max**
  : Array API compatible alternatives for `a_min` and `a_max`
    arguments. Either `a_min` and `a_max` or `min` and `max`
    can be passed at the same time. Default: `None`.
    <br/>
    #### Versionadded
    Added in version 2.1.0.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **clipped_array**
  : An array with the elements of a, but where values
    < a_min are replaced with a_min, and those > a_max
    with a_max.

#### SEE ALSO
[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

When a_min is greater than a_max, clip returns an
array in which all values are equal to a_max,
as shown in the second example.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> np.clip(a, 1, 8)
array([1, 1, 2, 3, 4, 5, 6, 7, 8, 8])
>>> np.clip(a, 8, 1)
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> np.clip(a, 3, 6, out=a)
array([3, 3, 3, 3, 4, 5, 6, 6, 6, 6])
>>> a
array([3, 3, 3, 3, 4, 5, 6, 6, 6, 6])
>>> a = np.arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> np.clip(a, [3, 4, 1, 1, 1, 4, 4, 4, 4, 4], 8)
array([3, 4, 2, 3, 4, 5, 6, 7, 8, 8])
```

<!-- !! processed by numpydoc !! -->


# dask.array.coarsen.html.md

# dask.array.coarsen

### dask.array.coarsen(reduction, x, axes, trim_excess=False, \*\*kwargs)

Coarsen array by applying reduction to fixed size neighborhoods

* **Parameters:**
  **reduction: function**
  : Reduction function (for example `np.sum` or `np.mean`).
    <br/>
    The function must accept:
    - an `array_like` positional input,
    - an `axis=` keyword containing a tuple of axes,
    - and any extra `**kwargs` forwarded by `coarsen`.
    <br/>
    In practice, NumPy-style reductions and Array-API-compatible
    reductions work well.

  **x: np.ndarray**
  : Array to be coarsened

  **axes: dict**
  : Mapping of axis to coarsening factor

### Examples

```pycon
>>> x = np.array([1, 2, 3, 4, 5, 6])
>>> coarsen(np.sum, x, {0: 2})
array([ 3,  7, 11])
>>> coarsen(np.max, x, {0: 3})
array([3, 6])
```

Provide dictionary of scale per dimension

```pycon
>>> x = np.arange(24).reshape((4, 6))
>>> x
array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17],
       [18, 19, 20, 21, 22, 23]])
```

```pycon
>>> coarsen(np.min, x, {0: 2, 1: 3})
array([[ 0,  3],
       [12, 15]])
```

You must avoid excess elements explicitly

```pycon
>>> x = np.array([1, 2, 3, 4, 5, 6, 7, 8])
>>> coarsen(np.min, x, {0: 3}, trim_excess=True)
array([1, 4])
```

<!-- !! processed by numpydoc !! -->


# dask.array.compress.html.md

# dask.array.compress

### dask.array.compress(condition, a, axis=None)

Return selected slices of an array along given axis.

This docstring was copied from numpy.compress.

Some inconsistencies with the Dask version may exist.

When working along a given axis, a slice along that axis is returned in
output for each index where condition evaluates to True. When
working on a 1-D array, compress is equivalent to extract.

* **Parameters:**
  **condition**
  : Array that selects which entries to return. If len(condition)
    is less than the size of a along the given axis, then output is
    truncated to the length of the condition array.

  **a**
  : Array from which to extract a part.

  **axis**
  : Axis along which to take slices. If None (default), work on the
    flattened array.

  **out**
  : Output array.  Its type is preserved and it must be of the right
    shape to hold the output.
* **Returns:**
  **compressed_array**
  : A copy of a without the slices along axis for which condition
    is false.

#### SEE ALSO
[`take`](dask.array.take.md#dask.array.take), [`choose`](dask.array.choose.md#dask.array.choose), [`diag`](dask.array.diag.md#dask.array.diag), [`diagonal`](dask.array.diagonal.md#dask.array.diagonal), [`select`](dask.array.select.md#dask.array.select)

`ndarray.compress`
: Equivalent method in ndarray

[`extract`](dask.array.extract.md#dask.array.extract)
: Equivalent method when working on 1-D arrays

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, 4], [5, 6]])
>>> a
array([[1, 2],
       [3, 4],
       [5, 6]])
>>> np.compress([0, 1], a, axis=0)
array([[3, 4]])
>>> np.compress([False, True, True], a, axis=0)
array([[3, 4],
       [5, 6]])
>>> np.compress([False, True], a, axis=1)
array([[2],
       [4],
       [6]])
```

Working on the flattened array does not return slices along an axis but
selects elements.

```pycon
>>> np.compress([False, True], a)
array([2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.concatenate.html.md

# dask.array.concatenate

### dask.array.concatenate(seq, axis=0, allow_unknown_chunksizes=False)

Concatenate arrays along an existing axis

Given a sequence of dask Arrays form a new dask Array by stacking them
along an existing dimension (axis=0 by default)

* **Parameters:**
  **seq: list of dask.arrays**

  **axis: int**
  : Dimension along which to align all of the arrays. If axis is None,
    arrays are flattened before use.

  **allow_unknown_chunksizes: bool**
  : Allow unknown chunksizes, such as come from converting from dask
    dataframes.  Dask.array is unable to verify that chunks line up.  If
    data comes from differently aligned sources then this can cause
    unexpected results.

#### SEE ALSO
[`stack`](dask.array.stack.md#dask.array.stack)

### Examples

Create slices

```pycon
>>> import dask.array as da
>>> import numpy as np
```

```pycon
>>> data = [da.from_array(np.ones((4, 4)), chunks=(2, 2))
...          for i in range(3)]
```

```pycon
>>> x = da.concatenate(data, axis=0)
>>> x.shape
(12, 4)
```

```pycon
>>> da.concatenate(data, axis=1).shape
(4, 12)
```

Result is a new dask Array

<!-- !! processed by numpydoc !! -->


# dask.array.conj.html.md

# dask.array.conj

### dask.array.conj(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'conjugate'>*

This docstring was copied from numpy.conjugate.

Some inconsistencies with the Dask version may exist.

Return the complex conjugate, element-wise.

The complex conjugate of a complex number is obtained by changing the
sign of its imaginary part.

* **Parameters:**
  **x**
  : Input value.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The complex conjugate of x, with same dtype as y.
    This is a scalar if x is a scalar.

### Notes

conj is an alias for conjugate:

```pycon
>>> np.conj is np.conjugate
True
```

### Examples

```pycon
>>> import numpy as np
>>> np.conjugate(1+2j)
(1-2j)
```

```pycon
>>> x = np.eye(2) + 1j * np.eye(2)
>>> np.conjugate(x)
array([[ 1.-1.j,  0.-0.j],
       [ 0.-0.j,  1.-1.j]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.copysign.html.md

# dask.array.copysign

### dask.array.copysign(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'copysign'>*

This docstring was copied from numpy.copysign.

Some inconsistencies with the Dask version may exist.

Change the sign of x1 to that of x2, element-wise.

If x2 is a scalar, its sign will be copied to all elements of x1.

* **Parameters:**
  **x1**
  : Values to change the sign of.

  **x2**
  : The sign of x2 is copied to x1.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : The values of x1 with the sign of x2.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`sign`](dask.array.sign.md#dask.array.sign)

[`signbit`](dask.array.signbit.md#dask.array.signbit)

### Examples

```pycon
>>> import numpy as np
>>> np.copysign(1.3, -1)
-1.3
>>> 1/np.copysign(0, 1)
inf
>>> 1/np.copysign(0, -1)
-inf
```

```pycon
>>> np.copysign([-1, 0, 1], -1.1)
array([-1., -0., -1.])
>>> np.copysign([-1, 0, 1], np.arange(3)-1)
array([-1.,  0.,  1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.core.PerformanceWarning.html.md

# dask.array.core.PerformanceWarning

### *exception* dask.array.core.PerformanceWarning

A warning given when bad chunking may cause poor performance

<!-- !! processed by numpydoc !! -->


# dask.array.core.blockwise.html.md

# dask.array.core.blockwise

### dask.array.core.blockwise(func, out_ind, \*args, name=None, token=None, dtype=None, adjust_chunks=None, new_axes=None, align_arrays=True, concatenate=None, meta=None, \*\*kwargs)

Tensor operation: Generalized inner and outer products

A broad class of blocked algorithms and patterns can be specified with a
concise multi-index notation.  The `blockwise` function applies an in-memory
function across multiple blocks of multiple inputs in a variety of ways.
Many dask.array operations are special cases of blockwise including
elementwise, broadcasting, reductions, tensordot, and transpose.

* **Parameters:**
  **func**
  : Function to apply to individual tuples of blocks

  **out_ind**
  : Block pattern of the output, something like ‘ijk’ or (1, 2, 3)

  **\*args**
  : You may also pass literal arguments, accompanied by None index
    e.g. (x, ‘ij’, y, ‘jk’, z, ‘i’, some_literal, None)

  **\*\*kwargs**
  : Extra keyword arguments to pass to function

  **dtype**
  : Datatype of resulting array.

  **concatenate**
  : If true concatenate arrays along dummy indices, else provide lists

  **adjust_chunks**
  : Dictionary mapping index to function to be applied to chunk sizes

  **new_axes**
  : New indexes and their dimension lengths

  **align_arrays: bool**
  : Whether or not to align chunks along equally sized dimensions when
    multiple arrays are provided.  This allows for larger chunks in some
    arrays to be broken into smaller ones that match chunk sizes in other
    arrays such that they are compatible for block function mapping. If
    this is false, then an error will be thrown if arrays do not already
    have the same number of blocks in each dimension.

### Examples

2D embarrassingly parallel operation from two arrays, x, and y.

```pycon
>>> import operator, numpy as np, dask.array as da
>>> x = da.from_array([[1, 2],
...                    [3, 4]], chunks=(1, 2))
>>> y = da.from_array([[10, 20],
...                    [0, 0]])
>>> z = blockwise(operator.add, 'ij', x, 'ij', y, 'ij', dtype='f8')
>>> z.compute()
array([[11, 22],
       [ 3,  4]])
```

Outer product multiplying a by b, two 1-d vectors

```pycon
>>> a = da.from_array([0, 1, 2], chunks=1)
>>> b = da.from_array([10, 50, 100], chunks=1)
>>> z = blockwise(np.outer, 'ij', a, 'i', b, 'j', dtype='f8')
>>> z.compute()
array([[  0,   0,   0],
       [ 10,  50, 100],
       [ 20, 100, 200]])
```

z = x.T

```pycon
>>> z = blockwise(np.transpose, 'ji', x, 'ij', dtype=x.dtype)
>>> z.compute()
array([[1, 3],
       [2, 4]])
```

The transpose case above is illustrative because it does transposition
both on each in-memory block by calling `np.transpose` and on the order
of the blocks themselves, by switching the order of the index `ij -> ji`.

We can compose these same patterns with more variables and more complex
in-memory functions

z = X + Y.T

```pycon
>>> z = blockwise(lambda x, y: x + y.T, 'ij', x, 'ij', y, 'ji', dtype='f8')
>>> z.compute()
array([[11,  2],
       [23,  4]])
```

Any index, like `i` missing from the output index is interpreted as a
contraction (note that this differs from Einstein convention; repeated
indices do not imply contraction.)  In the case of a contraction the passed
function should expect an iterable of blocks on any array that holds that
index.  To receive arrays concatenated along contracted dimensions instead
pass `concatenate=True`.

Inner product multiplying a by b, two 1-d vectors

```pycon
>>> def sequence_dot(a_blocks, b_blocks):
...     result = 0
...     for a, b in zip(a_blocks, b_blocks):
...         result += a.dot(b)
...     return result
```

```pycon
>>> z = blockwise(sequence_dot, '', a, 'i', b, 'i', dtype='f8')
>>> z.compute()
np.int64(250)
```

Add new single-chunk dimensions with the `new_axes=` keyword, including
the length of the new dimension.  New dimensions will always be in a single
chunk.

```pycon
>>> def f(a):
...     return a[:, None] * np.ones((1, 5))
```

```pycon
>>> z = blockwise(f, 'az', a, 'a', new_axes={'z': 5}, dtype=a.dtype)
```

New dimensions can also be multi-chunk by specifying a tuple of chunk
sizes.  This has limited utility as is (because the chunks are all the
same), but the resulting graph can be modified to achieve more useful
results (see `da.map_blocks`).

```pycon
>>> z = blockwise(f, 'az', a, 'a', new_axes={'z': (5, 5)}, dtype=x.dtype)
>>> z.chunks
((1, 1, 1), (5, 5))
```

If the applied function changes the size of each chunk you can specify this
with a `adjust_chunks={...}` dictionary holding a function for each index
that modifies the dimension size in that index.

```pycon
>>> def double(x):
...     return np.concatenate([x, x])
```

```pycon
>>> y = blockwise(double, 'ij', x, 'ij',
...               adjust_chunks={'i': lambda n: 2 * n}, dtype=x.dtype)
>>> y.chunks
((2, 2), (2,))
```

Include literals by indexing with None

```pycon
>>> z = blockwise(operator.add, 'ij', x, 'ij', 1234, None, dtype=x.dtype)
>>> z.compute()
array([[1235, 1236],
       [1237, 1238]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.core.normalize_chunks.html.md

# dask.array.core.normalize_chunks

### dask.array.core.normalize_chunks(chunks, shape=None, limit=None, dtype=None, previous_chunks=None)

Normalize chunks to tuple of tuples

This takes in a variety of input types and information and produces a full
tuple-of-tuples result for chunks, suitable to be passed to Array or
rechunk or any other operation that creates a Dask array.

* **Parameters:**
  **chunks: tuple, int, dict, or string**
  : The chunks to be normalized.  See examples below for more details

  **shape: Tuple[int]**
  : The shape of the array

  **limit: int (optional)**
  : The maximum block size to target in bytes,
    if freedom is given to choose

  **dtype: np.dtype**

  **previous_chunks: Tuple[Tuple[int]] optional**
  : Chunks from a previous array that we should use for inspiration when
    rechunking auto dimensions.  If not provided but auto-chunking exists
    then auto-dimensions will prefer square-like chunk shapes.

### Examples

Fully explicit tuple-of-tuples

```pycon
>>> from dask.array.core import normalize_chunks
>>> normalize_chunks(((2, 2, 1), (2, 2, 2)), shape=(5, 6))
((2, 2, 1), (2, 2, 2))
```

Specify uniform chunk sizes

```pycon
>>> normalize_chunks((2, 2), shape=(5, 6))
((2, 2, 1), (2, 2, 2))
```

Cleans up missing outer tuple

```pycon
>>> normalize_chunks((3, 2), (5,))
((3, 2),)
```

Cleans up lists to tuples

```pycon
>>> normalize_chunks([[2, 2], [3, 3]])
((2, 2), (3, 3))
```

Expands integer inputs 10 -> (10, 10)

```pycon
>>> normalize_chunks(10, shape=(30, 5))
((10, 10, 10), (5,))
```

Expands dict inputs

```pycon
>>> normalize_chunks({0: 2, 1: 3}, shape=(6, 6))
((2, 2, 2), (3, 3))
```

The values -1 and None get mapped to full size

```pycon
>>> normalize_chunks((5, -1), shape=(10, 10))
((5, 5), (10,))
>>> normalize_chunks((5, None), shape=(10, 10))
((5, 5), (10,))
```

Use the value “auto” to automatically determine chunk sizes along certain
dimensions.  This uses the `limit=` and `dtype=` keywords to
determine how large to make the chunks.  The term “auto” can be used
anywhere an integer can be used.  See array chunking documentation for more
information.

```pycon
>>> normalize_chunks(("auto",), shape=(20,), limit=5, dtype='uint8')
((5, 5, 5, 5),)
>>> normalize_chunks("auto", (2, 3), dtype=np.int32)
((2,), (3,))
```

You can also use byte sizes (see [`dask.utils.parse_bytes()`](../api.md#dask.utils.parse_bytes)) in place of
“auto” to ask for a particular size

```pycon
>>> normalize_chunks("1kiB", shape=(2000,), dtype='float32')
((256, 256, 256, 256, 256, 256, 256, 208),)
```

Respects null dimensions

```pycon
>>> normalize_chunks(())
()
>>> normalize_chunks((), ())
()
>>> normalize_chunks((1,), ())
()
>>> normalize_chunks((), shape=(0, 0))
((0,), (0,))
```

Handles NaNs

```pycon
>>> normalize_chunks((1, (np.nan,)), (1, np.nan))
((1,), (nan,))
```

<!-- !! processed by numpydoc !! -->


# dask.array.core.unify_chunks.html.md

# dask.array.core.unify_chunks

### dask.array.core.unify_chunks(\*args, \*\*kwargs)

Unify chunks across a sequence of arrays

This utility function is used within other common operations like
`dask.array.core.map_blocks()` and [`dask.array.core.blockwise()`](dask.array.core.blockwise.md#dask.array.core.blockwise).
It is not commonly used by end-users directly.

* **Parameters:**
  **\*args: sequence of Array, index pairs**
  : Sequence like (x, ‘ij’, y, ‘jk’, z, ‘i’)
* **Returns:**
  **chunkss**
  : Map like {index: chunks}.

  **arrays**
  : List of rechunked arrays.

#### SEE ALSO
`common_blockdim`

### Examples

```pycon
>>> import dask.array as da
>>> x = da.ones(10, chunks=((5, 2, 3),))
>>> y = da.ones(10, chunks=((2, 3, 5),))
>>> chunkss, arrays = unify_chunks(x, 'i', y, 'i')
>>> chunkss
{'i': (2, 3, 2, 3)}
```

```pycon
>>> x = da.ones((100, 10), chunks=(20, 5))
>>> y = da.ones((10, 100), chunks=(4, 50))
>>> chunkss, arrays = unify_chunks(x, 'ij', y, 'jk', 'constant', None)
>>> chunkss
{'k': (50, 50), 'i': (20, 20, 20, 20, 20), 'j': (4, 1, 3, 2)}
```

```pycon
>>> unify_chunks(0, None)
({}, [0])
```

<!-- !! processed by numpydoc !! -->


# dask.array.corrcoef.html.md

# dask.array.corrcoef

### dask.array.corrcoef(x, y=None, rowvar=1)

Return Pearson product-moment correlation coefficients.

This docstring was copied from numpy.corrcoef.

Some inconsistencies with the Dask version may exist.

Please refer to the documentation for cov for more detail.  The
relationship between the correlation coefficient matrix, R, and the
covariance matrix, C, is

$$
R_{ij} = \frac{ C_{ij} } { \sqrt{ C_{ii} C_{jj} } }

$$

The values of R are between -1 and 1, inclusive.

* **Parameters:**
  **x**
  : A 1-D or 2-D array containing multiple variables and observations.
    Each row of x represents a variable, and each column a single
    observation of all those variables. Also see rowvar below.

  **y**
  : An additional set of variables and observations. y has the same
    shape as x.

  **rowvar**
  : If rowvar is True (default), then each row represents a
    variable, with observations in the columns. Otherwise, the relationship
    is transposed: each column represents a variable, while the rows
    contain observations.

  **dtype**
  : Data-type of the result. By default, the return data-type will have
    at least numpy.float64 precision.
    <br/>
    #### Versionadded
    Added in version 1.20.
* **Returns:**
  **R**
  : The correlation coefficient matrix of the variables.

#### SEE ALSO
[`cov`](dask.array.cov.md#dask.array.cov)
: Covariance matrix

### Notes

Due to floating point rounding the resulting array may not be Hermitian,
the diagonal elements may not be 1, and the elements may not satisfy the
inequality abs(a) <= 1. The real and imaginary parts are clipped to the
interval [-1,  1] in an attempt to improve on that situation but is not
much help in the complex case.

### Examples

```pycon
>>> import numpy as np
```

In this example we generate two random arrays, `xarr` and `yarr`, and
compute the row-wise and column-wise Pearson correlation coefficients,
`R`. Since `rowvar` is  true by  default, we first find the row-wise
Pearson correlation coefficients between the variables of `xarr`.

```pycon
>>> import numpy as np
>>> rng = np.random.default_rng(seed=42)
>>> xarr = rng.random((3, 3))
>>> xarr
array([[0.77395605, 0.43887844, 0.85859792],
       [0.69736803, 0.09417735, 0.97562235],
       [0.7611397 , 0.78606431, 0.12811363]])
>>> R1 = np.corrcoef(xarr)
>>> R1
array([[ 1.        ,  0.99256089, -0.68080986],
       [ 0.99256089,  1.        , -0.76492172],
       [-0.68080986, -0.76492172,  1.        ]])
```

If we add another set of variables and observations `yarr`, we can
compute the row-wise Pearson correlation coefficients between the
variables in `xarr` and `yarr`.

```pycon
>>> yarr = rng.random((3, 3))
>>> yarr
array([[0.45038594, 0.37079802, 0.92676499],
       [0.64386512, 0.82276161, 0.4434142 ],
       [0.22723872, 0.55458479, 0.06381726]])
>>> R2 = np.corrcoef(xarr, yarr)
>>> R2
array([[ 1.        ,  0.99256089, -0.68080986,  0.75008178, -0.934284  ,
        -0.99004057],
       [ 0.99256089,  1.        , -0.76492172,  0.82502011, -0.97074098,
        -0.99981569],
       [-0.68080986, -0.76492172,  1.        , -0.99507202,  0.89721355,
         0.77714685],
       [ 0.75008178,  0.82502011, -0.99507202,  1.        , -0.93657855,
        -0.83571711],
       [-0.934284  , -0.97074098,  0.89721355, -0.93657855,  1.        ,
         0.97517215],
       [-0.99004057, -0.99981569,  0.77714685, -0.83571711,  0.97517215,
         1.        ]])
```

Finally if we use the option `rowvar=False`, the columns are now
being treated as the variables and we will find the column-wise Pearson
correlation coefficients between variables in `xarr` and `yarr`.

```pycon
>>> R3 = np.corrcoef(xarr, yarr, rowvar=False)
>>> R3
array([[ 1.        ,  0.77598074, -0.47458546, -0.75078643, -0.9665554 ,
         0.22423734],
       [ 0.77598074,  1.        , -0.92346708, -0.99923895, -0.58826587,
        -0.44069024],
       [-0.47458546, -0.92346708,  1.        ,  0.93773029,  0.23297648,
         0.75137473],
       [-0.75078643, -0.99923895,  0.93773029,  1.        ,  0.55627469,
         0.47536961],
       [-0.9665554 , -0.58826587,  0.23297648,  0.55627469,  1.        ,
        -0.46666491],
       [ 0.22423734, -0.44069024,  0.75137473,  0.47536961, -0.46666491,
         1.        ]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.cos.html.md

# dask.array.cos

### dask.array.cos(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'cos'>*

This docstring was copied from numpy.cos.

Some inconsistencies with the Dask version may exist.

Cosine element-wise.

* **Parameters:**
  **x**
  : Input array in radians.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding cosine values.
    This is a scalar if x is a scalar.

### Notes

If out is provided, the function writes the result into it,
and returns a reference to out.  (See Examples)

### References

M. Abramowitz and I. A. Stegun, Handbook of Mathematical Functions.
New York, NY: Dover, 1972.

### Examples

```pycon
>>> import numpy as np
>>> np.cos(np.array([0, np.pi/2, np.pi]))
array([  1.00000000e+00,   6.12303177e-17,  -1.00000000e+00])
>>>
>>> # Example of providing the optional output parameter
>>> out1 = np.array([0], dtype=np.float64)
>>> out2 = np.cos([0.1], out1)
>>> out2 is out1
True
>>>
>>> # Example of ValueError due to provision of shape mis-matched `out`
>>> np.cos(np.zeros((3,3)),np.zeros((2,2)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (3,3) (2,2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.cosh.html.md

# dask.array.cosh

### dask.array.cosh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'cosh'>*

This docstring was copied from numpy.cosh.

Some inconsistencies with the Dask version may exist.

Hyperbolic cosine, element-wise.

Equivalent to `1/2 * (np.exp(x) + np.exp(-x))` and `np.cos(1j*x)`.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array of same shape as x.
    This is a scalar if x is a scalar.

### Examples

```pycon
>>> import numpy as np
>>> np.cosh(0)
1.0
```

The hyperbolic cosine describes the shape of a hanging cable:

```pycon
>>> import matplotlib.pyplot as plt
>>> x = np.linspace(-4, 4, 1000)
>>> plt.plot(x, np.cosh(x))
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.count_nonzero.html.md

# dask.array.count_nonzero

### dask.array.count_nonzero(a, axis=None)

Counts the number of non-zero values in the array `a`.

This docstring was copied from numpy.count_nonzero.

Some inconsistencies with the Dask version may exist.

A non-zero value is one that evaluates to truthful in a boolean
context, including any non-zero number and any string that
is not empty. This function recursively counts how many elements
in `a` (and its sub-arrays) are non-zero values.

* **Parameters:**
  **a**
  : The array for which to count non-zeros.

  **axis**
  : Axis or tuple of axes along which to count non-zeros.
    Default is None, meaning that non-zeros will be counted
    along a flattened version of `a`.

  **keepdims**
  : If this is set to True, the axes that are counted are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **count**
  : Number of non-zero values in the array along a given axis.
    Otherwise, the total number of non-zero values in the array
    is returned.

#### SEE ALSO
[`nonzero`](dask.array.nonzero.md#dask.array.nonzero)
: Return the coordinates of all the non-zero values.

### Examples

```pycon
>>> import numpy as np
>>> np.count_nonzero(np.eye(4))
np.int64(4)
>>> a = np.array([[0, 1, 7, 0],
...               [3, 0, 2, 19]])
>>> np.count_nonzero(a)
np.int64(5)
>>> np.count_nonzero(a, axis=0)
array([1, 1, 2, 1])
>>> np.count_nonzero(a, axis=1)
array([2, 3])
>>> np.count_nonzero(a, axis=1, keepdims=True)
array([[2],
       [3]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.cov.html.md

# dask.array.cov

### dask.array.cov(m, y=None, rowvar=True, bias=False, ddof=None, fweights=None, aweights=None, , dtype=None)

Estimate a covariance matrix, given data and weights.

This docstring was copied from numpy.cov.

Some inconsistencies with the Dask version may exist.

Covariance indicates the level to which two variables vary together.
If we examine N-dimensional samples, $X = [x_1, x_2, ..., x_N]^T$,
then the covariance matrix element $C_{ij}$ is the covariance of
$x_i$ and $x_j$. The element $C_{ii}$ is the variance
of $x_i$.

See the notes for an outline of the algorithm.

* **Parameters:**
  **m**
  : A 1-D or 2-D array containing multiple variables and observations.
    Each row of m represents a variable, and each column a single
    observation of all those variables. Also see rowvar below.

  **y**
  : An additional set of variables and observations. y has the same form
    as that of m.

  **rowvar**
  : If rowvar is True (default), then each row represents a
    variable, with observations in the columns. Otherwise, the relationship
    is transposed: each column represents a variable, while the rows
    contain observations.

  **bias**
  : Default normalization (False) is by `(N - 1)`, where `N` is the
    number of observations given (unbiased estimate). If bias is True,
    then normalization is by `N`. These values can be overridden by using
    the keyword `ddof` in numpy versions >= 1.5.

  **ddof**
  : If not `None` the default value implied by bias is overridden.
    Note that `ddof=1` will return the unbiased estimate, even if both
    fweights and aweights are specified, and `ddof=0` will return
    the simple average. See the notes for the details. The default value
    is `None`.

  **fweights**
  : 1-D array of integer frequency weights; the number of times each
    observation vector should be repeated.

  **aweights**
  : 1-D array of observation vector weights. These relative weights are
    typically large for observations considered “important” and smaller for
    observations considered less “important”. If `ddof=0` the array of
    weights can be used to assign probabilities to observation vectors.

  **dtype**
  : Data-type of the result. By default, the return data-type will have
    at least numpy.float64 precision.
    <br/>
    #### Versionadded
    Added in version 1.20.
* **Returns:**
  **out**
  : The covariance matrix of the variables.

#### SEE ALSO
[`corrcoef`](dask.array.corrcoef.md#dask.array.corrcoef)
: Normalized covariance matrix

### Notes

Assume that the observations are in the columns of the observation
array m and let `f = fweights` and `a = aweights` for brevity. The
steps to compute the weighted covariance are as follows:

```default
>>> m = np.arange(10, dtype=np.float64)
>>> f = np.arange(10) * 2
>>> a = np.arange(10) ** 2.
>>> ddof = 1
>>> w = f * a
>>> v1 = np.sum(w)
>>> v2 = np.sum(w * a)
>>> m -= np.sum(m * w, axis=None, keepdims=True) / v1
>>> cov = np.dot(m * w, m.T) * v1 / (v1**2 - ddof * v2)
```

Note that when `a == 1`, the normalization factor
`v1 / (v1**2 - ddof * v2)` goes over to `1 / (np.sum(f) - ddof)`
as it should.

### Examples

```pycon
>>> import numpy as np
```

Consider two variables, $x_0$ and $x_1$, which
correlate perfectly, but in opposite directions:

```pycon
>>> x = np.array([[0, 2], [1, 1], [2, 0]]).T
>>> x
array([[0, 1, 2],
       [2, 1, 0]])
```

Note how $x_0$ increases while $x_1$ decreases. The covariance
matrix shows this clearly:

```pycon
>>> np.cov(x)
array([[ 1., -1.],
       [-1.,  1.]])
```

Note that element $C_{0,1}$, which shows the correlation between
$x_0$ and $x_1$, is negative.

Further, note how x and y are combined:

```pycon
>>> x = [-2.1, -1,  4.3]
>>> y = [3,  1.1,  0.12]
>>> X = np.stack((x, y), axis=0)
>>> np.cov(X)
array([[11.71      , -4.286     ], # may vary
       [-4.286     ,  2.144133]])
>>> np.cov(x, y)
array([[11.71      , -4.286     ], # may vary
       [-4.286     ,  2.144133]])
>>> np.cov(x)
array(11.71)
```

<!-- !! processed by numpydoc !! -->


# dask.array.cumprod.html.md

# dask.array.cumprod

### dask.array.cumprod(x, axis=None, dtype=None, out=None, method='sequential')

Return the cumulative product of elements along a given axis.

This docstring was copied from numpy.cumprod.

Some inconsistencies with the Dask version may exist.

Dask added an additional keyword-only argument `method`.

method
: Choose which method to use to perform the cumprod.  Default is ‘sequential’.
  <br/>
  * ‘sequential’ performs the cumprod of each prior block before the current block.
  * ‘blelloch’ is a work-efficient parallel cumprod.  It exposes parallelism by first
    taking the product of each block and combines the products via a binary tree.
    This method may be faster or more memory efficient depending on workload,
    scheduler, and hardware.  More benchmarking is necessary.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : Axis along which the cumulative product is computed.  By default
    the input is flattened.

  **dtype**
  : Type of the returned array, as well as of the accumulator in which
    the elements are multiplied.  If *dtype* is not specified, it
    defaults to the dtype of a, unless a has an integer dtype with
    a precision less than that of the default platform integer.  In
    that case, the default platform integer is used instead.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output
    but the type of the resulting values will be cast if necessary.
* **Returns:**
  **cumprod**
  : A new array holding the result is returned unless out is
    specified, in which case a reference to out is returned.

#### SEE ALSO
`cumulative_prod`
: Array API compatible alternative for `cumprod`.

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

Arithmetic is modular when using integer types, and no error is
raised on overflow.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1,2,3])
>>> np.cumprod(a) # intermediate results 1, 1*2
...               # total product 1*2*3 = 6
array([1, 2, 6])
>>> a = np.array([[1, 2, 3], [4, 5, 6]])
>>> np.cumprod(a, dtype=np.float64)  # specify type of output
array([   1.,    2.,    6.,   24.,  120.,  720.])
```

The cumulative product for each column (i.e., over the rows) of a:

```pycon
>>> np.cumprod(a, axis=0)
array([[ 1,  2,  3],
       [ 4, 10, 18]])
```

The cumulative product for each row (i.e. over the columns) of a:

```pycon
>>> np.cumprod(a,axis=1)
array([[  1,   2,   6],
       [  4,  20, 120]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.cumsum.html.md

# dask.array.cumsum

### dask.array.cumsum(x, axis=None, dtype=None, out=None, method='sequential')

Return the cumulative sum of the elements along a given axis.

This docstring was copied from numpy.cumsum.

Some inconsistencies with the Dask version may exist.

Dask added an additional keyword-only argument `method`.

method
: Choose which method to use to perform the cumsum.  Default is ‘sequential’.
  <br/>
  * ‘sequential’ performs the cumsum of each prior block before the current block.
  * ‘blelloch’ is a work-efficient parallel cumsum.  It exposes parallelism by
    first taking the sum of each block and combines the sums via a binary tree.
    This method may be faster or more memory efficient depending on workload,
    scheduler, and hardware.  More benchmarking is necessary.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : Axis along which the cumulative sum is computed. The default
    (None) is to compute the cumsum over the flattened array.

  **dtype**
  : Type of the returned array and of the accumulator in which the
    elements are summed.  If dtype is not specified, it defaults
    to the dtype of a, unless a has an integer dtype with a
    precision less than that of the default platform integer.  In
    that case, the default platform integer is used.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output
    but the type will be cast if necessary. See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)
    for more details.
* **Returns:**
  **cumsum_along_axis**
  : A new array holding the result is returned unless out is
    specified, in which case a reference to out is returned. The
    result has the same size as a, and the same shape as a if
    axis is not None or a is a 1-d array.

#### SEE ALSO
`cumulative_sum`
: Array API compatible alternative for `cumsum`.

[`sum`](dask.array.sum.md#dask.array.sum)
: Sum array elements.

`trapezoid`
: Integration of array values using composite trapezoidal rule.

[`diff`](dask.array.diff.md#dask.array.diff)
: Calculate the n-th discrete difference along given axis.

### Notes

Arithmetic is modular when using integer types, and no error is
raised on overflow.

`cumsum(a)[-1]` may not be equal to `sum(a)` for floating-point
values since `sum` may use a pairwise summation routine, reducing
the roundoff-error. See sum for more information.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1,2,3], [4,5,6]])
>>> a
array([[1, 2, 3],
       [4, 5, 6]])
>>> np.cumsum(a)
array([ 1,  3,  6, 10, 15, 21])
>>> np.cumsum(a, dtype=np.float64)  # specifies type of output value(s)
array([  1.,   3.,   6.,  10.,  15.,  21.])
```

```pycon
>>> np.cumsum(a,axis=0)      # sum over rows for each of the 3 columns
array([[1, 2, 3],
       [5, 7, 9]])
>>> np.cumsum(a,axis=1)      # sum over columns for each of the 2 rows
array([[ 1,  3,  6],
       [ 4,  9, 15]])
```

`cumsum(b)[-1]` may not be equal to `sum(b)`

```pycon
>>> b = np.array([1, 2e-9, 3e-9] * 1000000)
>>> b.cumsum()[-1]
1000000.0050045159
>>> b.sum()
1000000.0050000029
```

<!-- !! processed by numpydoc !! -->


# dask.array.deg2rad.html.md

# dask.array.deg2rad

### dask.array.deg2rad(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'deg2rad'>*

This docstring was copied from numpy.deg2rad.

Some inconsistencies with the Dask version may exist.

Convert angles from degrees to radians.

* **Parameters:**
  **x**
  : Angles in degrees.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding angle in radians.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`rad2deg`](dask.array.rad2deg.md#dask.array.rad2deg)
: Convert angles from radians to degrees.

`unwrap`
: Remove large jumps in angle by wrapping.

### Notes

`deg2rad(x)` is `x * pi / 180`.

### Examples

```pycon
>>> import numpy as np
>>> np.deg2rad(180)
3.1415926535897931
```

<!-- !! processed by numpydoc !! -->


# dask.array.degrees.html.md

# dask.array.degrees

### dask.array.degrees(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'degrees'>*

This docstring was copied from numpy.degrees.

Some inconsistencies with the Dask version may exist.

Convert angles from radians to degrees.

* **Parameters:**
  **x**
  : Input array in radians.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding degree values; if out was supplied this is a
    reference to it.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`rad2deg`](dask.array.rad2deg.md#dask.array.rad2deg)
: equivalent function

### Examples

Convert a radian array to degrees

```pycon
>>> import numpy as np
>>> rad = np.arange(12.)*np.pi/6
>>> np.degrees(rad)
array([   0.,   30.,   60.,   90.,  120.,  150.,  180.,  210.,  240.,
        270.,  300.,  330.])
```

```pycon
>>> out = np.zeros((rad.shape))
>>> r = np.degrees(rad, out)
>>> np.all(r == out)
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.delete.html.md

# dask.array.delete

### dask.array.delete(arr, obj, axis)

Return a new array with sub-arrays along an axis deleted. For a one
dimensional array, this returns those entries not returned by
arr[obj].

This docstring was copied from numpy.delete.

Some inconsistencies with the Dask version may exist.

NOTE: If `obj` is a dask array it is implicitly computed when this function
is called.

* **Parameters:**
  **arr**
  : Input array.

  **obj**
  : Indicate indices of sub-arrays to remove along the specified axis.
    <br/>
    #### Versionchanged
    Changed in version 1.19.0: Boolean indices are now treated as a mask of elements to remove,
    rather than being cast to the integers 0 and 1.

  **axis**
  : The axis along which to delete the subarray defined by obj.
    If axis is None, obj is applied to the flattened array.
* **Returns:**
  **out**
  : A copy of arr with the elements specified by obj removed. Note
    that delete does not occur in-place. If axis is None, out is
    a flattened array.

#### SEE ALSO
[`insert`](dask.array.insert.md#dask.array.insert)
: Insert elements into an array.

[`append`](dask.array.append.md#dask.array.append)
: Append elements at the end of an array.

### Notes

Often it is preferable to use a boolean mask. For example:

```pycon
>>> arr = np.arange(12) + 1
>>> mask = np.ones(len(arr), dtype=np.bool)
>>> mask[[0,2,4]] = False
>>> result = arr[mask,...]
```

Is equivalent to `np.delete(arr, [0,2,4], axis=0)`, but allows further
use of mask.

### Examples

```pycon
>>> import numpy as np
>>> arr = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
>>> arr
array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])
>>> np.delete(arr, 1, 0)
array([[ 1,  2,  3,  4],
       [ 9, 10, 11, 12]])
```

```pycon
>>> np.delete(arr, np.s_[::2], 1)
array([[ 2,  4],
       [ 6,  8],
       [10, 12]])
>>> np.delete(arr, [1,3,5], None)
array([ 1,  3,  5,  7,  8,  9, 10, 11, 12])
```

<!-- !! processed by numpydoc !! -->


# dask.array.diag.html.md

# dask.array.diag

### dask.array.diag(v, k=0)

Extract a diagonal or construct a diagonal array.

This docstring was copied from numpy.diag.

Some inconsistencies with the Dask version may exist.

See the more detailed documentation for `numpy.diagonal` if you use this
function to extract a diagonal and wish to write to the resulting array;
whether it returns a copy or a view depends on what version of numpy you
are using.

* **Parameters:**
  **v**
  : If v is a 2-D array, return a copy of its k-th diagonal.
    If v is a 1-D array, return a 2-D array with v on the k-th
    diagonal.

  **k**
  : Diagonal in question. The default is 0. Use k>0 for diagonals
    above the main diagonal, and k<0 for diagonals below the main
    diagonal.
* **Returns:**
  **out**
  : The extracted diagonal or constructed diagonal array.

#### SEE ALSO
[`diagonal`](dask.array.diagonal.md#dask.array.diagonal)
: Return specified diagonals.

`diagflat`
: Create a 2-D array with the flattened input as a diagonal.

[`trace`](dask.array.trace.md#dask.array.trace)
: Sum along diagonals.

[`triu`](dask.array.triu.md#dask.array.triu)
: Upper triangle of an array.

[`tril`](dask.array.tril.md#dask.array.tril)
: Lower triangle of an array.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(9).reshape((3,3))
>>> x
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
```

```pycon
>>> np.diag(x)
array([0, 4, 8])
>>> np.diag(x, k=1)
array([1, 5])
>>> np.diag(x, k=-1)
array([3, 7])
```

```pycon
>>> np.diag(np.diag(x))
array([[0, 0, 0],
       [0, 4, 0],
       [0, 0, 8]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.diagonal.html.md

# dask.array.diagonal

### dask.array.diagonal(a, offset=0, axis1=0, axis2=1)

Return specified diagonals.

This docstring was copied from numpy.diagonal.

Some inconsistencies with the Dask version may exist.

If a is 2-D, returns the diagonal of a with the given offset,
i.e., the collection of elements of the form `a[i, i+offset]`.  If
a has more than two dimensions, then the axes specified by axis1
and axis2 are used to determine the 2-D sub-array whose diagonal is
returned.  The shape of the resulting array can be determined by
removing axis1 and axis2 and appending an index to the right equal
to the size of the resulting diagonals.

In versions of NumPy prior to 1.7, this function always returned a new,
independent array containing a copy of the values in the diagonal.

In NumPy 1.7 and 1.8, it continues to return a copy of the diagonal,
but depending on this fact is deprecated. Writing to the resulting
array continues to work as it used to, but a FutureWarning is issued.

Starting in NumPy 1.9 it returns a read-only view on the original array.
Attempting to write to the resulting array will produce an error.

In some future release, it will return a read/write view and writing to
the returned array will alter your original array.  The returned array
will have the same type as the input array.

If you don’t write to the array returned by this function, then you can
just ignore all of the above.

If you depend on the current behavior, then we suggest copying the
returned array explicitly, i.e., use `np.diagonal(a).copy()` instead
of just `np.diagonal(a)`. This will work with both past and future
versions of NumPy.

* **Parameters:**
  **a**
  : Array from which the diagonals are taken.

  **offset**
  : Offset of the diagonal from the main diagonal.  Can be positive or
    negative.  Defaults to main diagonal (0).

  **axis1**
  : Axis to be used as the first axis of the 2-D sub-arrays from which
    the diagonals should be taken.  Defaults to first axis (0).

  **axis2**
  : Axis to be used as the second axis of the 2-D sub-arrays from
    which the diagonals should be taken. Defaults to second axis (1).
* **Returns:**
  **array_of_diagonals**
  : If a is 2-D, then a 1-D array containing the diagonal and of the
    same type as a is returned unless a is a matrix, in which case
    a 1-D array rather than a (2-D) matrix is returned in order to
    maintain backward compatibility.
    <br/>
    If `a.ndim > 2`, then the dimensions specified by axis1 and axis2
    are removed, and a new axis inserted at the end corresponding to the
    diagonal.
* **Raises:**
  ValueError
  : If the dimension of a is less than 2.

#### SEE ALSO
[`diag`](dask.array.diag.md#dask.array.diag)
: MATLAB work-a-like for 1-D and 2-D arrays.

`diagflat`
: Create diagonal arrays.

[`trace`](dask.array.trace.md#dask.array.trace)
: Sum along diagonals.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(4).reshape(2,2)
>>> a
array([[0, 1],
       [2, 3]])
>>> a.diagonal()
array([0, 3])
>>> a.diagonal(1)
array([1])
```

A 3-D example:

```pycon
>>> a = np.arange(8).reshape(2,2,2); a
array([[[0, 1],
        [2, 3]],
       [[4, 5],
        [6, 7]]])
>>> a.diagonal(0,  # Main diagonals of two arrays created by skipping
...            0,  # across the outer(left)-most axis last and
...            1)  # the "middle" (row) axis first.
array([[0, 6],
       [1, 7]])
```

The sub-arrays whose main diagonals we just obtained; note that each
corresponds to fixing the right-most (column) axis, and that the
diagonals are “packed” in rows.

```pycon
>>> a[:,:,0]  # main diagonal is [0 6]
array([[0, 2],
       [4, 6]])
>>> a[:,:,1]  # main diagonal is [1 7]
array([[1, 3],
       [5, 7]])
```

The anti-diagonal can be obtained by reversing the order of elements
using either numpy.flipud or numpy.fliplr.

```pycon
>>> a = np.arange(9).reshape(3, 3)
>>> a
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
>>> np.fliplr(a).diagonal()  # Horizontal flip
array([2, 4, 6])
>>> np.flipud(a).diagonal()  # Vertical flip
array([6, 4, 2])
```

Note that the order in which the diagonal is retrieved varies depending
on the flip function.

<!-- !! processed by numpydoc !! -->


# dask.array.diff.html.md

# dask.array.diff

### dask.array.diff(a, n=1, axis=-1, prepend=None, append=None)

Calculate the n-th discrete difference along the given axis.

This docstring was copied from numpy.diff.

Some inconsistencies with the Dask version may exist.

The first difference is given by `out[i] = a[i+1] - a[i]` along
the given axis, higher differences are calculated by using diff
recursively.

* **Parameters:**
  **a**
  : Input array

  **n**
  : The number of times values are differenced. If zero, the input
    is returned as-is.

  **axis**
  : The axis along which the difference is taken, default is the
    last axis.

  **prepend, append**
  : Values to prepend or append to a along axis prior to
    performing the difference.  Scalar values are expanded to
    arrays with length 1 in the direction of axis and the shape
    of the input array in along all other axes.  Otherwise the
    dimension and shape must match a except along axis.
* **Returns:**
  **diff**
  : The n-th differences. The shape of the output is the same as a
    except along axis where the dimension is smaller by n. The
    type of the output is the same as the type of the difference
    between any two elements of a. This is the same as the type of
    a in most cases. A notable exception is datetime64, which
    results in a timedelta64 output array.

#### SEE ALSO
[`gradient`](dask.array.gradient.md#dask.array.gradient), [`ediff1d`](dask.array.ediff1d.md#dask.array.ediff1d), [`cumsum`](dask.array.cumsum.md#dask.array.cumsum)

### Notes

Type is preserved for boolean arrays, so the result will contain
False when consecutive elements are the same and True when they
differ.

For unsigned integer arrays, the results will also be unsigned. This
should not be surprising, as the result is consistent with
calculating the difference directly:

```pycon
>>> u8_arr = np.array([1, 0], dtype=np.uint8)
>>> np.diff(u8_arr)
array([255], dtype=uint8)
>>> u8_arr[1,...] - u8_arr[0,...]
np.uint8(255)
```

If this is not desirable, then the array should be cast to a larger
integer type first:

```pycon
>>> i16_arr = u8_arr.astype(np.int16)
>>> np.diff(i16_arr)
array([-1], dtype=int16)
```

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([1, 2, 4, 7, 0])
>>> np.diff(x)
array([ 1,  2,  3, -7])
>>> np.diff(x, n=2)
array([  1,   1, -10])
```

```pycon
>>> x = np.array([[1, 3, 6, 10], [0, 5, 6, 8]])
>>> np.diff(x)
array([[2, 3, 4],
       [5, 1, 2]])
>>> np.diff(x, axis=0)
array([[-1,  2,  0, -2]])
```

```pycon
>>> x = np.arange('1066-10-13', '1066-10-16', dtype=np.datetime64)
>>> np.diff(x)
array([1, 1], dtype='timedelta64[D]')
```

<!-- !! processed by numpydoc !! -->


# dask.array.digitize.html.md

# dask.array.digitize

### dask.array.digitize(a, bins, right=False)

Return the indices of the bins to which each value in input array belongs.

This docstring was copied from numpy.digitize.

Some inconsistencies with the Dask version may exist.

| right   | order of bins   | returned index i satisfies   |
|---------|-----------------|------------------------------|
| `False` | increasing      | `bins[i-1] <= x < bins[i]`   |
| `True`  | increasing      | `bins[i-1] < x <= bins[i]`   |
| `False` | decreasing      | `bins[i-1] > x >= bins[i]`   |
| `True`  | decreasing      | `bins[i-1] >= x > bins[i]`   |

If values in x are beyond the bounds of bins, 0 or `len(bins)` is
returned as appropriate.

* **Parameters:**
  **x**
  : Input array to be binned. Prior to NumPy 1.10.0, this array had to
    be 1-dimensional, but can now have any shape.

  **bins**
  : Array of bins. It has to be 1-dimensional and monotonic.

  **right**
  : Indicating whether the intervals include the right or the left bin
    edge. Default behavior is (right==False) indicating that the interval
    does not include the right edge. The left bin end is open in this
    case, i.e., bins[i-1] <= x < bins[i] is the default behavior for
    monotonically increasing bins.
* **Returns:**
  **indices**
  : Output array of indices, of same shape as x.
* **Raises:**
  ValueError
  : If bins is not monotonic.

  TypeError
  : If the type of the input is complex.

#### SEE ALSO
[`bincount`](dask.array.bincount.md#dask.array.bincount), [`histogram`](dask.array.histogram.md#dask.array.histogram), [`unique`](dask.array.unique.md#dask.array.unique), [`searchsorted`](dask.array.searchsorted.md#dask.array.searchsorted)

### Notes

If values in x are such that they fall outside the bin range,
attempting to index bins with the indices that digitize returns
will result in an IndexError.

#### Versionadded
Added in version 1.10.0.

numpy.digitize is  implemented in terms of numpy.searchsorted.
This means that a binary search is used to bin the values, which scales
much better for larger number of bins than the previous linear search.
It also removes the requirement for the input array to be 1-dimensional.

For monotonically *increasing* bins, the following are equivalent:

```default
np.digitize(x, bins, right=True)
np.searchsorted(bins, x, side='left')
```

Note that as the order of the arguments are reversed, the side must be too.
The searchsorted call is marginally faster, as it does not do any
monotonicity checks. Perhaps more importantly, it supports all dtypes.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([0.2, 6.4, 3.0, 1.6])
>>> bins = np.array([0.0, 1.0, 2.5, 4.0, 10.0])
>>> inds = np.digitize(x, bins)
>>> inds
array([1, 4, 3, 2])
>>> for n in range(x.size):
...   print(bins[inds[n]-1], "<=", x[n], "<", bins[inds[n]])
...
0.0 <= 0.2 < 1.0
4.0 <= 6.4 < 10.0
2.5 <= 3.0 < 4.0
1.0 <= 1.6 < 2.5
```

```pycon
>>> x = np.array([1.2, 10.0, 12.4, 15.5, 20.])
>>> bins = np.array([0, 5, 10, 15, 20])
>>> np.digitize(x,bins,right=True)
array([1, 2, 3, 4, 4])
>>> np.digitize(x,bins,right=False)
array([1, 3, 3, 4, 5])
```

<!-- !! processed by numpydoc !! -->


# dask.array.divide.html.md

# dask.array.divide

### dask.array.divide(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'divide'>*

This docstring was copied from numpy.divide.

Some inconsistencies with the Dask version may exist.

Divide arguments element-wise.

* **Parameters:**
  **x1**
  : Dividend array.

  **x2**
  : Divisor array.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The quotient `x1/x2`, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
`seterr`
: Set whether to raise or warn on overflow, underflow and division by zero.

### Notes

Equivalent to `x1` / `x2` in terms of array-broadcasting.

The `true_divide(x1, x2)` function is an alias for
`divide(x1, x2)`.

### Examples

```pycon
>>> import numpy as np
>>> np.divide(2.0, 4.0)
0.5
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> np.divide(x1, x2)
array([[nan, 1. , 1. ],
       [inf, 4. , 2.5],
       [inf, 7. , 4. ]])
```

The `/` operator can be used as a shorthand for `np.divide` on
ndarrays.

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = 2 * np.ones(3)
>>> x1 / x2
array([[0. , 0.5, 1. ],
       [1.5, 2. , 2.5],
       [3. , 3.5, 4. ]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.divmod.html.md

# dask.array.divmod

### dask.array.divmod(x1, x2, /, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True)

This docstring was copied from numpy.divmod.

Some inconsistencies with the Dask version may exist.

Return element-wise quotient and remainder simultaneously.

`np.divmod(x, y)` is equivalent to `(x // y, x % y)`, but faster
because it avoids redundant work. It is used to implement the Python
built-in function `divmod` on NumPy arrays.

* **Parameters:**
  **x1**
  : Dividend array.

  **x2**
  : Divisor array.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out1**
  : Element-wise quotient resulting from floor division.
    This is a scalar if both x1 and x2 are scalars.

  **out2**
  : Element-wise remainder from floor division.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`floor_divide`](dask.array.floor_divide.md#dask.array.floor_divide)
: Equivalent to Python’s `//` operator.

[`remainder`](dask.array.remainder.md#dask.array.remainder)
: Equivalent to Python’s `%` operator.

[`modf`](dask.array.modf.md#dask.array.modf)
: Equivalent to `divmod(x, 1)` for positive `x` with the return values switched.

### Examples

```pycon
>>> import numpy as np
>>> np.divmod(np.arange(5), 3)
(array([0, 0, 0, 1, 1]), array([0, 1, 2, 0, 1]))
```

The divmod function can be used as a shorthand for `np.divmod` on
ndarrays.

```pycon
>>> x = np.arange(5)
>>> divmod(x, 3)
(array([0, 0, 0, 1, 1]), array([0, 1, 2, 0, 1]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.dot.html.md

# dask.array.dot

### dask.array.dot(a, b, out=None)

This docstring was copied from numpy.dot.

Some inconsistencies with the Dask version may exist.

Dot product of two arrays. Specifically,

- If both a and b are 1-D arrays, it is inner product of vectors
  (without complex conjugation).
- If both a and b are 2-D arrays, it is matrix multiplication,
  but using [`matmul()`](dask.array.matmul.md#dask.array.matmul) or `a @ b` is preferred.
- If either a or b is 0-D (scalar), it is equivalent to
  [`multiply()`](dask.array.multiply.md#dask.array.multiply) and using `numpy.multiply(a, b)` or `a * b` is
  preferred.
- If a is an N-D array and b is a 1-D array, it is a sum product over
  the last axis of a and b.
- If a is an N-D array and b is an M-D array (where `M>=2`), it is a
  sum product over the last axis of a and the second-to-last axis of
  b:
  ```default
  dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])
  ```

It uses an optimized BLAS library when possible (see numpy.linalg).

* **Parameters:**
  **a**
  : First argument.

  **b**
  : Second argument.

  **out**
  : Output argument. This must have the exact kind that would be returned
    if it was not used. In particular, it must have the right type, must be
    C-contiguous, and its dtype must be the dtype that would be returned
    for dot(a,b). This is a performance feature. Therefore, if these
    conditions are not met, an exception is raised, instead of attempting
    to be flexible.
* **Returns:**
  **output**
  : Returns the dot product of a and b.  If a and b are both
    scalars or both 1-D arrays then a scalar is returned; otherwise
    an array is returned.
    If out is given, then it is returned.
* **Raises:**
  ValueError
  : If the last dimension of a is not the same size as
    the second-to-last dimension of b.

#### SEE ALSO
[`vdot`](dask.array.vdot.md#dask.array.vdot)
: Complex-conjugating dot product.

`vecdot`
: Vector dot product of two arrays.

[`tensordot`](dask.array.tensordot.md#dask.array.tensordot)
: Sum products over arbitrary axes.

[`einsum`](dask.array.einsum.md#dask.array.einsum)
: Einstein summation convention.

[`matmul`](dask.array.matmul.md#dask.array.matmul)
: ‘@’ operator as method with out parameter.

`linalg.multi_dot`
: Chained dot product.

### Examples

```pycon
>>> import numpy as np
>>> np.dot(3, 4)
12
```

Neither argument is complex-conjugated:

```pycon
>>> np.dot([2j, 3j], [2j, 3j])
(-13+0j)
```

For 2-D arrays it is the matrix product:

```pycon
>>> a = [[1, 0], [0, 1]]
>>> b = [[4, 1], [2, 2]]
>>> np.dot(a, b)
array([[4, 1],
       [2, 2]])
```

```pycon
>>> a = np.arange(3*4*5*6).reshape((3,4,5,6))
>>> b = np.arange(3*4*5*6)[::-1].reshape((5,4,6,3))
>>> np.dot(a, b)[2,3,2,1,2,2]
499128
>>> sum(a[2,3,2,:] * b[1,2,:,2])
499128
```

<!-- !! processed by numpydoc !! -->


# dask.array.dstack.html.md

# dask.array.dstack

### dask.array.dstack(tup, allow_unknown_chunksizes=False)

Stack arrays in sequence depth wise (along third axis).

This docstring was copied from numpy.dstack.

Some inconsistencies with the Dask version may exist.

This is equivalent to concatenation along the third axis after 2-D arrays
of shape (M,N) have been reshaped to (M,N,1) and 1-D arrays of shape
(N,) have been reshaped to (1,N,1). Rebuilds arrays divided by
dsplit.

This function makes most sense for arrays with up to 3 dimensions. For
instance, for pixel-data with a height (first axis), width (second axis),
and r/g/b channels (third axis). The functions concatenate, stack and
block provide more general stacking and concatenation operations.

* **Parameters:**
  **tup**
  : The arrays must have the same shape along all but the third axis.
    1-D or 2-D arrays must have the same shape.
* **Returns:**
  **stacked**
  : The array formed by stacking the given arrays, will be at least 3-D.

#### SEE ALSO
[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)
: Join a sequence of arrays along an existing axis.

[`stack`](dask.array.stack.md#dask.array.stack)
: Join a sequence of arrays along a new axis.

[`block`](dask.array.block.md#dask.array.block)
: Assemble an nd-array from nested lists of blocks.

[`vstack`](dask.array.vstack.md#dask.array.vstack)
: Stack arrays in sequence vertically (row wise).

[`hstack`](dask.array.hstack.md#dask.array.hstack)
: Stack arrays in sequence horizontally (column wise).

`column_stack`
: Stack 1-D arrays as columns into a 2-D array.

`dsplit`
: Split array along third axis.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array((1,2,3))
>>> b = np.array((4,5,6))
>>> np.dstack((a,b))
array([[[1, 4],
        [2, 5],
        [3, 6]]])
```

```pycon
>>> a = np.array([[1],[2],[3]])
>>> b = np.array([[4],[5],[6]])
>>> np.dstack((a,b))
array([[[1, 4]],
       [[2, 5]],
       [[3, 6]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ediff1d.html.md

# dask.array.ediff1d

### dask.array.ediff1d(ary, to_end=None, to_begin=None)

The differences between consecutive elements of an array.

This docstring was copied from numpy.ediff1d.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ary**
  : If necessary, will be flattened before the differences are taken.

  **to_end**
  : Number(s) to append at the end of the returned differences.

  **to_begin**
  : Number(s) to prepend at the beginning of the returned differences.
* **Returns:**
  **ediff1d**
  : The differences. Loosely, this is `ary.flat[1:] - ary.flat[:-1]`.

#### SEE ALSO
[`diff`](dask.array.diff.md#dask.array.diff), [`gradient`](dask.array.gradient.md#dask.array.gradient)

### Notes

When applied to masked arrays, this function drops the mask information
if the to_begin and/or to_end parameters are used.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([1, 2, 4, 7, 0])
>>> np.ediff1d(x)
array([ 1,  2,  3, -7])
```

```pycon
>>> np.ediff1d(x, to_begin=-99, to_end=np.array([88, 99]))
array([-99,   1,   2, ...,  -7,  88,  99])
```

The returned array is always 1D.

```pycon
>>> y = [[1, 2, 4], [1, 6, 24]]
>>> np.ediff1d(y)
array([ 1,  2, -3,  5, 18])
```

<!-- !! processed by numpydoc !! -->


# dask.array.einsum.html.md

# dask.array.einsum

### dask.array.einsum(subscripts, \*operands, out=None, dtype=None, order='K', casting='safe', optimize=False)

This docstring was copied from numpy.einsum.

Some inconsistencies with the Dask version may exist.

Dask added an additional keyword-only argument `split_every`.

split_every: int >= 2 or dict(axis: int), optional
: Determines the depth of the recursive aggregation.
  Defaults to `None` which would let dask heuristically
  decide a good default.

Evaluates the Einstein summation convention on the operands.

Using the Einstein summation convention, many common multi-dimensional,
linear algebraic array operations can be represented in a simple fashion.
In *implicit* mode einsum computes these values.

In *explicit* mode, einsum provides further flexibility to compute
other array operations that might not be considered classical Einstein
summation operations, by disabling, or forcing summation over specified
subscript labels.

See the notes and examples for clarification.

* **Parameters:**
  **subscripts**
  : Specifies the subscripts for summation as comma separated list of
    subscript labels. An implicit (classical Einstein summation)
    calculation is performed unless the explicit indicator ‘->’ is
    included as well as subscript labels of the precise output form.

  **operands**
  : These are the arrays for the operation.

  **out**
  : If provided, the calculation is done into this array.

  **dtype**
  : If provided, forces the calculation to use the data type specified.
    Note that you may have to also give a more liberal casting
    parameter to allow the conversions. Default is None.

  **order**
  : Controls the memory layout of the output. ‘C’ means it should
    be C contiguous. ‘F’ means it should be Fortran contiguous,
    ‘A’ means it should be ‘F’ if the inputs are all ‘F’, ‘C’ otherwise.
    ‘K’ means it should be as close to the layout as the inputs as
    is possible, including arbitrarily permuted axes.
    Default is ‘K’.

  **casting**
  : Controls what kind of data casting may occur.  Setting this to
    ‘unsafe’ is not recommended, as it can adversely affect accumulations.
    * ‘no’ means the data types should not be cast at all.
    * ‘equiv’ means only byte-order changes are allowed.
    * ‘safe’ means only casts which can preserve values are allowed.
    * ‘same_kind’ means only safe casts or casts within a kind,
      like float64 to float32, are allowed.
    * ‘unsafe’ means any data conversions may be done.
    <br/>
    Default is ‘safe’.

  **optimize**
  : Controls if intermediate optimization should occur. No optimization
    will occur if False and True will default to the ‘greedy’ algorithm.
    Also accepts an explicit contraction list from the `np.einsum_path`
    function. See `np.einsum_path` for more details. Defaults to False.
* **Returns:**
  **output**
  : The calculation based on the Einstein summation convention.

#### SEE ALSO
`einsum_path`, [`dot`](dask.array.dot.md#dask.array.dot), `inner`, [`outer`](dask.array.outer.md#dask.array.outer), [`tensordot`](dask.array.tensordot.md#dask.array.tensordot), `linalg.multi_dot`

[`einsum`](#dask.array.einsum)
: Similar verbose interface is provided by the [einops](https://github.com/arogozhnikov/einops) package to cover additional operations: transpose, reshape/flatten, repeat/tile, squeeze/unsqueeze and reductions. The [opt_einsum](https://optimized-einsum.readthedocs.io/en/stable/) optimizes contraction order for einsum-like expressions in backend-agnostic manner.

### Notes

The Einstein summation convention can be used to compute
many multi-dimensional, linear algebraic array operations. einsum
provides a succinct way of representing these.

A non-exhaustive list of these operations,
which can be computed by einsum, is shown below along with examples:

* Trace of an array, [`numpy.trace()`](https://numpy.org/doc/stable/reference/generated/numpy.trace.html#numpy.trace).
* Return a diagonal, [`numpy.diag()`](https://numpy.org/doc/stable/reference/generated/numpy.diag.html#numpy.diag).
* Array axis summations, [`numpy.sum()`](https://numpy.org/doc/stable/reference/generated/numpy.sum.html#numpy.sum).
* Transpositions and permutations, [`numpy.transpose()`](https://numpy.org/doc/stable/reference/generated/numpy.transpose.html#numpy.transpose).
* Matrix multiplication and dot product, `numpy.matmul()`
  : [`numpy.dot()`](https://numpy.org/doc/stable/reference/generated/numpy.dot.html#numpy.dot).
* Vector inner and outer products, [`numpy.inner()`](https://numpy.org/doc/stable/reference/generated/numpy.inner.html#numpy.inner)
  : [`numpy.outer()`](https://numpy.org/doc/stable/reference/generated/numpy.outer.html#numpy.outer).
* Broadcasting, element-wise and scalar multiplication,
  : `numpy.multiply()`.
* Tensor contractions, [`numpy.tensordot()`](https://numpy.org/doc/stable/reference/generated/numpy.tensordot.html#numpy.tensordot).
* Chained array operations, in efficient calculation order,
  : [`numpy.einsum_path()`](https://numpy.org/doc/stable/reference/generated/numpy.einsum_path.html#numpy.einsum_path).

The subscripts string is a comma-separated list of subscript labels,
where each label refers to a dimension of the corresponding operand.
Whenever a label is repeated it is summed, so `np.einsum('i,i', a, b)`
is equivalent to [`np.inner(a,b)`](https://numpy.org/doc/stable/reference/generated/numpy.inner.html#numpy.inner). If a label
appears only once, it is not summed, so `np.einsum('i', a)`
produces a view of `a` with no changes. A further example
`np.einsum('ij,jk', a, b)` describes traditional matrix multiplication
and is equivalent to `np.matmul(a,b)`.
Repeated subscript labels in one operand take the diagonal.
For example, `np.einsum('ii', a)` is equivalent to
[`np.trace(a)`](https://numpy.org/doc/stable/reference/generated/numpy.trace.html#numpy.trace).

In *implicit mode*, the chosen subscripts are important
since the axes of the output are reordered alphabetically.  This
means that `np.einsum('ij', a)` doesn’t affect a 2D array, while
`np.einsum('ji', a)` takes its transpose. Additionally,
`np.einsum('ij,jk', a, b)` returns a matrix multiplication, while,
`np.einsum('ij,jh', a, b)` returns the transpose of the
multiplication since subscript ‘h’ precedes subscript ‘i’.

In *explicit mode* the output can be directly controlled by
specifying output subscript labels.  This requires the
identifier ‘->’ as well as the list of output subscript labels.
This feature increases the flexibility of the function since
summing can be disabled or forced when required. The call
`np.einsum('i->', a)` is like [`np.sum(a)`](https://numpy.org/doc/stable/reference/generated/numpy.sum.html#numpy.sum)
if `a` is a 1-D array, and `np.einsum('ii->i', a)`
is like [`np.diag(a)`](https://numpy.org/doc/stable/reference/generated/numpy.diag.html#numpy.diag) if `a` is a square 2-D array.
The difference is that einsum does not allow broadcasting by default.
Additionally `np.einsum('ij,jh->ih', a, b)` directly specifies the
order of the output subscript labels and therefore returns matrix
multiplication, unlike the example above in implicit mode.

To enable and control broadcasting, use an ellipsis.  Default
NumPy-style broadcasting is done by adding an ellipsis
to the left of each term, like `np.einsum('...ii->...i', a)`.
`np.einsum('...i->...', a)` is like
[`np.sum(a, axis=-1)`](https://numpy.org/doc/stable/reference/generated/numpy.sum.html#numpy.sum) for array `a` of any shape.
To take the trace along the first and last axes,
you can do `np.einsum('i...i', a)`, or to do a matrix-matrix
product with the left-most indices instead of rightmost, one can do
`np.einsum('ij...,jk...->ik...', a, b)`.

When there is only one operand, no axes are summed, and no output
parameter is provided, a view into the operand is returned instead
of a new array.  Thus, taking the diagonal as `np.einsum('ii->i', a)`
produces a view (changed in version 1.10.0).

einsum also provides an alternative way to provide the subscripts and
operands as `einsum(op0, sublist0, op1, sublist1, ..., [sublistout])`.
If the output shape is not provided in this format einsum will be
calculated in implicit mode, otherwise it will be performed explicitly.
The examples below have corresponding einsum calls with the two
parameter methods.

Views returned from einsum are now writeable whenever the input array
is writeable. For example, `np.einsum('ijk...->kji...', a)` will now
have the same effect as [`np.swapaxes(a, 0, 2)`](https://numpy.org/doc/stable/reference/generated/numpy.swapaxes.html#numpy.swapaxes)
and `np.einsum('ii->i', a)` will return a writeable view of the diagonal
of a 2D array.

Added the `optimize` argument which will optimize the contraction order
of an einsum expression. For a contraction with three or more operands
this can greatly increase the computational efficiency at the cost of
a larger memory footprint during computation.

Typically a ‘greedy’ algorithm is applied which empirical tests have shown
returns the optimal path in the majority of cases. In some cases ‘optimal’
will return the superlative path through a more expensive, exhaustive
search. For iterative calculations it may be advisable to calculate
the optimal path once and reuse that path by supplying it as an argument.
An example is given below.

See [`numpy.einsum_path()`](https://numpy.org/doc/stable/reference/generated/numpy.einsum_path.html#numpy.einsum_path) for more details.

### Examples

```pycon
>>> a = np.arange(25).reshape(5,5)
>>> b = np.arange(5)
>>> c = np.arange(6).reshape(2,3)
```

Trace of a matrix:

```pycon
>>> np.einsum('ii', a)
60
>>> np.einsum(a, [0,0])
60
>>> np.trace(a)
60
```

Extract the diagonal (requires explicit form):

```pycon
>>> np.einsum('ii->i', a)
array([ 0,  6, 12, 18, 24])
>>> np.einsum(a, [0,0], [0])
array([ 0,  6, 12, 18, 24])
>>> np.diag(a)
array([ 0,  6, 12, 18, 24])
```

Sum over an axis (requires explicit form):

```pycon
>>> np.einsum('ij->i', a)
array([ 10,  35,  60,  85, 110])
>>> np.einsum(a, [0,1], [0])
array([ 10,  35,  60,  85, 110])
>>> np.sum(a, axis=1)
array([ 10,  35,  60,  85, 110])
```

For higher dimensional arrays summing a single axis can be done
with ellipsis:

```pycon
>>> np.einsum('...j->...', a)
array([ 10,  35,  60,  85, 110])
>>> np.einsum(a, [Ellipsis,1], [Ellipsis])
array([ 10,  35,  60,  85, 110])
```

Compute a matrix transpose, or reorder any number of axes:

```pycon
>>> np.einsum('ji', c)
array([[0, 3],
       [1, 4],
       [2, 5]])
>>> np.einsum('ij->ji', c)
array([[0, 3],
       [1, 4],
       [2, 5]])
>>> np.einsum(c, [1,0])
array([[0, 3],
       [1, 4],
       [2, 5]])
>>> np.transpose(c)
array([[0, 3],
       [1, 4],
       [2, 5]])
```

Vector inner products:

```pycon
>>> np.einsum('i,i', b, b)
30
>>> np.einsum(b, [0], b, [0])
30
>>> np.inner(b,b)
30
```

Matrix vector multiplication:

```pycon
>>> np.einsum('ij,j', a, b)
array([ 30,  80, 130, 180, 230])
>>> np.einsum(a, [0,1], b, [1])
array([ 30,  80, 130, 180, 230])
>>> np.dot(a, b)
array([ 30,  80, 130, 180, 230])
>>> np.einsum('...j,j', a, b)
array([ 30,  80, 130, 180, 230])
```

Broadcasting and scalar multiplication:

```pycon
>>> np.einsum('..., ...', 3, c)
array([[ 0,  3,  6],
       [ 9, 12, 15]])
>>> np.einsum(',ij', 3, c)
array([[ 0,  3,  6],
       [ 9, 12, 15]])
>>> np.einsum(3, [Ellipsis], c, [Ellipsis])
array([[ 0,  3,  6],
       [ 9, 12, 15]])
>>> np.multiply(3, c)
array([[ 0,  3,  6],
       [ 9, 12, 15]])
```

Vector outer product:

```pycon
>>> np.einsum('i,j', np.arange(2)+1, b)
array([[0, 1, 2, 3, 4],
       [0, 2, 4, 6, 8]])
>>> np.einsum(np.arange(2)+1, [0], b, [1])
array([[0, 1, 2, 3, 4],
       [0, 2, 4, 6, 8]])
>>> np.outer(np.arange(2)+1, b)
array([[0, 1, 2, 3, 4],
       [0, 2, 4, 6, 8]])
```

Tensor contraction:

```pycon
>>> a = np.arange(60.).reshape(3,4,5)
>>> b = np.arange(24.).reshape(4,3,2)
>>> np.einsum('ijk,jil->kl', a, b)
array([[4400., 4730.],
       [4532., 4874.],
       [4664., 5018.],
       [4796., 5162.],
       [4928., 5306.]])
>>> np.einsum(a, [0,1,2], b, [1,0,3], [2,3])
array([[4400., 4730.],
       [4532., 4874.],
       [4664., 5018.],
       [4796., 5162.],
       [4928., 5306.]])
>>> np.tensordot(a,b, axes=([1,0],[0,1]))
array([[4400., 4730.],
       [4532., 4874.],
       [4664., 5018.],
       [4796., 5162.],
       [4928., 5306.]])
```

Writeable returned arrays (since version 1.10.0):

```pycon
>>> a = np.zeros((3, 3))
>>> np.einsum('ii->i', a)[:] = 1
>>> a
array([[1., 0., 0.],
       [0., 1., 0.],
       [0., 0., 1.]])
```

Example of ellipsis use:

```pycon
>>> a = np.arange(6).reshape((3,2))
>>> b = np.arange(12).reshape((4,3))
>>> np.einsum('ki,jk->ij', a, b)
array([[10, 28, 46, 64],
       [13, 40, 67, 94]])
>>> np.einsum('ki,...k->i...', a, b)
array([[10, 28, 46, 64],
       [13, 40, 67, 94]])
>>> np.einsum('k...,jk', a, b)
array([[10, 28, 46, 64],
       [13, 40, 67, 94]])
```

Chained array operations. For more complicated contractions, speed ups
might be achieved by repeatedly computing a ‘greedy’ path or pre-computing
the ‘optimal’ path and repeatedly applying it, using an einsum_path
insertion (since version 1.12.0). Performance improvements can be
particularly significant with larger arrays:

```pycon
>>> a = np.ones(64).reshape(2,4,8)
```

Basic einsum: ~1520ms  (benchmarked on 3.1GHz Intel i5.)

```pycon
>>> for iteration in range(500):
...     _ = np.einsum('ijk,ilm,njm,nlk,abc->',a,a,a,a,a)
```

Sub-optimal einsum (due to repeated path calculation time): ~330ms

```pycon
>>> for iteration in range(500):
...     _ = np.einsum('ijk,ilm,njm,nlk,abc->',a,a,a,a,a,
...         optimize='optimal')
```

Greedy einsum (faster optimal path approximation): ~160ms

```pycon
>>> for iteration in range(500):
...     _ = np.einsum('ijk,ilm,njm,nlk,abc->',a,a,a,a,a, optimize='greedy')
```

Optimal einsum (best usage pattern in some use cases): ~110ms

```pycon
>>> path = np.einsum_path('ijk,ilm,njm,nlk,abc->',a,a,a,a,a,
...     optimize='optimal')[0]
>>> for iteration in range(500):
...     _ = np.einsum('ijk,ilm,njm,nlk,abc->',a,a,a,a,a, optimize=path)
```

<!-- !! processed by numpydoc !! -->


# dask.array.empty.html.md

# dask.array.empty

### dask.array.empty(\*args, \*\*kwargs)

> Blocked variant of empty_like

> Follows the signature of empty_like exactly except that it also features
> optional keyword arguments `chunks: int, tuple, or dict` and `name: str`.

> Original signature follows below.
> Return a new array with the same shape and type as a given array.
* **Parameters:**
  **prototype**
  : The shape and data-type of prototype define these same attributes
    of the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if prototype is Fortran
    contiguous, ‘C’ otherwise. ‘K’ means match the layout of prototype
    as closely as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of prototype, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of uninitialized (arbitrary) data with the same
    shape and type as prototype.

#### SEE ALSO
[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`full_like`](dask.array.full_like.md#dask.array.full_like)
: Return a new array with shape of input filled with value.

[`empty`](#dask.array.empty)
: Return a new uninitialized array.

### Notes

Unlike other array creation functions (e.g. zeros_like, ones_like,
full_like), empty_like does not initialize the values of the array,
and may therefore be marginally faster. However, the values stored in the
newly allocated array are arbitrary. For reproducible behavior, be sure
to set each element of the array before reading.

### Examples

```pycon
>>> import numpy as np
>>> a = ([1,2,3], [4,5,6])                         # a is array-like
>>> np.empty_like(a)
array([[-1073741821, -1073741821,           3],    # uninitialized
       [          0,           0, -1073741821]])
>>> a = np.array([[1., 2., 3.],[4.,5.,6.]])
>>> np.empty_like(a)
array([[ -2.00000715e+000,   1.48219694e-323,  -2.00000572e+000], # uninitialized
       [  4.38791518e-305,  -2.00000715e+000,   4.17269252e-309]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.empty_like.html.md

# dask.array.empty_like

### dask.array.empty_like(a, dtype=None, order='C', chunks=None, name=None, shape=None)

Return a new array with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of the
    returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Whether to store multidimensional data in C- or Fortran-contiguous
    (row- or column-wise) order in memory.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if `len(array) % chunks != 0`.

  **name**
  : An optional keyname for the array. Defaults to hashing the input
    keyword arguments.

  **shape**
  : Overrides the shape of the result.
* **Returns:**
  **out**
  : Array of uninitialized (arbitrary) data with the same
    shape and type as a.

#### SEE ALSO
[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`empty`](dask.array.empty.md#dask.array.empty)
: Return a new uninitialized array.

[`ones`](dask.array.ones.md#dask.array.ones)
: Return a new array setting values to one.

[`zeros`](dask.array.zeros.md#dask.array.zeros)
: Return a new array setting values to zero.

### Notes

This function does *not* initialize the returned array; to do that use
zeros_like or ones_like instead.  It may be marginally faster than
the functions that do set the array values.

<!-- !! processed by numpydoc !! -->


# dask.array.equal.html.md

# dask.array.equal

### dask.array.equal(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'equal'>*

This docstring was copied from numpy.equal.

Some inconsistencies with the Dask version may exist.

Return (x1 == x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`not_equal`](dask.array.not_equal.md#dask.array.not_equal), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`greater`](dask.array.greater.md#dask.array.greater), [`less`](dask.array.less.md#dask.array.less)

### Examples

```pycon
>>> import numpy as np
>>> np.equal([0, 1, 3], np.arange(3))
array([ True,  True, False])
```

What is compared are values, not types. So an int (1) and an array of
length one can evaluate as True:

```pycon
>>> np.equal(1, np.ones(1))
array([ True])
```

The `==` operator can be used as a shorthand for `np.equal` on
ndarrays.

```pycon
>>> a = np.array([2, 4, 6])
>>> b = np.array([2, 4, 2])
>>> a == b
array([ True,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.exp.html.md

# dask.array.exp

### dask.array.exp(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'exp'>*

This docstring was copied from numpy.exp.

Some inconsistencies with the Dask version may exist.

Calculate the exponential of all elements in the input array.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise exponential of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`expm1`](dask.array.expm1.md#dask.array.expm1)
: Calculate `exp(x) - 1` for all elements in the array.

[`exp2`](dask.array.exp2.md#dask.array.exp2)
: Calculate `2**x` for all elements in the array.

### Notes

The irrational number `e` is also known as Euler’s number.  It is
approximately 2.718281, and is the base of the natural logarithm,
`ln` (this means that, if $x = \ln y = \log_e y$,
then $e^x = y$. For real input, `exp(x)` is always positive.

For complex arguments, `x = a + ib`, we can write
$e^x = e^a e^{ib}$.  The first term, $e^a$, is already
known (it is the real argument, described above).  The second term,
$e^{ib}$, is $\cos b + i \sin b$, a function with
magnitude 1 and a periodic phase.

### References

### Examples

Plot the magnitude and phase of `exp(x)` in the complex plane:

```pycon
>>> import numpy as np
```

```pycon
>>> import matplotlib.pyplot as plt
>>> import numpy as np
```

```pycon
>>> x = np.linspace(-2*np.pi, 2*np.pi, 100)
>>> xx = x + 1j * x[:, np.newaxis] # a + ib over complex plane
>>> out = np.exp(xx)
```

```pycon
>>> plt.subplot(121)
>>> plt.imshow(np.abs(out),
...            extent=[-2*np.pi, 2*np.pi, -2*np.pi, 2*np.pi], cmap='gray')
>>> plt.title('Magnitude of exp(x)')
```

```pycon
>>> plt.subplot(122)
>>> plt.imshow(np.angle(out),
...            extent=[-2*np.pi, 2*np.pi, -2*np.pi, 2*np.pi], cmap='hsv')
>>> plt.title('Phase (angle) of exp(x)')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.exp2.html.md

# dask.array.exp2

### dask.array.exp2(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'exp2'>*

This docstring was copied from numpy.exp2.

Some inconsistencies with the Dask version may exist.

Calculate 2\*\*p for all p in the input array.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Element-wise 2 to the power x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`power`](dask.array.power.md#dask.array.power)

### Examples

```pycon
>>> import numpy as np
>>> np.exp2([2, 3])
array([ 4.,  8.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.expand_dims.html.md

# dask.array.expand_dims

### dask.array.expand_dims(a, axis)

Expand the shape of an array.

This docstring was copied from numpy.expand_dims.

Some inconsistencies with the Dask version may exist.

Insert a new axis that will appear at the axis position in the expanded
array shape.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : Position in the expanded axes where the new axis (or axes) is placed.
    <br/>
    #### Deprecated
    Deprecated since version 1.13.0: Passing an axis where `axis > a.ndim` will be treated as
    `axis == a.ndim`, and passing `axis < -a.ndim - 1` will
    be treated as `axis == 0`. This behavior is deprecated.
* **Returns:**
  **result**
  : View of a with the number of dimensions increased.

#### SEE ALSO
[`squeeze`](dask.array.squeeze.md#dask.array.squeeze)
: The inverse operation, removing singleton dimensions

[`reshape`](dask.array.reshape.md#dask.array.reshape)
: Insert, remove, and combine dimensions, and resize existing ones

[`atleast_1d`](dask.array.atleast_1d.md#dask.array.atleast_1d), [`atleast_2d`](dask.array.atleast_2d.md#dask.array.atleast_2d), [`atleast_3d`](dask.array.atleast_3d.md#dask.array.atleast_3d)

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([1, 2])
>>> x.shape
(2,)
```

The following is equivalent to `x[np.newaxis, :]` or `x[np.newaxis]`:

```pycon
>>> y = np.expand_dims(x, axis=0)
>>> y
array([[1, 2]])
>>> y.shape
(1, 2)
```

The following is equivalent to `x[:, np.newaxis]`:

```pycon
>>> y = np.expand_dims(x, axis=1)
>>> y
array([[1],
       [2]])
>>> y.shape
(2, 1)
```

`axis` may also be a tuple:

```pycon
>>> y = np.expand_dims(x, axis=(0, 1))
>>> y
array([[[1, 2]]])
```

```pycon
>>> y = np.expand_dims(x, axis=(2, 0))
>>> y
array([[[1],
        [2]]])
```

Note that some examples may use `None` instead of `np.newaxis`.  These
are the same objects:

```pycon
>>> np.newaxis is None
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.expm1.html.md

# dask.array.expm1

### dask.array.expm1(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'expm1'>*

This docstring was copied from numpy.expm1.

Some inconsistencies with the Dask version may exist.

Calculate `exp(x) - 1` for all elements in the array.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Element-wise exponential minus one: `out = exp(x) - 1`.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`log1p`](dask.array.log1p.md#dask.array.log1p)
: `log(1 + x)`, the inverse of expm1.

### Notes

This function provides greater precision than `exp(x) - 1`
for small values of `x`.

### Examples

The true value of `exp(1e-10) - 1` is `1.00000000005e-10` to
about 32 significant digits. This example shows the superiority of
expm1 in this case.

```pycon
>>> import numpy as np
>>> np.expm1(1e-10)
1.00000000005e-10
>>> np.exp(1e-10) - 1
1.000000082740371e-10
```

<!-- !! processed by numpydoc !! -->


# dask.array.extract.html.md

# dask.array.extract

### dask.array.extract(condition, arr)

Return the elements of an array that satisfy some condition.

This docstring was copied from numpy.extract.

Some inconsistencies with the Dask version may exist.

This is equivalent to `np.compress(ravel(condition), ravel(arr))`.  If
condition is boolean `np.extract` is equivalent to `arr[condition]`.

Note that place does the exact opposite of extract.

* **Parameters:**
  **condition**
  : An array whose nonzero or True entries indicate the elements of arr
    to extract.

  **arr**
  : Input array of the same size as condition.
* **Returns:**
  **extract**
  : Rank 1 array of values from arr where condition is True.

#### SEE ALSO
[`take`](dask.array.take.md#dask.array.take), `put`, `copyto`, [`compress`](dask.array.compress.md#dask.array.compress), `place`

### Examples

```pycon
>>> import numpy as np
>>> arr = np.arange(12).reshape((3, 4))
>>> arr
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> condition = np.mod(arr, 3)==0
>>> condition
array([[ True, False, False,  True],
       [False, False,  True, False],
       [False,  True, False, False]])
>>> np.extract(condition, arr)
array([0, 3, 6, 9])
```

If condition is boolean:

```pycon
>>> arr[condition]
array([0, 3, 6, 9])
```

<!-- !! processed by numpydoc !! -->


# dask.array.eye.html.md

# dask.array.eye

### dask.array.eye(N, chunks='auto', M=None, k=0, dtype=<class 'float'>)

Return a 2-D Array with ones on the diagonal and zeros elsewhere.

* **Parameters:**
  **N**
  : Number of rows in the output.

  **chunks**
  : How to chunk the array. Must be one of the following forms:
    - A blocksize like 1000.
    - A size in bytes, like “100 MiB” which will choose a uniform
      block-like shape
    - The word “auto” which acts like the above, but uses a configuration
      value `array.chunk-size` for the chunk size

  **M**
  : Number of columns in the output. If None, defaults to N.

  **k**
  : Index of the diagonal: 0 (the default) refers to the main diagonal,
    a positive value refers to an upper diagonal, and a negative value
    to a lower diagonal.

  **dtype**
  : Data-type of the returned array.
* **Returns:**
  **I**
  : An array where all elements are equal to zero, except for the k-th
    diagonal, whose values are equal to one.

<!-- !! processed by numpydoc !! -->


# dask.array.fabs.html.md

# dask.array.fabs

### dask.array.fabs(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'fabs'>*

This docstring was copied from numpy.fabs.

Some inconsistencies with the Dask version may exist.

Compute the absolute values element-wise.

This function returns the absolute values (positive magnitude) of the
data in x. Complex values are not handled, use absolute to find the
absolute values of complex data.

* **Parameters:**
  **x**
  : The array of numbers for which the absolute values are required. If
    x is a scalar, the result y will also be a scalar.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The absolute values of x, the returned values are always floats.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`absolute`](dask.array.absolute.md#dask.array.absolute)
: Absolute values including complex types.

### Examples

```pycon
>>> import numpy as np
>>> np.fabs(-1)
1.0
>>> np.fabs([-1.2, 1.2])
array([ 1.2,  1.2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fft.html.md

# dask.array.fft.fft

### dask.array.fft.fft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.fft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.fft docstring follows below:

Compute the one-dimensional discrete Fourier Transform.

This function computes the one-dimensional *n*-point discrete Fourier
Transform (DFT) with the efficient Fast Fourier Transform (FFT)
algorithm [[CT]](#r83543ef959bc-ct).

* **Parameters:**
  **a**
  : Input array, can be complex.

  **n**
  : Length of the transformed axis of the output.
    If n is smaller than the length of the input, the input is cropped.
    If it is larger, the input is padded with zeros.  If n is not given,
    the length of the input along the axis specified by axis is used.

  **axis**
  : Axis over which to compute the FFT.  If not given, the last axis is
    used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
* **Raises:**
  IndexError
  : If axis is not a valid axis of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: for definition of the DFT and conventions used.

[`ifft`](dask.array.fft.ifft.md#dask.array.fft.ifft)
: The inverse of fft.

[`fft2`](dask.array.fft.fft2.md#dask.array.fft.fft2)
: The two-dimensional FFT.

[`fftn`](dask.array.fft.fftn.md#dask.array.fft.fftn)
: The *n*-dimensional FFT.

[`rfftn`](dask.array.fft.rfftn.md#dask.array.fft.rfftn)
: The *n*-dimensional FFT of real input.

[`fftfreq`](dask.array.fft.fftfreq.md#dask.array.fft.fftfreq)
: Frequency bins for given FFT parameters.

### Notes

FFT (Fast Fourier Transform) refers to a way the discrete Fourier
Transform (DFT) can be calculated efficiently, by using symmetries in the
calculated terms.  The symmetry is highest when n is a power of 2, and
the transform is therefore most efficient for these sizes.

The DFT is defined, with the conventions used in this implementation, in
the documentation for the numpy.fft module.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.fft.fft(np.exp(2j * np.pi * np.arange(8) / 8))
array([-2.33486982e-16+1.14423775e-17j,  8.00000000e+00-1.25557246e-15j,
        2.33486982e-16+2.33486982e-16j,  0.00000000e+00+1.22464680e-16j,
       -1.14423775e-17+2.33486982e-16j,  0.00000000e+00+5.20784380e-16j,
        1.14423775e-17+1.14423775e-17j,  0.00000000e+00+1.22464680e-16j])
```

In this example, real input has an FFT which is Hermitian, i.e., symmetric
in the real part and anti-symmetric in the imaginary part, as described in
the numpy.fft documentation:

```pycon
>>> import matplotlib.pyplot as plt
>>> t = np.arange(256)
>>> sp = np.fft.fft(np.sin(t))
>>> freq = np.fft.fftfreq(t.shape[-1])
>>> _ = plt.plot(freq, sp.real, freq, sp.imag)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fft2.html.md

# dask.array.fft.fft2

### dask.array.fft.fft2(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.fft2

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.fft2 docstring follows below:

Compute the 2-dimensional discrete Fourier Transform.

This function computes the *n*-dimensional discrete Fourier Transform
over any axes in an *M*-dimensional array by means of the
Fast Fourier Transform (FFT).  By default, the transform is computed over
the last two axes of the input array, i.e., a 2-dimensional FFT.

* **Parameters:**
  **a**
  : Input array, can be complex

  **s**
  : Shape (length of each transformed axis) of the output
    (`s[0]` refers to axis 0, `s[1]` to axis 1, etc.).
    This corresponds to `n` for `fft(x, n)`.
    Along each axis, if the given shape is smaller than that of the input,
    the input is cropped. If it is larger, the input is padded with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes specified
    by axes is used.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the FFT.  If not given, the last two
    axes are used.  A repeated index in axes means the transform over
    that axis is performed multiple times.  A one-element sequence means
    that a one-dimensional FFT is performed. Default: `(-2, -1)`.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must not be `None`.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for all axes (and hence only the
    last axis can have `s` not equal to the shape at that axis).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or the last two axes if axes is not given.
* **Raises:**
  ValueError
  : If s and axes have different length, or axes not given and
    `len(s) != 2`.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: Overall view of discrete Fourier transforms, with definitions and conventions used.

[`ifft2`](dask.array.fft.ifft2.md#dask.array.fft.ifft2)
: The inverse two-dimensional FFT.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT.

[`fftn`](dask.array.fft.fftn.md#dask.array.fft.fftn)
: The *n*-dimensional FFT.

[`fftshift`](dask.array.fft.fftshift.md#dask.array.fft.fftshift)
: Shifts zero-frequency terms to the center of the array. For two-dimensional input, swaps first and third quadrants, and second and fourth quadrants.

### Notes

fft2 is just fftn with a different default for axes.

The output, analogously to fft, contains the term for zero frequency in
the low-order corner of the transformed axes, the positive frequency terms
in the first half of these axes, the term for the Nyquist frequency in the
middle of the axes and the negative frequency terms in the second half of
the axes, in order of decreasingly negative frequency.

See fftn for details and a plotting example, and numpy.fft for
definitions and conventions used.

### Examples

```pycon
>>> import numpy as np
>>> a = np.mgrid[:5, :5][0]
>>> np.fft.fft2(a)
array([[ 50.  +0.j        ,   0.  +0.j        ,   0.  +0.j        , # may vary
          0.  +0.j        ,   0.  +0.j        ],
       [-12.5+17.20477401j,   0.  +0.j        ,   0.  +0.j        ,
          0.  +0.j        ,   0.  +0.j        ],
       [-12.5 +4.0614962j ,   0.  +0.j        ,   0.  +0.j        ,
          0.  +0.j        ,   0.  +0.j        ],
       [-12.5 -4.0614962j ,   0.  +0.j        ,   0.  +0.j        ,
          0.  +0.j        ,   0.  +0.j        ],
       [-12.5-17.20477401j,   0.  +0.j        ,   0.  +0.j        ,
          0.  +0.j        ,   0.  +0.j        ]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fft_wrap.html.md

# dask.array.fft.fft_wrap

### dask.array.fft.fft_wrap(fft_func, kind=None, dtype=None, allow_fftpack=False)

Wrap 1D, 2D, and ND real and complex FFT functions

Takes a function that behaves like `numpy.fft` functions and
a specified kind to match it to that are named after the functions
in the `numpy.fft` API.

Supported kinds include:

> * fft
> * fft2
> * fftn
> * ifft
> * ifft2
> * ifftn
> * rfft
> * rfft2
> * rfftn
> * irfft
> * irfft2
> * irfftn
> * hfft
> * ihfft

### Examples

```pycon
>>> import dask.array.fft as dff
>>> parallel_fft = dff.fft_wrap(np.fft.fft)
>>> parallel_ifft = dff.fft_wrap(np.fft.ifft)
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fftfreq.html.md

# dask.array.fft.fftfreq

### dask.array.fft.fftfreq(n, d=1.0, chunks='auto')

Return the Discrete Fourier Transform sample frequencies.

This docstring was copied from numpy.fft.fftfreq.

Some inconsistencies with the Dask version may exist.

The returned float array f contains the frequency bin centers in cycles
per unit of the sample spacing (with zero at the start).  For instance, if
the sample spacing is in seconds, then the frequency unit is cycles/second.

Given a window length n and a sample spacing d:

```default
f = [0, 1, ...,   n/2-1,     -n/2, ..., -1] / (d*n)   if n is even
f = [0, 1, ..., (n-1)/2, -(n-1)/2, ..., -1] / (d*n)   if n is odd
```

* **Parameters:**
  **n**
  : Window length.

  **d**
  : Sample spacing (inverse of the sampling rate). Defaults to 1.

  **device**
  : The device on which to place the created array. Default: `None`.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **f**
  : Array of length n containing the sample frequencies.

### Examples

```pycon
>>> import numpy as np
>>> signal = np.array([-2, 8, 6, 4, 1, 0, 3, 5], dtype=np.float64)
>>> fourier = np.fft.fft(signal)
>>> n = signal.size
>>> timestep = 0.1
>>> freq = np.fft.fftfreq(n, d=timestep)
>>> freq
array([ 0.  ,  1.25,  2.5 , ..., -3.75, -2.5 , -1.25])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fftn.html.md

# dask.array.fft.fftn

### dask.array.fft.fftn(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.fftn

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.fftn docstring follows below:

Compute the N-dimensional discrete Fourier Transform.

This function computes the *N*-dimensional discrete Fourier Transform over
any number of axes in an *M*-dimensional array by means of the Fast Fourier
Transform (FFT).

* **Parameters:**
  **a**
  : Input array, can be complex.

  **s**
  : Shape (length of each transformed axis) of the output
    (`s[0]` refers to axis 0, `s[1]` to axis 1, etc.).
    This corresponds to `n` for `fft(x, n)`.
    Along any axis, if the given shape is smaller than that of the input,
    the input is cropped. If it is larger, the input is padded with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes specified
    by axes is used.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the FFT.  If not given, the last `len(s)`
    axes are used, or all axes if s is also not specified.
    Repeated indices in axes means that the transform over that axis is
    performed multiple times.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must be explicitly specified too.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for all axes (and hence is
    incompatible with passing in all but the trivial `s`).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or by a combination of s and a,
    as explained in the parameters section above.
* **Raises:**
  ValueError
  : If s and axes have different length.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: Overall view of discrete Fourier transforms, with definitions and conventions used.

[`ifftn`](dask.array.fft.ifftn.md#dask.array.fft.ifftn)
: The inverse of fftn, the inverse *n*-dimensional FFT.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT, with definitions and conventions used.

[`rfftn`](dask.array.fft.rfftn.md#dask.array.fft.rfftn)
: The *n*-dimensional FFT of real input.

[`fft2`](dask.array.fft.fft2.md#dask.array.fft.fft2)
: The two-dimensional FFT.

[`fftshift`](dask.array.fft.fftshift.md#dask.array.fft.fftshift)
: Shifts zero-frequency terms to centre of array

### Notes

The output, analogously to fft, contains the term for zero frequency in
the low-order corner of all axes, the positive frequency terms in the
first half of all axes, the term for the Nyquist frequency in the middle
of all axes and the negative frequency terms in the second half of all
axes, in order of decreasingly negative frequency.

See numpy.fft for details, definitions and conventions used.

### Examples

```pycon
>>> import numpy as np
>>> a = np.mgrid[:3, :3, :3][0]
>>> np.fft.fftn(a, axes=(1, 2))
array([[[ 0.+0.j,   0.+0.j,   0.+0.j], # may vary
        [ 0.+0.j,   0.+0.j,   0.+0.j],
        [ 0.+0.j,   0.+0.j,   0.+0.j]],
       [[ 9.+0.j,   0.+0.j,   0.+0.j],
        [ 0.+0.j,   0.+0.j,   0.+0.j],
        [ 0.+0.j,   0.+0.j,   0.+0.j]],
       [[18.+0.j,   0.+0.j,   0.+0.j],
        [ 0.+0.j,   0.+0.j,   0.+0.j],
        [ 0.+0.j,   0.+0.j,   0.+0.j]]])
>>> np.fft.fftn(a, (2, 2), axes=(0, 1))
array([[[ 2.+0.j,  2.+0.j,  2.+0.j], # may vary
        [ 0.+0.j,  0.+0.j,  0.+0.j]],
       [[-2.+0.j, -2.+0.j, -2.+0.j],
        [ 0.+0.j,  0.+0.j,  0.+0.j]]])
```

```pycon
>>> import matplotlib.pyplot as plt
>>> [X, Y] = np.meshgrid(2 * np.pi * np.arange(200) / 12,
...                      2 * np.pi * np.arange(200) / 34)
>>> S = np.sin(X) + np.cos(Y) + np.random.uniform(0, 1, X.shape)
>>> FS = np.fft.fftn(S)
>>> plt.imshow(np.log(np.abs(np.fft.fftshift(FS))**2))
<matplotlib.image.AxesImage object at 0x...>
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.fftshift.html.md

# dask.array.fft.fftshift

### dask.array.fft.fftshift(x, axes=None)

Shift the zero-frequency component to the center of the spectrum.

This docstring was copied from numpy.fft.fftshift.

Some inconsistencies with the Dask version may exist.

This function swaps half-spaces for all axes listed (defaults to all).
Note that `y[0]` is the Nyquist component only if `len(x)` is even.

* **Parameters:**
  **x**
  : Input array.

  **axes**
  : Axes over which to shift.  Default is None, which shifts all axes.
* **Returns:**
  **y**
  : The shifted array.

#### SEE ALSO
[`ifftshift`](dask.array.fft.ifftshift.md#dask.array.fft.ifftshift)
: The inverse of fftshift.

### Examples

```pycon
>>> import numpy as np
>>> freqs = np.fft.fftfreq(10, 0.1)
>>> freqs
array([ 0.,  1.,  2., ..., -3., -2., -1.])
>>> np.fft.fftshift(freqs)
array([-5., -4., -3., -2., -1.,  0.,  1.,  2.,  3.,  4.])
```

Shift the zero-frequency component only along the second axis:

```pycon
>>> freqs = np.fft.fftfreq(9, d=1./9).reshape(3, 3)
>>> freqs
array([[ 0.,  1.,  2.],
       [ 3.,  4., -4.],
       [-3., -2., -1.]])
>>> np.fft.fftshift(freqs, axes=(1,))
array([[ 2.,  0.,  1.],
       [-4.,  3.,  4.],
       [-1., -3., -2.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.hfft.html.md

# dask.array.fft.hfft

### dask.array.fft.hfft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.hfft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.hfft docstring follows below:

Compute the FFT of a signal that has Hermitian symmetry, i.e., a real
spectrum.

* **Parameters:**
  **a**
  : The input array.

  **n**
  : Length of the transformed axis of the output. For n output
    points, `n//2 + 1` input points are necessary.  If the input is
    longer than this, it is cropped.  If it is shorter than this, it is
    padded with zeros.  If n is not given, it is taken to be `2*(m-1)`
    where `m` is the length of the input along the axis specified by
    axis.

  **axis**
  : Axis over which to compute the FFT. If not given, the last
    axis is used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
    The length of the transformed axis is n, or, if n is not given,
    `2*m - 2` where `m` is the length of the transformed axis of
    the input. To get an odd number of output points, n must be
    specified, for instance as `2*m - 1` in the typical case,
* **Raises:**
  IndexError
  : If axis is not a valid axis of a.

#### SEE ALSO
[`rfft`](dask.array.fft.rfft.md#dask.array.fft.rfft)
: Compute the one-dimensional FFT for real input.

[`ihfft`](dask.array.fft.ihfft.md#dask.array.fft.ihfft)
: The inverse of hfft.

### Notes

hfft/ihfft are a pair analogous to rfft/irfft, but for the
opposite case: here the signal has Hermitian symmetry in the time
domain and is real in the frequency domain. So here it’s hfft for
which you must supply the length of the result if it is to be odd.

* even: `ihfft(hfft(a, 2*len(a) - 2)) == a`, within roundoff error,
* odd: `ihfft(hfft(a, 2*len(a) - 1)) == a`, within roundoff error.

The correct interpretation of the hermitian input depends on the length of
the original data, as given by n. This is because each input shape could
correspond to either an odd or even length signal. By default, hfft
assumes an even output length which puts the last entry at the Nyquist
frequency; aliasing with its symmetric counterpart. By Hermitian symmetry,
the value is thus treated as purely real. To avoid losing information, the
shape of the full signal **must** be given.

### Examples

```pycon
>>> import numpy as np
>>> signal = np.array([1, 2, 3, 4, 3, 2])
>>> np.fft.fft(signal)
array([15.+0.j,  -4.+0.j,   0.+0.j,  -1.-0.j,   0.+0.j,  -4.+0.j]) # may vary
>>> np.fft.hfft(signal[:4]) # Input first half of signal
array([15.,  -4.,   0.,  -1.,   0.,  -4.])
>>> np.fft.hfft(signal, 6)  # Input entire signal and truncate
array([15.,  -4.,   0.,  -1.,   0.,  -4.])
```

```pycon
>>> signal = np.array([[1, 1.j], [-1.j, 2]])
>>> np.conj(signal.T) - signal   # check Hermitian symmetry
array([[ 0.-0.j,  -0.+0.j], # may vary
       [ 0.+0.j,  0.-0.j]])
>>> freq_spectrum = np.fft.hfft(signal)
>>> freq_spectrum
array([[ 1.,  1.],
       [ 2., -2.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.ifft.html.md

# dask.array.fft.ifft

### dask.array.fft.ifft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.ifft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.ifft docstring follows below:

Compute the one-dimensional inverse discrete Fourier Transform.

This function computes the inverse of the one-dimensional *n*-point
discrete Fourier transform computed by fft.  In other words,
`ifft(fft(a)) == a` to within numerical accuracy.
For a general description of the algorithm and definitions,
see numpy.fft.

The input should be ordered in the same way as is returned by fft,
i.e.,

* `a[0]` should contain the zero frequency term,
* `a[1:n//2]` should contain the positive-frequency terms,
* `a[n//2 + 1:]` should contain the negative-frequency terms, in
  increasing order starting from the most negative frequency.

For an even number of input points, `A[n//2]` represents the sum of
the values at the positive and negative Nyquist frequencies, as the two
are aliased together. See numpy.fft for details.

* **Parameters:**
  **a**
  : Input array, can be complex.

  **n**
  : Length of the transformed axis of the output.
    If n is smaller than the length of the input, the input is cropped.
    If it is larger, the input is padded with zeros.  If n is not given,
    the length of the input along the axis specified by axis is used.
    See notes about padding issues.

  **axis**
  : Axis over which to compute the inverse DFT.  If not given, the last
    axis is used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
* **Raises:**
  IndexError
  : If axis is not a valid axis of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: An introduction, with definitions and general explanations.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional (forward) FFT, of which ifft is the inverse

[`ifft2`](dask.array.fft.ifft2.md#dask.array.fft.ifft2)
: The two-dimensional inverse FFT.

[`ifftn`](dask.array.fft.ifftn.md#dask.array.fft.ifftn)
: The n-dimensional inverse FFT.

### Notes

If the input parameter n is larger than the size of the input, the input
is padded by appending zeros at the end.  Even though this is the common
approach, it might lead to surprising results.  If a different padding is
desired, it must be performed before calling ifft.

### Examples

```pycon
>>> import numpy as np
>>> np.fft.ifft([0, 4, 0, 0])
array([ 1.+0.j,  0.+1.j, -1.+0.j,  0.-1.j]) # may vary
```

Create and plot a band-limited signal with random phases:

```pycon
>>> import matplotlib.pyplot as plt
>>> t = np.arange(400)
>>> n = np.zeros((400,), dtype=np.complex128)
>>> n[40:60] = np.exp(1j*np.random.uniform(0, 2*np.pi, (20,)))
>>> s = np.fft.ifft(n)
>>> plt.plot(t, s.real, label='real')
[<matplotlib.lines.Line2D object at ...>]
>>> plt.plot(t, s.imag, '--', label='imaginary')
[<matplotlib.lines.Line2D object at ...>]
>>> plt.legend()
<matplotlib.legend.Legend object at ...>
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.ifft2.html.md

# dask.array.fft.ifft2

### dask.array.fft.ifft2(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.ifft2

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.ifft2 docstring follows below:

Compute the 2-dimensional inverse discrete Fourier Transform.

This function computes the inverse of the 2-dimensional discrete Fourier
Transform over any number of axes in an M-dimensional array by means of
the Fast Fourier Transform (FFT).  In other words, `ifft2(fft2(a)) == a`
to within numerical accuracy.  By default, the inverse transform is
computed over the last two axes of the input array.

The input, analogously to ifft, should be ordered in the same way as is
returned by fft2, i.e. it should have the term for zero frequency
in the low-order corner of the two axes, the positive frequency terms in
the first half of these axes, the term for the Nyquist frequency in the
middle of the axes and the negative frequency terms in the second half of
both axes, in order of decreasingly negative frequency.

* **Parameters:**
  **a**
  : Input array, can be complex.

  **s**
  : Shape (length of each axis) of the output (`s[0]` refers to axis 0,
    `s[1]` to axis 1, etc.).  This corresponds to n for `ifft(x, n)`.
    Along each axis, if the given shape is smaller than that of the input,
    the input is cropped. If it is larger, the input is padded with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes specified
    by axes is used.  See notes for issue on ifft zero padding.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the FFT.  If not given, the last two
    axes are used.  A repeated index in axes means the transform over
    that axis is performed multiple times.  A one-element sequence means
    that a one-dimensional FFT is performed. Default: `(-2, -1)`.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must not be `None`.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for all axes (and hence is
    incompatible with passing in all but the trivial `s`).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or the last two axes if axes is not given.
* **Raises:**
  ValueError
  : If s and axes have different length, or axes not given and
    `len(s) != 2`.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: Overall view of discrete Fourier transforms, with definitions and conventions used.

[`fft2`](dask.array.fft.fft2.md#dask.array.fft.fft2)
: The forward 2-dimensional FFT, of which ifft2 is the inverse.

[`ifftn`](dask.array.fft.ifftn.md#dask.array.fft.ifftn)
: The inverse of the *n*-dimensional FFT.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT.

[`ifft`](dask.array.fft.ifft.md#dask.array.fft.ifft)
: The one-dimensional inverse FFT.

### Notes

ifft2 is just ifftn with a different default for axes.

See ifftn for details and a plotting example, and numpy.fft for
definition and conventions used.

Zero-padding, analogously with ifft, is performed by appending zeros to
the input along the specified dimension.  Although this is the common
approach, it might lead to surprising results.  If another form of zero
padding is desired, it must be performed before ifft2 is called.

### Examples

```pycon
>>> import numpy as np
>>> a = 4 * np.eye(4)
>>> np.fft.ifft2(a)
array([[1.+0.j,  0.+0.j,  0.+0.j,  0.+0.j], # may vary
       [0.+0.j,  0.+0.j,  0.+0.j,  1.+0.j],
       [0.+0.j,  0.+0.j,  1.+0.j,  0.+0.j],
       [0.+0.j,  1.+0.j,  0.+0.j,  0.+0.j]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.ifftn.html.md

# dask.array.fft.ifftn

### dask.array.fft.ifftn(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.ifftn

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.ifftn docstring follows below:

Compute the N-dimensional inverse discrete Fourier Transform.

This function computes the inverse of the N-dimensional discrete
Fourier Transform over any number of axes in an M-dimensional array by
means of the Fast Fourier Transform (FFT).  In other words,
`ifftn(fftn(a)) == a` to within numerical accuracy.
For a description of the definitions and conventions used, see numpy.fft.

The input, analogously to ifft, should be ordered in the same way as is
returned by fftn, i.e. it should have the term for zero frequency
in all axes in the low-order corner, the positive frequency terms in the
first half of all axes, the term for the Nyquist frequency in the middle
of all axes and the negative frequency terms in the second half of all
axes, in order of decreasingly negative frequency.

* **Parameters:**
  **a**
  : Input array, can be complex.

  **s**
  : Shape (length of each transformed axis) of the output
    (`s[0]` refers to axis 0, `s[1]` to axis 1, etc.).
    This corresponds to `n` for `ifft(x, n)`.
    Along any axis, if the given shape is smaller than that of the input,
    the input is cropped. If it is larger, the input is padded with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes specified
    by axes is used. See notes for issue on ifft zero padding.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the IFFT.  If not given, the last `len(s)`
    axes are used, or all axes if s is also not specified.
    Repeated indices in axes means that the inverse transform over that
    axis is performed multiple times.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must be explicitly specified too.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for all axes (and hence is
    incompatible with passing in all but the trivial `s`).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or by a combination of s or a,
    as explained in the parameters section above.
* **Raises:**
  ValueError
  : If s and axes have different length.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: Overall view of discrete Fourier transforms, with definitions and conventions used.

[`fftn`](dask.array.fft.fftn.md#dask.array.fft.fftn)
: The forward *n*-dimensional FFT, of which ifftn is the inverse.

[`ifft`](dask.array.fft.ifft.md#dask.array.fft.ifft)
: The one-dimensional inverse FFT.

[`ifft2`](dask.array.fft.ifft2.md#dask.array.fft.ifft2)
: The two-dimensional inverse FFT.

[`ifftshift`](dask.array.fft.ifftshift.md#dask.array.fft.ifftshift)
: Undoes fftshift, shifts zero-frequency terms to beginning of array.

### Notes

See numpy.fft for definitions and conventions used.

Zero-padding, analogously with ifft, is performed by appending zeros to
the input along the specified dimension.  Although this is the common
approach, it might lead to surprising results.  If another form of zero
padding is desired, it must be performed before ifftn is called.

### Examples

```pycon
>>> import numpy as np
>>> a = np.eye(4)
>>> np.fft.ifftn(np.fft.fftn(a, axes=(0,)), axes=(1,))
array([[1.+0.j,  0.+0.j,  0.+0.j,  0.+0.j], # may vary
       [0.+0.j,  1.+0.j,  0.+0.j,  0.+0.j],
       [0.+0.j,  0.+0.j,  1.+0.j,  0.+0.j],
       [0.+0.j,  0.+0.j,  0.+0.j,  1.+0.j]])
```

Create and plot an image with band-limited frequency content:

```pycon
>>> import matplotlib.pyplot as plt
>>> n = np.zeros((200,200), dtype=np.complex128)
>>> n[60:80, 20:40] = np.exp(1j*np.random.uniform(0, 2*np.pi, (20, 20)))
>>> im = np.fft.ifftn(n).real
>>> plt.imshow(im)
<matplotlib.image.AxesImage object at 0x...>
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.ifftshift.html.md

# dask.array.fft.ifftshift

### dask.array.fft.ifftshift(x, axes=None)

The inverse of fftshift. Although identical for even-length x, the
functions differ by one sample for odd-length x.

This docstring was copied from numpy.fft.ifftshift.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **x**
  : Input array.

  **axes**
  : Axes over which to calculate.  Defaults to None, which shifts all axes.
* **Returns:**
  **y**
  : The shifted array.

#### SEE ALSO
[`fftshift`](dask.array.fft.fftshift.md#dask.array.fft.fftshift)
: Shift zero-frequency component to the center of the spectrum.

### Examples

```pycon
>>> import numpy as np
>>> freqs = np.fft.fftfreq(9, d=1./9).reshape(3, 3)
>>> freqs
array([[ 0.,  1.,  2.],
       [ 3.,  4., -4.],
       [-3., -2., -1.]])
>>> np.fft.ifftshift(np.fft.fftshift(freqs))
array([[ 0.,  1.,  2.],
       [ 3.,  4., -4.],
       [-3., -2., -1.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.ihfft.html.md

# dask.array.fft.ihfft

### dask.array.fft.ihfft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.ihfft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.ihfft docstring follows below:

Compute the inverse FFT of a signal that has Hermitian symmetry.

* **Parameters:**
  **a**
  : Input array.

  **n**
  : Length of the inverse FFT, the number of points along
    transformation axis in the input to use.  If n is smaller than
    the length of the input, the input is cropped.  If it is larger,
    the input is padded with zeros. If n is not given, the length of
    the input along the axis specified by axis is used.

  **axis**
  : Axis over which to compute the inverse FFT. If not given, the last
    axis is used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
    The length of the transformed axis is `n//2 + 1`.

#### SEE ALSO
[`hfft`](dask.array.fft.hfft.md#dask.array.fft.hfft), [`irfft`](dask.array.fft.irfft.md#dask.array.fft.irfft)

### Notes

hfft/ihfft are a pair analogous to rfft/irfft, but for the
opposite case: here the signal has Hermitian symmetry in the time
domain and is real in the frequency domain. So here it’s hfft for
which you must supply the length of the result if it is to be odd:

* even: `ihfft(hfft(a, 2*len(a) - 2)) == a`, within roundoff error,
* odd: `ihfft(hfft(a, 2*len(a) - 1)) == a`, within roundoff error.

### Examples

```pycon
>>> import numpy as np
>>> spectrum = np.array([ 15, -4, 0, -1, 0, -4])
>>> np.fft.ifft(spectrum)
array([1.+0.j,  2.+0.j,  3.+0.j,  4.+0.j,  3.+0.j,  2.+0.j]) # may vary
>>> np.fft.ihfft(spectrum)
array([ 1.-0.j,  2.-0.j,  3.-0.j,  4.-0.j]) # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.irfft.html.md

# dask.array.fft.irfft

### dask.array.fft.irfft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.irfft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.irfft docstring follows below:

Computes the inverse of rfft.

This function computes the inverse of the one-dimensional *n*-point
discrete Fourier Transform of real input computed by rfft.
In other words, `irfft(rfft(a), len(a)) == a` to within numerical
accuracy. (See Notes below for why `len(a)` is necessary here.)

The input is expected to be in the form returned by rfft, i.e. the
real zero-frequency term followed by the complex positive frequency terms
in order of increasing frequency.  Since the discrete Fourier Transform of
real input is Hermitian-symmetric, the negative frequency terms are taken
to be the complex conjugates of the corresponding positive frequency terms.

* **Parameters:**
  **a**
  : The input array.

  **n**
  : Length of the transformed axis of the output.
    For n output points, `n//2+1` input points are necessary.  If the
    input is longer than this, it is cropped.  If it is shorter than this,
    it is padded with zeros.  If n is not given, it is taken to be
    `2*(m-1)` where `m` is the length of the input along the axis
    specified by axis.

  **axis**
  : Axis over which to compute the inverse FFT. If not given, the last
    axis is used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
    The length of the transformed axis is n, or, if n is not given,
    `2*(m-1)` where `m` is the length of the transformed axis of the
    input. To get an odd number of output points, n must be specified.
* **Raises:**
  IndexError
  : If axis is not a valid axis of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: For definition of the DFT and conventions used.

[`rfft`](dask.array.fft.rfft.md#dask.array.fft.rfft)
: The one-dimensional FFT of real input, of which irfft is inverse.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT.

[`irfft2`](dask.array.fft.irfft2.md#dask.array.fft.irfft2)
: The inverse of the two-dimensional FFT of real input.

[`irfftn`](dask.array.fft.irfftn.md#dask.array.fft.irfftn)
: The inverse of the *n*-dimensional FFT of real input.

### Notes

Returns the real valued n-point inverse discrete Fourier transform
of a, where a contains the non-negative frequency terms of a
Hermitian-symmetric sequence. n is the length of the result, not the
input.

If you specify an n such that a must be zero-padded or truncated, the
extra/removed values will be added/removed at high frequencies. One can
thus resample a series to m points via Fourier interpolation by:
`a_resamp = irfft(rfft(a), m)`.

The correct interpretation of the hermitian input depends on the length of
the original data, as given by n. This is because each input shape could
correspond to either an odd or even length signal. By default, irfft
assumes an even output length which puts the last entry at the Nyquist
frequency; aliasing with its symmetric counterpart. By Hermitian symmetry,
the value is thus treated as purely real. To avoid losing information, the
correct length of the real input **must** be given.

### Examples

```pycon
>>> import numpy as np
>>> np.fft.ifft([1, -1j, -1, 1j])
array([0.+0.j,  1.+0.j,  0.+0.j,  0.+0.j]) # may vary
>>> np.fft.irfft([1, -1j, -1])
array([0.,  1.,  0.,  0.])
```

Notice how the last term in the input to the ordinary ifft is the
complex conjugate of the second term, and the output has zero imaginary
part everywhere.  When calling irfft, the negative frequencies are not
specified, and the output array is purely real.

<!-- !! processed by numpydoc !! -->


# dask.array.fft.irfft2.html.md

# dask.array.fft.irfft2

### dask.array.fft.irfft2(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.irfft2

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.irfft2 docstring follows below:

Computes the inverse of rfft2.

* **Parameters:**
  **a**
  : The input array

  **s**
  : Shape of the real output to the inverse FFT.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : The axes over which to compute the inverse fft.
    Default: `(-2, -1)`, the last two axes.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must not be `None`.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for the last transformation.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The result of the inverse real 2-D FFT.

#### SEE ALSO
[`rfft2`](dask.array.fft.rfft2.md#dask.array.fft.rfft2)
: The forward two-dimensional FFT of real input, of which irfft2 is the inverse.

[`rfft`](dask.array.fft.rfft.md#dask.array.fft.rfft)
: The one-dimensional FFT for real input.

[`irfft`](dask.array.fft.irfft.md#dask.array.fft.irfft)
: The inverse of the one-dimensional FFT of real input.

[`irfftn`](dask.array.fft.irfftn.md#dask.array.fft.irfftn)
: Compute the inverse of the N-dimensional FFT of real input.

### Notes

This is really irfftn with different defaults.
For more details see irfftn.

### Examples

```pycon
>>> import numpy as np
>>> a = np.mgrid[:5, :5][0]
>>> A = np.fft.rfft2(a)
>>> np.fft.irfft2(A, s=a.shape)
array([[0., 0., 0., 0., 0.],
       [1., 1., 1., 1., 1.],
       [2., 2., 2., 2., 2.],
       [3., 3., 3., 3., 3.],
       [4., 4., 4., 4., 4.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.irfftn.html.md

# dask.array.fft.irfftn

### dask.array.fft.irfftn(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.irfftn

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.irfftn docstring follows below:

Computes the inverse of rfftn.

This function computes the inverse of the N-dimensional discrete
Fourier Transform for real input over any number of axes in an
M-dimensional array by means of the Fast Fourier Transform (FFT).  In
other words, `irfftn(rfftn(a), a.shape) == a` to within numerical
accuracy. (The `a.shape` is necessary like `len(a)` is for irfft,
and for the same reason.)

The input should be ordered in the same way as is returned by rfftn,
i.e. as for irfft for the final transformation axis, and as for ifftn
along all the other axes.

* **Parameters:**
  **a**
  : Input array.

  **s**
  : Shape (length of each transformed axis) of the output
    (`s[0]` refers to axis 0, `s[1]` to axis 1, etc.). s is also the
    number of input points used along this axis, except for the last axis,
    where `s[-1]//2+1` points of the input are used.
    Along any axis, if the shape indicated by s is smaller than that of
    the input, the input is cropped.  If it is larger, the input is padded
    with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes
    specified by axes is used. Except for the last axis which is taken to
    be `2*(m-1)` where `m` is the length of the input along that axis.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the inverse FFT. If not given, the last
    len(s) axes are used, or all axes if s is also not specified.
    Repeated indices in axes means that the inverse transform over that
    axis is performed multiple times.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must be explicitly specified too.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for the last transformation.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or by a combination of s or a,
    as explained in the parameters section above.
    The length of each transformed axis is as given by the corresponding
    element of s, or the length of the input in every axis except for the
    last one if s is not given.  In the final transformed axis the length
    of the output when s is not given is `2*(m-1)` where `m` is the
    length of the final transformed axis of the input.  To get an odd
    number of output points in the final axis, s must be specified.
* **Raises:**
  ValueError
  : If s and axes have different length.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`rfftn`](dask.array.fft.rfftn.md#dask.array.fft.rfftn)
: The forward n-dimensional FFT of real input, of which ifftn is the inverse.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT, with definitions and conventions used.

[`irfft`](dask.array.fft.irfft.md#dask.array.fft.irfft)
: The inverse of the one-dimensional FFT of real input.

[`irfft2`](dask.array.fft.irfft2.md#dask.array.fft.irfft2)
: The inverse of the two-dimensional FFT of real input.

### Notes

See fft for definitions and conventions used.

See rfft for definitions and conventions used for real input.

The correct interpretation of the hermitian input depends on the shape of
the original data, as given by s. This is because each input shape could
correspond to either an odd or even length signal. By default, irfftn
assumes an even output length which puts the last entry at the Nyquist
frequency; aliasing with its symmetric counterpart. When performing the
final complex to real transform, the last value is thus treated as purely
real. To avoid losing information, the correct shape of the real input
**must** be given.

### Examples

```pycon
>>> import numpy as np
>>> a = np.zeros((3, 2, 2))
>>> a[0, 0, 0] = 3 * 2 * 2
>>> np.fft.irfftn(a)
array([[[1.,  1.],
        [1.,  1.]],
       [[1.,  1.],
        [1.,  1.]],
       [[1.,  1.],
        [1.,  1.]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.rfft.html.md

# dask.array.fft.rfft

### dask.array.fft.rfft(a, n=None, axis=None, norm=None)

> Wrapping of numpy.fft.rfft

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.rfft docstring follows below:

Compute the one-dimensional discrete Fourier Transform for real input.

This function computes the one-dimensional *n*-point discrete Fourier
Transform (DFT) of a real-valued array by means of an efficient algorithm
called the Fast Fourier Transform (FFT).

* **Parameters:**
  **a**
  : Input array

  **n**
  : Number of points along transformation axis in the input to use.
    If n is smaller than the length of the input, the input is cropped.
    If it is larger, the input is padded with zeros. If n is not given,
    the length of the input along the axis specified by axis is used.

  **axis**
  : Axis over which to compute the FFT. If not given, the last axis is
    used.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axis
    indicated by axis, or the last one if axis is not specified.
    If n is even, the length of the transformed axis is `(n/2)+1`.
    If n is odd, the length is `(n+1)/2`.
* **Raises:**
  IndexError
  : If axis is not a valid axis of a.

#### SEE ALSO
[`numpy.fft`](https://numpy.org/doc/stable/reference/routines.fft.html#module-numpy.fft)
: For definition of the DFT and conventions used.

[`irfft`](dask.array.fft.irfft.md#dask.array.fft.irfft)
: The inverse of rfft.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT of general (complex) input.

[`fftn`](dask.array.fft.fftn.md#dask.array.fft.fftn)
: The *n*-dimensional FFT.

[`rfftn`](dask.array.fft.rfftn.md#dask.array.fft.rfftn)
: The *n*-dimensional FFT of real input.

### Notes

When the DFT is computed for purely real input, the output is
Hermitian-symmetric, i.e. the negative frequency terms are just the complex
conjugates of the corresponding positive-frequency terms, and the
negative-frequency terms are therefore redundant.  This function does not
compute the negative frequency terms, and the length of the transformed
axis of the output is therefore `n//2 + 1`.

When `A = rfft(a)` and fs is the sampling frequency, `A[0]` contains
the zero-frequency term 0\*fs, which is real due to Hermitian symmetry.

If n is even, `A[-1]` contains the term representing both positive
and negative Nyquist frequency (+fs/2 and -fs/2), and must also be purely
real. If n is odd, there is no term at fs/2; `A[-1]` contains
the largest positive frequency (fs/2\*(n-1)/n), and is complex in the
general case.

If the input a contains an imaginary part, it is silently discarded.

### Examples

```pycon
>>> import numpy as np
>>> np.fft.fft([0, 1, 0, 0])
array([ 1.+0.j,  0.-1.j, -1.+0.j,  0.+1.j]) # may vary
>>> np.fft.rfft([0, 1, 0, 0])
array([ 1.+0.j,  0.-1.j, -1.+0.j]) # may vary
```

Notice how the final element of the fft output is the complex conjugate
of the second element, for real input. For rfft, this symmetry is
exploited to compute only the non-negative frequency terms.

<!-- !! processed by numpydoc !! -->


# dask.array.fft.rfft2.html.md

# dask.array.fft.rfft2

### dask.array.fft.rfft2(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.rfft2

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.rfft2 docstring follows below:

Compute the 2-dimensional FFT of a real array.

* **Parameters:**
  **a**
  : Input array, taken to be real.

  **s**
  : Shape of the FFT.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the FFT. Default: `(-2, -1)`.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must not be `None`.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for the last inverse transform.
    incompatible with passing in all but the trivial `s`).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The result of the real 2-D FFT.

#### SEE ALSO
[`rfftn`](dask.array.fft.rfftn.md#dask.array.fft.rfftn)
: Compute the N-dimensional discrete Fourier Transform for real input.

### Notes

This is really just rfftn with different default behavior.
For more details see rfftn.

### Examples

```pycon
>>> import numpy as np
>>> a = np.mgrid[:5, :5][0]
>>> np.fft.rfft2(a)
array([[ 50.  +0.j        ,   0.  +0.j        ,   0.  +0.j        ],
       [-12.5+17.20477401j,   0.  +0.j        ,   0.  +0.j        ],
       [-12.5 +4.0614962j ,   0.  +0.j        ,   0.  +0.j        ],
       [-12.5 -4.0614962j ,   0.  +0.j        ,   0.  +0.j        ],
       [-12.5-17.20477401j,   0.  +0.j        ,   0.  +0.j        ]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.rfftfreq.html.md

# dask.array.fft.rfftfreq

### dask.array.fft.rfftfreq(n, d=1.0, chunks='auto')

Return the Discrete Fourier Transform sample frequencies
(for usage with rfft, irfft).

This docstring was copied from numpy.fft.rfftfreq.

Some inconsistencies with the Dask version may exist.

The returned float array f contains the frequency bin centers in cycles
per unit of the sample spacing (with zero at the start).  For instance, if
the sample spacing is in seconds, then the frequency unit is cycles/second.

Given a window length n and a sample spacing d:

```default
f = [0, 1, ...,     n/2-1,     n/2] / (d*n)   if n is even
f = [0, 1, ..., (n-1)/2-1, (n-1)/2] / (d*n)   if n is odd
```

Unlike fftfreq (but like scipy.fftpack.rfftfreq)
the Nyquist frequency component is considered to be positive.

* **Parameters:**
  **n**
  : Window length.

  **d**
  : Sample spacing (inverse of the sampling rate). Defaults to 1.

  **device**
  : The device on which to place the created array. Default: `None`.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **f**
  : Array of length `n//2 + 1` containing the sample frequencies.

### Examples

```pycon
>>> import numpy as np
>>> signal = np.array([-2, 8, 6, 4, 1, 0, 3, 5, -3, 4], dtype=np.float64)
>>> fourier = np.fft.rfft(signal)
>>> n = signal.size
>>> sample_rate = 100
>>> freq = np.fft.fftfreq(n, d=1./sample_rate)
>>> freq
array([  0.,  10.,  20., ..., -30., -20., -10.])
>>> freq = np.fft.rfftfreq(n, d=1./sample_rate)
>>> freq
array([  0.,  10.,  20.,  30.,  40.,  50.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fft.rfftn.html.md

# dask.array.fft.rfftn

### dask.array.fft.rfftn(a, s=None, axes=None, norm=None)

> Wrapping of numpy.fft.rfftn

> The axis along which the FFT is applied must have only one chunk. To change
> the array’s chunking use dask.Array.rechunk.

> The numpy.fft.rfftn docstring follows below:

Compute the N-dimensional discrete Fourier Transform for real input.

This function computes the N-dimensional discrete Fourier Transform over
any number of axes in an M-dimensional real array by means of the Fast
Fourier Transform (FFT).  By default, all axes are transformed, with the
real transform performed over the last axis, while the remaining
transforms are complex.

* **Parameters:**
  **a**
  : Input array, taken to be real.

  **s**
  : Shape (length along each transformed axis) to use from the input.
    (`s[0]` refers to axis 0, `s[1]` to axis 1, etc.).
    The final element of s corresponds to n for `rfft(x, n)`, while
    for the remaining axes, it corresponds to n for `fft(x, n)`.
    Along any axis, if the given shape is smaller than that of the input,
    the input is cropped. If it is larger, the input is padded with zeros.
    <br/>
    #### Versionchanged
    Changed in version 2.0: If it is `-1`, the whole input is used (no padding/trimming).
    <br/>
    If s is not given, the shape of the input along the axes specified
    by axes is used.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is not `None`, axes must not be `None` either.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: s must contain only `int` s, not `None` values. `None`
    values currently mean that the default value for `n` is used
    in the corresponding 1-D transform, but this behaviour is
    deprecated.

  **axes**
  : Axes over which to compute the FFT.  If not given, the last `len(s)`
    axes are used, or all axes if s is also not specified.
    <br/>
    #### Deprecated
    Deprecated since version 2.0: If s is specified, the corresponding axes to be transformed
    must be explicitly specified too.

  **norm**
  : Normalization mode (see numpy.fft). Default is “backward”.
    Indicates which direction of the forward/backward pair of transforms
    is scaled and with what normalization factor.
    <br/>
    #### Versionadded
    Added in version 1.20.0: The “backward”, “forward” values were added.

  **out**
  : If provided, the result will be placed in this array. It should be
    of the appropriate shape and dtype for all axes (and hence is
    incompatible with passing in all but the trivial `s`).
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : The truncated or zero-padded input, transformed along the axes
    indicated by axes, or by a combination of s and a,
    as explained in the parameters section above.
    The length of the last axis transformed will be `s[-1]//2+1`,
    while the remaining transformed axes will have lengths according to
    s, or unchanged from the input.
* **Raises:**
  ValueError
  : If s and axes have different length.

  IndexError
  : If an element of axes is larger than than the number of axes of a.

#### SEE ALSO
[`irfftn`](dask.array.fft.irfftn.md#dask.array.fft.irfftn)
: The inverse of rfftn, i.e. the inverse of the n-dimensional FFT of real input.

[`fft`](dask.array.fft.fft.md#dask.array.fft.fft)
: The one-dimensional FFT, with definitions and conventions used.

[`rfft`](dask.array.fft.rfft.md#dask.array.fft.rfft)
: The one-dimensional FFT of real input.

[`fftn`](dask.array.fft.fftn.md#dask.array.fft.fftn)
: The n-dimensional FFT.

[`rfft2`](dask.array.fft.rfft2.md#dask.array.fft.rfft2)
: The two-dimensional FFT of real input.

### Notes

The transform for real input is performed over the last transformation
axis, as by rfft, then the transform over the remaining axes is
performed as by fftn.  The order of the output is as for rfft for the
final transformation axis, and as for fftn for the remaining
transformation axes.

See fft for details, definitions and conventions used.

### Examples

```pycon
>>> import numpy as np
>>> a = np.ones((2, 2, 2))
>>> np.fft.rfftn(a)
array([[[8.+0.j,  0.+0.j], # may vary
        [0.+0.j,  0.+0.j]],
       [[0.+0.j,  0.+0.j],
        [0.+0.j,  0.+0.j]]])
```

```pycon
>>> np.fft.rfftn(a, axes=(2, 0))
array([[[4.+0.j,  0.+0.j], # may vary
        [4.+0.j,  0.+0.j]],
       [[0.+0.j,  0.+0.j],
        [0.+0.j,  0.+0.j]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fix.html.md

# dask.array.fix

### dask.array.fix(\*args, \*\*kwargs)

Round to nearest integer towards zero.

This docstring was copied from numpy.fix.

Some inconsistencies with the Dask version may exist.

#### Deprecated
Deprecated since version 2.5: numpy.fix is deprecated. Use numpy.trunc instead,
which is faster and follows the Array API standard.

Round an array of floats element-wise to nearest integer towards zero.
The rounded values have the same data-type as the input.

* **Parameters:**
  **x**
  : An array to be rounded

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the input broadcasts to. If not provided or None, a
    freshly-allocated array is returned.
* **Returns:**
  **out**
  : An array with the same dimensions and data-type as the input.
    If second argument is not supplied then a new array is returned
    with the rounded values.
    <br/>
    If a second argument is supplied the result is stored there.
    The return value `out` is then a reference to that array.

#### SEE ALSO
[`rint`](dask.array.rint.md#dask.array.rint), [`trunc`](dask.array.trunc.md#dask.array.trunc), [`floor`](dask.array.floor.md#dask.array.floor), [`ceil`](dask.array.ceil.md#dask.array.ceil)

[`around`](dask.array.around.md#dask.array.around)
: Round to given number of decimals

### Examples

```pycon
>>> import numpy as np
>>> np.fix(3.14)
3.0
>>> np.fix(3)
3
>>> np.fix([2.1, 2.9, -2.1, -2.9])
array([ 2.,  2., -2., -2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.flatnonzero.html.md

# dask.array.flatnonzero

### dask.array.flatnonzero(a)

Return indices that are non-zero in the flattened version of a.

This docstring was copied from numpy.flatnonzero.

Some inconsistencies with the Dask version may exist.

This is equivalent to `np.nonzero(np.ravel(a))[0]`.

* **Parameters:**
  **a**
  : Input data.
* **Returns:**
  **res**
  : Output array, containing the indices of the elements of `a.ravel()`
    that are non-zero.

#### SEE ALSO
[`nonzero`](dask.array.nonzero.md#dask.array.nonzero)
: Return the indices of the non-zero elements of the input array.

[`ravel`](dask.array.ravel.md#dask.array.ravel)
: Return a 1-D array containing the elements of the input array.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(-2, 3)
>>> x
array([-2, -1,  0,  1,  2])
>>> np.flatnonzero(x)
array([0, 1, 3, 4])
```

Use the indices of the non-zero elements as an index array to extract
these elements:

```pycon
>>> x.ravel()[np.flatnonzero(x)]
array([-2, -1,  1,  2])
```

<!-- !! processed by numpydoc !! -->


# dask.array.flip.html.md

# dask.array.flip

### dask.array.flip(m, axis=None)

Reverse element order along axis.

* **Parameters:**
  **m**
  : Input array.

  **axis**
  : Axis or axes to reverse element order of. None will reverse all axes.
* **Returns:**
  dask.array.Array
  : The flipped array.

<!-- !! processed by numpydoc !! -->


# dask.array.fliplr.html.md

# dask.array.fliplr

### dask.array.fliplr(m)

Reverse the order of elements along axis 1 (left/right).

This docstring was copied from numpy.fliplr.

Some inconsistencies with the Dask version may exist.

For a 2-D array, this flips the entries in each row in the left/right
direction. Columns are preserved, but appear in a different order than
before.

* **Parameters:**
  **m**
  : Input array, must be at least 2-D.
* **Returns:**
  **f**
  : A view of m with the columns reversed.  Since a view
    is returned, this operation is $\mathcal O(1)$.

#### SEE ALSO
[`flipud`](dask.array.flipud.md#dask.array.flipud)
: Flip array in the up/down direction.

[`flip`](dask.array.flip.md#dask.array.flip)
: Flip array in one or more dimensions.

[`rot90`](dask.array.rot90.md#dask.array.rot90)
: Rotate array counterclockwise.

### Notes

Equivalent to `m[:,::-1]` or `np.flip(m, axis=1)`.
Requires the array to be at least 2-D.

### Examples

```pycon
>>> import numpy as np
>>> A = np.diag([1.,2.,3.])
>>> A
array([[1.,  0.,  0.],
       [0.,  2.,  0.],
       [0.,  0.,  3.]])
>>> np.fliplr(A)
array([[0.,  0.,  1.],
       [0.,  2.,  0.],
       [3.,  0.,  0.]])
```

```pycon
>>> rng = np.random.default_rng()
>>> A = rng.normal(size=(2,3,5))
>>> np.all(np.fliplr(A) == A[:,::-1,...])
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.flipud.html.md

# dask.array.flipud

### dask.array.flipud(m)

Reverse the order of elements along axis 0 (up/down).

This docstring was copied from numpy.flipud.

Some inconsistencies with the Dask version may exist.

For a 2-D array, this flips the entries in each column in the up/down
direction. Rows are preserved, but appear in a different order than before.

* **Parameters:**
  **m**
  : Input array.
* **Returns:**
  **out**
  : A view of m with the rows reversed.  Since a view is
    returned, this operation is $\mathcal O(1)$.

#### SEE ALSO
[`fliplr`](dask.array.fliplr.md#dask.array.fliplr)
: Flip array in the left/right direction.

[`flip`](dask.array.flip.md#dask.array.flip)
: Flip array in one or more dimensions.

[`rot90`](dask.array.rot90.md#dask.array.rot90)
: Rotate array counterclockwise.

### Notes

Equivalent to `m[::-1, ...]` or `np.flip(m, axis=0)`.
Requires the array to be at least 1-D.

### Examples

```pycon
>>> import numpy as np
>>> A = np.diag([1.0, 2, 3])
>>> A
array([[1.,  0.,  0.],
       [0.,  2.,  0.],
       [0.,  0.,  3.]])
>>> np.flipud(A)
array([[0.,  0.,  3.],
       [0.,  2.,  0.],
       [1.,  0.,  0.]])
```

```pycon
>>> rng = np.random.default_rng()
>>> A = rng.normal(size=(2,3,5))
>>> np.all(np.flipud(A) == A[::-1,...])
True
```

```pycon
>>> np.flipud([1,2])
array([2, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.float_power.html.md

# dask.array.float_power

### dask.array.float_power(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'float_power'>*

This docstring was copied from numpy.float_power.

Some inconsistencies with the Dask version may exist.

First array elements raised to powers from second array, element-wise.

Raise each base in x1 to the positionally-corresponding power in x2.
x1 and x2 must be broadcastable to the same shape. This differs from
the power function in that integers, float16, and float32  are promoted to
floats with a minimum precision of float64 so that the result is always
inexact.  The intent is that the function will return a usable result for
negative powers and seldom overflow for positive powers.

Negative values raised to a non-integral value will return `nan`.
To get complex results, cast the input to complex, or specify the
`dtype` to be `complex` (see the example below).

* **Parameters:**
  **x1**
  : The bases.

  **x2**
  : The exponents.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The bases in x1 raised to the exponents in x2.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`power`](dask.array.power.md#dask.array.power)
: power function that preserves type

### Examples

```pycon
>>> import numpy as np
```

Cube each element in a list.

```pycon
>>> x1 = range(6)
>>> x1
[0, 1, 2, 3, 4, 5]
>>> np.float_power(x1, 3)
array([   0.,    1.,    8.,   27.,   64.,  125.])
```

Raise the bases to different exponents.

```pycon
>>> x2 = [1.0, 2.0, 3.0, 3.0, 2.0, 1.0]
>>> np.float_power(x1, x2)
array([  0.,   1.,   8.,  27.,  16.,   5.])
```

The effect of broadcasting.

```pycon
>>> x2 = np.array([[1, 2, 3, 3, 2, 1], [1, 2, 3, 3, 2, 1]])
>>> x2
array([[1, 2, 3, 3, 2, 1],
       [1, 2, 3, 3, 2, 1]])
>>> np.float_power(x1, x2)
array([[  0.,   1.,   8.,  27.,  16.,   5.],
       [  0.,   1.,   8.,  27.,  16.,   5.]])
```

Negative values raised to a non-integral value will result in `nan`
(and a warning will be generated).

```pycon
>>> x3 = np.array([-1, -4])
>>> with np.errstate(invalid='ignore'):
...     p = np.float_power(x3, 1.5)
...
>>> p
array([nan, nan])
```

To get complex results, give the argument `dtype=np.complex128`.

```pycon
>>> np.float_power(x3, 1.5, dtype=np.complex128)
array([-1.83697020e-16-1.j, -1.46957616e-15-8.j])
```

<!-- !! processed by numpydoc !! -->


# dask.array.floor.html.md

# dask.array.floor

### dask.array.floor(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'floor'>*

This docstring was copied from numpy.floor.

Some inconsistencies with the Dask version may exist.

Return the floor of the input, element-wise.

The floor of the scalar x is the largest integer i, such that
i <= x.  It is often denoted as $\lfloor x \rfloor$.

* **Parameters:**
  **x**
  : Input data.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The floor of each element in x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`ceil`](dask.array.ceil.md#dask.array.ceil), [`trunc`](dask.array.trunc.md#dask.array.trunc), [`rint`](dask.array.rint.md#dask.array.rint), [`fix`](dask.array.fix.md#dask.array.fix)

### Notes

Some spreadsheet programs calculate the “floor-towards-zero”, where
`floor(-2.5) == -2`.  NumPy instead uses the definition of
floor where floor(-2.5) == -3. The “floor-towards-zero”
function is called `fix` in NumPy.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([-1.7, -1.5, -0.2, 0.2, 1.5, 1.7, 2.0])
>>> np.floor(a)
array([-2., -2., -1.,  0.,  1.,  1.,  2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.floor_divide.html.md

# dask.array.floor_divide

### dask.array.floor_divide(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'floor_divide'>*

This docstring was copied from numpy.floor_divide.

Some inconsistencies with the Dask version may exist.

Return the largest integer smaller or equal to the division of the inputs.
It is equivalent to the Python `//` operator and pairs with the
Python `%` (remainder), function so that `a = a % b + b * (a // b)`
up to roundoff.

* **Parameters:**
  **x1**
  : Numerator.

  **x2**
  : Denominator.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : y = floor(x1/x2)
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`remainder`](dask.array.remainder.md#dask.array.remainder)
: Remainder complementary to floor_divide.

[`divmod`](dask.array.divmod.md#dask.array.divmod)
: Simultaneous floor division and remainder.

[`divide`](dask.array.divide.md#dask.array.divide)
: Standard division.

[`floor`](dask.array.floor.md#dask.array.floor)
: Round a number to the nearest integer toward minus infinity.

[`ceil`](dask.array.ceil.md#dask.array.ceil)
: Round a number to the nearest integer toward infinity.

### Examples

```pycon
>>> import numpy as np
>>> np.floor_divide(7,3)
2
>>> np.floor_divide([1., 2., 3., 4.], 2.5)
array([ 0.,  0.,  1.,  1.])
```

The `//` operator can be used as a shorthand for `np.floor_divide`
on ndarrays.

```pycon
>>> x1 = np.array([1., 2., 3., 4.])
>>> x1 // 2.5
array([0., 0., 1., 1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fmax.html.md

# dask.array.fmax

### dask.array.fmax(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'fmax'>*

This docstring was copied from numpy.fmax.

Some inconsistencies with the Dask version may exist.

Element-wise maximum of array elements.

Compare two arrays and return a new array containing the element-wise
maxima. If one of the elements being compared is a NaN, then the
non-nan element is returned. If both elements are NaNs then the first
is returned.  The latter distinction is important for complex NaNs,
which are defined as at least one of the real or imaginary parts being
a NaN. The net effect is that NaNs are ignored when possible.

* **Parameters:**
  **x1, x2**
  : The arrays holding the elements to be compared.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The maximum of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`fmin`](dask.array.fmin.md#dask.array.fmin)
: Element-wise minimum of two arrays, ignores NaNs.

[`maximum`](dask.array.maximum.md#dask.array.maximum)
: Element-wise maximum of two arrays, propagates NaNs.

`amax`
: The maximum value of an array along a given axis, propagates NaNs.

[`nanmax`](dask.array.nanmax.md#dask.array.nanmax)
: The maximum value of an array along a given axis, ignores NaNs.

[`minimum`](dask.array.minimum.md#dask.array.minimum), `amin`, [`nanmin`](dask.array.nanmin.md#dask.array.nanmin)

### Notes

The fmax is equivalent to `np.where(x1 >= x2, x1, x2)` when neither
x1 nor x2 are NaNs, but it is faster and does proper broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.fmax([2, 3, 4], [1, 5, 2])
array([ 2,  5,  4])
```

```pycon
>>> np.fmax(np.eye(2), [0.5, 2])
array([[ 1. ,  2. ],
       [ 0.5,  2. ]])
```

```pycon
>>> np.fmax([np.nan, 0, np.nan],[0, np.nan, np.nan])
array([ 0.,  0., nan])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fmin.html.md

# dask.array.fmin

### dask.array.fmin(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'fmin'>*

This docstring was copied from numpy.fmin.

Some inconsistencies with the Dask version may exist.

Element-wise minimum of array elements.

Compare two arrays and return a new array containing the element-wise
minima. If one of the elements being compared is a NaN, then the
non-nan element is returned. If both elements are NaNs then the first
is returned.  The latter distinction is important for complex NaNs,
which are defined as at least one of the real or imaginary parts being
a NaN. The net effect is that NaNs are ignored when possible.

* **Parameters:**
  **x1, x2**
  : The arrays holding the elements to be compared.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The minimum of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`fmax`](dask.array.fmax.md#dask.array.fmax)
: Element-wise maximum of two arrays, ignores NaNs.

[`minimum`](dask.array.minimum.md#dask.array.minimum)
: Element-wise minimum of two arrays, propagates NaNs.

`amin`
: The minimum value of an array along a given axis, propagates NaNs.

[`nanmin`](dask.array.nanmin.md#dask.array.nanmin)
: The minimum value of an array along a given axis, ignores NaNs.

[`maximum`](dask.array.maximum.md#dask.array.maximum), `amax`, [`nanmax`](dask.array.nanmax.md#dask.array.nanmax)

### Notes

The fmin is equivalent to `np.where(x1 <= x2, x1, x2)` when neither
x1 nor x2 are NaNs, but it is faster and does proper broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.fmin([2, 3, 4], [1, 5, 2])
array([1, 3, 2])
```

```pycon
>>> np.fmin(np.eye(2), [0.5, 2])
array([[ 0.5,  0. ],
       [ 0. ,  1. ]])
```

```pycon
>>> np.fmin([np.nan, 0, np.nan],[0, np.nan, np.nan])
array([ 0.,  0., nan])
```

<!-- !! processed by numpydoc !! -->


# dask.array.fmod.html.md

# dask.array.fmod

### dask.array.fmod(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'fmod'>*

This docstring was copied from numpy.fmod.

Some inconsistencies with the Dask version may exist.

Returns the element-wise remainder of division.

This is the NumPy implementation of the C library function fmod, the
remainder has the same sign as the dividend x1. It is equivalent to
the Matlab(TM) `rem` function and should not be confused with the
Python modulus operator `x1 % x2`.

* **Parameters:**
  **x1**
  : Dividend.

  **x2**
  : Divisor.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The remainder of the division of x1 by x2.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`remainder`](dask.array.remainder.md#dask.array.remainder)
: Equivalent to the Python `%` operator.

[`divide`](dask.array.divide.md#dask.array.divide)

### Notes

The result of the modulo operation for negative dividend and divisors
is bound by conventions. For fmod, the sign of result is the sign of
the dividend, while for remainder the sign of the result is the sign
of the divisor. The fmod function is equivalent to the Matlab(TM)
`rem` function.

### Examples

```pycon
>>> import numpy as np
>>> np.fmod([-3, -2, -1, 1, 2, 3], 2)
array([-1,  0, -1,  1,  0,  1])
>>> np.remainder([-3, -2, -1, 1, 2, 3], 2)
array([1, 0, 1, 1, 0, 1])
```

```pycon
>>> np.fmod([5, 3], [2, 2.])
array([ 1.,  1.])
>>> a = np.arange(-3, 3).reshape(3, 2)
>>> a
array([[-3, -2],
       [-1,  0],
       [ 1,  2]])
>>> np.fmod(a, [2,2])
array([[-1,  0],
       [-1,  0],
       [ 1,  0]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.frexp.html.md

# dask.array.frexp

### dask.array.frexp(x, /, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True)

This docstring was copied from numpy.frexp.

Some inconsistencies with the Dask version may exist.

Decompose the elements of x into mantissa and twos exponent.

Returns (mantissa, exponent), where `x = mantissa * 2**exponent`.
The mantissa lies in the open interval(-1, 1), while the twos
exponent is a signed integer.

* **Parameters:**
  **x**
  : Array of numbers to be decomposed.

  **out1**
  : Output array for the mantissa. Must have the same shape as x.

  **out2**
  : Output array for the exponent. Must have the same shape as x.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **mantissa**
  : Floating values between -1 and 1.
    This is a scalar if x is a scalar.

  **exponent**
  : Integer exponents of 2.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`ldexp`](dask.array.ldexp.md#dask.array.ldexp)
: Compute `y = x1 * 2**x2`, the inverse of frexp.

### Notes

Complex dtypes are not supported, they will raise a TypeError.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(9)
>>> y1, y2 = np.frexp(x)
>>> y1
array([ 0.   ,  0.5  ,  0.5  ,  0.75 ,  0.5  ,  0.625,  0.75 ,  0.875,
        0.5  ])
>>> y2
array([0, 1, 2, 2, 3, 3, 3, 3, 4], dtype=int32)
>>> y1 * 2**y2
array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.from_array.html.md

# dask.array.from_array

### dask.array.from_array(x, chunks='auto', name=None, lock=False, asarray=None, fancy=True, getitem=None, meta=None, inline_array=False)

Create dask array from something that looks like an array.

Input must have a `.shape`, `.ndim`, `.dtype` and support numpy-style slicing.

* **Parameters:**
  **x**

  **chunks**
  : How to chunk the array. Must be one of the following forms:
    - A blocksize like 1000.
    - A blockshape like (1000, 1000).
    - Explicit sizes of all blocks along all dimensions like
      ((1000, 1000, 500), (400, 400)).
    - A size in bytes, like “100 MiB” which will choose a uniform
      block-like shape
    - The word “auto” which acts like the above, but uses a configuration
      value `array.chunk-size` for the chunk size
    <br/>
    -1 or None as a blocksize indicate the size of the corresponding
    dimension.

  **name**
  : The key name to use for the array. Defaults to a hash of `x`.
    <br/>
    Hashing is useful if the same value of `x` is used to create multiple
    arrays, as Dask can then recognise that they’re the same and
    avoid duplicate computations. However, it can also be slow, and if the
    array is not contiguous it is copied for hashing. If the array uses
    stride tricks (such as [`numpy.broadcast_to()`](https://numpy.org/doc/stable/reference/generated/numpy.broadcast_to.html#numpy.broadcast_to) or
    [`skimage.util.view_as_windows()`](https://scikit-image.org/docs/dev/api/skimage.util.html#skimage.util.view_as_windows)) to have a larger logical
    than physical size, this copy can cause excessive memory usage.
    <br/>
    If you don’t need the deduplication provided by hashing, use
    `name=False` to generate a random name instead of hashing, which
    avoids the pitfalls described above. Using `name=True` is
    equivalent to the default.
    <br/>
    By default, hashing uses python’s standard sha1. This behaviour can be
    changed by installing cityhash, xxhash or murmurhash. If installed,
    a large-factor speedup can be obtained in the tokenisation step.
    <br/>
    #### NOTE
    Because this `name` is used as the key in task graphs, you should
    ensure that it uniquely identifies the data contained within. If
    you’d like to provide a descriptive name that is still unique, combine
    the descriptive name with `dask.base.tokenize()` of the
    `array_like`. See [Task Graphs](../graphs.md#graphs) for more.

  **lock**
  : If `x` doesn’t support concurrent reads then provide a lock here, or
    pass in True to have dask.array create one for you.

  **asarray**
  : If True then call np.asarray on chunks to convert them to numpy arrays.
    If False then chunks are passed through unchanged.
    If None (default) then we use True if the `__array_function__` method
    is undefined.
    <br/>
    #### NOTE
    Dask does not preserve the memory layout of the original array when
    the array is created using Fortran rather than C ordering.

  **fancy**
  : If `x` doesn’t support fancy indexing (e.g. indexing with lists or
    arrays) then set to False. Default is True.

  **meta**
  : The metadata for the resulting dask array.  This is the kind of array
    that will result from slicing the input array.
    Defaults to the input array.

  **inline_array**
  : How to include the array in the task graph. By default
    (`inline_array=False`) the array is included in a task by itself,
    and each chunk refers to that task by its key.
    ```python
    >>> x = h5py.File("data.h5")["/x"]
    >>> a = da.from_array(x, chunks=500)
    >>> dict(a.dask)
    {
       'array-original-<name>': <HDF5 dataset ...>,
       ('array-<name>', 0): (getitem, "array-original-<name>", ...),
       ('array-<name>', 1): (getitem, "array-original-<name>", ...)
    }
    ```
    <br/>
    With `inline_array=True`, Dask will instead inline the array directly
    in the values of the task graph.
    ```python
    >>> a = da.from_array(x, chunks=500, inline_array=True)
    >>> dict(a.dask)
    {
       ('array-<name>', 0): (getitem, <HDF5 dataset ...>, ...),
       ('array-<name>', 1): (getitem, <HDF5 dataset ...>, ...)
    }
    ```
    <br/>
    Note that there’s no key in the task graph with just the array x
    anymore. Instead it’s placed directly in the values.
    <br/>
    The right choice for `inline_array` depends on several factors,
    including the size of `x`, how expensive it is to create, which
    scheduler you’re using, and the pattern of downstream computations.
    As a heuristic, `inline_array=True` may be the right choice when
    the array `x` is cheap to serialize and deserialize (since it’s
    included in the graph many times) and if you’re experiencing ordering
    issues (see [Ordering](../order.md#order) for more).
    <br/>
    This has no effect when `x` is a NumPy array.

### Examples

```pycon
>>> x = h5py.File('...')['/data/path']
>>> a = da.from_array(x, chunks=(1000, 1000))
```

If your underlying datastore does not support concurrent reads then include
the `lock=True` keyword argument or `lock=mylock` if you want multiple
arrays to coordinate around the same lock.

```pycon
>>> a = da.from_array(x, chunks=(1000, 1000), lock=True)
```

If your underlying datastore has a `.chunks` attribute (as h5py and zarr
datasets do) then a multiple of that chunk shape will be used if you
do not provide a chunk shape.

```pycon
>>> a = da.from_array(x, chunks='auto')
>>> a = da.from_array(x, chunks='100 MiB')
>>> a = da.from_array(x)
```

If providing a name, ensure that it is unique

```pycon
>>> import dask.base
>>> token = dask.base.tokenize(x)
>>> a = da.from_array('myarray-' + token)
```

NumPy ndarrays are eagerly sliced and then embedded in the graph.

```pycon
>>> import dask.array
>>> a = dask.array.from_array(np.array([[1, 2], [3, 4]]), chunks=(1,1))
>>> a.dask[a.name, 0, 0][0]
array([1])
```

Chunks with exactly-specified, different sizes can be created.

```pycon
>>> import numpy as np
>>> import dask.array as da
>>> rng = np.random.default_rng()
>>> x = rng.random((100, 6))
>>> a = da.from_array(x, chunks=((67, 33), (6,)))
```

<!-- !! processed by numpydoc !! -->


# dask.array.from_delayed.html.md

# dask.array.from_delayed

### dask.array.from_delayed(value, shape, dtype=None, meta=None, name=None)

Create a dask array from a dask delayed value

This routine is useful for constructing dask arrays in an ad-hoc fashion
using dask delayed, particularly when combined with stack and concatenate.

The dask array will consist of a single chunk.

### Examples

```pycon
>>> import dask
>>> import dask.array as da
>>> import numpy as np
>>> value = dask.delayed(np.ones)(5)
>>> array = da.from_delayed(value, (5,), dtype=float)
>>> array
dask.array<from-value, shape=(5,), dtype=float64, chunksize=(5,), chunktype=numpy.ndarray>
>>> array.compute()
array([1., 1., 1., 1., 1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.from_npy_stack.html.md

# dask.array.from_npy_stack

### dask.array.from_npy_stack(dirname, mmap_mode='r')

Load dask array from stack of npy files

* **Parameters:**
  **dirname: string**
  : Directory of .npy files

  **mmap_mode: (None or ‘r’)**
  : Read data in memory map mode

#### SEE ALSO
[`to_npy_stack`](dask.array.to_npy_stack.md#dask.array.to_npy_stack)

<!-- !! processed by numpydoc !! -->


# dask.array.from_tiledb.html.md

# dask.array.from_tiledb

### dask.array.from_tiledb(uri, attribute=None, chunks=None, storage_options=None, \*\*kwargs)

Load array from the TileDB storage format

See [https://docs.tiledb.io](https://docs.tiledb.io) for more information about TileDB.

* **Parameters:**
  **uri: TileDB array or str**
  : Location to save the data

  **attribute: str or None**
  : Attribute selection (single-attribute view on multi-attribute array)
* **Returns:**
  A Dask Array

### Examples

```pycon
>>> import tempfile, tiledb
>>> import dask.array as da, numpy as np
>>> uri = tempfile.NamedTemporaryFile().name
>>> _ = tiledb.from_numpy(uri, np.arange(0,9).reshape(3,3))  # create a tiledb array
>>> tdb_ar = da.from_tiledb(uri)  # read back the array
>>> tdb_ar.shape
(3, 3)
>>> tdb_ar.mean().compute()
4.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.from_zarr.html.md

# dask.array.from_zarr

### dask.array.from_zarr(url, component=None, storage_options=None, chunks=None, name=None, inline_array=False, \*\*kwargs)

Load array from the zarr storage format

See [https://zarr.readthedocs.io](https://zarr.readthedocs.io) for details about the format.

* **Parameters:**
  **url: Zarr Array or str or MutableMapping**
  : Location of the data. A URL can include a protocol specifier like s3://
    for remote data. Can also be any MutableMapping instance, which should
    be serializable if used in multiple processes.

  **component: str or None**
  : If the location is a zarr group rather than an array, this is the
    subcomponent that should be loaded, something like `'foo/bar'`.

  **storage_options: dict**
  : Any additional parameters for the storage backend (ignored for local
    paths)

  **chunks: tuple of ints or tuples of ints**
  : Passed to [`dask.array.from_array()`](dask.array.from_array.md#dask.array.from_array), allows setting the chunks on
    initialisation, if the chunking scheme in the on-disc dataset is not
    optimal for the calculations to follow.

  **name**
  : An optional keyname for the array.  Defaults to hashing the input

  **kwargs:**
  : Passed to `zarr.core.Array`.

  **inline_array**
  : Whether to inline the zarr Array in the values of the task graph.
    See [`dask.array.from_array()`](dask.array.from_array.md#dask.array.from_array) for an explanation.

#### SEE ALSO
[`from_array`](dask.array.from_array.md#dask.array.from_array)

<!-- !! processed by numpydoc !! -->


# dask.array.fromfunction.html.md

# dask.array.fromfunction

### dask.array.fromfunction(func, chunks='auto', shape=None, dtype=None, \*\*kwargs)

Construct an array by executing a function over each coordinate.

This docstring was copied from numpy.fromfunction.

Some inconsistencies with the Dask version may exist.

The resulting array therefore has a value `fn(x, y, z)` at
coordinate `(x, y, z)`.

* **Parameters:**
  **function**
  : The function is called with N parameters, where N is the rank of
    shape.  Each parameter represents the coordinates of the array
    varying along a specific axis.  For example, if shape
    were `(2, 2)`, then the parameters would be
    `array([[0, 0], [1, 1]])` and `array([[0, 1], [0, 1]])`

  **shape**
  : Shape of the output array, which also determines the shape of
    the coordinate arrays passed to function.

  **dtype**
  : Data-type of the coordinate arrays passed to function.
    By default, dtype is float.

  **like**
  : > Reference object to allow the creation of arrays which are not
    > NumPy arrays. If an array-like passed in as `like` supports
    > the `__array_function__` protocol, the result will be defined
    > by it. In this case, it ensures the creation of an array object
    > compatible with that passed in via this argument.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **fromfunction**
  : The result of the call to function is passed back directly.
    Therefore the shape of fromfunction is completely determined by
    function.  If function returns a scalar value, the shape of
    fromfunction would not match the shape parameter.

#### SEE ALSO
[`indices`](dask.array.indices.md#dask.array.indices), [`meshgrid`](dask.array.meshgrid.md#dask.array.meshgrid)

### Notes

Keywords other than dtype and like are passed to function.

### Examples

```pycon
>>> import numpy as np
>>> np.fromfunction(lambda i, j: i, (2, 2), dtype=np.float64)
array([[0., 0.],
       [1., 1.]])
```

```pycon
>>> np.fromfunction(lambda i, j: j, (2, 2), dtype=np.float64)
array([[0., 1.],
       [0., 1.]])
```

```pycon
>>> np.fromfunction(lambda i, j: i == j, (3, 3), dtype=np.int_)
array([[ True, False, False],
       [False,  True, False],
       [False, False,  True]])
```

```pycon
>>> np.fromfunction(lambda i, j: i + j, (3, 3), dtype=np.int_)
array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.frompyfunc.html.md

# dask.array.frompyfunc

### dask.array.frompyfunc(func, /, nin, nout, \*)

This docstring was copied from numpy.frompyfunc.

Some inconsistencies with the Dask version may exist.

Takes an arbitrary Python function and returns a NumPy ufunc.

Can be used, for example, to add broadcasting to a built-in Python
function (see Examples section).

* **Parameters:**
  **func**
  : An arbitrary Python function.

  **nin**
  : The number of input arguments.

  **nout**
  : The number of objects returned by func.

  **identity**
  : The value to use for the ~numpy.ufunc.identity attribute of the resulting
    object. If specified, this is equivalent to setting the underlying
    C `identity` field to `PyUFunc_IdentityValue`.
    If omitted, the identity is set to `PyUFunc_None`. Note that this is
    \_not_ equivalent to setting the identity to `None`, which implies the
    operation is reorderable.
* **Returns:**
  **out**
  : Returns a NumPy universal function (`ufunc`) object.

#### SEE ALSO
`vectorize`
: Evaluates pyfunc over input arrays using broadcasting rules of numpy.

### Notes

The returned ufunc always returns PyObject arrays.

### Examples

Use frompyfunc to add broadcasting to the Python function `oct`:

```pycon
>>> import numpy as np
>>> oct_array = np.frompyfunc(oct, 1, 1)
>>> oct_array(np.array((10, 30, 100)))
array(['0o12', '0o36', '0o144'], dtype=object)
>>> np.array((oct(10), oct(30), oct(100))) # for comparison
array(['0o12', '0o36', '0o144'], dtype='<U5')
```

<!-- !! processed by numpydoc !! -->


# dask.array.full.html.md

# dask.array.full

### dask.array.full(shape, fill_value, \*args, \*\*kwargs)

> Blocked variant of full_like

> Follows the signature of full_like exactly except that it also features
> optional keyword arguments `chunks: int, tuple, or dict` and `name: str`.

> Original signature follows below.

Return a full array with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **fill_value**
  : Fill value.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if a is Fortran contiguous,
    ‘C’ otherwise. ‘K’ means match the layout of a as closely
    as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of a, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of fill_value with the same shape and type as a.

#### SEE ALSO
[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`full`](#dask.array.full)
: Return a new array of given shape filled with value.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6, dtype=np.int_)
>>> np.full_like(x, 1)
array([1, 1, 1, 1, 1, 1])
>>> np.full_like(x, 0.1)
array([0, 0, 0, 0, 0, 0])
>>> np.full_like(x, 0.1, dtype=np.float64)
array([0.1, 0.1, 0.1, 0.1, 0.1, 0.1])
>>> np.full_like(x, np.nan, dtype=np.float64)
array([nan, nan, nan, nan, nan, nan])
```

```pycon
>>> y = np.arange(6, dtype=np.float64)
>>> np.full_like(y, 0.1)
array([0.1, 0.1, 0.1, 0.1, 0.1, 0.1])
```

```pycon
>>> y = np.zeros([2, 2, 3], dtype=np.int_)
>>> np.full_like(y, [0, 0, 255])
array([[[  0,   0, 255],
        [  0,   0, 255]],
       [[  0,   0, 255],
        [  0,   0, 255]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.full_like.html.md

# dask.array.full_like

### dask.array.full_like(a, fill_value, order='C', dtype=None, chunks=None, name=None, shape=None)

Return a full array with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **fill_value**
  : Fill value.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Whether to store multidimensional data in C- or Fortran-contiguous
    (row- or column-wise) order in memory.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if `len(array) % chunks != 0`.

  **name**
  : An optional keyname for the array. Defaults to hashing the input
    keyword arguments.

  **shape**
  : Overrides the shape of the result.
* **Returns:**
  **out**
  : Array of fill_value with the same shape and type as a.

#### SEE ALSO
[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`zeros`](dask.array.zeros.md#dask.array.zeros)
: Return a new array setting values to zero.

[`ones`](dask.array.ones.md#dask.array.ones)
: Return a new array setting values to one.

[`empty`](dask.array.empty.md#dask.array.empty)
: Return a new uninitialized array.

[`full`](dask.array.full.md#dask.array.full)
: Fill a new array.

<!-- !! processed by numpydoc !! -->


# dask.array.gradient.html.md

# dask.array.gradient

### dask.array.gradient(f, \*varargs, axis=None, \*\*kwargs)

Return the gradient of an N-dimensional array.

This docstring was copied from numpy.gradient.

Some inconsistencies with the Dask version may exist.

The gradient is computed using second order accurate central differences
in the interior points and either first or second order accurate one-sides
(forward or backwards) differences at the boundaries.
The returned gradient hence has the same shape as the input array.

* **Parameters:**
  **f**
  : An N-dimensional array containing samples of a scalar function.

  **varargs**
  : Spacing between f values. Default unitary spacing for all dimensions.
    Spacing can be specified using:
    1. Single scalar to specify a sample distance for all dimensions.
    2. N scalars to specify a constant sample distance for each dimension.
       i.e. dx, dy, dz, …
    3. N arrays to specify the coordinates of the values along each
       dimension of F. The length of the array must match the size of
       the corresponding dimension
    4. Any combination of N scalars/arrays with the meaning of 2. and 3.
    <br/>
    If axis is given, the number of varargs must equal the number of axes
    specified in the axis parameter.
    Default: 1. (see Examples below).

  **edge_order**
  : Gradient is calculated using N-th order accurate differences
    at the boundaries. Default: 1.

  **axis**
  : Gradient is calculated only along the given axis or axes.
    The default (axis = None) is to calculate the gradient for all the axes
    of the input array. axis may be negative, in which case it counts from
    the last to the first axis.
* **Returns:**
  **gradient**
  : A tuple of ndarrays (or a single ndarray if there is only one
    dimension) corresponding to the derivatives of f with respect
    to each dimension. Each derivative has the same shape as f.

### Notes

Assuming that $f\in C^{3}$ (i.e., $f$ has at least 3 continuous
derivatives) and let $h_{*}$ be a non-homogeneous stepsize, we
minimize the “consistency error” $\eta_{i}$ between the true gradient
and its estimate from a linear combination of the neighboring grid-points:

$$
\eta_{i} = f_{i}^{\left(1\right)} -
            \left[ \alpha f\left(x_{i}\right) +
                    \beta f\left(x_{i} + h_{d}\right) +
                    \gamma f\left(x_{i}-h_{s}\right)
            \right]
$$

By substituting $f(x_{i} + h_{d})$ and $f(x_{i} - h_{s})$
with their Taylor series expansion, this translates into solving
the following the linear system:

$$
\left\{
    \begin{array}{r}
        \alpha+\beta+\gamma=0 \\
        \beta h_{d}-\gamma h_{s}=1 \\
        \beta h_{d}^{2}+\gamma h_{s}^{2}=0
    \end{array}
\right.
$$

The resulting approximation of $f_{i}^{(1)}$ is the following:

$$
\hat f_{i}^{(1)} =
    \frac{
        h_{s}^{2}f\left(x_{i} + h_{d}\right)
        + \left(h_{d}^{2} - h_{s}^{2}\right)f\left(x_{i}\right)
        - h_{d}^{2}f\left(x_{i}-h_{s}\right)}
        { h_{s}h_{d}\left(h_{d} + h_{s}\right)}
    + \mathcal{O}\left(\frac{h_{d}h_{s}^{2}
                        + h_{s}h_{d}^{2}}{h_{d}
                        + h_{s}}\right)
$$

It is worth noting that if $h_{s}=h_{d}$
(i.e., data are evenly spaced)
we find the standard second order approximation:

$$
\hat f_{i}^{(1)}=
    \frac{f\left(x_{i+1}\right) - f\left(x_{i-1}\right)}{2h}
    + \mathcal{O}\left(h^{2}\right)
$$

With a similar procedure the forward/backward approximations used for
boundaries can be derived.

### References

### Examples

```pycon
>>> import numpy as np
>>> f = np.array([1, 2, 4, 7, 11, 16])
>>> np.gradient(f)
array([1. , 1.5, 2.5, 3.5, 4.5, 5. ])
>>> np.gradient(f, 2)
array([0.5 ,  0.75,  1.25,  1.75,  2.25,  2.5 ])
```

Spacing can be also specified with an array that represents the coordinates
of the values F along the dimensions.
For instance a uniform spacing:

```pycon
>>> x = np.arange(f.size)
>>> np.gradient(f, x)
array([1. ,  1.5,  2.5,  3.5,  4.5,  5. ])
```

Or a non uniform one:

```pycon
>>> x = np.array([0., 1., 1.5, 3.5, 4., 6.])
>>> np.gradient(f, x)
array([1. ,  3. ,  3.5,  6.7,  6.9,  2.5])
```

For two dimensional arrays, the return will be two arrays ordered by
axis. In this example the first array stands for the gradient in
rows and the second one in columns direction:

```pycon
>>> np.gradient(np.array([[1, 2, 6], [3, 4, 5]]))
(array([[ 2.,  2., -1.],
        [ 2.,  2., -1.]]),
 array([[1. , 2.5, 4. ],
        [1. , 1. , 1. ]]))
```

In this example the spacing is also specified:
uniform for axis=0 and non uniform for axis=1

```pycon
>>> dx = 2.
>>> y = [1., 1.5, 3.5]
>>> np.gradient(np.array([[1, 2, 6], [3, 4, 5]]), dx, y)
(array([[ 1. ,  1. , -0.5],
        [ 1. ,  1. , -0.5]]),
 array([[2. , 2. , 2. ],
        [2. , 1.7, 0.5]]))
```

It is possible to specify how boundaries are treated using edge_order

```pycon
>>> x = np.array([0, 1, 2, 3, 4])
>>> f = x**2
>>> np.gradient(f, edge_order=1)
array([1.,  2.,  4.,  6.,  7.])
>>> np.gradient(f, edge_order=2)
array([0., 2., 4., 6., 8.])
```

The axis keyword can be used to specify a subset of axes of which the
gradient is calculated

```pycon
>>> np.gradient(np.array([[1, 2, 6], [3, 4, 5]]), axis=0)
array([[ 2.,  2., -1.],
       [ 2.,  2., -1.]])
```

The varargs argument defines the spacing between sample points in the
input array. It can take two forms:

1. An array, specifying coordinates, which may be unevenly spaced:

```pycon
>>> x = np.array([0., 2., 3., 6., 8.])
>>> y = x ** 2
>>> np.gradient(y, x, edge_order=2)
array([ 0.,  4.,  6., 12., 16.])
```

1. A scalar, representing the fixed sample distance:

```pycon
>>> dx = 2
>>> x = np.array([0., 2., 4., 6., 8.])
>>> y = x ** 2
>>> np.gradient(y, dx, edge_order=2)
array([ 0.,  4.,  8., 12., 16.])
```

It’s possible to provide different data for spacing along each dimension.
The number of arguments must match the number of dimensions in the input
data.

```pycon
>>> dx = 2
>>> dy = 3
>>> x = np.arange(0, 6, dx)
>>> y = np.arange(0, 9, dy)
>>> xs, ys = np.meshgrid(x, y)
>>> zs = xs + 2 * ys
>>> np.gradient(zs, dy, dx)  # Passing two scalars
(array([[2., 2., 2.],
        [2., 2., 2.],
        [2., 2., 2.]]),
 array([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]]))
```

Mixing scalars and arrays is also allowed:

```pycon
>>> np.gradient(zs, y, dx)  # Passing one array and one scalar
(array([[2., 2., 2.],
        [2., 2., 2.],
        [2., 2., 2.]]),
 array([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.greater.html.md

# dask.array.greater

### dask.array.greater(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'greater'>*

This docstring was copied from numpy.greater.

Some inconsistencies with the Dask version may exist.

Return the truth value of (x1 > x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`less`](dask.array.less.md#dask.array.less), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`equal`](dask.array.equal.md#dask.array.equal), [`not_equal`](dask.array.not_equal.md#dask.array.not_equal)

### Examples

```pycon
>>> import numpy as np
>>> np.greater([4,2],[2,2])
array([ True, False])
```

The `>` operator can be used as a shorthand for `np.greater` on
ndarrays.

```pycon
>>> a = np.array([4, 2])
>>> b = np.array([2, 2])
>>> a > b
array([ True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.greater_equal.html.md

# dask.array.greater_equal

### dask.array.greater_equal(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'greater_equal'>*

This docstring was copied from numpy.greater_equal.

Some inconsistencies with the Dask version may exist.

Return the truth value of (x1 >= x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`greater`](dask.array.greater.md#dask.array.greater), [`less`](dask.array.less.md#dask.array.less), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`equal`](dask.array.equal.md#dask.array.equal), [`not_equal`](dask.array.not_equal.md#dask.array.not_equal)

### Examples

```pycon
>>> import numpy as np
>>> np.greater_equal([4, 2, 1], [2, 2, 2])
array([ True, True, False])
```

The `>=` operator can be used as a shorthand for `np.greater_equal`
on ndarrays.

```pycon
>>> a = np.array([4, 2, 1])
>>> b = np.array([2, 2, 2])
>>> a >= b
array([ True,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.gufunc.apply_gufunc.html.md

# dask.array.gufunc.apply_gufunc

### dask.array.gufunc.apply_gufunc(func, signature, \*args, axes=None, axis=None, keepdims=False, output_dtypes=None, output_sizes=None, vectorize=None, allow_rechunk=False, meta=None, \*\*kwargs)

Apply a generalized ufunc or similar python function to arrays.

`signature` determines if the function consumes or produces core
dimensions. The remaining dimensions in given input arrays (`*args`)
are considered loop dimensions and are required to broadcast
naturally against each other.

In other terms, this function is like `np.vectorize`, but for
the blocks of dask arrays. If the function itself shall also
be vectorized use `vectorize=True` for convenience.

* **Parameters:**
  **func**
  : Function to call like `func(*args, **kwargs)` on input arrays
    (`*args`) that returns an array or tuple of arrays. If multiple
    arguments with non-matching dimensions are supplied, this function is
    expected to vectorize (broadcast) over axes of positional arguments in
    the style of NumPy universal functions [[1]](#rb8504d1ef58d-1) (if this is not the case,
    set `vectorize=True`). If this function returns multiple outputs,
    `output_core_dims` has to be set as well.

  **signature: string**
  : Specifies what core dimensions are consumed and produced by `func`.
    According to the specification of numpy.gufunc signature [[2]](#rb8504d1ef58d-2)

  **\*args**
  : Input arrays or scalars to the callable function.

  **axes: List of tuples, optional, keyword only**
  : A list of tuples with indices of axes a generalized ufunc should operate on.
    For instance, for a signature of `"(i,j),(j,k)->(i,k)"` appropriate for
    matrix multiplication, the base elements are two-dimensional matrices
    and these are taken to be stored in the two last axes of each argument. The
    corresponding axes keyword would be `[(-2, -1), (-2, -1), (-2, -1)]`.
    For simplicity, for generalized ufuncs that operate on 1-dimensional arrays
    (vectors), a single integer is accepted instead of a single-element tuple,
    and for generalized ufuncs for which all outputs are scalars, the output
    tuples can be omitted.

  **axis: int, optional, keyword only**
  : A single axis over which a generalized ufunc should operate. This is a short-cut
    for ufuncs that operate over a single, shared core dimension, equivalent to passing
    in axes with entries of (axis,) for each single-core-dimension argument and `()` for
    all others. For instance, for a signature `"(i),(i)->()"`, it is equivalent to passing
    in `axes=[(axis,), (axis,), ()]`.

  **keepdims: bool, optional, keyword only**
  : If this is set to True, axes which are reduced over will be left in the result as
    a dimension with size one, so that the result will broadcast correctly against the
    inputs. This option can only be used for generalized ufuncs that operate on inputs
    that all have the same number of core dimensions and with outputs that have no core
    dimensions , i.e., with signatures like `"(i),(i)->()"` or `"(m,m)->()"`.
    If used, the location of the dimensions in the output can be controlled with axes
    and axis.

  **output_dtypes**
  : Valid numpy dtype specification or list thereof.
    If not given, a call of `func` with a small set of data
    is performed in order to try to automatically determine the
    output dtypes.

  **output_sizes**
  : Optional mapping from dimension names to sizes for outputs. Only used if
    new core dimensions (not found on inputs) appear on outputs.

  **vectorize: bool, keyword only**
  : If set to `True`, `np.vectorize` is applied to `func` for
    convenience. Defaults to `False`.

  **allow_rechunk: Optional, bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.
    Warning: enabling this can increase memory usage significantly.
    Defaults to `False`.

  **meta: Optional, tuple, keyword only**
  : tuple of empty ndarrays describing the shape and dtype of the output of the gufunc.
    Defaults to `None`.

  **\*\*kwargs**
  : Extra keyword arguments to pass to func
* **Returns:**
  Single dask.array.Array or tuple of dask.array.Array

### References

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> def stats(x):
...     return np.mean(x, axis=-1), np.std(x, axis=-1)
>>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30))
>>> mean, std = da.apply_gufunc(stats, "(i)->(),()", a)
>>> mean.compute().shape
(10, 20)
```

```pycon
>>> def outer_product(x, y):
...     return np.einsum("i,j->ij", x, y)
>>> a = da.random.normal(size=(   20,30), chunks=(10, 30))
>>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40))
>>> c = da.apply_gufunc(outer_product, "(i),(j)->(i,j)", a, b, vectorize=True)
>>> c.compute().shape
(10, 20, 30, 40)
```

<!-- !! processed by numpydoc !! -->


# dask.array.gufunc.as_gufunc.html.md

# dask.array.gufunc.as_gufunc

### dask.array.gufunc.as_gufunc(signature=None, \*\*kwargs)

Decorator for `dask.array.gufunc`.

* **Parameters:**
  **signature**
  : Specifies what core dimensions are consumed and produced by `func`.
    According to the specification of numpy.gufunc signature [[2]](#red860e6aea48-2)

  **axes: List of tuples, optional, keyword only**
  : A list of tuples with indices of axes a generalized ufunc should operate on.
    For instance, for a signature of `"(i,j),(j,k)->(i,k)"` appropriate for
    matrix multiplication, the base elements are two-dimensional matrices
    and these are taken to be stored in the two last axes of each argument. The
    corresponding axes keyword would be `[(-2, -1), (-2, -1), (-2, -1)]`.
    For simplicity, for generalized ufuncs that operate on 1-dimensional arrays
    (vectors), a single integer is accepted instead of a single-element tuple,
    and for generalized ufuncs for which all outputs are scalars, the output
    tuples can be omitted.

  **axis: int, optional, keyword only**
  : A single axis over which a generalized ufunc should operate. This is a short-cut
    for ufuncs that operate over a single, shared core dimension, equivalent to passing
    in axes with entries of (axis,) for each single-core-dimension argument and `()` for
    all others. For instance, for a signature `"(i),(i)->()"`, it is equivalent to passing
    in `axes=[(axis,), (axis,), ()]`.

  **keepdims: bool, optional, keyword only**
  : If this is set to True, axes which are reduced over will be left in the result as
    a dimension with size one, so that the result will broadcast correctly against the
    inputs. This option can only be used for generalized ufuncs that operate on inputs
    that all have the same number of core dimensions and with outputs that have no core
    dimensions , i.e., with signatures like `"(i),(i)->()"` or `"(m,m)->()"`.
    If used, the location of the dimensions in the output can be controlled with axes
    and axis.

  **output_dtypes**
  : Valid numpy dtype specification or list thereof.
    If not given, a call of `func` with a small set of data
    is performed in order to try to automatically determine the
    output dtypes.

  **output_sizes**
  : Optional mapping from dimension names to sizes for outputs. Only used if
    new core dimensions (not found on inputs) appear on outputs.

  **vectorize: bool, keyword only**
  : If set to `True`, `np.vectorize` is applied to `func` for
    convenience. Defaults to `False`.

  **allow_rechunk: Optional, bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.
    Warning: enabling this can increase memory usage significantly.
    Defaults to `False`.

  **meta: Optional, tuple, keyword only**
  : tuple of empty ndarrays describing the shape and dtype of the output of the gufunc.
    Defaults to `None`.
* **Returns:**
  Decorator for pyfunc that itself returns a gufunc.

### References

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30))
>>> @da.as_gufunc("(i)->(),()", output_dtypes=(float, float))
... def stats(x):
...     return np.mean(x, axis=-1), np.std(x, axis=-1)
>>> mean, std = stats(a)
>>> mean.compute().shape
(10, 20)
```

```pycon
>>> a = da.random.normal(size=(   20,30), chunks=(10, 30))
>>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40))
>>> @da.as_gufunc("(i),(j)->(i,j)", output_dtypes=float, vectorize=True)
... def outer_product(x, y):
...     return np.einsum("i,j->ij", x, y)
>>> c = outer_product(a, b)
>>> c.compute().shape
(10, 20, 30, 40)
```

<!-- !! processed by numpydoc !! -->


# dask.array.gufunc.gufunc.html.md

# dask.array.gufunc.gufunc

### *class* dask.array.gufunc.gufunc(pyfunc, , signature=None, vectorize=False, axes=None, axis=None, keepdims=False, output_sizes=None, output_dtypes=None, allow_rechunk=False, meta=None)

Binds pyfunc into `dask.array.apply_gufunc` when called.

* **Parameters:**
  **pyfunc**
  : Function to call like `func(*args, **kwargs)` on input arrays
    (`*args`) that returns an array or tuple of arrays. If multiple
    arguments with non-matching dimensions are supplied, this function is
    expected to vectorize (broadcast) over axes of positional arguments in
    the style of NumPy universal functions [[1]](#r02163d2dfbb7-1) (if this is not the case,
    set `vectorize=True`). If this function returns multiple outputs,
    `output_core_dims` has to be set as well.

  **signature**
  : Specifies what core dimensions are consumed and produced by `func`.
    According to the specification of numpy.gufunc signature [[2]](#r02163d2dfbb7-2)

  **axes: List of tuples, optional, keyword only**
  : A list of tuples with indices of axes a generalized ufunc should operate on.
    For instance, for a signature of `"(i,j),(j,k)->(i,k)"` appropriate for
    matrix multiplication, the base elements are two-dimensional matrices
    and these are taken to be stored in the two last axes of each argument. The
    corresponding axes keyword would be `[(-2, -1), (-2, -1), (-2, -1)]`.
    For simplicity, for generalized ufuncs that operate on 1-dimensional arrays
    (vectors), a single integer is accepted instead of a single-element tuple,
    and for generalized ufuncs for which all outputs are scalars, the output
    tuples can be omitted.

  **axis: int, optional, keyword only**
  : A single axis over which a generalized ufunc should operate. This is a short-cut
    for ufuncs that operate over a single, shared core dimension, equivalent to passing
    in axes with entries of (axis,) for each single-core-dimension argument and `()` for
    all others. For instance, for a signature `"(i),(i)->()"`, it is equivalent to passing
    in `axes=[(axis,), (axis,), ()]`.

  **keepdims: bool, optional, keyword only**
  : If this is set to True, axes which are reduced over will be left in the result as
    a dimension with size one, so that the result will broadcast correctly against the
    inputs. This option can only be used for generalized ufuncs that operate on inputs
    that all have the same number of core dimensions and with outputs that have no core
    dimensions , i.e., with signatures like `"(i),(i)->()"` or `"(m,m)->()"`.
    If used, the location of the dimensions in the output can be controlled with axes
    and axis.

  **output_dtypes**
  : Valid numpy dtype specification or list thereof.
    If not given, a call of `func` with a small set of data
    is performed in order to try to automatically determine the
    output dtypes.

  **output_sizes**
  : Optional mapping from dimension names to sizes for outputs. Only used if
    new core dimensions (not found on inputs) appear on outputs.

  **vectorize: bool, keyword only**
  : If set to `True`, `np.vectorize` is applied to `func` for
    convenience. Defaults to `False`.

  **allow_rechunk: Optional, bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.
    Warning: enabling this can increase memory usage significantly.
    Defaults to `False`.

  **meta: Optional, tuple, keyword only**
  : tuple of empty ndarrays describing the shape and dtype of the output of the gufunc.
    Defaults to `None`.
* **Returns:**
  Wrapped function

### References

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30))
>>> def stats(x):
...     return np.mean(x, axis=-1), np.std(x, axis=-1)
>>> gustats = da.gufunc(stats, signature="(i)->(),()", output_dtypes=(float, float))
>>> mean, std = gustats(a)
>>> mean.compute().shape
(10, 20)
```

```pycon
>>> a = da.random.normal(size=(   20,30), chunks=(10, 30))
>>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40))
>>> def outer_product(x, y):
...     return np.einsum("i,j->ij", x, y)
>>> guouter_product = da.gufunc(outer_product, signature="(i),(j)->(i,j)", output_dtypes=float, vectorize=True)
>>> c = guouter_product(a, b)
>>> c.compute().shape
(10, 20, 30, 40)
```

```pycon
>>> a = da.ones((1, 5, 10), chunks=(-1, -1, -1))
>>> def stats(x):
...     return np.atleast_1d(x.mean()), np.atleast_1d(x.max())
>>> meta = (np.array((), dtype=np.float64), np.array((), dtype=np.float64))
>>> gustats = da.gufunc(stats, signature="(i,j)->(),()", meta=meta)
>>> result = gustats(a)
>>> result[0].compute().shape
(1,)
>>> result[1].compute().shape
(1,)
```

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(pyfunc, , signature=None, vectorize=False, axes=None, axis=None, keepdims=False, output_sizes=None, output_dtypes=None, allow_rechunk=False, meta=None)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.array.gufunc.gufunc.__init__)(pyfunc, \*[, signature, vectorize, ...])   |    |
|---------------------------------------------------------------------------------------------|----|


# dask.array.histogram.html.md

# dask.array.histogram

### dask.array.histogram(a, bins=None, range=None, normed=False, weights=None, density=None)

Blocked variant of [`numpy.histogram()`](https://numpy.org/doc/stable/reference/generated/numpy.histogram.html#numpy.histogram).

* **Parameters:**
  **a**
  : Input data; the histogram is computed over the flattened
    array. If the `weights` argument is used, the chunks of
    `a` are accessed to check chunking compatibility between
    `a` and `weights`. If `weights` is `None`, a
    [`dask.dataframe.Series`](dask.dataframe.Series.md#dask.dataframe.Series) object can be passed as
    input data.

  **bins**
  : Either an iterable specifying the `bins` or the number of `bins`
    and a `range` argument is required as computing `min` and `max`
    over blocked arrays is an expensive operation that must be performed
    explicitly.
    If bins is an int, it defines the number of equal-width
    bins in the given range (10, by default). If bins is a
    sequence, it defines a monotonically increasing array of bin edges,
    including the rightmost edge, allowing for non-uniform bin widths.

  **range**
  : The lower and upper range of the bins.  If not provided, range
    is simply `(a.min(), a.max())`.  Values outside the range are
    ignored. The first element of the range must be less than or
    equal to the second. range affects the automatic bin
    computation as well. While bin width is computed to be optimal
    based on the actual data within range, the bin count will fill
    the entire range including portions containing no data.

  **normed**
  : This is equivalent to the `density` argument, but produces incorrect
    results for unequal bin widths. It should not be used.

  **weights**
  : A dask.array.Array of weights, of the same block structure as `a`.  Each value in
    `a` only contributes its associated weight towards the bin count
    (instead of 1). If `density` is True, the weights are
    normalized, so that the integral of the density over the range
    remains 1.

  **density**
  : If `False`, the result will contain the number of samples in
    each bin. If `True`, the result is the value of the
    probability *density* function at the bin, normalized such that
    the *integral* over the range is 1. Note that the sum of the
    histogram values will not be equal to 1 unless bins of unity
    width are chosen; it is not a probability *mass* function.
    Overrides the `normed` keyword if given.
    If `density` is True, `bins` cannot be a single-number delayed
    value. It must be a concrete number, or a (possibly-delayed)
    array/sequence of the bin edges.
* **Returns:**
  **hist**
  : The values of the histogram. See density and weights for a
    description of the possible semantics.

  **bin_edges**
  : Return the bin edges `(length(hist)+1)`.

### Examples

Using number of bins and range:

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> x = da.from_array(np.arange(10000), chunks=10)
>>> h, bins = da.histogram(x, bins=10, range=[0, 10000])
>>> bins
array([    0.,  1000.,  2000.,  3000.,  4000.,  5000.,  6000.,  7000.,
        8000.,  9000., 10000.])
>>> h.compute()
array([1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000])
```

Explicitly specifying the bins:

```pycon
>>> h, bins = da.histogram(x, bins=np.array([0, 5000, 10000]))
>>> bins
array([    0,  5000, 10000])
>>> h.compute()
array([5000, 5000])
```

<!-- !! processed by numpydoc !! -->


# dask.array.histogram2d.html.md

# dask.array.histogram2d

### dask.array.histogram2d(x, y, bins=10, range=None, normed=None, weights=None, density=None)

Blocked variant of [`numpy.histogram2d()`](https://numpy.org/doc/stable/reference/generated/numpy.histogram2d.html#numpy.histogram2d).

* **Parameters:**
  **x**
  : An array containing the x-coordinates of the points to be
    histogrammed.

  **y**
  : An array containing the y-coordinates of the points to be
    histogrammed.

  **bins**
  : The bin specification. See the bins argument description for
    [`histogramdd()`](dask.array.histogramdd.md#dask.array.histogramdd) for a complete description of all
    possible bin configurations (this function is a 2D specific
    version of histogramdd).

  **range**
  : The leftmost and rightmost edges of the bins along each
    dimension when integers are passed to bins; of the form:
    ((xmin, xmax), (ymin, ymax)).

  **normed**
  : An alias for the density argument that behaves identically. To
    avoid confusion with the broken argument in the histogram
    function, density should be preferred.

  **weights**
  : An array of values weighing each sample in the input data. The
    chunks of the weights must be identical to the chunking along
    the 0th (row) axis of the data sample.

  **density**
  : If False (the default) return the number of samples in each
    bin. If True, the returned array represents the probability
    density function at each bin.
* **Returns:**
  dask.array.Array
  : The values of the histogram.

  dask.array.Array
  : The edges along the x-dimension.

  dask.array.Array
  : The edges along the y-dimension.

#### SEE ALSO
[`histogram`](dask.array.histogram.md#dask.array.histogram)

[`histogramdd`](dask.array.histogramdd.md#dask.array.histogramdd)

### Examples

```pycon
>>> import dask.array as da
>>> x = da.array([2, 4, 2, 4, 2, 4])
>>> y = da.array([2, 2, 4, 4, 2, 4])
>>> bins = 2
>>> range = ((0, 6), (0, 6))
>>> h, xedges, yedges = da.histogram2d(x, y, bins=bins, range=range)
>>> h
dask.array<sum-aggregate, shape=(2, 2), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>
>>> xedges
dask.array<array, shape=(3,), dtype=float64, chunksize=(3,), chunktype=numpy.ndarray>
>>> h.compute()
array([[2., 1.],
       [1., 2.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.histogramdd.html.md

# dask.array.histogramdd

### dask.array.histogramdd(sample, bins, range=None, normed=None, weights=None, density=None)

Blocked variant of [`numpy.histogramdd()`](https://numpy.org/doc/stable/reference/generated/numpy.histogramdd.html#numpy.histogramdd).

Chunking of the input data (`sample`) is only allowed along the
0th (row) axis (the axis corresponding to the total number of
samples). Data chunked along the 1st axis (column) axis is not
compatible with this function. If weights are used, they must be
chunked along the 0th axis identically to the input sample.

An example setup for a three dimensional histogram, where the
sample shape is `(8, 3)` and weights are shape `(8,)`, sample
chunks would be `((4, 4), (3,))` and the weights chunks would be
`((4, 4),)` a table of the structure:

|       |     |    |    |    | sample (8 x 3)   | weights   |
|-------|-----|----|----|----|------------------|-----------|
| chunk | row | x  | y  | z  | row              | w         |
| 0     | 0   | 5  | 6  | 6  | 0                | 0.5       |
| 1     | 8   | 9  | 2  | 1  | 0.8              |           |
| 2     | 3   | 3  | 1  | 2  | 0.3              |           |
| 3     | 2   | 5  | 6  | 3  | 0.7              |           |
| 1     | 4   | 3  | 1  | 1  | 4                | 0.3       |
| 5     | 3   | 2  | 9  | 5  | 1.3              |           |
| 6     | 8   | 1  | 5  | 6  | 0.8              |           |
| 7     | 3   | 5  | 3  | 7  | 0.7              |           |

If the sample 0th dimension and weight 0th (row) dimension are
chunked differently, a `ValueError` will be raised. If
coordinate groupings ((x, y, z) trios) are separated by a chunk
boundary, then a `ValueError` will be raised. We suggest that you
rechunk your data if it is of that form.

The chunks property of the data (and optional weights) are used to
check for compatibility with the blocked algorithm (as described
above); therefore, you must call to_dask_array on a collection
from `dask.dataframe`, i.e. [`dask.dataframe.Series`](dask.dataframe.Series.md#dask.dataframe.Series) or
[`dask.dataframe.DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame).

The function is also compatible with x, y, and z being
individual 1D arrays with equal chunking. In that case, the data
should be passed as a tuple: `histogramdd((x, y, z), ...)`

* **Parameters:**
  **sample**
  : Multidimensional data to be histogrammed.
    <br/>
    Note the unusual interpretation of a sample when it is a
    sequence of dask Arrays:
    * When a (N, D) dask Array, each row is an entry in the sample
      (coordinate in D dimensional space).
    * When a sequence of dask Arrays, each element in the sequence
      is the array of values for a single coordinate.

  **bins**
  : The bin specification.
    <br/>
    The possible binning configurations are:
    * A sequence of arrays describing the monotonically increasing
      bin edges along each dimension.
    * A single int describing the total number of bins that will
      be used in each dimension (this requires the `range`
      argument to be defined).
    * A sequence of ints describing the total number of bins to be
      used in each dimension (this requires the `range` argument
      to be defined).
    <br/>
    When bins are described by arrays, the rightmost edge is
    included. Bins described by arrays also allows for non-uniform
    bin widths.

  **range**
  : A sequence of length D, each a (min, max) tuple giving the
    outer bin edges to be used if the edges are not given
    explicitly in bins. If defined, this argument is required to
    have an entry for each dimension. Unlike
    [`numpy.histogramdd()`](https://numpy.org/doc/stable/reference/generated/numpy.histogramdd.html#numpy.histogramdd), if bins does not define bin
    edges, this argument is required (this function will not
    automatically use the min and max of the value in a given
    dimension because the input data may be lazy in dask).

  **normed**
  : An alias for the density argument that behaves identically. To
    avoid confusion with the broken argument to histogram,
    density should be preferred.

  **weights**
  : An array of values weighing each sample in the input data. The
    chunks of the weights must be identical to the chunking along
    the 0th (row) axis of the data sample.

  **density**
  : If `False` (default), the returned array represents the
    number of samples in each bin. If `True`, the returned array
    represents the probability density function at each bin.
* **Returns:**
  dask.array.Array
  : The values of the histogram.

  list(dask.array.Array)
  : Sequence of arrays representing the bin edges along each
    dimension.

#### SEE ALSO
[`histogram`](dask.array.histogram.md#dask.array.histogram)

### Examples

Computing the histogram in 5 blocks using different bin edges
along each dimension:

```pycon
>>> import dask.array as da
>>> x = da.random.uniform(0, 1, size=(1000, 3), chunks=(200, 3))
>>> edges = [
...     np.linspace(0, 1, 5), # 4 bins in 1st dim
...     np.linspace(0, 1, 6), # 5 in the 2nd
...     np.linspace(0, 1, 4), # 3 in the 3rd
... ]
>>> h, edges = da.histogramdd(x, bins=edges)
>>> result = h.compute()
>>> result.shape
(4, 5, 3)
```

Defining the bins by total number and their ranges, along with
using weights:

```pycon
>>> bins = (4, 5, 3)
>>> ranges = ((0, 1),) * 3  # expands to ((0, 1), (0, 1), (0, 1))
>>> w = da.random.uniform(0, 1, size=(1000,), chunks=x.chunksize[0])
>>> h, edges = da.histogramdd(x, bins=bins, range=ranges, weights=w)
>>> np.isclose(h.sum().compute(), w.sum().compute())
np.True_
```

Using a sequence of 1D arrays as the input:

```pycon
>>> x = da.array([2, 4, 2, 4, 2, 4])
>>> y = da.array([2, 2, 4, 4, 2, 4])
>>> z = da.array([4, 2, 4, 2, 4, 2])
>>> bins = ([0, 3, 6],) * 3
>>> h, edges = da.histogramdd((x, y, z), bins)
>>> h
dask.array<sum-aggregate, shape=(2, 2, 2), dtype=float64, chunksize=(2, 2, 2), chunktype=numpy.ndarray>
>>> edges[0]
dask.array<array, shape=(3,), dtype=int64, chunksize=(3,), chunktype=numpy.ndarray>
>>> h.compute()
array([[[0., 2.],
        [0., 1.]],

       [[1., 0.],
        [2., 0.]]])
>>> edges[0].compute()
array([0, 3, 6])
>>> edges[1].compute()
array([0, 3, 6])
>>> edges[2].compute()
array([0, 3, 6])
```

<!-- !! processed by numpydoc !! -->


# dask.array.hstack.html.md

# dask.array.hstack

### dask.array.hstack(tup, allow_unknown_chunksizes=False)

Stack arrays in sequence horizontally (column wise).

This docstring was copied from numpy.hstack.

Some inconsistencies with the Dask version may exist.

This is equivalent to concatenation along the second axis, except for 1-D
arrays where it concatenates along the first axis. Rebuilds arrays divided
by hsplit.

This function makes most sense for arrays with up to 3 dimensions. For
instance, for pixel-data with a height (first axis), width (second axis),
and r/g/b channels (third axis). The functions concatenate, stack and
block provide more general stacking and concatenation operations.

* **Parameters:**
  **tup**
  : The arrays must have the same shape along all but the second axis,
    except 1-D arrays which can be any length. In the case of a single
    array_like input, it will be treated as a sequence of arrays; i.e.,
    each element along the zeroth axis is treated as a separate array.

  **dtype**
  : If provided, the destination array will have this dtype. Cannot be
    provided together with out.
    <br/>
    #### Versionadded
    Added in version 1.24.

  **casting**
  : Controls what kind of data casting may occur. Defaults to ‘same_kind’.
    <br/>
    #### Versionadded
    Added in version 1.24.
* **Returns:**
  **stacked**
  : The array formed by stacking the given arrays.

#### SEE ALSO
[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)
: Join a sequence of arrays along an existing axis.

[`stack`](dask.array.stack.md#dask.array.stack)
: Join a sequence of arrays along a new axis.

[`block`](dask.array.block.md#dask.array.block)
: Assemble an nd-array from nested lists of blocks.

[`vstack`](dask.array.vstack.md#dask.array.vstack)
: Stack arrays in sequence vertically (row wise).

[`dstack`](dask.array.dstack.md#dask.array.dstack)
: Stack arrays in sequence depth wise (along third axis).

`column_stack`
: Stack 1-D arrays as columns into a 2-D array.

`hsplit`
: Split an array into multiple sub-arrays horizontally (column-wise).

`unstack`
: Split an array into a tuple of sub-arrays along an axis.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array((1,2,3))
>>> b = np.array((4,5,6))
>>> np.hstack((a,b))
array([1, 2, 3, 4, 5, 6])
>>> a = np.array([[1],[2],[3]])
>>> b = np.array([[4],[5],[6]])
>>> np.hstack((a,b))
array([[1, 4],
       [2, 5],
       [3, 6]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.hypot.html.md

# dask.array.hypot

### dask.array.hypot(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'hypot'>*

This docstring was copied from numpy.hypot.

Some inconsistencies with the Dask version may exist.

Given the “legs” of a right triangle, return its hypotenuse.

Equivalent to `sqrt(x1**2 + x2**2)`, element-wise.  If x1 or
x2 is scalar_like (i.e., unambiguously cast-able to a scalar type),
it is broadcast for use with each element of the other argument.
(See Examples)

* **Parameters:**
  **x1, x2**
  : Leg of the triangle(s).
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **z**
  : The hypotenuse of the triangle(s).
    This is a scalar if both x1 and x2 are scalars.

### Examples

```pycon
>>> import numpy as np
>>> np.hypot(3*np.ones((3, 3)), 4*np.ones((3, 3)))
array([[ 5.,  5.,  5.],
       [ 5.,  5.,  5.],
       [ 5.,  5.,  5.]])
```

Example showing broadcast of scalar_like argument:

```pycon
>>> np.hypot(3*np.ones((3, 3)), [4])
array([[ 5.,  5.,  5.],
       [ 5.,  5.,  5.],
       [ 5.,  5.,  5.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.i0.html.md

# dask.array.i0

### dask.array.i0(\*args, \*\*kwargs)

Modified Bessel function of the first kind, order 0.

This docstring was copied from numpy.i0.

Some inconsistencies with the Dask version may exist.

Usually denoted $I_0$.

* **Parameters:**
  **x**
  : Argument of the Bessel function.
* **Returns:**
  **out**
  : The modified Bessel function evaluated at each of the elements of x.

#### SEE ALSO
[`scipy.special.i0`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.i0.html#scipy.special.i0), [`scipy.special.iv`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.iv.html#scipy.special.iv), [`scipy.special.ive`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.ive.html#scipy.special.ive)

### Notes

The scipy implementation is recommended over this function: it is a
proper ufunc written in C, and more than an order of magnitude faster.

We use the algorithm published by Clenshaw [[1]](#r357ebadae9e0-1) and referenced by
Abramowitz and Stegun [[2]](#r357ebadae9e0-2), for which the function domain is
partitioned into the two intervals [0,8] and (8,inf), and Chebyshev
polynomial expansions are employed in each interval. Relative error on
the domain [0,30] using IEEE arithmetic is documented [[3]](#r357ebadae9e0-3) as having a
peak of 5.8e-16 with an rms of 1.4e-16 (n = 30000).

### References

### Examples

```pycon
>>> import numpy as np
>>> np.i0(0.)
array(1.0)
>>> np.i0([0, 1, 2, 3])
array([1.        , 1.26606588, 2.2795853 , 4.88079259])
```

<!-- !! processed by numpydoc !! -->


# dask.array.imag.html.md

# dask.array.imag

### dask.array.imag(\*args, \*\*kwargs)

Return the imaginary part of the complex argument.

This docstring was copied from numpy.imag.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **val**
  : Input array.
* **Returns:**
  **out**
  : The imaginary component of the complex argument. If val is real,
    the type of val is used for the output.  If val has complex
    elements, the returned type is float.

#### SEE ALSO
[`real`](dask.array.real.md#dask.array.real), [`angle`](dask.array.angle.md#dask.array.angle), `real_if_close`

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1+2j, 3+4j, 5+6j])
>>> a.imag
array([2.,  4.,  6.])
>>> a.imag = np.array([8, 10, 12])
>>> a
array([1. +8.j,  3.+10.j,  5.+12.j])
>>> np.imag(1 + 1j)
1.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.image.imread.html.md

# dask.array.image.imread

### dask.array.image.imread(filename, imread=None, preprocess=None)

Read a stack of images into a dask array

* **Parameters:**
  **filename: string**
  : A globstring like ‘myfile.\*.png’

  **imread: function (optional)**
  : Optionally provide custom imread function.
    Function should expect a filename and produce a numpy array.
    Defaults to `skimage.io.imread`.

  **preprocess: function (optional)**
  : Optionally provide custom function to preprocess the image.
    Function should expect a numpy array for a single image.
* **Returns:**
  Dask array of all images stacked along the first dimension.

  Each separate image file will be treated as an individual chunk.

### Examples

```pycon
>>> from dask.array.image import imread
>>> im = imread('2015-*-*.png')
>>> im.shape
(365, 1000, 1000, 3)
```

<!-- !! processed by numpydoc !! -->


# dask.array.indices.html.md

# dask.array.indices

### dask.array.indices(dimensions, dtype=<class 'int'>, chunks='auto')

Implements NumPy’s `indices` for Dask Arrays.

Generates a grid of indices covering the dimensions provided.

The final array has the shape `(len(dimensions), *dimensions)`. The
chunks are used to specify the chunking for axis 1 up to
`len(dimensions)`. The 0th axis always has chunks of length 1.

* **Parameters:**
  **dimensions**
  : The shape of the index grid.

  **dtype**
  : Type to use for the array. Default is `int`.

  **chunks**
  : The size of each block.  Must be one of the following forms:
    - A blocksize like (500, 1000)
    - A size in bytes, like “100 MiB” which will choose a uniform
      block-like shape
    - The word “auto” which acts like the above, but uses a configuration
      value `array.chunk-size` for the chunk size
    <br/>
    Note that the last block will have fewer samples if `len(array) % chunks != 0`.
* **Returns:**
  **grid**

<!-- !! processed by numpydoc !! -->


# dask.array.insert.html.md

# dask.array.insert

### dask.array.insert(arr, obj, values, axis)

Insert values along the given axis before the given indices.

This docstring was copied from numpy.insert.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **arr**
  : Input array.

  **obj**
  : Object that defines the index or indices before which values is
    inserted.
    <br/>
    #### Versionchanged
    Changed in version 2.1.2: Boolean indices are now treated as a mask of elements to insert,
    rather than being cast to the integers 0 and 1.
    <br/>
    Support for multiple insertions when obj is a single scalar or a
    sequence with one element (similar to calling insert multiple
    times).

  **values**
  : Values to insert into arr. If the type of values is different
    from that of arr, values is converted to the type of arr.
    values should be shaped so that `arr[...,obj,...] = values`
    is legal.

  **axis**
  : Axis along which to insert values.  If axis is None then arr
    is flattened first.
* **Returns:**
  **out**
  : A copy of arr with values inserted.  Note that insert
    does not occur in-place: a new array is returned. If
    axis is None, out is a flattened array.

#### SEE ALSO
[`append`](dask.array.append.md#dask.array.append)
: Append elements at the end of an array.

[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)
: Join a sequence of arrays along an existing axis.

[`delete`](dask.array.delete.md#dask.array.delete)
: Delete elements from an array.

### Notes

Note that for higher dimensional inserts `obj=0` behaves very different
from `obj=[0]` just like `arr[:,0,:] = values` is different from
`arr[:,[0],:] = values`. This is because of the difference between basic
and advanced [indexing](https://numpy.org/doc/stable/user/basics.indexing.html#basics-indexing).

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(6).reshape(3, 2)
>>> a
array([[0, 1],
       [2, 3],
       [4, 5]])
>>> np.insert(a, 1, 6)
array([0, 6, 1, 2, 3, 4, 5])
>>> np.insert(a, 1, 6, axis=1)
array([[0, 6, 1],
       [2, 6, 3],
       [4, 6, 5]])
```

Difference between sequence and scalars,
showing how `obj=[1]` behaves different from `obj=1`:

```pycon
>>> np.insert(a, [1], [[7],[8],[9]], axis=1)
array([[0, 7, 1],
       [2, 8, 3],
       [4, 9, 5]])
>>> np.insert(a, 1, [[7],[8],[9]], axis=1)
array([[0, 7, 8, 9, 1],
       [2, 7, 8, 9, 3],
       [4, 7, 8, 9, 5]])
>>> np.array_equal(np.insert(a, 1, [7, 8, 9], axis=1),
...                np.insert(a, [1], [[7],[8],[9]], axis=1))
True
```

```pycon
>>> b = a.flatten()
>>> b
array([0, 1, 2, 3, 4, 5])
>>> np.insert(b, [2, 2], [6, 7])
array([0, 1, 6, 7, 2, 3, 4, 5])
```

```pycon
>>> np.insert(b, slice(2, 4), [7, 8])
array([0, 1, 7, 2, 8, 3, 4, 5])
```

```pycon
>>> np.insert(b, [2, 2], [7.13, False]) # type casting
array([0, 1, 7, 0, 2, 3, 4, 5])
```

```pycon
>>> x = np.arange(8).reshape(2, 4)
>>> idx = (1, 3)
>>> np.insert(x, idx, 999, axis=1)
array([[  0, 999,   1,   2, 999,   3],
       [  4, 999,   5,   6, 999,   7]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.invert.html.md

# dask.array.invert

### dask.array.invert(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'invert'>*

This docstring was copied from numpy.invert.

Some inconsistencies with the Dask version may exist.

Compute bit-wise inversion, or bit-wise NOT, element-wise.

Computes the bit-wise NOT of the underlying binary representation of
the integers in the input arrays. This ufunc implements the C/Python
operator `~`.

For signed integer inputs, the bit-wise NOT of the absolute value is
returned. In a two’s-complement system, this operation effectively flips
all the bits, resulting in a representation that corresponds to the
negative of the input plus one. This is the most common method of
representing signed integers on computers [[1]](#r046109bd3458-1). An N-bit two’s-complement
system can represent every integer in the range $-2^{N-1}$ to
$+2^{N-1}-1$.

* **Parameters:**
  **x**
  : Only integer and boolean types are handled.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Result.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`bitwise_and`](dask.array.bitwise_and.md#dask.array.bitwise_and), [`bitwise_or`](dask.array.bitwise_or.md#dask.array.bitwise_or), [`bitwise_xor`](dask.array.bitwise_xor.md#dask.array.bitwise_xor)

[`logical_not`](dask.array.logical_not.md#dask.array.logical_not)

`binary_repr`
: Return the binary representation of the input number as a string.

### Notes

`numpy.bitwise_not` is an alias for invert:

```pycon
>>> np.bitwise_not is np.invert
True
```

### References

### Examples

```pycon
>>> import numpy as np
```

We’ve seen that 13 is represented by `00001101`.
The invert or bit-wise NOT of 13 is then:

```pycon
>>> x = np.invert(np.array(13, dtype=np.uint8))
>>> x
np.uint8(242)
>>> np.binary_repr(x, width=8)
'11110010'
```

The result depends on the bit-width:

```pycon
>>> x = np.invert(np.array(13, dtype=np.uint16))
>>> x
np.uint16(65522)
>>> np.binary_repr(x, width=16)
'1111111111110010'
```

When using signed integer types, the result is the bit-wise NOT of
the unsigned type, interpreted as a signed integer:

```pycon
>>> np.invert(np.array([13], dtype=np.int8))
array([-14], dtype=int8)
>>> np.binary_repr(-14, width=8)
'11110010'
```

Booleans are accepted as well:

```pycon
>>> np.invert(np.array([True, False]))
array([False,  True])
```

The `~` operator can be used as a shorthand for `np.invert` on
ndarrays.

```pycon
>>> x1 = np.array([True, False])
>>> ~x1
array([False,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isclose.html.md

# dask.array.isclose

### dask.array.isclose(arr1, arr2, rtol=1e-05, atol=1e-08, equal_nan=False)

Returns a boolean array where two arrays are element-wise equal within a
tolerance.

This docstring was copied from numpy.isclose.

Some inconsistencies with the Dask version may exist.

The tolerance values are positive, typically very small numbers.  The
relative difference (rtol \* abs(b)) and the absolute difference
atol are added together to compare against the absolute difference
between a and b.

#### WARNING
The default atol is not appropriate for comparing numbers
with magnitudes much smaller than one (see Notes).

* **Parameters:**
  **a, b**
  : Input arrays to compare.

  **rtol**
  : The relative tolerance parameter (see Notes).

  **atol**
  : The absolute tolerance parameter (see Notes).

  **equal_nan**
  : Whether to compare NaN’s as equal.  If True, NaN’s in a will be
    considered equal to NaN’s in b in the output array.
* **Returns:**
  **y**
  : Returns a boolean array of where a and b are equal within the
    given tolerance. If both a and b are scalars, returns a single
    boolean value.

#### SEE ALSO
[`allclose`](dask.array.allclose.md#dask.array.allclose)

[`math.isclose`](https://docs.python.org/3/library/math.html#math.isclose)

### Notes

For finite values, isclose uses the following equation to test whether
two floating point values are equivalent.:

```default
absolute(a - b) <= (atol + rtol * absolute(b))
```

Unlike the built-in math.isclose, the above equation is not symmetric
in a and b – it assumes b is the reference value – so that
isclose(a, b) might be different from isclose(b, a).

The default value of atol is not appropriate when the reference value
b has magnitude smaller than one. For example, it is unlikely that
`a = 1e-9` and `b = 2e-9` should be considered “close”, yet
`isclose(1e-9, 2e-9)` is `True` with default settings. Be sure
to select atol for the use case at hand, especially for defining the
threshold below which a non-zero value in a will be considered “close”
to a very small or zero value in b.

isclose is not defined for non-numeric data types.
[`bool`](https://docs.python.org/3/library/functions.html#bool) is considered a numeric data-type for this purpose.

### Examples

```pycon
>>> import numpy as np
>>> np.isclose([1e10,1e-7], [1.00001e10,1e-8])
array([ True, False])
```

```pycon
>>> np.isclose([1e10,1e-8], [1.00001e10,1e-9])
array([ True, True])
```

```pycon
>>> np.isclose([1e10,1e-8], [1.0001e10,1e-9])
array([False,  True])
```

```pycon
>>> np.isclose([1.0, np.nan], [1.0, np.nan])
array([ True, False])
```

```pycon
>>> np.isclose([1.0, np.nan], [1.0, np.nan], equal_nan=True)
array([ True, True])
```

```pycon
>>> np.isclose([1e-8, 1e-7], [0.0, 0.0])
array([ True, False])
```

```pycon
>>> np.isclose([1e-100, 1e-7], [0.0, 0.0], atol=0.0)
array([False, False])
```

```pycon
>>> np.isclose([1e-10, 1e-10], [1e-20, 0.0])
array([ True,  True])
```

```pycon
>>> np.isclose([1e-10, 1e-10], [1e-20, 0.999999e-10], atol=0.0)
array([False,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.iscomplex.html.md

# dask.array.iscomplex

### dask.array.iscomplex(\*args, \*\*kwargs)

Returns a bool array, where True if input element is complex.

This docstring was copied from numpy.iscomplex.

Some inconsistencies with the Dask version may exist.

What is tested is whether the input has a non-zero imaginary part, not if
the input type is complex.

* **Parameters:**
  **x**
  : Input array.
* **Returns:**
  **out**
  : Output array.

#### SEE ALSO
[`isreal`](dask.array.isreal.md#dask.array.isreal)

`iscomplexobj`
: Return True if x is a complex type or an array of complex numbers.

### Examples

```pycon
>>> import numpy as np
>>> np.iscomplex([1+1j, 1+0j, 4.5, 3, 2, 2j])
array([ True, False, False, False, False,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isfinite.html.md

# dask.array.isfinite

### dask.array.isfinite(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'isfinite'>*

This docstring was copied from numpy.isfinite.

Some inconsistencies with the Dask version may exist.

Test element-wise for finiteness (not infinity and not Not a Number).

The result is returned as a boolean array.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : True where `x` is not positive infinity, negative infinity,
    or NaN; false otherwise.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`isinf`](dask.array.isinf.md#dask.array.isinf), [`isneginf`](dask.array.isneginf.md#dask.array.isneginf), [`isposinf`](dask.array.isposinf.md#dask.array.isposinf), [`isnan`](dask.array.isnan.md#dask.array.isnan)

### Notes

Not a Number, positive infinity and negative infinity are considered
to be non-finite.

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754). This means that Not a Number is not equivalent to infinity.
Also that positive infinity is not equivalent to negative infinity. But
infinity is equivalent to positive infinity.  Errors result if the
second argument is also supplied when x is a scalar input, or if
first and second arguments have different shapes.

### Examples

```pycon
>>> import numpy as np
>>> np.isfinite(1)
True
>>> np.isfinite(0)
True
>>> np.isfinite(np.nan)
False
>>> np.isfinite(np.inf)
False
>>> np.isfinite(-np.inf)
False
>>> np.isfinite([np.log(-1.),1.,np.log(0)])
array([False,  True, False])
```

```pycon
>>> x = np.array([-np.inf, 0., np.inf])
>>> y = np.array([2, 2, 2])
>>> np.isfinite(x, y)
array([0, 1, 0])
>>> y
array([0, 1, 0])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isin.html.md

# dask.array.isin

### dask.array.isin(element, test_elements, assume_unique=False, invert=False, , kind=None)

Calculates `element in test_elements`, broadcasting over element only.
Returns a boolean array of the same shape as element that is True
where an element of element is in test_elements and False otherwise.

* **Parameters:**
  **element**
  : Input array.

  **test_elements**
  : The values against which to test each value of element.
    This argument is flattened if it is an array or array_like.
    See notes for behavior with non-array-like parameters.

  **assume_unique**
  : If True, the input arrays are both assumed to be unique, which
    can speed up the calculation.  Default is False.

  **invert**
  : If True, the values in the returned array are inverted, as if
    calculating element not in test_elements. Default is False.
    `np.isin(a, b, invert=True)` is equivalent to (but faster
    than) `np.invert(np.isin(a, b))`.

  **kind**
  : The algorithm to use. This will not affect the final result,
    but will affect the speed and memory use. The default, None,
    will select automatically based on memory considerations.
    * If ‘sort’, will use a mergesort-based approach. This will have
      a memory usage of roughly 6 times the sum of the sizes of
      element and test_elements, not accounting for size of dtypes.
    * If ‘table’, will use a lookup table approach similar
      to a counting sort. This is only available for boolean and
      integer arrays. This will have a memory usage of the
      size of element plus the max-min value of test_elements.
      assume_unique has no effect when the ‘table’ option is used.
    * If None, will automatically choose ‘table’ if
      the required memory allocation is less than or equal to
      6 times the sum of the sizes of element and test_elements,
      otherwise will use ‘sort’. This is done to not use
      a large amount of memory by default, even though
      ‘table’ may be faster in most cases. If ‘table’ is chosen,
      assume_unique will have no effect.
* **Returns:**
  **isin**
  : Has the same shape as element. The values element[isin]
    are in test_elements.

### Notes

isin is an element-wise function version of the python keyword in.
`isin(a, b)` is roughly equivalent to
`np.array([item in b for item in a])` if a and b are 1-D sequences.

element and test_elements are converted to arrays if they are not
already. If test_elements is a set (or other non-sequence collection)
it will be converted to an object array with one element, rather than an
array of the values contained in test_elements. This is a consequence
of the array constructor’s way of handling non-sequence collections.
Converting the set to a list usually gives the desired behavior.

Using `kind='table'` tends to be faster than kind=’sort’ if the
following relationship is true:
`log10(len(test_elements)) >
(log10(max(test_elements)-min(test_elements)) - 2.27) / 0.927`,
but may use greater memory. The default value for kind will
be automatically selected based only on memory usage, so one may
manually set `kind='table'` if memory constraints can be relaxed.

### Examples

```pycon
>>> import numpy as np
>>> element = 2*np.arange(4).reshape((2, 2))
>>> element
array([[0, 2],
       [4, 6]])
>>> test_elements = [1, 2, 4, 8]
>>> mask = np.isin(element, test_elements)
>>> mask
array([[False,  True],
       [ True, False]])
>>> element[mask]
array([2, 4])
```

The indices of the matched values can be obtained with nonzero:

```pycon
>>> np.nonzero(mask)
(array([0, 1]), array([1, 0]))
```

The test can also be inverted:

```pycon
>>> mask = np.isin(element, test_elements, invert=True)
>>> mask
array([[ True, False],
       [False,  True]])
>>> element[mask]
array([0, 6])
```

Because of how array handles sets, the following does not
work as expected:

```pycon
>>> test_set = {1, 2, 4, 8}
>>> np.isin(element, test_set)
array([[False, False],
       [False, False]])
```

Casting the set to a list gives the expected result:

```pycon
>>> np.isin(element, list(test_set))
array([[False,  True],
       [ True, False]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isinf.html.md

# dask.array.isinf

### dask.array.isinf(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'isinf'>*

This docstring was copied from numpy.isinf.

Some inconsistencies with the Dask version may exist.

Test element-wise for positive or negative infinity.

Returns a boolean array of the same shape as x, True where `x ==
+/-inf`, otherwise False.

* **Parameters:**
  **x**
  : Input values

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : True where `x` is positive or negative infinity, false otherwise.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`isneginf`](dask.array.isneginf.md#dask.array.isneginf), [`isposinf`](dask.array.isposinf.md#dask.array.isposinf), [`isnan`](dask.array.isnan.md#dask.array.isnan), [`isfinite`](dask.array.isfinite.md#dask.array.isfinite)

### Notes

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754).

Errors result if the second argument is supplied when the first
argument is a scalar, or if the first and second arguments have
different shapes.

### Examples

```pycon
>>> import numpy as np
>>> np.isinf(np.inf)
True
>>> np.isinf(np.nan)
False
>>> np.isinf(-np.inf)
True
>>> np.isinf([np.inf, -np.inf, 1.0, np.nan])
array([ True,  True, False, False])
```

```pycon
>>> x = np.array([-np.inf, 0., np.inf])
>>> y = np.array([2, 2, 2])
>>> np.isinf(x, y)
array([1, 0, 1])
>>> y
array([1, 0, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isnan.html.md

# dask.array.isnan

### dask.array.isnan(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'isnan'>*

This docstring was copied from numpy.isnan.

Some inconsistencies with the Dask version may exist.

Test element-wise for NaN and return result as a boolean array.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : True where `x` is NaN, false otherwise.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`isinf`](dask.array.isinf.md#dask.array.isinf), [`isneginf`](dask.array.isneginf.md#dask.array.isneginf), [`isposinf`](dask.array.isposinf.md#dask.array.isposinf), [`isfinite`](dask.array.isfinite.md#dask.array.isfinite), `isnat`

### Notes

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754). This means that Not a Number is not equivalent to infinity.

### Examples

```pycon
>>> import numpy as np
>>> np.isnan(np.nan)
True
>>> np.isnan(np.inf)
False
>>> np.isnan([np.log(-1.),1.,np.log(0)])
array([ True, False, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isneginf.html.md

# dask.array.isneginf

### dask.array.isneginf *= functools.partial(<ufunc 'equal'>, -inf)*

This docstring was copied from numpy.equal.

Some inconsistencies with the Dask version may exist.

Return (x1 == x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`not_equal`](dask.array.not_equal.md#dask.array.not_equal), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`greater`](dask.array.greater.md#dask.array.greater), [`less`](dask.array.less.md#dask.array.less)

### Examples

```pycon
>>> import numpy as np
>>> np.equal([0, 1, 3], np.arange(3))
array([ True,  True, False])
```

What is compared are values, not types. So an int (1) and an array of
length one can evaluate as True:

```pycon
>>> np.equal(1, np.ones(1))
array([ True])
```

The `==` operator can be used as a shorthand for `np.equal` on
ndarrays.

```pycon
>>> a = np.array([2, 4, 6])
>>> b = np.array([2, 4, 2])
>>> a == b
array([ True,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isnull.html.md

# dask.array.isnull

### dask.array.isnull(values)

pandas.isnull for dask arrays

<!-- !! processed by numpydoc !! -->


# dask.array.isposinf.html.md

# dask.array.isposinf

### dask.array.isposinf *= functools.partial(<ufunc 'equal'>, inf)*

This docstring was copied from numpy.equal.

Some inconsistencies with the Dask version may exist.

Return (x1 == x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`not_equal`](dask.array.not_equal.md#dask.array.not_equal), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`greater`](dask.array.greater.md#dask.array.greater), [`less`](dask.array.less.md#dask.array.less)

### Examples

```pycon
>>> import numpy as np
>>> np.equal([0, 1, 3], np.arange(3))
array([ True,  True, False])
```

What is compared are values, not types. So an int (1) and an array of
length one can evaluate as True:

```pycon
>>> np.equal(1, np.ones(1))
array([ True])
```

The `==` operator can be used as a shorthand for `np.equal` on
ndarrays.

```pycon
>>> a = np.array([2, 4, 6])
>>> b = np.array([2, 4, 2])
>>> a == b
array([ True,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.isreal.html.md

# dask.array.isreal

### dask.array.isreal(\*args, \*\*kwargs)

Returns a bool array, where True if input element is real.

This docstring was copied from numpy.isreal.

Some inconsistencies with the Dask version may exist.

If element has complex type with zero imaginary part, the return value
for that element is True.

* **Parameters:**
  **x**
  : Input array.
* **Returns:**
  **out**
  : Boolean array of same shape as x.

#### SEE ALSO
[`iscomplex`](dask.array.iscomplex.md#dask.array.iscomplex)

`isrealobj`
: Return True if x is not a complex type.

### Notes

isreal may behave unexpectedly for string or object arrays (see examples)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1+1j, 1+0j, 4.5, 3, 2, 2j], dtype=np.complex128)
>>> np.isreal(a)
array([False,  True,  True,  True,  True, False])
```

The function does not work on string arrays.

```pycon
>>> a = np.array([2j, "a"], dtype=np.str_)
>>> np.isreal(a)  # returns the result of `"" == 0` currently.
array([False, False])
```

Returns True for all elements that either have no `.imag` attribute
or for which that attribute is zero:

```pycon
>>> a = np.array([1, "2", 3+4j], dtype=np.object_)
>>> np.isreal(a)
array([ True,  True,  False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ldexp.html.md

# dask.array.ldexp

### dask.array.ldexp(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'ldexp'>*

This docstring was copied from numpy.ldexp.

Some inconsistencies with the Dask version may exist.

Returns x1 \* 2\*\*x2, element-wise.

The mantissas x1 and twos exponents x2 are used to construct
floating point numbers `x1 * 2**x2`.

* **Parameters:**
  **x1**
  : Array of multipliers.

  **x2**
  : Array of twos exponents.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The result of `x1 * 2**x2`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`frexp`](dask.array.frexp.md#dask.array.frexp)
: Return (y1, y2) from `x = y1 * 2**y2`, inverse to ldexp.

### Notes

Complex dtypes are not supported, they will raise a TypeError.

ldexp is useful as the inverse of frexp, if used by itself it is
more clear to simply use the expression `x1 * 2**x2`.

### Examples

```pycon
>>> import numpy as np
>>> np.ldexp(5, np.arange(4))
array([ 5., 10., 20., 40.], dtype=float16)
```

```pycon
>>> x = np.arange(6)
>>> np.ldexp(*np.frexp(x))
array([ 0.,  1.,  2.,  3.,  4.,  5.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.left_shift.html.md

# dask.array.left_shift

### dask.array.left_shift(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'left_shift'>*

This docstring was copied from numpy.left_shift.

Some inconsistencies with the Dask version may exist.

Shift the bits of an integer to the left.

Bits are shifted to the left by appending x2 0s at the right of x1.
Since the internal representation of numbers is in binary format, this
operation is equivalent to multiplying x1 by `2**x2`.

* **Parameters:**
  **x1**
  : Input values.

  **x2**
  : Number of zeros to append to x1. Has to be non-negative.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Return x1 with bits shifted x2 times to the left.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`right_shift`](dask.array.right_shift.md#dask.array.right_shift)
: Shift the bits of an integer to the right.

`binary_repr`
: Return the binary representation of the input number as a string.

### Examples

```pycon
>>> import numpy as np
>>> np.binary_repr(5)
'101'
>>> np.left_shift(5, 2)
20
>>> np.binary_repr(20)
'10100'
```

```pycon
>>> np.left_shift(5, [1,2,3])
array([10, 20, 40])
```

Note that the dtype of the second argument may change the dtype of the
result and can lead to unexpected results in some cases (see
[Casting Rules](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-casting)):

```pycon
>>> a = np.left_shift(np.uint8(255), np.int64(1))  # Expect 254
>>> print(a, type(a)) # Unexpected result due to upcasting
510 <class 'numpy.int64'>
>>> b = np.left_shift(np.uint8(255), np.uint8(1))
>>> print(b, type(b))
254 <class 'numpy.uint8'>
```

The `<<` operator can be used as a shorthand for `np.left_shift` on
ndarrays.

```pycon
>>> x1 = 5
>>> x2 = np.array([1, 2, 3])
>>> x1 << x2
array([10, 20, 40])
```

<!-- !! processed by numpydoc !! -->


# dask.array.less.html.md

# dask.array.less

### dask.array.less(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'less'>*

This docstring was copied from numpy.less.

Some inconsistencies with the Dask version may exist.

Return the truth value of (x1 < x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`greater`](dask.array.greater.md#dask.array.greater), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`equal`](dask.array.equal.md#dask.array.equal), [`not_equal`](dask.array.not_equal.md#dask.array.not_equal)

### Examples

```pycon
>>> import numpy as np
>>> np.less([1, 2], [2, 2])
array([ True, False])
```

The `<` operator can be used as a shorthand for `np.less` on ndarrays.

```pycon
>>> a = np.array([1, 2])
>>> b = np.array([2, 2])
>>> a < b
array([ True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.less_equal.html.md

# dask.array.less_equal

### dask.array.less_equal(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'less_equal'>*

This docstring was copied from numpy.less_equal.

Some inconsistencies with the Dask version may exist.

Return the truth value of (x1 <= x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`greater`](dask.array.greater.md#dask.array.greater), [`less`](dask.array.less.md#dask.array.less), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`equal`](dask.array.equal.md#dask.array.equal), [`not_equal`](dask.array.not_equal.md#dask.array.not_equal)

### Examples

```pycon
>>> import numpy as np
>>> np.less_equal([4, 2, 1], [2, 2, 2])
array([False,  True,  True])
```

The `<=` operator can be used as a shorthand for `np.less_equal` on
ndarrays.

```pycon
>>> a = np.array([4, 2, 1])
>>> b = np.array([2, 2, 2])
>>> a <= b
array([False,  True,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.lib.stride_tricks.sliding_window_view.html.md

# dask.array.lib.stride_tricks.sliding_window_view

### dask.array.lib.stride_tricks.sliding_window_view(x, window_shape, axis=None, automatic_rechunk=True)

Create a sliding window view into the array with the given window shape.

This docstring was copied from numpy.lib.stride_tricks.sliding_window_view.

Some inconsistencies with the Dask version may exist.

Also known as rolling or moving window, the window slides across all
dimensions of the array and extracts subsets of the array at all window
positions.

#### Versionadded
Added in version 1.20.0.

* **Parameters:**
  **x**
  : Array to create the sliding window view from.

  **window_shape**
  : Size of window over each axis that takes part in the sliding window.
    If axis is not present, must have same length as the number of input
    array dimensions. Single integers i are treated as if they were the
    tuple (i,).

  **axis**
  : Axis or axes along which the sliding window is applied.
    By default, the sliding window is applied to all axes and
    window_shape[i] will refer to axis i of x.
    If axis is given as a tuple of int, window_shape[i] will refer to
    the axis axis[i] of x.
    Single integers i are treated as if they were the tuple (i,).

  **subok**
  : If True, sub-classes will be passed-through, otherwise the returned
    array will be forced to be a base-class array (default).

  **writeable**
  : When true, allow writing to the returned view. The default is false,
    as this should be used with caution: the returned view contains the
    same memory location multiple times, so writing to one location will
    cause others to change.
* **Returns:**
  **view**
  : Sliding window view of the array. The sliding window dimensions are
    inserted at the end, and the original dimensions are trimmed as
    required by the size of the sliding window.
    That is, `view.shape = x_shape_trimmed + window_shape`, where
    `x_shape_trimmed` is `x.shape` with every entry reduced by one less
    than the corresponding window size.

#### SEE ALSO
`lib.stride_tricks.as_strided`
: A lower-level and less safe routine for creating arbitrary views from custom shape and strides. Use the `check_bounds` parameter for bounds validation.

`broadcast_to`
: broadcast an array to a given shape.

### Notes

#### WARNING
This function creates views with overlapping memory. When
`writeable=True`, writing to the view will modify the original array
and may affect multiple view positions. See the examples below and
this guide
about the difference between copies and views.

For many applications using a sliding window view can be convenient, but
potentially very slow. Often specialized solutions exist, for example:

- scipy.signal.fftconvolve
- filtering functions in scipy.ndimage
- moving window functions provided by
  [bottleneck](https://github.com/pydata/bottleneck).

As a rough estimate, a sliding window approach with an input size of N
and a window size of W will scale as O(N\*W) where frequently a special
algorithm can achieve O(N). That means that the sliding window variant
for a window size of 100 can be a 100 times slower than a more specialized
version.

Nevertheless, for small window sizes, when no custom algorithm exists, or
as a prototyping and developing tool, this function can be a good solution.

### Examples

```pycon
>>> import numpy as np
>>> from numpy.lib.stride_tricks import sliding_window_view
>>> x = np.arange(6)
>>> x.shape
(6,)
>>> v = sliding_window_view(x, 3)
>>> v.shape
(4, 3)
>>> v
array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4],
       [3, 4, 5]])
```

This also works in more dimensions, e.g.

```pycon
>>> i, j = np.ogrid[:3, :4]
>>> x = 10*i + j
>>> x.shape
(3, 4)
>>> x
array([[ 0,  1,  2,  3],
       [10, 11, 12, 13],
       [20, 21, 22, 23]])
>>> shape = (2,2)
>>> v = sliding_window_view(x, shape)
>>> v.shape
(2, 3, 2, 2)
>>> v
array([[[[ 0,  1],
         [10, 11]],
        [[ 1,  2],
         [11, 12]],
        [[ 2,  3],
         [12, 13]]],
       [[[10, 11],
         [20, 21]],
        [[11, 12],
         [21, 22]],
        [[12, 13],
         [22, 23]]]])
```

The axis can be specified explicitly:

```pycon
>>> v = sliding_window_view(x, 3, 0)
>>> v.shape
(1, 4, 3)
>>> v
array([[[ 0, 10, 20],
        [ 1, 11, 21],
        [ 2, 12, 22],
        [ 3, 13, 23]]])
```

The same axis can be used several times. In that case, every use reduces
the corresponding original dimension:

```pycon
>>> v = sliding_window_view(x, (2, 3), (1, 1))
>>> v.shape
(3, 1, 2, 3)
>>> v
array([[[[ 0,  1,  2],
         [ 1,  2,  3]]],
       [[[10, 11, 12],
         [11, 12, 13]]],
       [[[20, 21, 22],
         [21, 22, 23]]]])
```

Combining with stepped slicing (::step), this can be used to take sliding
views which skip elements:

```pycon
>>> x = np.arange(7)
>>> sliding_window_view(x, 5)[:, ::2]
array([[0, 2, 4],
       [1, 3, 5],
       [2, 4, 6]])
```

or views which move by multiple elements

```pycon
>>> x = np.arange(7)
>>> sliding_window_view(x, 3)[::2, :]
array([[0, 1, 2],
       [2, 3, 4],
       [4, 5, 6]])
```

A common application of sliding_window_view is the calculation of running
statistics. The simplest example is the
[moving average](https://en.wikipedia.org/wiki/Moving_average):

```pycon
>>> x = np.arange(6)
>>> x.shape
(6,)
>>> v = sliding_window_view(x, 3)
>>> v.shape
(4, 3)
>>> v
array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4],
       [3, 4, 5]])
>>> moving_average = v.mean(axis=-1)
>>> moving_average
array([1., 2., 3., 4.])
```

To adjust the step size of the sliding window, index the output view along
the desired dimension(s). Using the array shown above:

```pycon
>>> v[::2]
array([[0, 1, 2],
       [2, 3, 4]])
```

You can slide in the reverse direction using the same technique:

```pycon
>>> v[::-1]
array([[3, 4, 5],
       [2, 3, 4],
       [1, 2, 3],
       [0, 1, 2]])
```

The two examples below demonstrate the effect of `writeable=True`.

Creating a view with the default `writeable=False` and then writing to
it raises an error.

```pycon
>>> v = sliding_window_view(x, 3)
>>> v[0,1] = 10
Traceback (most recent call last):
...
ValueError: assignment destination is read-only
```

Creating a view with `writeable=True` and then writing to it changes
the original array and multiple view positions.

```pycon
>>> x = np.arange(6)  # reset x for the second example
>>> v = sliding_window_view(x, 3, writeable=True)
>>> v[0,1] = 10
>>> x
array([ 0, 10,  2,  3,  4,  5])
>>> v
array([[ 0, 10,  2],
       [10,  2,  3],
       [ 2,  3,  4],
       [ 3,  4,  5]])
```

Note that a sliding window approach is often **not** optimal (see Notes).

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.cholesky.html.md

# dask.array.linalg.cholesky

### dask.array.linalg.cholesky(a, lower=False)

Returns the Cholesky decomposition, $A = L L^*$ or
$A = U^* U$ of a Hermitian positive-definite matrix A.

* **Parameters:**
  **a**
  : Matrix to be decomposed

  **lower**
  : Whether to compute the upper or lower triangular Cholesky
    factorization.  Default is upper-triangular.
* **Returns:**
  **c**
  : Upper- or lower-triangular Cholesky factor of a.

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.inv.html.md

# dask.array.linalg.inv

### dask.array.linalg.inv(a)

Compute the inverse of a matrix with LU decomposition and
forward / backward substitutions.

* **Parameters:**
  **a**
  : Square matrix to be inverted.
* **Returns:**
  **ainv**
  : Inverse of the matrix a.

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.lstsq.html.md

# dask.array.linalg.lstsq

### dask.array.linalg.lstsq(a, b)

Return the least-squares solution to a linear matrix equation using
QR decomposition.

Solves the equation a x = b by computing a vector x that
minimizes the Euclidean 2-norm || b - a x ||^2.  The equation may
be under-, well-, or over- determined (i.e., the number of
linearly independent rows of a can be less than, equal to, or
greater than its number of linearly independent columns).  If a
is square and of full rank, then x (but for round-off error) is
the “exact” solution of the equation.

* **Parameters:**
  **a**
  : “Coefficient” matrix.

  **b**
  : Ordinate or “dependent variable” values. If b is two-dimensional,
    the least-squares solution is calculated for each of the K columns
    of b.
* **Returns:**
  **x**
  : Least-squares solution. If b is two-dimensional,
    the solutions are in the K columns of x.

  **residuals**
  : Sums of residuals; squared Euclidean 2-norm for each column in
    `b - a*x`.
    If b is 1-dimensional, this is a (1,) shape array.
    Otherwise the shape is (K,).

  **rank**
  : Rank of matrix a.

  **s**
  : Singular values of a.

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.lu.html.md

# dask.array.linalg.lu

### dask.array.linalg.lu(a)

Compute the lu decomposition of a matrix.

* **Returns:**
  p:  Array, permutation matrix

  l:  Array, lower triangular matrix with unit diagonal.

  u:  Array, upper triangular matrix

### Examples

```pycon
>>> p, l, u = da.linalg.lu(x)
```

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.norm.html.md

# dask.array.linalg.norm

### dask.array.linalg.norm(x, ord=None, axis=None, keepdims=False)

Matrix or vector norm.

This docstring was copied from numpy.linalg.norm.

Some inconsistencies with the Dask version may exist.

This function is able to return one of eight different matrix norms,
or one of an infinite number of vector norms (described below), depending
on the value of the `ord` parameter.

* **Parameters:**
  **x**
  : Input array.  If axis is None, x must be 1-D or 2-D, unless ord
    is None. If both axis and ord are None, the 2-norm of
    `x.ravel` will be returned.

  **ord**
  : Order of the norm (see table under `Notes` for what values are
    supported for matrices and vectors respectively). inf means numpy’s
    inf object. The default is None.

  **axis**
  : If axis is an integer, it specifies the axis of x along which to
    compute the vector norms.  If axis is a 2-tuple, it specifies the
    axes that hold 2-D matrices, and the matrix norms of these matrices
    are computed.  If axis is None then either a vector norm (when x
    is 1-D) or a matrix norm (when x is 2-D) is returned. The default
    is None.

  **keepdims**
  : If this is set to True, the axes which are normed over are left in the
    result as dimensions with size one.  With this option the result will
    broadcast correctly against the original x.
* **Returns:**
  **n**
  : Norm of the matrix or vector(s).

#### SEE ALSO
[`scipy.linalg.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html#scipy.linalg.norm)
: Similar function in SciPy.

### Notes

For values of `ord < 1`, the result is, strictly speaking, not a
mathematical ‘norm’, but it may still be useful for various numerical
purposes.

The following norms can be calculated:

| ord   | norm for matrices            | norm for vectors               |
|-------|------------------------------|--------------------------------|
| None  | Frobenius norm               | 2-norm                         |
| ‘fro’ | Frobenius norm               | –                              |
| ‘nuc’ | nuclear norm                 | –                              |
| inf   | max(sum(abs(x), axis=1))     | max(abs(x))                    |
| -inf  | min(sum(abs(x), axis=1))     | min(abs(x))                    |
| 0     | –                            | sum(x != 0)                    |
| 1     | max(sum(abs(x), axis=0))     | as below                       |
| -1    | min(sum(abs(x), axis=0))     | as below                       |
| 2     | 2-norm (largest sing. value) | as below                       |
| -2    | smallest singular value      | as below                       |
| other | –                            | sum(abs(x)\*\*ord)\*\*(1./ord) |

The Frobenius norm is given by [[1]](#r1bae488c141b-1):

$||A||_F = [\sum_{i,j} abs(a_{i,j})^2]^{1/2}$

The nuclear norm is the sum of the singular values.

Both the Frobenius and nuclear norm orders are only defined for
matrices and raise a ValueError when `x.ndim != 2`.

### References

### Examples

```pycon
>>> import numpy as np
>>> from numpy import linalg as LA
>>> a = np.arange(9) - 4
>>> a
array([-4, -3, -2, ...,  2,  3,  4])
>>> b = a.reshape((3, 3))
>>> b
array([[-4, -3, -2],
       [-1,  0,  1],
       [ 2,  3,  4]])
```

```pycon
>>> LA.norm(a)
7.745966692414834
>>> LA.norm(b)
7.745966692414834
>>> LA.norm(b, 'fro')
7.745966692414834
>>> LA.norm(a, np.inf)
4.0
>>> LA.norm(b, np.inf)
9.0
>>> LA.norm(a, -np.inf)
0.0
>>> LA.norm(b, -np.inf)
2.0
```

```pycon
>>> LA.norm(a, 1)
20.0
>>> LA.norm(b, 1)
7.0
>>> LA.norm(a, -1)
-4.6566128774142013e-010
>>> LA.norm(b, -1)
6.0
>>> LA.norm(a, 2)
7.745966692414834
>>> LA.norm(b, 2)
7.3484692283495345
```

```pycon
>>> LA.norm(a, -2)
0.0
>>> LA.norm(b, -2)
1.8570331885190563e-016 # may vary
>>> LA.norm(a, 3)
5.8480354764257312 # may vary
>>> LA.norm(a, -3)
0.0
```

Using the axis argument to compute vector norms:

```pycon
>>> c = np.array([[ 1, 2, 3],
...               [-1, 1, 4]])
>>> LA.norm(c, axis=0)
array([ 1.41421356,  2.23606798,  5.        ])
>>> LA.norm(c, axis=1)
array([ 3.74165739,  4.24264069])
>>> LA.norm(c, ord=1, axis=1)
array([ 6.,  6.])
```

Using the axis argument to compute matrix norms:

```pycon
>>> m = np.arange(8).reshape(2,2,2)
>>> LA.norm(m, axis=(1,2))
array([  3.74165739,  11.22497216])
>>> LA.norm(m[0, :, :]), LA.norm(m[1, :, :])
(3.7416573867739413, 11.224972160321824)
```

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.qr.html.md

# dask.array.linalg.qr

### dask.array.linalg.qr(a)

Compute the qr factorization of a matrix.

* **Parameters:**
  **a**
* **Returns:**
  q:  Array, orthonormal

  r:  Array, upper-triangular

#### SEE ALSO
[`numpy.linalg.qr`](https://numpy.org/doc/stable/reference/generated/numpy.linalg.qr.html#numpy.linalg.qr)
: Equivalent NumPy Operation

[`dask.array.linalg.tsqr`](dask.array.linalg.tsqr.md#dask.array.linalg.tsqr)
: Implementation for tall-and-skinny arrays

[`dask.array.linalg.sfqr`](dask.array.linalg.sfqr.md#dask.array.linalg.sfqr)
: Implementation for short-and-fat arrays

### Examples

```pycon
>>> q, r = da.linalg.qr(x)
```

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.sfqr.html.md

# dask.array.linalg.sfqr

### dask.array.linalg.sfqr(data, name=None)

Direct Short-and-Fat QR

Currently, this is a quick hack for non-tall-and-skinny matrices which
are one chunk tall and (unless they are one chunk wide) have chunks
that are wider than they are tall

Q [R_1 R_2 …] = [A_1 A_2 …]

it computes the factorization Q R_1 = A_1, then computes the other
R_k’s in parallel.

* **Parameters:**
  **data: Array**

#### SEE ALSO
[`dask.array.linalg.qr`](dask.array.linalg.qr.md#dask.array.linalg.qr)
: Main user API that uses this function

[`dask.array.linalg.tsqr`](dask.array.linalg.tsqr.md#dask.array.linalg.tsqr)
: Variant for tall-and-skinny case

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.solve.html.md

# dask.array.linalg.solve

### dask.array.linalg.solve(a, b, sym_pos=None, assume_a='gen')

Solve the equation `a x = b` for `x`. By default, use LU
decomposition and forward / backward substitutions. When `assume_a = "pos"`
use Cholesky decomposition.

* **Parameters:**
  **a**
  : A square matrix.

  **b**
  : Right-hand side matrix in `a x = b`.

  **sym_pos**
  : Assume a is symmetric and positive definite. If `True`, use Cholesky
    decomposition.
    <br/>
    #### NOTE
    `sym_pos` is deprecated and will be removed in a future version.
    Use `assume_a = 'pos'` instead.

  **assume_a**
  : Type of data matrix. It is used to choose the dedicated solver.
    Note that Dask does not support ‘her’ and ‘sym’ types.
    <br/>
    #### Versionchanged
    Changed in version 2022.8.0: `assume_a = 'pos'` was previously defined as `sym_pos = True`.
* **Returns:**
  **x**
  : Solution to the system `a x = b`.  Shape of the return matches the
    shape of b.

#### SEE ALSO
[`scipy.linalg.solve`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.solve.html#scipy.linalg.solve)

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.solve_triangular.html.md

# dask.array.linalg.solve_triangular

### dask.array.linalg.solve_triangular(a, b, lower=False)

Solve the equation a x = b for x, assuming a is a triangular matrix.

* **Parameters:**
  **a**
  : A triangular matrix

  **b**
  : Right-hand side matrix in a x = b

  **lower**
  : Use only data contained in the lower triangle of a.
    Default is to use upper triangle.
* **Returns:**
  **x**
  : Solution to the system a x = b. Shape of return matches b.

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.svd.html.md

# dask.array.linalg.svd

### dask.array.linalg.svd(a, coerce_signs=True, full_matrices=False)

Compute the singular value decomposition of a matrix.

* **Parameters:**
  **a**

  **coerce_signs**
  : Whether or not to apply sign coercion to singular vectors in
    order to maintain deterministic results, by default True.

  **full_matrices**
  : If True, raises `NotImplementedError`. Only `full_matrices=False`
    (reduced SVD) is currently supported. Default is `False`.
    <br/>
    #### Versionadded
    Added in version 2026.3.0.
* **Returns:**
  **u**
  : Left-singular vectors of a (in columns) with shape (M, K)
    where K = min(M, N).

  **s**
  : Singular values of a.

  **v**
  : Right-singular vectors of a (in rows) with shape (K, N)
    where K = min(M, N).

#### WARNING
SVD is only supported for arrays with chunking in one dimension.
This requires that all inputs either contain a single column
of chunks (tall-and-skinny) or a single row of chunks (short-and-fat).
For arrays with chunking in both dimensions, see da.linalg.svd_compressed.

#### SEE ALSO
`np.linalg.svd`
: Equivalent NumPy Operation

`da.linalg.svd_compressed`
: Randomized SVD for fully chunked arrays

[`dask.array.linalg.tsqr`](dask.array.linalg.tsqr.md#dask.array.linalg.tsqr)
: QR factorization for tall-and-skinny arrays

`dask.array.utils.svd_flip`
: Sign normalization for singular vectors

### Examples

```pycon
>>> u, s, v = da.linalg.svd(x)
```

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.svd_compressed.html.md

# dask.array.linalg.svd_compressed

### dask.array.linalg.svd_compressed(a, k, iterator='power', n_power_iter=0, n_oversamples=10, seed=None, compute=False, coerce_signs=True)

Randomly compressed rank-k thin Singular Value Decomposition.

This computes the approximate singular value decomposition of a large
array.  This algorithm is generally faster than the normal algorithm
but does not provide exact results.  One can balance between
performance and accuracy with input parameters (see below).

* **Parameters:**
  **a: Array**
  : Input array

  **k: int**
  : Rank of the desired thin SVD decomposition.

  **iterator: {‘power’, ‘QR’}, default=’power’**
  : Define the technique used for iterations to cope with flat
    singular spectra or when the input matrix is very large.

  **n_power_iter: int, default=0**
  : Number of power iterations, useful when the singular values
    decay slowly. Error decreases exponentially as n_power_iter
    increases. In practice, set n_power_iter <= 4.

  **n_oversamples: int, default=10**
  : Number of oversamples used for generating the sampling matrix.
    This value increases the size of the subspace computed, which is more
    accurate at the cost of efficiency.  Results are rarely sensitive to this choice
    though and in practice a value of 10 is very commonly high enough.

  **compute**
  : Whether or not to compute data at each use.
    Recomputing the input while performing several passes reduces memory
    pressure, but means that we have to compute the input multiple times.
    This is a good choice if the data is larger than memory and cheap to
    recreate.

  **coerce_signs**
  : Whether or not to apply sign coercion to singular vectors in
    order to maintain deterministic results, by default True.
* **Returns:**
  u:  Array, unitary / orthogonal

  s:  Array, singular values in decreasing order (largest first)

  v:  Array, unitary / orthogonal

### References

N. Halko, P. G. Martinsson, and J. A. Tropp.
Finding structure with randomness: Probabilistic algorithms for
constructing approximate matrix decompositions.
SIAM Rev., Survey and Review section, Vol. 53, num. 2,
pp. 217-288, June 2011
[https://arxiv.org/abs/0909.4061](https://arxiv.org/abs/0909.4061)

### Examples

```pycon
>>> u, s, v = svd_compressed(x, 20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.linalg.tsqr.html.md

# dask.array.linalg.tsqr

### dask.array.linalg.tsqr(data, compute_svd=False, \_max_vchunk_size=None)

Direct Tall-and-Skinny QR algorithm

As presented in:

> A. Benson, D. Gleich, and J. Demmel.
> Direct QR factorizations for tall-and-skinny matrices in
> MapReduce architectures.
> IEEE International Conference on Big Data, 2013.
> [https://arxiv.org/abs/1301.1071](https://arxiv.org/abs/1301.1071)

This algorithm is used to compute both the QR decomposition and the
Singular Value Decomposition.  It requires that the input array have a
single column of blocks, each of which fit in memory.

* **Parameters:**
  **data: Array**

  **compute_svd: bool**
  : Whether to compute the SVD rather than the QR decomposition

  **\_max_vchunk_size: Integer**
  : Used internally in recursion to set the maximum row dimension
    of chunks in subsequent recursive calls.

#### SEE ALSO
[`dask.array.linalg.qr`](dask.array.linalg.qr.md#dask.array.linalg.qr)
: Powered by this algorithm

[`dask.array.linalg.svd`](dask.array.linalg.svd.md#dask.array.linalg.svd)
: Powered by this algorithm

[`dask.array.linalg.sfqr`](dask.array.linalg.sfqr.md#dask.array.linalg.sfqr)
: Variant for short-and-fat arrays

### Notes

With `k` blocks of size `(m, n)`, this algorithm has memory use that
scales as `k * n * n`.

The implementation here is the recursive variant due to the ultimate
need for one “single core” QR decomposition. In the non-recursive version
of the algorithm, given `k` blocks, after `k` `m * n` QR
decompositions, there will be a “single core” QR decomposition that will
have to work with a `(k * n, n)` matrix.

Here, recursion is applied as necessary to ensure that `k * n` is not
larger than `m` (if `m / n >= 2`). In particular, this is done
to ensure that single core computations do not have to work on blocks
larger than `(m, n)`.

Where blocks are irregular, the above logic is applied with the “height” of
the “tallest” block used in place of `m`.

Consider use of the `rechunk` method to control this behavior.
Taller blocks will reduce overall memory use (assuming that many of them
still fit in memory at once).

<!-- !! processed by numpydoc !! -->


# dask.array.linspace.html.md

# dask.array.linspace

### dask.array.linspace(start, stop, num=50, endpoint=True, retstep=False, chunks='auto', dtype=None)

Return num evenly spaced values over the closed interval [start,
stop].

* **Parameters:**
  **start**
  : The starting value of the sequence.

  **stop**
  : The last value of the sequence.

  **num**
  : Number of samples to include in the returned dask array, including the
    endpoints. Default is 50.

  **endpoint**
  : If True, `stop` is the last sample. Otherwise, it is not included.
    Default is True.

  **retstep**
  : If True, return (samples, step), where step is the spacing between
    samples. Default is False.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if num % blocksize != 0

  **dtype**
  : The type of the output array.
* **Returns:**
  **samples**

  **step**
  : Only returned if `retstep` is True. Size of spacing between samples.

#### SEE ALSO
[`dask.array.arange`](dask.array.arange.md#dask.array.arange)

<!-- !! processed by numpydoc !! -->


# dask.array.log.html.md

# dask.array.log

### dask.array.log(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'log'>*

This docstring was copied from numpy.log.

Some inconsistencies with the Dask version may exist.

Natural logarithm, element-wise.

The natural logarithm log is the inverse of the exponential function,
so that log(exp(x)) = x. The natural logarithm is logarithm in base
e.

* **Parameters:**
  **x**
  : Input value.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The natural logarithm of x, element-wise.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`log10`](dask.array.log10.md#dask.array.log10), [`log2`](dask.array.log2.md#dask.array.log2), [`log1p`](dask.array.log1p.md#dask.array.log1p), `emath.log`

### Notes

Logarithm is a multivalued function: for each x there is an infinite
number of z such that exp(z) = x. The convention is to return the
z whose imaginary part lies in (-pi, pi].

For real-valued input data types, log always returns real output. For
each value that cannot be expressed as a real number or infinity, it
yields `nan` and sets the invalid floating point error flag.

For complex-valued input, log is a complex analytical function that
has a branch cut [-inf, 0] and is continuous from above on it. log
handles the floating-point negative zero as an infinitesimal negative
number, conforming to the C99 standard.

In the cases where the input has a negative real part and a very small
negative complex part (approaching 0), the result is so close to -pi
that it evaluates to exactly -pi.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.log([1, np.e, np.e**2, 0])
array([  0.,   1.,   2., -inf])
```

<!-- !! processed by numpydoc !! -->


# dask.array.log10.html.md

# dask.array.log10

### dask.array.log10(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'log10'>*

This docstring was copied from numpy.log10.

Some inconsistencies with the Dask version may exist.

Return the base 10 logarithm of the input array, element-wise.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The logarithm to the base 10 of x, element-wise. NaNs are
    returned where x is negative.
    This is a scalar if x is a scalar.

#### SEE ALSO
`emath.log10`

### Notes

Logarithm is a multivalued function: for each x there is an infinite
number of z such that 10\*\*z = x. The convention is to return the
z whose imaginary part lies in (-pi, pi].

For real-valued input data types, log10 always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, log10 is a complex analytical function that
has a branch cut [-inf, 0] and is continuous from above on it.
log10 handles the floating-point negative zero as an infinitesimal
negative number, conforming to the C99 standard.

In the cases where the input has a negative real part and a very small
negative complex part (approaching 0), the result is so close to -pi
that it evaluates to exactly -pi.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.log10([1e-15, -3.])
array([-15.,  nan])
```

<!-- !! processed by numpydoc !! -->


# dask.array.log1p.html.md

# dask.array.log1p

### dask.array.log1p(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'log1p'>*

This docstring was copied from numpy.log1p.

Some inconsistencies with the Dask version may exist.

Return the natural logarithm of one plus the input array, element-wise.

Calculates `log(1 + x)`.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Natural logarithm of 1 + x, element-wise.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`expm1`](dask.array.expm1.md#dask.array.expm1)
: `exp(x) - 1`, the inverse of log1p.

### Notes

For real-valued input, log1p is accurate also for x so small
that 1 + x == 1 in floating-point accuracy.

Logarithm is a multivalued function: for each x there is an infinite
number of z such that exp(z) = 1 + x. The convention is to return
the z whose imaginary part lies in [-pi, pi].

For real-valued input data types, log1p always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, log1p is a complex analytical function that
has a branch cut [-inf, -1] and is continuous from above on it.
log1p handles the floating-point negative zero as an infinitesimal
negative number, conforming to the C99 standard.

### References

### Examples

```pycon
>>> import numpy as np
>>> np.log1p(1e-99)
1e-99
>>> np.log(1 + 1e-99)
0.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.log2.html.md

# dask.array.log2

### dask.array.log2(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'log2'>*

This docstring was copied from numpy.log2.

Some inconsistencies with the Dask version may exist.

Base-2 logarithm of x.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Base-2 logarithm of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`log`](dask.array.log.md#dask.array.log), [`log10`](dask.array.log10.md#dask.array.log10), [`log1p`](dask.array.log1p.md#dask.array.log1p), `emath.log2`

### Notes

Logarithm is a multivalued function: for each x there is an infinite
number of z such that 2\*\*z = x. The convention is to return the z
whose imaginary part lies in (-pi, pi].

For real-valued input data types, log2 always returns real output.
For each value that cannot be expressed as a real number or infinity,
it yields `nan` and sets the invalid floating point error flag.

For complex-valued input, log2 is a complex analytical function that
has a branch cut [-inf, 0] and is continuous from above on it. log2
handles the floating-point negative zero as an infinitesimal negative
number, conforming to the C99 standard.

In the cases where the input has a negative real part and a very small
negative complex part (approaching 0), the result is so close to -pi
that it evaluates to exactly -pi.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([0, 1, 2, 2**4])
>>> np.log2(x)
array([-inf,   0.,   1.,   4.])
```

```pycon
>>> xi = np.array([0+1.j, 1, 2+0.j, 4.j])
>>> np.log2(xi)
array([ 0.+2.26618007j,  0.+0.j        ,  1.+0.j        ,  2.+2.26618007j])
```

<!-- !! processed by numpydoc !! -->


# dask.array.logaddexp.html.md

# dask.array.logaddexp

### dask.array.logaddexp(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logaddexp'>*

This docstring was copied from numpy.logaddexp.

Some inconsistencies with the Dask version may exist.

Logarithm of the sum of exponentiations of the inputs.

Calculates `log(exp(x1) + exp(x2))`. This function is useful in
statistics where the calculated probabilities of events may be so small
as to exceed the range of normal floating point numbers.  In such cases
the logarithm of the calculated probability is stored. This function
allows adding probabilities stored in such a fashion.

* **Parameters:**
  **x1, x2**
  : Input values.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **result**
  : Logarithm of `exp(x1) + exp(x2)`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logaddexp2`](dask.array.logaddexp2.md#dask.array.logaddexp2)
: Logarithm of the sum of exponentiations of inputs in base 2.

### Examples

```pycon
>>> import numpy as np
>>> prob1 = np.log(1e-50)
>>> prob2 = np.log(2.5e-50)
>>> prob12 = np.logaddexp(prob1, prob2)
>>> prob12
-113.87649168120691
>>> np.exp(prob12)
3.5000000000000057e-50
```

<!-- !! processed by numpydoc !! -->


# dask.array.logaddexp2.html.md

# dask.array.logaddexp2

### dask.array.logaddexp2(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logaddexp2'>*

This docstring was copied from numpy.logaddexp2.

Some inconsistencies with the Dask version may exist.

Logarithm of the sum of exponentiations of the inputs in base-2.

Calculates `log2(2**x1 + 2**x2)`. This function is useful in machine
learning when the calculated probabilities of events may be so small as
to exceed the range of normal floating point numbers.  In such cases
the base-2 logarithm of the calculated probability can be used instead.
This function allows adding probabilities stored in such a fashion.

* **Parameters:**
  **x1, x2**
  : Input values.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **result**
  : Base-2 logarithm of `2**x1 + 2**x2`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logaddexp`](dask.array.logaddexp.md#dask.array.logaddexp)
: Logarithm of the sum of exponentiations of the inputs.

### Examples

```pycon
>>> import numpy as np
>>> prob1 = np.log2(1e-50)
>>> prob2 = np.log2(2.5e-50)
>>> prob12 = np.logaddexp2(prob1, prob2)
>>> prob1, prob2, prob12
(-166.09640474436813, -164.77447664948076, -164.28904982231052)
>>> 2**prob12
3.4999999999999914e-50
```

<!-- !! processed by numpydoc !! -->


# dask.array.logical_and.html.md

# dask.array.logical_and

### dask.array.logical_and(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logical_and'>*

This docstring was copied from numpy.logical_and.

Some inconsistencies with the Dask version may exist.

Compute the truth value of x1 AND x2 element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Boolean result of the logical AND operation applied to the elements
    of x1 and x2; the shape is determined by broadcasting.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_or`](dask.array.logical_or.md#dask.array.logical_or), [`logical_not`](dask.array.logical_not.md#dask.array.logical_not), [`logical_xor`](dask.array.logical_xor.md#dask.array.logical_xor)

[`bitwise_and`](dask.array.bitwise_and.md#dask.array.bitwise_and)

### Examples

```pycon
>>> import numpy as np
>>> np.logical_and(True, False)
False
>>> np.logical_and([True, False], [False, False])
array([False, False])
```

```pycon
>>> x = np.arange(5)
>>> np.logical_and(x>1, x<4)
array([False, False,  True,  True, False])
```

The `&` operator can be used as a shorthand for `np.logical_and` on
boolean ndarrays.

```pycon
>>> a = np.array([True, False])
>>> b = np.array([False, False])
>>> a & b
array([False, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.logical_not.html.md

# dask.array.logical_not

### dask.array.logical_not(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logical_not'>*

This docstring was copied from numpy.logical_not.

Some inconsistencies with the Dask version may exist.

Compute the truth value of NOT x element-wise.

* **Parameters:**
  **x**
  : Logical NOT is applied to the elements of x.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Boolean result with the same shape as x of the NOT operation
    on elements of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`logical_and`](dask.array.logical_and.md#dask.array.logical_and), [`logical_or`](dask.array.logical_or.md#dask.array.logical_or), [`logical_xor`](dask.array.logical_xor.md#dask.array.logical_xor)

### Examples

```pycon
>>> import numpy as np
>>> np.logical_not(3)
False
>>> np.logical_not([True, False, 0, 1])
array([False,  True,  True, False])
```

```pycon
>>> x = np.arange(5)
>>> np.logical_not(x<3)
array([False, False, False,  True,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.logical_or.html.md

# dask.array.logical_or

### dask.array.logical_or(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logical_or'>*

This docstring was copied from numpy.logical_or.

Some inconsistencies with the Dask version may exist.

Compute the truth value of x1 OR x2 element-wise.

* **Parameters:**
  **x1, x2**
  : Logical OR is applied to the elements of x1 and x2.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Boolean result of the logical OR operation applied to the elements
    of x1 and x2; the shape is determined by broadcasting.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_and`](dask.array.logical_and.md#dask.array.logical_and), [`logical_not`](dask.array.logical_not.md#dask.array.logical_not), [`logical_xor`](dask.array.logical_xor.md#dask.array.logical_xor)

[`bitwise_or`](dask.array.bitwise_or.md#dask.array.bitwise_or)

### Examples

```pycon
>>> import numpy as np
>>> np.logical_or(True, False)
True
>>> np.logical_or([True, False], [False, False])
array([ True, False])
```

```pycon
>>> x = np.arange(5)
>>> np.logical_or(x < 1, x > 3)
array([ True, False, False, False,  True])
```

The `|` operator can be used as a shorthand for `np.logical_or` on
boolean ndarrays.

```pycon
>>> a = np.array([True, False])
>>> b = np.array([False, False])
>>> a | b
array([ True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.logical_xor.html.md

# dask.array.logical_xor

### dask.array.logical_xor(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'logical_xor'>*

This docstring was copied from numpy.logical_xor.

Some inconsistencies with the Dask version may exist.

Compute the truth value of x1 XOR x2, element-wise.

* **Parameters:**
  **x1, x2**
  : Logical XOR is applied to the elements of x1 and x2.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Boolean result of the logical XOR operation applied to the elements
    of x1 and x2; the shape is determined by broadcasting.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`logical_and`](dask.array.logical_and.md#dask.array.logical_and), [`logical_or`](dask.array.logical_or.md#dask.array.logical_or), [`logical_not`](dask.array.logical_not.md#dask.array.logical_not), [`bitwise_xor`](dask.array.bitwise_xor.md#dask.array.bitwise_xor)

### Examples

```pycon
>>> import numpy as np
>>> np.logical_xor(True, False)
True
>>> np.logical_xor([True, True, False, False], [True, False, True, False])
array([False,  True,  True, False])
```

```pycon
>>> x = np.arange(5)
>>> np.logical_xor(x < 1, x > 3)
array([ True, False, False, False,  True])
```

Simple example showing support of broadcasting

```pycon
>>> np.logical_xor(0, np.eye(2))
array([[ True, False],
       [False,  True]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.average.html.md

# dask.array.ma.average

### dask.array.ma.average(a, axis=None, weights=None, returned=False, keepdims=False)

Return the weighted average of array over the given axis.

This docstring was copied from numpy.ma.average.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Data to be averaged.
    Masked entries are not taken into account in the computation.

  **axis**
  : Axis or axes along which to average a.  The default,
    axis=None, will average over all of the elements of the input array.
    If axis is a tuple of ints, averaging is performed on all of the axes
    specified in the tuple instead of a single axis or all the axes as
    before.

  **weights**
  : An array of weights associated with the values in a. Each value in
    a contributes to the average according to its associated weight.
    The array of weights must be the same shape as a if no axis is
    specified, otherwise the weights must have dimensions and shape
    consistent with a along the specified axis.
    If weights=None, then all data in a are assumed to have a
    weight equal to one.
    The calculation is:
    ```default
    avg = sum(a * weights) / sum(weights)
    ```
    <br/>
    where the sum is over all included elements.
    The only constraint on the values of weights is that sum(weights)
    must not be 0.

  **returned**
  : Flag indicating whether a tuple `(result, sum of weights)`
    should be returned as output (True), or just the result (False).
    Default is False.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    *Note:* keepdims will not work with instances of numpy.matrix
    or other classes whose methods do not support keepdims.
    <br/>
    #### Versionadded
    Added in version 1.23.0.
* **Returns:**
  **average, [sum_of_weights]**
  : The average along the specified axis. When returned is True,
    return a tuple with the average as the first element and the sum
    of the weights as the second element. The return type is np.float64
    if a is of integer type and floats smaller than float64, or the
    input data-type, otherwise. If returned, sum_of_weights is always
    float64.
* **Raises:**
  ZeroDivisionError
  : When all weights along axis are zero. See numpy.ma.average for a
    version robust to this type of error.

  TypeError
  : When weights does not have the same shape as a, and axis=None.

  ValueError
  : When weights does not have dimensions and shape consistent with a
    along specified axis.

### Examples

```pycon
>>> import numpy as np
>>> a = np.ma.array([1., 2., 3., 4.], mask=[False, False, True, True])
>>> np.ma.average(a, weights=[3, 1, 0, 0])
1.25
```

```pycon
>>> x = np.ma.arange(6.).reshape(3, 2)
>>> x
masked_array(
  data=[[0., 1.],
        [2., 3.],
        [4., 5.]],
  mask=False,
  fill_value=1e+20)
>>> data = np.arange(8).reshape((2, 2, 2))
>>> data
array([[[0, 1],
        [2, 3]],
       [[4, 5],
        [6, 7]]])
>>> np.ma.average(data, axis=(0, 1), weights=[[1./4, 3./4], [1., 1./2]])
masked_array(data=[3.4, 4.4],
         mask=[False, False],
   fill_value=1e+20)
>>> np.ma.average(data, axis=0, weights=[[1./4, 3./4], [1., 1./2]])
Traceback (most recent call last):
    ...
ValueError: Shape of weights must be consistent
with shape of a along specified axis.
```

```pycon
>>> avg, sumweights = np.ma.average(x, axis=0, weights=[1, 2, 3],
...                                 returned=True)
>>> avg
masked_array(data=[2.6666666666666665, 3.6666666666666665],
             mask=[False, False],
       fill_value=1e+20)
```

With `keepdims=True`, the following result has shape (3, 1).

```pycon
>>> np.ma.average(x, axis=1, keepdims=True)
masked_array(
  data=[[0.5],
        [2.5],
        [4.5]],
  mask=False,
  fill_value=1e+20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.empty_like.html.md

# dask.array.ma.empty_like

### dask.array.ma.empty_like(a, \*\*kwargs)

Return a new array with the same shape and type as a given array.

This docstring was copied from numpy.ma.core.empty_like.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **prototype**
  : The shape and data-type of prototype define these same attributes
    of the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if prototype is Fortran
    contiguous, ‘C’ otherwise. ‘K’ means match the layout of prototype
    as closely as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of prototype, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of uninitialized (arbitrary) data with the same
    shape and type as prototype.

#### SEE ALSO
[`ones_like`](dask.array.ma.ones_like.md#dask.array.ma.ones_like)
: Return an array of ones with shape and type of input.

[`zeros_like`](dask.array.ma.zeros_like.md#dask.array.ma.zeros_like)
: Return an array of zeros with shape and type of input.

`full_like`
: Return a new array with shape of input filled with value.

`empty`
: Return a new uninitialized array.

### Notes

Unlike other array creation functions (e.g. zeros_like, ones_like,
full_like), empty_like does not initialize the values of the array,
and may therefore be marginally faster. However, the values stored in the
newly allocated array are arbitrary. For reproducible behavior, be sure
to set each element of the array before reading.

### Examples

```pycon
>>> import numpy as np
>>> a = ([1,2,3], [4,5,6])                         # a is array-like
>>> np.empty_like(a)
array([[-1073741821, -1073741821,           3],    # uninitialized
       [          0,           0, -1073741821]])
>>> a = np.array([[1., 2., 3.],[4.,5.,6.]])
>>> np.empty_like(a)
array([[ -2.00000715e+000,   1.48219694e-323,  -2.00000572e+000], # uninitialized
       [  4.38791518e-305,  -2.00000715e+000,   4.17269252e-309]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.filled.html.md

# dask.array.ma.filled

### dask.array.ma.filled(a, fill_value=None)

Return input as an ~numpy.ndarray, with masked values replaced by
fill_value.

This docstring was copied from numpy.ma.filled.

Some inconsistencies with the Dask version may exist.

If a is not a MaskedArray, a itself is returned.
If a is a MaskedArray with no masked values, then `a.data` is
returned.
If a is a MaskedArray and fill_value is None, fill_value is set to
`a.fill_value`.

* **Parameters:**
  **a**
  : An input object.

  **fill_value**
  : Can be scalar or non-scalar. If non-scalar, the
    resulting filled array should be broadcastable
    over input array. Default is None.
* **Returns:**
  **a**
  : The filled array.

#### SEE ALSO
`compressed`

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> x = ma.array(np.arange(9).reshape(3, 3), mask=[[1, 0, 0],
...                                                [1, 0, 0],
...                                                [0, 0, 0]])
>>> x.filled()
array([[999999,      1,      2],
       [999999,      4,      5],
       [     6,      7,      8]])
>>> x.filled(fill_value=333)
array([[333,   1,   2],
       [333,   4,   5],
       [  6,   7,   8]])
>>> x.filled(fill_value=np.arange(3))
array([[0, 1, 2],
       [0, 4, 5],
       [6, 7, 8]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.fix_invalid.html.md

# dask.array.ma.fix_invalid

### dask.array.ma.fix_invalid(a, fill_value=None)

Return input with invalid data masked and replaced by a fill value.

This docstring was copied from numpy.ma.fix_invalid.

Some inconsistencies with the Dask version may exist.

Invalid data means values of nan, inf, etc.

* **Parameters:**
  **a**
  : Input array, a (subclass of) ndarray.

  **mask**
  : Mask. Must be convertible to an array of booleans with the same
    shape as data. True indicates a masked (i.e. invalid) data.

  **copy**
  : Whether to use a copy of a (True) or to fix a in place (False).
    Default is True.

  **fill_value**
  : Value used for fixing invalid data. Default is None, in which case
    the `a.fill_value` is used.
* **Returns:**
  **b**
  : The input array with invalid entries fixed.

### Notes

A copy is performed by default.

### Examples

```pycon
>>> import numpy as np
>>> x = np.ma.array([1., -1, np.nan, np.inf], mask=[1] + [0]*3)
>>> x
masked_array(data=[--, -1.0, nan, inf],
             mask=[ True, False, False, False],
       fill_value=1e+20)
>>> np.ma.fix_invalid(x)
masked_array(data=[--, -1.0, --, --],
             mask=[ True, False,  True,  True],
       fill_value=1e+20)
```

```pycon
>>> fixed = np.ma.fix_invalid(x)
>>> fixed.data
array([ 1.e+00, -1.e+00,  1.e+20,  1.e+20])
>>> x.data
array([ 1., -1., nan, inf])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.getdata.html.md

# dask.array.ma.getdata

### dask.array.ma.getdata(a)

Return the data of a masked array as an ndarray.

This docstring was copied from numpy.ma.getdata.

Some inconsistencies with the Dask version may exist.

Return the data of a (if any) as an ndarray if a is a `MaskedArray`,
else return a as an ndarray or subclass (depending on subok) if not.

* **Parameters:**
  **a**
  : Input `MaskedArray`, alternatively an ndarray or a subclass thereof.

  **subok**
  : Whether to force the output to be a pure ndarray (False) or to
    return a subclass of ndarray if appropriate (True, default).

#### SEE ALSO
`getmask`
: Return the mask of a masked array, or nomask.

[`getmaskarray`](dask.array.ma.getmaskarray.md#dask.array.ma.getmaskarray)
: Return the mask of a masked array, or full array of False.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = ma.masked_equal([[1,2],[3,4]], 2)
>>> a
masked_array(
  data=[[1, --],
        [3, 4]],
  mask=[[False,  True],
        [False, False]],
  fill_value=2)
>>> ma.getdata(a)
array([[1, 2],
       [3, 4]])
```

Equivalently use the `MaskedArray` data attribute.

```pycon
>>> a.data
array([[1, 2],
       [3, 4]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.getmaskarray.html.md

# dask.array.ma.getmaskarray

### dask.array.ma.getmaskarray(a)

Return the mask of a masked array, or full boolean array of False.

This docstring was copied from numpy.ma.getmaskarray.

Some inconsistencies with the Dask version may exist.

Return the mask of arr as an ndarray if arr is a MaskedArray and
the mask is not nomask, else return a full boolean array of False of
the same shape as arr.

* **Parameters:**
  **arr**
  : Input MaskedArray for which the mask is required.

#### SEE ALSO
`getmask`
: Return the mask of a masked array, or nomask.

[`getdata`](dask.array.ma.getdata.md#dask.array.ma.getdata)
: Return the data of a masked array as an ndarray.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = ma.masked_equal([[1,2],[3,4]], 2)
>>> a
masked_array(
  data=[[1, --],
        [3, 4]],
  mask=[[False,  True],
        [False, False]],
  fill_value=2)
>>> ma.getmaskarray(a)
array([[False,  True],
       [False, False]])
```

Result when mask == `nomask`

```pycon
>>> b = ma.masked_array([[1,2],[3,4]])
>>> b
masked_array(
  data=[[1, 2],
        [3, 4]],
  mask=False,
  fill_value=999999)
>>> ma.getmaskarray(b)
array([[False, False],
       [False, False]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_array.html.md

# dask.array.ma.masked_array

### dask.array.ma.masked_array(data, mask=np.False_, fill_value=None, \*\*kwargs)

An array class with possibly masked values.

This docstring was copied from numpy.ma.masked_array.

Some inconsistencies with the Dask version may exist.

Masked values of True exclude the corresponding element from any
computation.

Construction:

```default
x = MaskedArray(data, mask=nomask, dtype=None, copy=False, subok=True,
                ndmin=0, fill_value=None, keep_mask=True, hard_mask=None,
                shrink=True, order=None)
```

* **Parameters:**
  **data**
  : Input data.

  **mask**
  : Mask. Must be convertible to an array of booleans with the same
    shape as data. True indicates a masked (i.e. invalid) data.

  **dtype**
  : Data type of the output.
    If dtype is None, the type of the data argument (`data.dtype`)
    is used. If dtype is not None and different from `data.dtype`,
    a copy is performed.

  **copy**
  : Whether to copy the input data (True), or to use a reference instead.
    Default is False.

  **subok**
  : Whether to return a subclass of MaskedArray if possible (True) or a
    plain MaskedArray. Default is True.

  **ndmin**
  : Minimum number of dimensions. Default is 0.

  **fill_value**
  : Value used to fill in the masked values when necessary.
    If None, a default based on the data-type is used.

  **keep_mask**
  : Whether to combine mask with the mask of the input data, if any
    (True), or to use only mask for the output (False). Default is True.

  **hard_mask**
  : Whether to use a hard mask or not. With a hard mask, masked values
    cannot be unmasked. Default is False.

  **shrink**
  : Whether to force compression of an empty mask. Default is True.

  **order**
  : Specify the order of the array.  If order is ‘C’, then the array
    will be in C-contiguous order (last-index varies the fastest).
    If order is ‘F’, then the returned array will be in
    Fortran-contiguous order (first-index varies the fastest).
    If order is ‘A’ (default), then the returned array may be
    in any order (either C-, Fortran-contiguous, or even discontiguous),
    unless a copy is required, in which case it will be C-contiguous.

### Examples

```pycon
>>> import numpy as np
```

The `mask` can be initialized with an array of boolean values
with the same shape as `data`.

```pycon
>>> data = np.arange(6).reshape((2, 3))
>>> np.ma.MaskedArray(data, mask=[[False, True, False],
...                               [False, False, True]])
masked_array(
  data=[[0, --, 2],
        [3, 4, --]],
  mask=[[False,  True, False],
        [False, False,  True]],
  fill_value=999999)
```

Alternatively, the `mask` can be initialized to homogeneous boolean
array with the same shape as `data` by passing in a scalar
boolean value:

```pycon
>>> np.ma.MaskedArray(data, mask=False)
masked_array(
  data=[[0, 1, 2],
        [3, 4, 5]],
  mask=[[False, False, False],
        [False, False, False]],
  fill_value=999999)
```

```pycon
>>> np.ma.MaskedArray(data, mask=True)
masked_array(
  data=[[--, --, --],
        [--, --, --]],
  mask=[[ True,  True,  True],
        [ True,  True,  True]],
  fill_value=999999,
  dtype=int64)
```

#### NOTE
The recommended practice for initializing `mask` with a scalar
boolean value is to use `True`/`False` rather than
`np.True_`/`np.False_`. The reason is `nomask`
is represented internally as `np.False_`.

```pycon
>>> np.False_ is np.ma.nomask
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_equal.html.md

# dask.array.ma.masked_equal

### dask.array.ma.masked_equal(a, value)

Mask an array where equal to a given value.

This docstring was copied from numpy.ma.masked_equal.

Some inconsistencies with the Dask version may exist.

Return a MaskedArray, masked where the data in array x are
equal to value. The fill_value of the returned MaskedArray
is set to value.

For floating point arrays, consider using `masked_values(x, value)`.

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

[`masked_values`](dask.array.ma.masked_values.md#dask.array.ma.masked_values)
: Mask using floating point equality.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_equal(a, 2)
masked_array(data=[0, 1, --, 3],
             mask=[False, False,  True, False],
       fill_value=2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_greater.html.md

# dask.array.ma.masked_greater

### dask.array.ma.masked_greater(x, value, copy=True)

Mask an array where greater than a given value.

This function is a shortcut to `masked_where`, with
condition = (x > value).

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_greater(a, 2)
masked_array(data=[0, 1, 2, --],
             mask=[False, False, False,  True],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_greater_equal.html.md

# dask.array.ma.masked_greater_equal

### dask.array.ma.masked_greater_equal(x, value, copy=True)

Mask an array where greater than or equal to a given value.

This function is a shortcut to `masked_where`, with
condition = (x >= value).

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_greater_equal(a, 2)
masked_array(data=[0, 1, --, --],
             mask=[False, False,  True,  True],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_inside.html.md

# dask.array.ma.masked_inside

### dask.array.ma.masked_inside(x, v1, v2)

Mask an array inside a given interval.

This docstring was copied from numpy.ma.masked_inside.

Some inconsistencies with the Dask version may exist.

Shortcut to `masked_where`, where condition is True for x inside
the interval [v1,v2] (v1 <= x <= v2).  The boundaries v1 and v2
can be given in either order.

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Notes

The array x is prefilled with its filling value.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> x = [0.31, 1.2, 0.01, 0.2, -0.4, -1.1]
>>> ma.masked_inside(x, -0.3, 0.3)
masked_array(data=[0.31, 1.2, --, --, -0.4, -1.1],
             mask=[False, False,  True,  True, False, False],
       fill_value=1e+20)
```

The order of v1 and v2 doesn’t matter.

```pycon
>>> ma.masked_inside(x, 0.3, -0.3)
masked_array(data=[0.31, 1.2, --, --, -0.4, -1.1],
             mask=[False, False,  True,  True, False, False],
       fill_value=1e+20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_invalid.html.md

# dask.array.ma.masked_invalid

### dask.array.ma.masked_invalid(a)

Mask an array where invalid values occur (NaNs or infs).

This docstring was copied from numpy.ma.masked_invalid.

Some inconsistencies with the Dask version may exist.

This function is a shortcut to `masked_where`, with
condition = ~(np.isfinite(a)). Any pre-existing mask is conserved.
Only applies to arrays with a dtype where NaNs or infs make sense
(i.e. floating point types), but accepts any array_like object.

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(5, dtype=np.float64)
>>> a[2] = np.nan
>>> a[3] = np.inf
>>> a
array([ 0.,  1., nan, inf,  4.])
>>> ma.masked_invalid(a)
masked_array(data=[0.0, 1.0, --, --, 4.0],
             mask=[False, False,  True,  True, False],
       fill_value=1e+20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_less.html.md

# dask.array.ma.masked_less

### dask.array.ma.masked_less(x, value, copy=True)

Mask an array where less than a given value.

This function is a shortcut to `masked_where`, with
condition = (x < value).

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_less(a, 2)
masked_array(data=[--, --, 2, 3],
             mask=[ True,  True, False, False],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_less_equal.html.md

# dask.array.ma.masked_less_equal

### dask.array.ma.masked_less_equal(x, value, copy=True)

Mask an array where less than or equal to a given value.

This function is a shortcut to `masked_where`, with
condition = (x <= value).

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_less_equal(a, 2)
masked_array(data=[--, --, --, 3],
             mask=[ True,  True,  True, False],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_not_equal.html.md

# dask.array.ma.masked_not_equal

### dask.array.ma.masked_not_equal(x, value, copy=True)

Mask an array where *not* equal to a given value.

This function is a shortcut to `masked_where`, with
condition = (x != value).

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_not_equal(a, 2)
masked_array(data=[--, --, 2, --],
             mask=[ True,  True, False,  True],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_outside.html.md

# dask.array.ma.masked_outside

### dask.array.ma.masked_outside(x, v1, v2)

Mask an array outside a given interval.

This docstring was copied from numpy.ma.masked_outside.

Some inconsistencies with the Dask version may exist.

Shortcut to `masked_where`, where condition is True for x outside
the interval [v1,v2] (x < v1)|(x > v2).
The boundaries v1 and v2 can be given in either order.

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

### Notes

The array x is prefilled with its filling value.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> x = [0.31, 1.2, 0.01, 0.2, -0.4, -1.1]
>>> ma.masked_outside(x, -0.3, 0.3)
masked_array(data=[--, --, 0.01, 0.2, --, --],
             mask=[ True,  True, False, False,  True,  True],
       fill_value=1e+20)
```

The order of v1 and v2 doesn’t matter.

```pycon
>>> ma.masked_outside(x, 0.3, -0.3)
masked_array(data=[--, --, 0.01, 0.2, --, --],
             mask=[ True,  True, False, False,  True,  True],
       fill_value=1e+20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_values.html.md

# dask.array.ma.masked_values

### dask.array.ma.masked_values(x, value, rtol=1e-05, atol=1e-08, shrink=True)

Mask using floating point equality.

This docstring was copied from numpy.ma.masked_values.

Some inconsistencies with the Dask version may exist.

Return a MaskedArray, masked where the data in array x are approximately
equal to value, determined using isclose. The default tolerances for
masked_values are the same as those for isclose.

For integer types, exact equality is used, in the same way as
masked_equal.

The fill_value is set to value and the mask is set to `nomask` if
possible.

* **Parameters:**
  **x**
  : Array to mask.

  **value**
  : Masking value.

  **rtol, atol**
  : Tolerance parameters passed on to isclose

  **copy**
  : Whether to return a copy of x.

  **shrink**
  : Whether to collapse a mask full of False to `nomask`.
* **Returns:**
  **result**
  : The result of masking x where approximately equal to value.

#### SEE ALSO
[`masked_where`](dask.array.ma.masked_where.md#dask.array.ma.masked_where)
: Mask where a condition is met.

[`masked_equal`](dask.array.ma.masked_equal.md#dask.array.ma.masked_equal)
: Mask where equal to a given value (integers).

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> x = np.array([1, 1.1, 2, 1.1, 3])
>>> ma.masked_values(x, 1.1)
masked_array(data=[1.0, --, 2.0, --, 3.0],
             mask=[False,  True, False,  True, False],
       fill_value=1.1)
```

Note that mask is set to `nomask` if possible.

```pycon
>>> ma.masked_values(x, 2.1)
masked_array(data=[1. , 1.1, 2. , 1.1, 3. ],
             mask=False,
       fill_value=2.1)
```

Unlike masked_equal, masked_values can perform approximate equalities.

```pycon
>>> ma.masked_values(x, 2.1, atol=1e-1)
masked_array(data=[1.0, 1.1, --, 1.1, 3.0],
             mask=[False, False,  True, False, False],
       fill_value=2.1)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.masked_where.html.md

# dask.array.ma.masked_where

### dask.array.ma.masked_where(condition, a)

Mask an array where a condition is met.

This docstring was copied from numpy.ma.masked_where.

Some inconsistencies with the Dask version may exist.

Return a as an array masked where condition is True.
Any masked values of a or condition are also masked in the output.

* **Parameters:**
  **condition**
  : Masking condition.  When condition tests floating point values for
    equality, consider using `masked_values` instead.

  **a**
  : Array to mask.

  **copy**
  : If True (default) make a copy of a in the result.  If False modify
    a in place and return a view.
* **Returns:**
  **result**
  : The result of masking a where condition is True.

#### SEE ALSO
[`masked_values`](dask.array.ma.masked_values.md#dask.array.ma.masked_values)
: Mask using floating point equality.

[`masked_equal`](dask.array.ma.masked_equal.md#dask.array.ma.masked_equal)
: Mask where equal to a given value.

[`masked_not_equal`](dask.array.ma.masked_not_equal.md#dask.array.ma.masked_not_equal)
: Mask where *not* equal to a given value.

[`masked_less_equal`](dask.array.ma.masked_less_equal.md#dask.array.ma.masked_less_equal)
: Mask where less than or equal to a given value.

[`masked_greater_equal`](dask.array.ma.masked_greater_equal.md#dask.array.ma.masked_greater_equal)
: Mask where greater than or equal to a given value.

[`masked_less`](dask.array.ma.masked_less.md#dask.array.ma.masked_less)
: Mask where less than a given value.

[`masked_greater`](dask.array.ma.masked_greater.md#dask.array.ma.masked_greater)
: Mask where greater than a given value.

[`masked_inside`](dask.array.ma.masked_inside.md#dask.array.ma.masked_inside)
: Mask inside a given interval.

[`masked_outside`](dask.array.ma.masked_outside.md#dask.array.ma.masked_outside)
: Mask outside a given interval.

[`masked_invalid`](dask.array.ma.masked_invalid.md#dask.array.ma.masked_invalid)
: Mask invalid values (NaNs or infs).

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(4)
>>> a
array([0, 1, 2, 3])
>>> ma.masked_where(a <= 2, a)
masked_array(data=[--, --, --, 3],
             mask=[ True,  True,  True, False],
       fill_value=999999)
```

Mask array b conditional on a.

```pycon
>>> b = ['a', 'b', 'c', 'd']
>>> ma.masked_where(a == 2, b)
masked_array(data=['a', 'b', --, 'd'],
             mask=[False, False,  True, False],
       fill_value='N/A',
            dtype='<U1')
```

Effect of the copy argument.

```pycon
>>> c = ma.masked_where(a <= 2, a)
>>> c
masked_array(data=[--, --, --, 3],
             mask=[ True,  True,  True, False],
       fill_value=999999)
>>> c[0] = 99
>>> c
masked_array(data=[99, --, --, 3],
             mask=[False,  True,  True, False],
       fill_value=999999)
>>> a
array([0, 1, 2, 3])
>>> c = ma.masked_where(a <= 2, a, copy=False)
>>> c[0] = 99
>>> c
masked_array(data=[99, --, --, 3],
             mask=[False,  True,  True, False],
       fill_value=999999)
>>> a
array([99,  1,  2,  3])
```

When condition or a contain masked values.

```pycon
>>> a = np.arange(4)
>>> a = ma.masked_where(a == 2, a)
>>> a
masked_array(data=[0, 1, --, 3],
             mask=[False, False,  True, False],
       fill_value=999999)
>>> b = np.arange(4)
>>> b = ma.masked_where(b == 0, b)
>>> b
masked_array(data=[--, 1, 2, 3],
             mask=[ True, False, False, False],
       fill_value=999999)
>>> ma.masked_where(a == 3, b)
masked_array(data=[--, 1, --, --],
             mask=[ True, False,  True,  True],
       fill_value=999999)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.nonzero.html.md

# dask.array.ma.nonzero

### dask.array.ma.nonzero(self)

This docstring was copied from numpy.ma.core.nonzero.

Some inconsistencies with the Dask version may exist.

Return the indices of unmasked elements that are not zero.

Returns a tuple of arrays, one for each dimension, containing the
indices of the non-zero elements in that dimension. The corresponding
non-zero values can be obtained with:

```default
a[a.nonzero()]
```

To group the indices by element, rather than dimension, use
instead:

```default
np.transpose(a.nonzero())
```

The result of this is always a 2d array, with a row for each non-zero
element.

* **Parameters:**
  **None**
* **Returns:**
  **tuple_of_arrays**
  : Indices of elements that are non-zero.

#### SEE ALSO
[`numpy.nonzero`](https://numpy.org/doc/stable/reference/generated/numpy.nonzero.html#numpy.nonzero)
: Function operating on ndarrays.

`flatnonzero`
: Return indices that are non-zero in the flattened version of the input array.

[`numpy.ndarray.nonzero`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.nonzero.html#numpy.ndarray.nonzero)
: Equivalent ndarray method.

`count_nonzero`
: Counts the number of non-zero elements in the input array.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> x = ma.array(np.eye(3))
>>> x
masked_array(
  data=[[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.]],
  mask=False,
  fill_value=1e+20)
>>> x.nonzero()
(array([0, 1, 2]), array([0, 1, 2]))
```

Masked elements are ignored.

```pycon
>>> x[1, 1] = ma.masked
>>> x
masked_array(
  data=[[1.0, 0.0, 0.0],
        [0.0, --, 0.0],
        [0.0, 0.0, 1.0]],
  mask=[[False, False, False],
        [False,  True, False],
        [False, False, False]],
  fill_value=1e+20)
>>> x.nonzero()
(array([0, 2]), array([0, 2]))
```

Indices can also be grouped by element.

```pycon
>>> np.transpose(x.nonzero())
array([[0, 0],
       [2, 2]])
```

A common use for `nonzero` is to find the indices of an array, where
a condition is True.  Given an array a, the condition a > 3 is a
boolean array and since False is interpreted as 0, ma.nonzero(a > 3)
yields the indices of the a where the condition is true.

```pycon
>>> a = ma.array([[1,2,3],[4,5,6],[7,8,9]])
>>> a > 3
masked_array(
  data=[[False, False, False],
        [ True,  True,  True],
        [ True,  True,  True]],
  mask=False,
  fill_value=True)
>>> ma.nonzero(a > 3)
(array([1, 1, 1, 2, 2, 2]), array([0, 1, 2, 0, 1, 2]))
```

The `nonzero` method of the condition array can also be called.

```pycon
>>> (a > 3).nonzero()
(array([1, 1, 1, 2, 2, 2]), array([0, 1, 2, 0, 1, 2]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.ones_like.html.md

# dask.array.ma.ones_like

### dask.array.ma.ones_like(a, \*\*kwargs)

Return an array of ones with the same shape and type as a given array.

This docstring was copied from numpy.ma.core.ones_like.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if a is Fortran contiguous,
    ‘C’ otherwise. ‘K’ means match the layout of a as closely
    as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of a, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of ones with the same shape and type as a.

#### SEE ALSO
[`empty_like`](dask.array.ma.empty_like.md#dask.array.ma.empty_like)
: Return an empty array with shape and type of input.

[`zeros_like`](dask.array.ma.zeros_like.md#dask.array.ma.zeros_like)
: Return an array of zeros with shape and type of input.

`full_like`
: Return a new array with shape of input filled with value.

`ones`
: Return a new array setting values to one.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6)
>>> x = x.reshape((2, 3))
>>> x
array([[0, 1, 2],
       [3, 4, 5]])
>>> np.ones_like(x)
array([[1, 1, 1],
       [1, 1, 1]])
```

```pycon
>>> y = np.arange(3, dtype=np.float64)
>>> y
array([0., 1., 2.])
>>> np.ones_like(y)
array([1.,  1.,  1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.set_fill_value.html.md

# dask.array.ma.set_fill_value

### dask.array.ma.set_fill_value(a, fill_value)

Set the filling value of a, if a is a masked array.

This docstring was copied from numpy.ma.set_fill_value.

Some inconsistencies with the Dask version may exist.

This function changes the fill value of the masked array a in place.
If a is not a masked array, the function returns silently, without
doing anything.

* **Parameters:**
  **a**
  : Input array.

  **fill_value**
  : Filling value. A consistency test is performed to make sure
    the value is compatible with the dtype of a.
* **Returns:**
  None
  : Nothing returned by this function.

#### SEE ALSO
`maximum_fill_value`
: Return the default fill value for a dtype.

`MaskedArray.fill_value`
: Return current fill value.

`MaskedArray.set_fill_value`
: Equivalent method.

### Examples

```pycon
>>> import numpy as np
>>> import numpy.ma as ma
>>> a = np.arange(5)
>>> a
array([0, 1, 2, 3, 4])
>>> a = ma.masked_where(a < 3, a)
>>> a
masked_array(data=[--, --, --, 3, 4],
             mask=[ True,  True,  True, False, False],
       fill_value=999999)
>>> ma.set_fill_value(a, -999)
>>> a
masked_array(data=[--, --, --, 3, 4],
             mask=[ True,  True,  True, False, False],
       fill_value=-999)
```

Nothing happens if a is not a masked array.

```pycon
>>> a = list(range(5))
>>> a
[0, 1, 2, 3, 4]
>>> ma.set_fill_value(a, 100)
>>> a
[0, 1, 2, 3, 4]
>>> a = np.arange(5)
>>> a
array([0, 1, 2, 3, 4])
>>> ma.set_fill_value(a, 100)
>>> a
array([0, 1, 2, 3, 4])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.where.html.md

# dask.array.ma.where

### dask.array.ma.where(condition, x=None, y=None)

Return a masked array with elements from x or y, depending on condition.

This docstring was copied from numpy.ma.core.where.

Some inconsistencies with the Dask version may exist.

#### NOTE
When only condition is provided, this function is identical to
nonzero. The rest of this documentation covers only the case where
all three arguments are provided.

* **Parameters:**
  **condition**
  : Where True, yield x, otherwise yield y.

  **x, y**
  : Values from which to choose. x, y and condition need to be
    broadcastable to some shape.
* **Returns:**
  **out**
  : A masked array with masked elements where the condition is masked,
    elements from x where condition is True, and elements from y
    elsewhere.

#### SEE ALSO
[`numpy.where`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where)
: Equivalent function in the top-level NumPy module.

[`nonzero`](dask.array.ma.nonzero.md#dask.array.ma.nonzero)
: The function that is called when x and y are omitted

### Examples

```pycon
>>> import numpy as np
>>> x = np.ma.array(np.arange(9.).reshape(3, 3), mask=[[0, 1, 0],
...                                                    [1, 0, 1],
...                                                    [0, 1, 0]])
>>> x
masked_array(
  data=[[0.0, --, 2.0],
        [--, 4.0, --],
        [6.0, --, 8.0]],
  mask=[[False,  True, False],
        [ True, False,  True],
        [False,  True, False]],
  fill_value=1e+20)
>>> np.ma.where(x > 5, x, -3.1416)
masked_array(
  data=[[-3.1416, --, -3.1416],
        [--, -3.1416, --],
        [6.0, --, 8.0]],
  mask=[[False,  True, False],
        [ True, False,  True],
        [False,  True, False]],
  fill_value=1e+20)
```

<!-- !! processed by numpydoc !! -->


# dask.array.ma.zeros_like.html.md

# dask.array.ma.zeros_like

### dask.array.ma.zeros_like(a, \*\*kwargs)

Return an array of zeros with the same shape and type as a given array.

This docstring was copied from numpy.ma.core.zeros_like.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if a is Fortran contiguous,
    ‘C’ otherwise. ‘K’ means match the layout of a as closely
    as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of a, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of zeros with the same shape and type as a.

#### SEE ALSO
[`empty_like`](dask.array.ma.empty_like.md#dask.array.ma.empty_like)
: Return an empty array with shape and type of input.

[`ones_like`](dask.array.ma.ones_like.md#dask.array.ma.ones_like)
: Return an array of ones with shape and type of input.

`full_like`
: Return a new array with shape of input filled with value.

`zeros`
: Return a new array setting values to zero.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6)
>>> x = x.reshape((2, 3))
>>> x
array([[0, 1, 2],
       [3, 4, 5]])
>>> np.zeros_like(x)
array([[0, 0, 0],
       [0, 0, 0]])
```

```pycon
>>> y = np.arange(3, dtype=np.float64)
>>> y
array([0., 1., 2.])
>>> np.zeros_like(y)
array([0.,  0.,  0.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.map_blocks.html.md

# dask.array.map_blocks

### dask.array.map_blocks(func, \*args, name=None, token=None, dtype=None, chunks=None, drop_axis=None, new_axis=None, enforce_ndim=False, meta=None, \*\*kwargs)

Map a function across all blocks of a dask array.

Note that `map_blocks` will attempt to automatically determine the output
array type by calling `func` on 0-d versions of the inputs. Please refer to
the `meta` keyword argument below if you expect that the function will not
succeed when operating on 0-d arrays.

* **Parameters:**
  **func**
  : Function to apply to every block in the array.
    If `func` accepts `block_info=` or `block_id=`
    as keyword arguments, these will be passed dictionaries
    containing information about input and output chunks/arrays
    during computation. See examples for details.

  **args**

  **dtype**
  : The `dtype` of the output array. It is recommended to provide this.
    If not provided, will be inferred by applying the function to a small
    set of fake data.

  **chunks**
  : Chunk shape of resulting blocks if the function does not preserve
    shape. If not provided, the resulting array is assumed to have the same
    block structure as the first input array.

  **drop_axis**
  : Dimensions lost by the function.

  **new_axis**
  : New dimensions created by the function. Note that these are applied
    after `drop_axis` (if present). The size of each chunk along this
    dimension will be set to 1. Please specify `chunks` if the individual
    chunks have a different size.

  **enforce_ndim**
  : Whether to enforce at runtime that the dimensionality of the array
    produced by `func` actually matches that of the array returned by
    `map_blocks`.
    If True, this will raise an error when there is a mismatch.

  **token**
  : The key prefix to use for the output array. If not provided, will be
    determined from the function name.

  **name**
  : The key name to use for the output array. Note that this fully
    specifies the output key name, and must be unique. If not provided,
    will be determined by a hash of the arguments.

  **meta**
  : The `meta` of the output array, when specified is expected to be an
    array of the same type and dtype of that returned when calling `.compute()`
    on the array returned by this function. When not provided, `meta` will be
    inferred by applying the function to a small set of fake data, usually a
    0-d array. It’s important to ensure that `func` can successfully complete
    computation without raising exceptions when 0-d is passed to it, providing
    `meta` will be required otherwise. If the output type is known beforehand
    (e.g., `np.ndarray`, `cupy.ndarray`), an empty array of such type dtype
    can be passed, for example: `meta=np.array((), dtype=np.int32)`.

  **\*\*kwargs**
  : Other keyword arguments to pass to function. Values must be constants
    (not dask.arrays)

#### SEE ALSO
[`dask.array.map_overlap`](dask.array.map_overlap.md#dask.array.map_overlap)
: Generalized operation with overlap between neighbors.

[`dask.array.blockwise`](dask.array.blockwise.md#dask.array.blockwise)
: Generalized operation with control over block alignment.

### Examples

```pycon
>>> import dask.array as da
>>> x = da.arange(6, chunks=3)
```

```pycon
>>> x.map_blocks(lambda x: x * 2).compute()
array([ 0,  2,  4,  6,  8, 10])
```

The `da.map_blocks` function can also accept multiple arrays.

```pycon
>>> d = da.arange(5, chunks=2)
>>> e = da.arange(5, chunks=2)
```

```pycon
>>> f = da.map_blocks(lambda a, b: a + b**2, d, e)
>>> f.compute()
array([ 0,  2,  6, 12, 20])
```

If the function changes shape of the blocks then you must provide chunks
explicitly.

```pycon
>>> y = x.map_blocks(lambda x: x[::2], chunks=((2, 2),))
```

You have a bit of freedom in specifying chunks.  If all of the output chunk
sizes are the same, you can provide just that chunk size as a single tuple.

```pycon
>>> a = da.arange(18, chunks=(6,))
>>> b = a.map_blocks(lambda x: x[:3], chunks=(3,))
```

If the function changes the dimension of the blocks you must specify the
created or destroyed dimensions.

```pycon
>>> b = a.map_blocks(lambda x: x[None, :, None], chunks=(1, 6, 1),
...                  new_axis=[0, 2])
```

If `chunks` is specified but `new_axis` is not, then it is inferred to
add the necessary number of axes on the left.

Note that `map_blocks()` will concatenate chunks along axes specified by
the keyword parameter `drop_axis` prior to applying the function.
This is illustrated in the figure below:

![image](images/map_blocks_drop_axis.png)

Due to memory-size-constraints, it is often not advisable to use `drop_axis`
on an axis that is chunked.  In that case, it is better not to use
`map_blocks` but rather
`dask.array.reduction(..., axis=dropped_axes, concatenate=False)` which
maintains a leaner memory footprint while it drops any axis.

Map_blocks aligns blocks by block positions without regard to shape. In the
following example we have two arrays with the same number of blocks but
with different shape and chunk sizes.

```pycon
>>> x = da.arange(1000, chunks=(100,))
>>> y = da.arange(100, chunks=(10,))
```

The relevant attribute to match is numblocks.

```pycon
>>> x.numblocks
(10,)
>>> y.numblocks
(10,)
```

If these match (up to broadcasting rules) then we can map arbitrary
functions across blocks

```pycon
>>> def func(a, b):
...     return np.array([a.max(), b.max()])
```

```pycon
>>> da.map_blocks(func, x, y, chunks=(2,), dtype='i8')
dask.array<func, shape=(20,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>
```

```pycon
>>> _.compute()
array([ 99,   9, 199,  19, 299,  29, 399,  39, 499,  49, 599,  59, 699,
        69, 799,  79, 899,  89, 999,  99])
```

Your block function can get information about where it is in the array by
accepting a special `block_info` or `block_id` keyword argument.
During computation, they will contain information about each of the input
and output chunks (and dask arrays) relevant to each call of `func`.

```pycon
>>> def func(block_info=None):
...     pass
```

This will receive the following information:

```pycon
>>> block_info
{0: {'shape': (1000,),
     'num-chunks': (10,),
     'chunk-location': (4,),
     'array-location': [(400, 500)]},
 None: {'shape': (1000,),
        'num-chunks': (10,),
        'chunk-location': (4,),
        'array-location': [(400, 500)],
        'chunk-shape': (100,),
        'dtype': dtype('float64')}}
```

The keys to the `block_info` dictionary indicate which is the input and
output Dask array:

- **Input Dask array(s):** `block_info[0]` refers to the first input Dask array.
  The dictionary key is `0` because that is the argument index corresponding
  to the first input Dask array.
  In cases where multiple Dask arrays have been passed as input to the function,
  you can access them with the number corresponding to the input argument,
  eg: `block_info[1]`, `block_info[2]`, etc.
  (Note that if you pass multiple Dask arrays as input to map_blocks,
  the arrays must match each other by having matching numbers of chunks,
  along corresponding dimensions up to broadcasting rules.)
- **Output Dask array:** `block_info[None]` refers to the output Dask array,
  and contains information about the output chunks.
  The output chunk shape and dtype may may be different than the input chunks.

For each dask array, `block_info` describes:

- `shape`: the shape of the full Dask array,
- `num-chunks`: the number of chunks of the full array in each dimension,
- `chunk-location`: the chunk location (for example the fourth chunk over
  in the first dimension), and
- `array-location`: the array location within the full Dask array
  (for example the slice corresponding to `40:50`).

In addition to these, there are two extra parameters described by
`block_info` for the output array (in `block_info[None]`):

- `chunk-shape`: the output chunk shape, and
- `dtype`: the output dtype.

These features can be combined to synthesize an array from scratch, for
example:

```pycon
>>> def func(block_info=None):
...     loc = block_info[None]['array-location'][0]
...     return np.arange(loc[0], loc[1])
```

```pycon
>>> da.map_blocks(func, chunks=((4, 4),), dtype=np.float64)
dask.array<func, shape=(8,), dtype=float64, chunksize=(4,), chunktype=numpy.ndarray>
```

```pycon
>>> _.compute()
array([0, 1, 2, 3, 4, 5, 6, 7])
```

`block_id` is similar to `block_info` but contains only the `chunk_location`:

```pycon
>>> def func(block_id=None):
...     pass
```

This will receive the following information:

```pycon
>>> block_id
(4, 3)
```

You may specify the key name prefix of the resulting task in the graph with
the optional `token` keyword argument.

```pycon
>>> x.map_blocks(lambda x: x + 1, name='increment')
dask.array<increment, shape=(1000,), dtype=int64, chunksize=(100,), chunktype=numpy.ndarray>
```

For functions that may not handle 0-d arrays, it’s also possible to specify
`meta` with an empty array matching the type of the expected result. In
the example below, `func` will result in an `IndexError` when computing
`meta`:

```pycon
>>> rng = da.random.default_rng()
>>> da.map_blocks(lambda x: x[2], rng.random(5), meta=np.array(()))
dask.array<lambda, shape=(5,), dtype=float64, chunksize=(5,), chunktype=numpy.ndarray>
```

Similarly, it’s possible to specify a non-NumPy array to `meta`, and provide
a `dtype`:

```pycon
>>> import cupy
>>> rng = da.random.default_rng(cupy.random.default_rng())
>>> dt = np.float32
>>> da.map_blocks(lambda x: x[2], rng.random(5, dtype=dt), meta=cupy.array((), dtype=dt))
dask.array<lambda, shape=(5,), dtype=float32, chunksize=(5,), chunktype=cupy.ndarray>
```

<!-- !! processed by numpydoc !! -->


# dask.array.map_overlap.html.md

# dask.array.map_overlap

### dask.array.map_overlap(func, \*args, depth=None, boundary=None, trim=True, align_arrays=True, allow_rechunk=True, \*\*kwargs)

Map a function over blocks of arrays with some overlap

We share neighboring zones between blocks of the array, map a
function, and then trim away the neighboring strips. If depth is
larger than any chunk along a particular axis, then the array is
rechunked.

Note that this function will attempt to automatically determine the output
array type before computing it, please refer to the `meta` keyword argument
in `map_blocks` if you expect that the function will not succeed when
operating on 0-d arrays.

* **Parameters:**
  **func: function**
  : The function to apply to each extended block.
    If multiple arrays are provided, then the function should expect to
    receive chunks of each array in the same order.

  **args**

  **depth: int, tuple, dict or list, keyword only**
  : The number of elements that each block should share with its neighbors
    If a tuple or dict then this can be different per axis.
    If a list then each element of that list must be an int, tuple or dict
    defining depth for the corresponding array in args.
    Asymmetric depths may be specified using a dict value of (-/+) tuples.
    Note that asymmetric depths are currently only supported when
    `boundary` is ‘none’.
    The default value is 0.

  **boundary: str, tuple, dict or list, keyword only**
  : How to handle the boundaries.
    Values include ‘reflect’, ‘periodic’, ‘nearest’, ‘none’,
    or any constant value like 0 or np.nan.
    If a list then each element must be a str, tuple or dict defining the
    boundary for the corresponding array in args.

  **trim: bool, keyword only**
  : Whether or not to trim `depth` elements from each block after
    calling the map function.
    Set this to False if your mapping function already does this for you

  **align_arrays: bool, keyword only**
  : Whether or not to align chunks along equally sized dimensions when
    multiple arrays are provided.  This allows for larger chunks in some
    arrays to be broken into smaller ones that match chunk sizes in other
    arrays such that they are compatible for block function mapping. If
    this is false, then an error will be thrown if arrays do not already
    have the same number of blocks in each dimension.

  **allow_rechunk: bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.

  **\*\*kwargs:**
  : Other keyword arguments valid in `map_blocks`

### Examples

```pycon
>>> import numpy as np
>>> import dask.array as da
```

```pycon
>>> x = np.array([1, 1, 2, 3, 3, 3, 2, 1, 1])
>>> x = da.from_array(x, chunks=5)
>>> def derivative(x):
...     return x - np.roll(x, 1)
```

```pycon
>>> y = x.map_overlap(derivative, depth=1, boundary=0)
>>> y.compute()
array([ 1,  0,  1,  1,  0,  0, -1, -1,  0])
```

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> d.map_overlap(lambda x: x + x.size, depth=1, boundary='reflect').compute()
array([[16, 17, 18, 19],
       [20, 21, 22, 23],
       [24, 25, 26, 27],
       [28, 29, 30, 31]])
```

```pycon
>>> func = lambda x: x + x.size
>>> depth = {0: 1, 1: 1}
>>> boundary = {0: 'reflect', 1: 'none'}
>>> d.map_overlap(func, depth, boundary).compute()
array([[12,  13,  14,  15],
       [16,  17,  18,  19],
       [20,  21,  22,  23],
       [24,  25,  26,  27]])
```

The `da.map_overlap` function can also accept multiple arrays.

```pycon
>>> func = lambda x, y: x + y
>>> x = da.arange(8).reshape(2, 4).rechunk((1, 2))
>>> y = da.arange(4).rechunk(2)
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect').compute()
array([[ 0,  2,  4,  6],
       [ 4,  6,  8,  10]])
```

When multiple arrays are given, they do not need to have the
same number of dimensions but they must broadcast together.
Arrays are aligned block by block (just as in `da.map_blocks`)
so the blocks must have a common chunk size.  This common chunking
is determined automatically as long as `align_arrays` is True.

```pycon
>>> x = da.arange(8, chunks=4)
>>> y = da.arange(8, chunks=2)
>>> r = da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=True)
>>> len(r.to_delayed())
4
```

```pycon
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=False).compute()
Traceback (most recent call last):
    ...
ValueError: Shapes do not align {'.0': {2, 4}}
```

Note also that this function is equivalent to `map_blocks`
by default.  A non-zero `depth` must be defined for any
overlap to appear in the arrays provided to `func`.

```pycon
>>> func = lambda x: x.sum()
>>> x = da.ones(10, dtype='int')
>>> block_args = dict(chunks=(), drop_axis=0)
>>> da.map_blocks(func, x, **block_args).compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, boundary='reflect').compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, depth=1, boundary='reflect').compute()
np.int64(12)
```

For functions that may not handle 0-d arrays, it’s also possible to specify
`meta` with an empty array matching the type of the expected result. In
the example below, `func` will result in an `IndexError` when computing
`meta`:

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=np.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

Similarly, it’s possible to specify a non-NumPy array to `meta`:

```pycon
>>> import cupy
>>> x = cupy.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=cupy.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=cupy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.matmul.html.md

# dask.array.matmul

### dask.array.matmul(x1, x2, /, out=None, \*, casting='same_kind', order='K', dtype=None, subok=True)

This docstring was copied from numpy.matmul.

Some inconsistencies with the Dask version may exist.

Matrix product of two arrays.

* **Parameters:**
  **x1, x2**
  : Input arrays, scalars not allowed.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that matches the signature (n,k),(k,m)->(n,m). If not
    provided or None, a freshly-allocated array is returned.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The matrix product of the inputs.
    This is a scalar only when both x1, x2 are 1-d vectors.
* **Raises:**
  ValueError
  : If the last dimension of x1 is not the same size as
    the second-to-last dimension of x2.
    <br/>
    If a scalar value is passed in.

#### SEE ALSO
`vecdot`
: Complex-conjugating dot product for stacks of vectors.

`matvec`
: Matrix-vector product for stacks of matrices and vectors.

`vecmat`
: Vector-matrix product for stacks of vectors and matrices.

[`tensordot`](dask.array.tensordot.md#dask.array.tensordot)
: Sum products over arbitrary axes.

[`einsum`](dask.array.einsum.md#dask.array.einsum)
: Einstein summation convention.

[`dot`](dask.array.dot.md#dask.array.dot)
: alternative matrix product with different broadcasting rules.

### Notes

The behavior depends on the arguments in the following way.

- If both arguments are 2-D they are multiplied like conventional
  matrices.
- If either argument is N-D, N > 2, it is treated as a stack of
  matrices residing in the last two indexes and broadcast accordingly.
- If the first argument is 1-D, it is promoted to a matrix by
  prepending a 1 to its dimensions. After matrix multiplication
  the prepended 1 is removed. (For stacks of vectors, use `vecmat`.)
- If the second argument is 1-D, it is promoted to a matrix by
  appending a 1 to its dimensions. After matrix multiplication
  the appended 1 is removed. (For stacks of vectors, use `matvec`.)

`matmul` differs from `dot` in two important ways:

- Multiplication by scalars is not allowed, use `*` instead.
- Stacks of matrices are broadcast together as if the matrices
  were elements, respecting the signature `(n,k),(k,m)->(n,m)`:
  ```pycon
  >>> a = np.ones([9, 5, 7, 4])
  >>> c = np.ones([9, 5, 4, 3])
  >>> np.dot(a, c).shape
  (9, 5, 7, 9, 5, 3)
  >>> np.matmul(a, c).shape
  (9, 5, 7, 3)
  >>> # n is 7, k is 4, m is 3
  ```

The matmul function implements the semantics of the `@` operator
defined in [**PEP 465**](https://peps.python.org/pep-0465/).

It uses an optimized BLAS library when possible (see numpy.linalg).

### Examples

For 2-D arrays it is the matrix product:

```pycon
>>> import numpy as np
>>> a = np.array([[1, 0],
...               [0, 1]])
>>> b = np.array([[4, 1],
...               [2, 2]])
>>> np.matmul(a, b)
array([[4, 1],
       [2, 2]])
```

For 2-D mixed with 1-D, the result is the usual.

```pycon
>>> a = np.array([[1, 0],
...               [0, 1]])
>>> b = np.array([1, 2])
>>> np.matmul(a, b)
array([1, 2])
>>> np.matmul(b, a)
array([1, 2])
```

Broadcasting is conventional for stacks of arrays

```pycon
>>> a = np.arange(2 * 2 * 4).reshape((2, 2, 4))
>>> b = np.arange(2 * 2 * 4).reshape((2, 4, 2))
>>> np.matmul(a,b).shape
(2, 2, 2)
>>> np.matmul(a, b)[0, 1, 1]
98
>>> sum(a[0, 1, :] * b[0 , :, 1])
98
```

Vector, vector returns the scalar inner product, but neither argument
is complex-conjugated:

```pycon
>>> np.matmul([2j, 3j], [2j, 3j])
(-13+0j)
```

Scalar multiplication raises an error.

```pycon
>>> np.matmul([1,2], 3)
Traceback (most recent call last):
...
ValueError: matmul: Input operand 1 does not have enough dimensions ...
```

The `@` operator can be used as a shorthand for `np.matmul` on
ndarrays.

```pycon
>>> x1 = np.array([2j, 3j])
>>> x2 = np.array([2j, 3j])
>>> x1 @ x2
(-13+0j)
```

<!-- !! processed by numpydoc !! -->


# dask.array.max.html.md

# dask.array.max

### dask.array.max(a, axis=None, keepdims=False, split_every=None, out=None)

Return the maximum of an array or maximum along an axis.

This docstring was copied from numpy.max.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Axis or axes along which to operate.  By default, flattened input is
    used. If this is a tuple of ints, the maximum is selected over
    multiple axes, instead of a single axis or all the axes as before.

  **out**
  : Alternative output array in which to place the result.  Must
    be of the same shape and buffer length as the expected output.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the `max` method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **initial**
  : The minimum value of an output element. Must be present to allow
    computation on empty slice. See ~numpy.ufunc.reduce for details.

  **where**
  : Elements to compare for the maximum. See ~numpy.ufunc.reduce
    for details.
* **Returns:**
  **max**
  : Maximum of a. If axis is None, the result is a scalar value.
    If axis is an int, the result is an array of dimension
    `a.ndim - 1`. If axis is a tuple, the result is an array of
    dimension `a.ndim - len(axis)`.

#### SEE ALSO
[`min`](dask.array.min.md#dask.array.min)
: The minimum value of an array along a given axis, propagating any NaNs.

[`nanmax`](dask.array.nanmax.md#dask.array.nanmax)
: The maximum value of an array along a given axis, ignoring any NaNs.

[`maximum`](dask.array.maximum.md#dask.array.maximum)
: Element-wise maximum of two arrays, propagating any NaNs.

[`fmax`](dask.array.fmax.md#dask.array.fmax)
: Element-wise maximum of two arrays, ignoring any NaNs.

[`argmax`](dask.array.argmax.md#dask.array.argmax)
: Return the indices of the maximum values.

[`nanmin`](dask.array.nanmin.md#dask.array.nanmin), [`minimum`](dask.array.minimum.md#dask.array.minimum), [`fmin`](dask.array.fmin.md#dask.array.fmin)

### Notes

NaN values are propagated, that is if at least one item is NaN, the
corresponding max value will be NaN as well. To ignore NaN values
(MATLAB behavior), please use nanmax.

Don’t use ~numpy.max for element-wise comparison of 2 arrays; when
`a.shape[0]` is 2, `maximum(a[0], a[1])` is faster than
`max(a, axis=0)`.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(4).reshape((2,2))
>>> a
array([[0, 1],
       [2, 3]])
>>> np.max(a)           # Maximum of the flattened array
3
>>> np.max(a, axis=0)   # Maxima along the first axis
array([2, 3])
>>> np.max(a, axis=1)   # Maxima along the second axis
array([1, 3])
>>> np.max(a, where=[False, True], initial=-1, axis=0)
array([-1,  3])
>>> b = np.arange(5, dtype=np.float64)
>>> b[2] = np.nan
>>> np.max(b)
np.float64(nan)
>>> np.max(b, where=~np.isnan(b), initial=-1)
4.0
>>> np.nanmax(b)
4.0
```

You can use an initial value to compute the maximum of an empty slice, or
to initialize it to a different value:

```pycon
>>> np.max([[-50], [10]], axis=-1, initial=0)
array([ 0, 10])
```

Notice that the initial value is used as one of the elements for which the
maximum is determined, unlike for the default argument Python’s max
function, which is only used for empty iterables.

```pycon
>>> np.max([5], initial=6)
6
>>> max([5], default=6)
5
```

<!-- !! processed by numpydoc !! -->


# dask.array.maximum.html.md

# dask.array.maximum

### dask.array.maximum(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'maximum'>*

This docstring was copied from numpy.maximum.

Some inconsistencies with the Dask version may exist.

Element-wise maximum of array elements.

Compare two arrays and return a new array containing the element-wise
maxima. If one of the elements being compared is a NaN, then that
element is returned. If both elements are NaNs then the first is
returned. The latter distinction is important for complex NaNs, which
are defined as at least one of the real or imaginary parts being a NaN.
The net effect is that NaNs are propagated.

* **Parameters:**
  **x1, x2**
  : The arrays holding the elements to be compared.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The maximum of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`minimum`](dask.array.minimum.md#dask.array.minimum)
: Element-wise minimum of two arrays, propagates NaNs.

[`fmax`](dask.array.fmax.md#dask.array.fmax)
: Element-wise maximum of two arrays, ignores NaNs.

`amax`
: The maximum value of an array along a given axis, propagates NaNs.

[`nanmax`](dask.array.nanmax.md#dask.array.nanmax)
: The maximum value of an array along a given axis, ignores NaNs.

[`fmin`](dask.array.fmin.md#dask.array.fmin), `amin`, [`nanmin`](dask.array.nanmin.md#dask.array.nanmin)

### Notes

The maximum is equivalent to `np.where(x1 >= x2, x1, x2)` when
neither x1 nor x2 are nans, but it is faster and does proper
broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.maximum([2, 3, 4], [1, 5, 2])
array([2, 5, 4])
```

```pycon
>>> np.maximum(np.eye(2), [0.5, 2]) # broadcasting
array([[ 1. ,  2. ],
       [ 0.5,  2. ]])
```

```pycon
>>> np.maximum([np.nan, 0, np.nan], [0, np.nan, np.nan])
array([nan, nan, nan])
>>> np.maximum(np.inf, 1)
inf
```

<!-- !! processed by numpydoc !! -->


# dask.array.mean.html.md

# dask.array.mean

### dask.array.mean(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Compute the arithmetic mean along the specified axis.

This docstring was copied from numpy.mean.

Some inconsistencies with the Dask version may exist.

Returns the average of the array elements.  The average is taken over
the flattened array by default, otherwise over the specified axis.
float64 intermediate and return values are used for integer inputs.

* **Parameters:**
  **a**
  : Array containing numbers whose mean is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the means are computed. The default is to
    compute the mean of the flattened array.
    <br/>
    If this is a tuple of ints, a mean is performed over multiple axes,
    instead of a single axis or all the axes as before.

  **dtype**
  : Type to use in computing the mean.  For integer inputs, the default
    is float64; for floating point inputs, it is the same as the
    input dtype.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`; if provided, it must have the same shape as the
    expected output, but the type will be cast if necessary.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the mean method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **where**
  : Elements to include in the mean. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **m**
  : If out=None, returns a new array containing the mean values,
    otherwise a reference to the output array is returned.

#### SEE ALSO
[`average`](dask.array.average.md#dask.array.average)
: Weighted average

[`std`](dask.array.std.md#dask.array.std), [`var`](dask.array.var.md#dask.array.var), [`nanmean`](dask.array.nanmean.md#dask.array.nanmean), [`nanstd`](dask.array.nanstd.md#dask.array.nanstd), [`nanvar`](dask.array.nanvar.md#dask.array.nanvar)

### Notes

The arithmetic mean is the sum of the elements along the axis divided
by the number of elements.

Note that for floating-point input, the mean is computed using the
same precision the input has.  Depending on the input data, this can
cause the results to be inaccurate, especially for float32 (see
example below).  Specifying a higher-precision accumulator using the
dtype keyword can alleviate this issue.

By default, float16 results are computed using float32 intermediates
for extra precision.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, 4]])
>>> np.mean(a)
2.5
>>> np.mean(a, axis=0)
array([2., 3.])
>>> np.mean(a, axis=1)
array([1.5, 3.5])
```

In single precision, mean can be inaccurate:

```pycon
>>> a = np.zeros((2, 512*512), dtype=np.float32)
>>> a[0, :] = 1.0
>>> a[1, :] = 0.1
>>> np.mean(a)
np.float32(0.54999924)
```

Computing the mean in float64 is more accurate:

```pycon
>>> np.mean(a, dtype=np.float64)
0.55000000074505806 # may vary
```

Computing the mean in timedelta64 is available:

```pycon
>>> b = np.array([1, 3], dtype="timedelta64[D]")
>>> np.mean(b)
np.timedelta64(2,'D')
```

Specifying a where argument:

```pycon
>>> a = np.array([[5, 9, 13], [14, 10, 12], [11, 15, 19]])
>>> np.mean(a)
12.0
>>> np.mean(a, where=[[True], [False], [False]])
9.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.median.html.md

# dask.array.median

### dask.array.median(a, axis=None, keepdims=False, out=None)

Compute the median along the specified axis.

This docstring was copied from numpy.median.

Some inconsistencies with the Dask version may exist.

This works by automatically chunking the reduced axes to a single chunk if necessary
and then calling `numpy.median` function across the remaining dimensions

Returns the median of the array elements.

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array.

  **axis**
  : Axis or axes along which the medians are computed. The default,
    axis=None, will compute the median along a flattened version of
    the array. If a sequence of axes, the array is first flattened
    along the given axes, then the median is computed along the
    resulting flattened axis.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output,
    but the type (of the output) will be cast if necessary.

  **overwrite_input**
  : If True, then allow use of memory of input array a for
    calculations. The input array will be modified by the call to
    median. This will save memory when you do not need to preserve
    the contents of the input array. Treat the input as undefined,
    but it will probably be fully or partially sorted. Default is
    False. If overwrite_input is `True` and a is not already an
    ndarray, an error will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original arr.
* **Returns:**
  **median**
  : A new array holding the result. If the input contains integers
    or floats smaller than `float64`, then the output data-type is
    `np.float64`.  Otherwise, the data-type of the output is the
    same as that of the input. If out is specified, that array is
    returned instead.

#### SEE ALSO
[`mean`](dask.array.mean.md#dask.array.mean), [`percentile`](dask.array.percentile.md#dask.array.percentile)

### Notes

Given a vector `V` of length `N`, the median of `V` is the
middle value of a sorted copy of `V`, `V_sorted` - i
e., `V_sorted[(N-1)/2]`, when `N` is odd, and the average of the
two middle values of `V_sorted` when `N` is even.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[10, 7, 4], [3, 2, 1]])
>>> a
array([[10,  7,  4],
       [ 3,  2,  1]])
>>> np.median(a)
np.float64(3.5)
>>> np.median(a, axis=0)
array([6.5, 4.5, 2.5])
>>> np.median(a, axis=1)
array([7.,  2.])
>>> np.median(a, axis=(0, 1))
np.float64(3.5)
>>> m = np.median(a, axis=0)
>>> out = np.zeros_like(m)
>>> np.median(a, axis=0, out=m)
array([6.5,  4.5,  2.5])
>>> m
array([6.5,  4.5,  2.5])
>>> b = a.copy()
>>> np.median(b, axis=1, overwrite_input=True)
array([7.,  2.])
>>> assert not np.all(a==b)
>>> b = a.copy()
>>> np.median(b, axis=None, overwrite_input=True)
np.float64(3.5)
>>> assert not np.all(a==b)
```

<!-- !! processed by numpydoc !! -->


# dask.array.meshgrid.html.md

# dask.array.meshgrid

### dask.array.meshgrid(\*xi, sparse=False, indexing='xy', \*\*kwargs)

Return a tuple of coordinate matrices from coordinate vectors.

This docstring was copied from numpy.meshgrid.

Some inconsistencies with the Dask version may exist.

Make N-D coordinate arrays for vectorized evaluations of
N-D scalar/vector fields over N-D grids, given
one-dimensional coordinate arrays x1, x2,…, xn.

* **Parameters:**
  **x1, x2,…, xn**
  : 1-D arrays representing the coordinates of a grid.

  **indexing**
  : Cartesian (‘xy’, default) or matrix (‘ij’) indexing of output.
    See Notes for more details.

  **sparse**
  : If True the shape of the returned coordinate array for dimension *i*
    is reduced from `(N1, ..., Ni, ... Nn)` to
    `(1, ..., 1, Ni, 1, ..., 1)`.  These sparse coordinate grids are
    intended to be used with [Broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html#basics-broadcasting).  When all
    coordinates are used in an expression, broadcasting still leads to a
    fully-dimensional result array.
    <br/>
    Default is False.

  **copy**
  : If False, a view into the original arrays are returned in order to
    conserve memory.  Default is True.  Please note that
    `sparse=False, copy=False` will likely return non-contiguous
    arrays.  Furthermore, more than one element of a broadcast array
    may refer to a single memory location.  If you need to write to the
    arrays, make copies first.
* **Returns:**
  **X1, X2,…, XN**
  : For vectors x1, x2,…, xn with lengths `Ni=len(xi)`,
    returns `(N1, N2, N3,..., Nn)` shaped arrays if indexing=’ij’
    or `(N2, N1, N3,..., Nn)` shaped arrays if indexing=’xy’
    with the elements of xi repeated to fill the matrix along
    the first dimension for x1, the second for x2 and so on.

#### SEE ALSO
`mgrid`
: Construct a multi-dimensional “meshgrid” using indexing notation.

`ogrid`
: Construct an open multi-dimensional “meshgrid” using indexing notation.

[How to index ndarrays](https://numpy.org/doc/stable/user/how-to-index.html#how-to-index)

### Notes

This function supports both indexing conventions through the indexing
keyword argument.  Giving the string ‘ij’ returns a meshgrid with
matrix indexing, while ‘xy’ returns a meshgrid with Cartesian indexing.
In the 2-D case with inputs of length M and N, the outputs are of shape
(N, M) for ‘xy’ indexing and (M, N) for ‘ij’ indexing.  In the 3-D case
with inputs of length M, N and P, outputs are of shape (N, M, P) for
‘xy’ indexing and (M, N, P) for ‘ij’ indexing.  The difference is
illustrated by the following code snippet:

```default
xv, yv = np.meshgrid(x, y, indexing='ij')
for i in range(nx):
    for j in range(ny):
        # treat xv[i,j], yv[i,j]

xv, yv = np.meshgrid(x, y, indexing='xy')
for i in range(nx):
    for j in range(ny):
        # treat xv[j,i], yv[j,i]
```

In the 1-D and 0-D case, the indexing and sparse keywords have no effect.

### Examples

```pycon
>>> import numpy as np
>>> nx, ny = (3, 2)
>>> x = np.linspace(0, 1, nx)
>>> y = np.linspace(0, 1, ny)
>>> xv, yv = np.meshgrid(x, y)
>>> xv
array([[0. , 0.5, 1. ],
       [0. , 0.5, 1. ]])
>>> yv
array([[0.,  0.,  0.],
       [1.,  1.,  1.]])
```

The result of meshgrid is a coordinate grid:

```pycon
>>> import matplotlib.pyplot as plt
>>> plt.plot(xv, yv, marker='o', color='k', linestyle='none')
>>> plt.show()
```

You can create sparse output arrays to save memory and computation time.

```pycon
>>> xv, yv = np.meshgrid(x, y, sparse=True)
>>> xv
array([[0. ,  0.5,  1. ]])
>>> yv
array([[0.],
       [1.]])
```

meshgrid is very useful to evaluate functions on a grid. If the
function depends on all coordinates, both dense and sparse outputs can be
used.

```pycon
>>> x = np.linspace(-5, 5, 101)
>>> y = np.linspace(-5, 5, 101)
>>> # full coordinate arrays
>>> xx, yy = np.meshgrid(x, y)
>>> zz = np.sqrt(xx**2 + yy**2)
>>> xx.shape, yy.shape, zz.shape
((101, 101), (101, 101), (101, 101))
>>> # sparse coordinate arrays
>>> xs, ys = np.meshgrid(x, y, sparse=True)
>>> zs = np.sqrt(xs**2 + ys**2)
>>> xs.shape, ys.shape, zs.shape
((1, 101), (101, 1), (101, 101))
>>> np.array_equal(zz, zs)
True
```

```pycon
>>> h = plt.contourf(x, y, zs)
>>> plt.axis('scaled')
>>> plt.colorbar()
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.min.html.md

# dask.array.min

### dask.array.min(a, axis=None, keepdims=False, split_every=None, out=None)

Return the minimum of an array or minimum along an axis.

This docstring was copied from numpy.min.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Axis or axes along which to operate.  By default, flattened input is
    used.
    <br/>
    If this is a tuple of ints, the minimum is selected over multiple axes,
    instead of a single axis or all the axes as before.

  **out**
  : Alternative output array in which to place the result.  Must
    be of the same shape and buffer length as the expected output.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the `min` method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **initial**
  : The maximum value of an output element. Must be present to allow
    computation on empty slice. See ~numpy.ufunc.reduce for details.

  **where**
  : Elements to compare for the minimum. See ~numpy.ufunc.reduce
    for details.
* **Returns:**
  **min**
  : Minimum of a. If axis is None, the result is a scalar value.
    If axis is an int, the result is an array of dimension
    `a.ndim - 1`.  If axis is a tuple, the result is an array of
    dimension `a.ndim - len(axis)`.

#### SEE ALSO
[`max`](dask.array.max.md#dask.array.max)
: The maximum value of an array along a given axis, propagating any NaNs.

[`nanmin`](dask.array.nanmin.md#dask.array.nanmin)
: The minimum value of an array along a given axis, ignoring any NaNs.

[`minimum`](dask.array.minimum.md#dask.array.minimum)
: Element-wise minimum of two arrays, propagating any NaNs.

[`fmin`](dask.array.fmin.md#dask.array.fmin)
: Element-wise minimum of two arrays, ignoring any NaNs.

[`argmin`](dask.array.argmin.md#dask.array.argmin)
: Return the indices of the minimum values.

[`nanmax`](dask.array.nanmax.md#dask.array.nanmax), [`maximum`](dask.array.maximum.md#dask.array.maximum), [`fmax`](dask.array.fmax.md#dask.array.fmax)

### Notes

NaN values are propagated, that is if at least one item is NaN, the
corresponding min value will be NaN as well. To ignore NaN values
(MATLAB behavior), please use nanmin.

Don’t use ~numpy.min for element-wise comparison of 2 arrays; when
`a.shape[0]` is 2, `minimum(a[0], a[1])` is faster than
`min(a, axis=0)`.

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(4).reshape((2,2))
>>> a
array([[0, 1],
       [2, 3]])
>>> np.min(a)           # Minimum of the flattened array
0
>>> np.min(a, axis=0)   # Minima along the first axis
array([0, 1])
>>> np.min(a, axis=1)   # Minima along the second axis
array([0, 2])
>>> np.min(a, where=[False, True], initial=10, axis=0)
array([10,  1])
```

```pycon
>>> b = np.arange(5, dtype=np.float64)
>>> b[2] = np.nan
>>> np.min(b)
np.float64(nan)
>>> np.min(b, where=~np.isnan(b), initial=10)
0.0
>>> np.nanmin(b)
0.0
```

```pycon
>>> np.min([[-50], [10]], axis=-1, initial=0)
array([-50,   0])
```

Notice that the initial value is used as one of the elements for which the
minimum is determined, unlike for the default argument Python’s max
function, which is only used for empty iterables.

Notice that this isn’t the same as Python’s `default` argument.

```pycon
>>> np.min([6], initial=5)
5
>>> min([6], default=5)
6
```

<!-- !! processed by numpydoc !! -->


# dask.array.minimum.html.md

# dask.array.minimum

### dask.array.minimum(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'minimum'>*

This docstring was copied from numpy.minimum.

Some inconsistencies with the Dask version may exist.

Element-wise minimum of array elements.

Compare two arrays and return a new array containing the element-wise
minima. If one of the elements being compared is a NaN, then that
element is returned. If both elements are NaNs then the first is
returned. The latter distinction is important for complex NaNs, which
are defined as at least one of the real or imaginary parts being a NaN.
The net effect is that NaNs are propagated.

* **Parameters:**
  **x1, x2**
  : The arrays holding the elements to be compared.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The minimum of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`maximum`](dask.array.maximum.md#dask.array.maximum)
: Element-wise maximum of two arrays, propagates NaNs.

[`fmin`](dask.array.fmin.md#dask.array.fmin)
: Element-wise minimum of two arrays, ignores NaNs.

`amin`
: The minimum value of an array along a given axis, propagates NaNs.

[`nanmin`](dask.array.nanmin.md#dask.array.nanmin)
: The minimum value of an array along a given axis, ignores NaNs.

[`fmax`](dask.array.fmax.md#dask.array.fmax), `amax`, [`nanmax`](dask.array.nanmax.md#dask.array.nanmax)

### Notes

The minimum is equivalent to `np.where(x1 <= x2, x1, x2)` when
neither x1 nor x2 are NaNs, but it is faster and does proper
broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.minimum([2, 3, 4], [1, 5, 2])
array([1, 3, 2])
```

```pycon
>>> np.minimum(np.eye(2), [0.5, 2]) # broadcasting
array([[ 0.5,  0. ],
       [ 0. ,  1. ]])
```

```pycon
>>> np.minimum([np.nan, 0, np.nan],[0, np.nan, np.nan])
array([nan, nan, nan])
>>> np.minimum(-np.inf, 1)
-inf
```

<!-- !! processed by numpydoc !! -->


# dask.array.mod.html.md

# dask.array.mod

### dask.array.mod(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'remainder'>*

This docstring was copied from numpy.remainder.

Some inconsistencies with the Dask version may exist.

Returns the element-wise remainder of division.

Computes the remainder complementary to the floor_divide function.  It is
equivalent to the Python modulus operator `x1 % x2` and has the same sign
as the divisor x2. The MATLAB function equivalent to `np.remainder`
is `mod`.

#### WARNING
This should not be confused with:

* Python’s math.remainder and C’s `remainder`, which
  compute the IEEE remainder, which are the complement to
  `round(x1 / x2)`.
* The MATLAB `rem` function and or the C `%` operator which is the
  complement to `int(x1 / x2)`.

* **Parameters:**
  **x1**
  : Dividend array.

  **x2**
  : Divisor array.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The element-wise remainder of the quotient `floor_divide(x1, x2)`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`floor_divide`](dask.array.floor_divide.md#dask.array.floor_divide)
: Equivalent of Python `//` operator.

[`divmod`](dask.array.divmod.md#dask.array.divmod)
: Simultaneous floor division and remainder.

[`fmod`](dask.array.fmod.md#dask.array.fmod)
: Equivalent of the MATLAB `rem` function.

[`divide`](dask.array.divide.md#dask.array.divide), [`floor`](dask.array.floor.md#dask.array.floor)

### Notes

Returns 0 when x2 is 0 and both x1 and x2 are (arrays of)
integers.
`mod` is an alias of `remainder`.

### Examples

```pycon
>>> import numpy as np
>>> np.remainder([4, 7], [2, 3])
array([0, 1])
>>> np.remainder(np.arange(7), 5)
array([0, 1, 2, 3, 4, 0, 1])
```

The `%` operator can be used as a shorthand for `np.remainder` on
ndarrays.

```pycon
>>> x1 = np.arange(7)
>>> x1 % 5
array([0, 1, 2, 3, 4, 0, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.modf.html.md

# dask.array.modf

### dask.array.modf(x, /, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True)

This docstring was copied from numpy.modf.

Some inconsistencies with the Dask version may exist.

Return the fractional and integral parts of an array, element-wise.

The fractional and integral parts are negative if the given number is
negative.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y1**
  : Fractional part of x.
    This is a scalar if x is a scalar.

  **y2**
  : Integral part of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`divmod`](dask.array.divmod.md#dask.array.divmod)
: `divmod(x, 1)` is equivalent to `modf` with the return values switched, except it always has a positive remainder.

### Notes

For integer input the return values are floats.

### Examples

```pycon
>>> import numpy as np
>>> np.modf([0, 3.5])
(array([ 0. ,  0.5]), array([ 0.,  3.]))
>>> np.modf(-0.5)
(-0.5, -0)
```

<!-- !! processed by numpydoc !! -->


# dask.array.moment.html.md

# dask.array.moment

### dask.array.moment(a, order, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Calculate the nth centralized moment.

* **Parameters:**
  **a**
  : Data over which to compute moment

  **order**
  : Order of the moment that is returned, must be >= 2.

  **axis**
  : Axis along which the central moment is computed. The default is to
    compute the moment of the flattened array.

  **dtype**
  : Type to use in computing the moment. For arrays of integer type the
    default is float64; for arrays of float types it is the same as the
    array type.

  **keepdims**
  : If this is set to True, the axes which are reduced are left in the
    result as dimensions with size one. With this option, the result
    will broadcast correctly against the original array.

  **ddof**
  : “Delta Degrees of Freedom”: the divisor used in the calculation is
    N - ddof, where N represents the number of elements. By default
    ddof is zero.
* **Returns:**
  **moment**

### References

<!-- !! processed by numpydoc !! -->


# dask.array.moveaxis.html.md

# dask.array.moveaxis

### dask.array.moveaxis(a, source, destination)

Move axes of an array to new positions.

This docstring was copied from numpy.moveaxis.

Some inconsistencies with the Dask version may exist.

Other axes remain in their original order.

* **Parameters:**
  **a**
  : The array whose axes should be reordered.

  **source**
  : Original positions of the axes to move. These must be unique.

  **destination**
  : Destination positions for each of the original axes. These must also be
    unique.
* **Returns:**
  **result**
  : Array with moved axes. This array is a view of the input array.

#### SEE ALSO
[`transpose`](dask.array.transpose.md#dask.array.transpose)
: Permute the dimensions of an array.

[`swapaxes`](dask.array.swapaxes.md#dask.array.swapaxes)
: Interchange two axes of an array.

### Examples

```pycon
>>> import numpy as np
>>> x = np.zeros((3, 4, 5))
>>> np.moveaxis(x, 0, -1).shape
(4, 5, 3)
>>> np.moveaxis(x, -1, 0).shape
(5, 3, 4)
```

These all achieve the same result:

```pycon
>>> np.transpose(x).shape
(5, 4, 3)
>>> np.swapaxes(x, 0, -1).shape
(5, 4, 3)
>>> np.moveaxis(x, [0, 1], [-1, -2]).shape
(5, 4, 3)
>>> np.moveaxis(x, [0, 1, 2], [-1, -2, -3]).shape
(5, 4, 3)
```

<!-- !! processed by numpydoc !! -->


# dask.array.multiply.html.md

# dask.array.multiply

### dask.array.multiply(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'multiply'>*

This docstring was copied from numpy.multiply.

Some inconsistencies with the Dask version may exist.

Multiply arguments element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays to be multiplied.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The product of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

### Notes

Equivalent to x1 \* x2 in terms of array broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.multiply(2.0, 4.0)
8.0
```

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> np.multiply(x1, x2)
array([[  0.,   1.,   4.],
       [  0.,   4.,  10.],
       [  0.,   7.,  16.]])
```

The `*` operator can be used as a shorthand for `np.multiply` on
ndarrays.

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> x1 * x2
array([[  0.,   1.,   4.],
       [  0.,   4.,  10.],
       [  0.,   7.,  16.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nan_to_num.html.md

# dask.array.nan_to_num

### dask.array.nan_to_num(\*args, \*\*kwargs)

Replace NaN with zero and infinity with large finite numbers (default
behaviour) or with the numbers defined by the user using the nan,
posinf and/or neginf keywords.

This docstring was copied from numpy.nan_to_num.

Some inconsistencies with the Dask version may exist.

If x is inexact, NaN is replaced by zero or by the user defined value in
nan keyword, infinity is replaced by the largest finite floating point
values representable by `x.dtype` or by the user defined value in
posinf keyword and -infinity is replaced by the most negative finite
floating point values representable by `x.dtype` or by the user defined
value in neginf keyword.

For complex dtypes, the above is applied to each of the real and
imaginary components of x separately.

If x is not inexact, then no replacements are made.

* **Parameters:**
  **x**
  : Input data.

  **copy**
  : Whether to create a copy of x (True) or to replace values
    in-place (False). The in-place operation only occurs if
    casting to an array does not require a copy.
    Default is True.

  **nan**
  : Values to be used to fill NaN values. If no values are passed
    then NaN values will be replaced with 0.0.

  **posinf**
  : Values to be used to fill positive infinity values. If no values are
    passed then positive infinity values will be replaced with a very
    large number.

  **neginf**
  : Values to be used to fill negative infinity values. If no values are
    passed then negative infinity values will be replaced with a very
    small (or negative) number.
* **Returns:**
  **out**
  : x, with the non-finite values replaced. If copy is False, this may
    be x itself.

#### SEE ALSO
[`isinf`](dask.array.isinf.md#dask.array.isinf)
: Shows which elements are positive or negative infinity.

[`isneginf`](dask.array.isneginf.md#dask.array.isneginf)
: Shows which elements are negative infinity.

[`isposinf`](dask.array.isposinf.md#dask.array.isposinf)
: Shows which elements are positive infinity.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Shows which elements are Not a Number (NaN).

[`isfinite`](dask.array.isfinite.md#dask.array.isfinite)
: Shows which elements are finite (not NaN, not infinity)

### Notes

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754). This means that Not a Number is not equivalent to infinity.

### Examples

```pycon
>>> import numpy as np
>>> np.nan_to_num(np.inf)
1.7976931348623157e+308
>>> np.nan_to_num(-np.inf)
-1.7976931348623157e+308
>>> np.nan_to_num(np.nan)
0.0
>>> x = np.array([np.inf, -np.inf, np.nan, -128, 128])
>>> np.nan_to_num(x)
array([ 1.79769313e+308, -1.79769313e+308,  0.00000000e+000, # may vary
       -1.28000000e+002,  1.28000000e+002])
>>> np.nan_to_num(x, nan=-9999, posinf=33333333, neginf=33333333)
array([ 3.3333333e+07,  3.3333333e+07, -9.9990000e+03,
       -1.2800000e+02,  1.2800000e+02])
>>> nan = np.array([11, 12, -9999, 13, 14])
>>> posinf = np.array([33333333, 11, 12, 13, 14])
>>> neginf = np.array([11, 33333333, 12, 13, 14])
>>> np.nan_to_num(x, nan=nan, posinf=posinf, neginf=neginf)
array([ 3.3333333e+07,  3.3333333e+07, -9.9990000e+03, -1.2800000e+02,
        1.2800000e+02])
>>> y = np.array([complex(np.inf, np.nan), np.nan, complex(np.nan, np.inf)])
array([  1.79769313e+308,  -1.79769313e+308,   0.00000000e+000, # may vary
     -1.28000000e+002,   1.28000000e+002])
>>> np.nan_to_num(y)
array([  1.79769313e+308 +0.00000000e+000j, # may vary
         0.00000000e+000 +0.00000000e+000j,
         0.00000000e+000 +1.79769313e+308j])
>>> np.nan_to_num(y, nan=111111, posinf=222222)
array([222222.+111111.j, 111111.     +0.j, 111111.+222222.j])
>>> nan = np.array([11, 12, 13])
>>> posinf = np.array([21, 22, 23])
>>> neginf = np.array([31, 32, 33])
>>> np.nan_to_num(y, nan=nan, posinf=posinf, neginf=neginf)
array([21.+11.j, 12. +0.j, 13.+23.j])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanargmax.html.md

# dask.array.nanargmax

### dask.array.nanargmax(a, axis=None, keepdims=False, split_every=None, out=None)

Return the indices of the maximum values in the specified axis ignoring
NaNs. For all-NaN slices `ValueError` is raised. Warning: the
results cannot be trusted if a slice contains only NaNs and -Infs.

This docstring was copied from numpy.nanargmax.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Axis along which to operate.  By default flattened input is used.

  **out**
  : If provided, the result will be inserted into this array. It should
    be of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the array.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **index_array**
  : An array of indices or a single index value.

#### SEE ALSO
[`argmax`](dask.array.argmax.md#dask.array.argmax), [`nanargmin`](dask.array.nanargmin.md#dask.array.nanargmin)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[np.nan, 4], [2, 3]])
>>> np.argmax(a)
0
>>> np.nanargmax(a)
1
>>> np.nanargmax(a, axis=0)
array([1, 0])
>>> np.nanargmax(a, axis=1)
array([1, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanargmin.html.md

# dask.array.nanargmin

### dask.array.nanargmin(a, axis=None, keepdims=False, split_every=None, out=None)

Return the indices of the minimum values in the specified axis ignoring
NaNs. For all-NaN slices `ValueError` is raised. Warning: the results
cannot be trusted if a slice contains only NaNs and Infs.

This docstring was copied from numpy.nanargmin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Axis along which to operate.  By default flattened input is used.

  **out**
  : If provided, the result will be inserted into this array. It should
    be of the appropriate shape and dtype.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the array.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **index_array**
  : An array of indices or a single index value.

#### SEE ALSO
[`argmin`](dask.array.argmin.md#dask.array.argmin), [`nanargmax`](dask.array.nanargmax.md#dask.array.nanargmax)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[np.nan, 4], [2, 3]])
>>> np.argmin(a)
0
>>> np.nanargmin(a)
2
>>> np.nanargmin(a, axis=0)
array([1, 1])
>>> np.nanargmin(a, axis=1)
array([1, 0])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nancumprod.html.md

# dask.array.nancumprod

### dask.array.nancumprod(x, axis, dtype=None, out=None, , method='sequential')

Return the cumulative product of array elements over a given axis treating Not a
Numbers (NaNs) as one.  The cumulative product does not change when NaNs are
encountered and leading NaNs are replaced by ones.

This docstring was copied from numpy.nancumprod.

Some inconsistencies with the Dask version may exist.

Dask added an additional keyword-only argument `method`.

method
: Choose which method to use to perform the cumprod.  Default is ‘sequential’.
  <br/>
  * ‘sequential’ performs the cumprod of each prior block before the current block.
  * ‘blelloch’ is a work-efficient parallel cumprod.  It exposes parallelism by first
    : taking the product of each block and combines the products via a binary tree.
      This method may be faster or more memory efficient depending on workload,
      scheduler, and hardware.  More benchmarking is necessary.

Ones are returned for slices that are all-NaN or empty.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : Axis along which the cumulative product is computed.  By default
    the input is flattened.

  **dtype**
  : Type of the returned array, as well as of the accumulator in which
    the elements are multiplied.  If *dtype* is not specified, it
    defaults to the dtype of a, unless a has an integer dtype with
    a precision less than that of the default platform integer.  In
    that case, the default platform integer is used instead.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output
    but the type of the resulting values will be cast if necessary.
* **Returns:**
  **nancumprod**
  : A new array holding the result is returned unless out is
    specified, in which case it is returned.

#### SEE ALSO
[`numpy.cumprod`](https://numpy.org/doc/stable/reference/generated/numpy.cumprod.html#numpy.cumprod)
: Cumulative product across array propagating NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Show which elements are NaN.

### Examples

```pycon
>>> import numpy as np
>>> np.nancumprod(1)
array([1])
>>> np.nancumprod([1])
array([1])
>>> np.nancumprod([1, np.nan])
array([1.,  1.])
>>> a = np.array([[1, 2], [3, np.nan]])
>>> np.nancumprod(a)
array([1.,  2.,  6.,  6.])
>>> np.nancumprod(a, axis=0)
array([[1.,  2.],
       [3.,  2.]])
>>> np.nancumprod(a, axis=1)
array([[1.,  2.],
       [3.,  3.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nancumsum.html.md

# dask.array.nancumsum

### dask.array.nancumsum(x, axis, dtype=None, out=None, , method='sequential')

Return the cumulative sum of array elements over a given axis treating Not a
Numbers (NaNs) as zero.  The cumulative sum does not change when NaNs are
encountered and leading NaNs are replaced by zeros.

This docstring was copied from numpy.nancumsum.

Some inconsistencies with the Dask version may exist.

Dask added an additional keyword-only argument `method`.

method
: Choose which method to use to perform the cumsum.  Default is ‘sequential’.
  <br/>
  * ‘sequential’ performs the cumsum of each prior block before the current block.
  * ‘blelloch’ is a work-efficient parallel cumsum.  It exposes parallelism by
    : first taking the sum of each block and combines the sums via a binary tree.
      This method may be faster or more memory efficient depending on workload,
      scheduler, and hardware.  More benchmarking is necessary.

Zeros are returned for slices that are all-NaN or empty.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : Axis along which the cumulative sum is computed. The default
    (None) is to compute the cumsum over the flattened array.

  **dtype**
  : Type of the returned array and of the accumulator in which the
    elements are summed.  If dtype is not specified, it defaults
    to the dtype of a, unless a has an integer dtype with a
    precision less than that of the default platform integer.  In
    that case, the default platform integer is used.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output
    but the type will be cast if necessary. See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for
    more details.
* **Returns:**
  **nancumsum**
  : A new array holding the result is returned unless out is
    specified, in which it is returned. The result has the same
    size as a, and the same shape as a if axis is not None
    or a is a 1-d array.

#### SEE ALSO
[`numpy.cumsum`](https://numpy.org/doc/stable/reference/generated/numpy.cumsum.html#numpy.cumsum)
: Cumulative sum across array propagating NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Show which elements are NaN.

### Examples

```pycon
>>> import numpy as np
>>> np.nancumsum(1)
array([1])
>>> np.nancumsum([1])
array([1])
>>> np.nancumsum([1, np.nan])
array([1.,  1.])
>>> a = np.array([[1, 2], [3, np.nan]])
>>> np.nancumsum(a)
array([1.,  3.,  6.,  6.])
>>> np.nancumsum(a, axis=0)
array([[1.,  2.],
       [4.,  2.]])
>>> np.nancumsum(a, axis=1)
array([[1.,  3.],
       [3.,  3.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanmax.html.md

# dask.array.nanmax

### dask.array.nanmax(a, axis=None, keepdims=False, split_every=None, out=None)

Return the maximum of an array or maximum along an axis, ignoring any
NaNs.  When all-NaN slices are encountered a `RuntimeWarning` is
raised and NaN is returned for that slice.

This docstring was copied from numpy.nanmax.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Array containing numbers whose maximum is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the maximum is computed. The default is to compute
    the maximum of the flattened array.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`; if provided, it must have the same shape as the
    expected output, but the type will be cast if necessary. See
    [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    If the value is anything but the default, then
    keepdims will be passed through to the max method
    of sub-classes of ndarray.  If the sub-classes methods
    does not implement keepdims any exceptions will be raised.

  **initial**
  : The minimum value of an output element. Must be present to allow
    computation on empty slice. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **where**
  : Elements to compare for the maximum. See ~numpy.ufunc.reduce
    for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **nanmax**
  : An array with the same shape as a, with the specified axis removed.
    If a is a 0-d array, or if axis is None, an ndarray scalar is
    returned.  The same dtype as a is returned.

#### SEE ALSO
[`nanmin`](dask.array.nanmin.md#dask.array.nanmin)
: The minimum value of an array along a given axis, ignoring any NaNs.

`amax`
: The maximum value of an array along a given axis, propagating any NaNs.

[`fmax`](dask.array.fmax.md#dask.array.fmax)
: Element-wise maximum of two arrays, ignoring any NaNs.

[`maximum`](dask.array.maximum.md#dask.array.maximum)
: Element-wise maximum of two arrays, propagating any NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Shows which elements are Not a Number (NaN).

[`isfinite`](dask.array.isfinite.md#dask.array.isfinite)
: Shows which elements are neither NaN nor infinity.

`amin`, [`fmin`](dask.array.fmin.md#dask.array.fmin), [`minimum`](dask.array.minimum.md#dask.array.minimum)

### Notes

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754). This means that Not a Number is not equivalent to infinity.
Positive infinity is treated as a very large number and negative
infinity is treated as a very small (i.e. negative) number.

If the input has a integer type the function is equivalent to np.max.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, np.nan]])
>>> np.nanmax(a)
3.0
>>> np.nanmax(a, axis=0)
array([3.,  2.])
>>> np.nanmax(a, axis=1)
array([2.,  3.])
```

When positive infinity and negative infinity are present:

```pycon
>>> np.nanmax([1, 2, np.nan, -np.inf])
2.0
>>> np.nanmax([1, 2, np.nan, np.inf])
inf
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanmean.html.md

# dask.array.nanmean

### dask.array.nanmean(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Compute the arithmetic mean along the specified axis, ignoring NaNs.

This docstring was copied from numpy.nanmean.

Some inconsistencies with the Dask version may exist.

Returns the average of the array elements.  The average is taken over
the flattened array by default, otherwise over the specified axis.
float64 intermediate and return values are used for integer inputs.

For all-NaN slices, NaN is returned and a RuntimeWarning is raised.

* **Parameters:**
  **a**
  : Array containing numbers whose mean is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the means are computed. The default is to compute
    the mean of the flattened array.

  **dtype**
  : Type to use in computing the mean.  For integer inputs, the default
    is float64; for inexact inputs, it is the same as the input
    dtype.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`; if provided, it must have the same shape as the
    expected output, but the type will be cast if necessary.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    <br/>
    If the value is anything but the default, then
    keepdims will be passed through to the mean or sum methods
    of sub-classes of ndarray.  If the sub-classes methods
    does not implement keepdims any exceptions will be raised.

  **where**
  : Elements to include in the mean. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **m**
  : If out=None, returns a new array containing the mean values,
    otherwise a reference to the output array is returned. Nan is
    returned for slices that contain only NaNs.

#### SEE ALSO
[`average`](dask.array.average.md#dask.array.average)
: Weighted average

[`mean`](dask.array.mean.md#dask.array.mean)
: Arithmetic mean taken while not ignoring NaNs

[`var`](dask.array.var.md#dask.array.var), [`nanvar`](dask.array.nanvar.md#dask.array.nanvar)

### Notes

The arithmetic mean is the sum of the non-NaN elements along the axis
divided by the number of non-NaN elements.

Note that for floating-point input, the mean is computed using the same
precision the input has.  Depending on the input data, this can cause
the results to be inaccurate, especially for float32.  Specifying a
higher-precision accumulator using the dtype keyword can alleviate
this issue.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, np.nan], [3, 4]])
>>> np.nanmean(a)
2.6666666666666665
>>> np.nanmean(a, axis=0)
array([2.,  4.])
>>> np.nanmean(a, axis=1)
array([1.,  3.5]) # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanmedian.html.md

# dask.array.nanmedian

### dask.array.nanmedian(a, axis=None, keepdims=False, out=None)

Compute the median along the specified axis, while ignoring NaNs.

This docstring was copied from numpy.nanmedian.

Some inconsistencies with the Dask version may exist.

This works by automatically chunking the reduced axes to a single chunk
and then calling `numpy.nanmedian` function across the remaining dimensions

Returns the median of the array elements.

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array.

  **axis**
  : Axis or axes along which the medians are computed. The default
    is to compute the median along a flattened version of the array.
    A sequence of axes is supported since version 1.9.0.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output,
    but the type (of the output) will be cast if necessary.

  **overwrite_input**
  : If True, then allow use of memory of input array a for
    calculations. The input array will be modified by the call to
    median. This will save memory when you do not need to preserve
    the contents of the input array. Treat the input as undefined,
    but it will probably be fully or partially sorted. Default is
    False. If overwrite_input is `True` and a is not already an
    ndarray, an error will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    <br/>
    If this is anything but the default value it will be passed
    through (in the special case of an empty array) to the
    mean function of the underlying array.  If the array is
    a sub-class and mean does not have the kwarg keepdims this
    will raise a RuntimeError.
* **Returns:**
  **median**
  : A new array holding the result. If the input contains integers
    or floats smaller than `float64`, then the output data-type is
    `np.float64`.  Otherwise, the data-type of the output is the
    same as that of the input. If out is specified, that array is
    returned instead.

#### SEE ALSO
[`mean`](dask.array.mean.md#dask.array.mean), [`median`](dask.array.median.md#dask.array.median), [`percentile`](dask.array.percentile.md#dask.array.percentile)

### Notes

Given a vector `V` of length `N`, the median of `V` is the
middle value of a sorted copy of `V`, `V_sorted` - i.e.,
`V_sorted[(N-1)/2]`, when `N` is odd and the average of the two
middle values of `V_sorted` when `N` is even.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[10.0, 7, 4], [3, 2, 1]])
>>> a[0, 1] = np.nan
>>> a
array([[10., nan,  4.],
       [ 3.,  2.,  1.]])
>>> np.median(a)
np.float64(nan)
>>> np.nanmedian(a)
3.0
>>> np.nanmedian(a, axis=0)
array([6.5, 2. , 2.5])
>>> np.median(a, axis=1)
array([nan,  2.])
>>> b = a.copy()
>>> np.nanmedian(b, axis=1, overwrite_input=True)
array([7.,  2.])
>>> assert not np.all(a==b)
>>> b = a.copy()
>>> np.nanmedian(b, axis=None, overwrite_input=True)
3.0
>>> assert not np.all(a==b)
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanmin.html.md

# dask.array.nanmin

### dask.array.nanmin(a, axis=None, keepdims=False, split_every=None, out=None)

Return minimum of an array or minimum along an axis, ignoring any NaNs.
When all-NaN slices are encountered a `RuntimeWarning` is raised and
Nan is returned for that slice.

This docstring was copied from numpy.nanmin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Array containing numbers whose minimum is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the minimum is computed. The default is to compute
    the minimum of the flattened array.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`; if provided, it must have the same shape as the
    expected output, but the type will be cast if necessary. See
    [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    <br/>
    If the value is anything but the default, then
    keepdims will be passed through to the min method
    of sub-classes of ndarray.  If the sub-classes methods
    does not implement keepdims any exceptions will be raised.

  **initial**
  : The maximum value of an output element. Must be present to allow
    computation on empty slice. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **where**
  : Elements to compare for the minimum. See ~numpy.ufunc.reduce
    for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **nanmin**
  : An array with the same shape as a, with the specified axis
    removed.  If a is a 0-d array, or if axis is None, an ndarray
    scalar is returned.  The same dtype as a is returned.

#### SEE ALSO
[`nanmax`](dask.array.nanmax.md#dask.array.nanmax)
: The maximum value of an array along a given axis, ignoring any NaNs.

`amin`
: The minimum value of an array along a given axis, propagating any NaNs.

[`fmin`](dask.array.fmin.md#dask.array.fmin)
: Element-wise minimum of two arrays, ignoring any NaNs.

[`minimum`](dask.array.minimum.md#dask.array.minimum)
: Element-wise minimum of two arrays, propagating any NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Shows which elements are Not a Number (NaN).

[`isfinite`](dask.array.isfinite.md#dask.array.isfinite)
: Shows which elements are neither NaN nor infinity.

`amax`, [`fmax`](dask.array.fmax.md#dask.array.fmax), [`maximum`](dask.array.maximum.md#dask.array.maximum)

### Notes

NumPy uses the IEEE Standard for Binary Floating-Point for Arithmetic
(IEEE 754). This means that Not a Number is not equivalent to infinity.
Positive infinity is treated as a very large number and negative
infinity is treated as a very small (i.e. negative) number.

If the input has a integer type the function is equivalent to np.min.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, np.nan]])
>>> np.nanmin(a)
1.0
>>> np.nanmin(a, axis=0)
array([1.,  2.])
>>> np.nanmin(a, axis=1)
array([1.,  3.])
```

When positive infinity and negative infinity are present:

```pycon
>>> np.nanmin([1, 2, np.nan, np.inf])
1.0
>>> np.nanmin([1, 2, np.nan, -np.inf])
-inf
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanpercentile.html.md

# dask.array.nanpercentile

### dask.array.nanpercentile(a, q, \*\*kwargs)

Compute the qth percentile of the data along the specified axis,
while ignoring nan values.

This docstring was copied from numpy.nanpercentile.

Some inconsistencies with the Dask version may exist.

Returns the qth percentile(s) of the array elements.

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array, containing
    nan values to be ignored.

  **q**
  : Percentile or sequence of percentiles to compute, which must be
    between 0 and 100 inclusive.

  **axis**
  : Axis or axes along which the percentiles are computed. The default
    is to compute the percentile(s) along a flattened version of the
    array.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape and buffer length as the expected output, but the
    type (of the output) will be cast if necessary.

  **overwrite_input**
  : If True, then allow the input array a to be modified by
    intermediate calculations, to save memory. In this case, the
    contents of the input a after this function completes is
    undefined.

  **method**
  : This parameter specifies the method to use for estimating the
    percentile.  There are many different methods, some unique to NumPy.
    See the notes for explanation.  The options sorted by their R type
    as summarized in the H&F paper [[1]](#r96a794b740f8-1) are:
    1. ‘inverted_cdf’
    2. ‘averaged_inverted_cdf’
    3. ‘closest_observation’
    4. ‘interpolated_inverted_cdf’
    5. ‘hazen’
    6. ‘weibull’
    7. ‘linear’  (default)
    8. ‘median_unbiased’
    9. ‘normal_unbiased’
    <br/>
    The first three methods are discontinuous.  NumPy further defines the
    following discontinuous variations of the default ‘linear’ (7.) option:
    * ‘lower’
    * ‘higher’,
    * ‘midpoint’
    * ‘nearest’
    <br/>
    #### Versionchanged
    Changed in version 1.22.0: This argument was previously called “interpolation” and only
    offered the “linear” default and last four options.

  **keepdims**
  : If this is set to True, the axes which are reduced are left in
    the result as dimensions with size one. With this option, the
    result will broadcast correctly against the original array a.
    <br/>
    If this is anything but the default value it will be passed
    through (in the special case of an empty array) to the
    mean function of the underlying array.  If the array is
    a sub-class and mean does not have the kwarg keepdims this
    will raise a RuntimeError.

  **weights**
  : An array of weights associated with the values in a. Each value in
    a contributes to the percentile according to its associated weight.
    The weights array can either be 1-D (in which case its length must be
    the size of a along the given axis) or of the same shape as a.
    If weights=None, then all data in a are assumed to have a
    weight equal to one.
    Only method=”inverted_cdf” supports weights.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **percentile**
  : If q is a single percentile and axis=None, then the result
    is a scalar. If multiple percentiles are given, first axis of
    the result corresponds to the percentiles. The other axes are
    the axes that remain after the reduction of a. If the input
    contains integers or floats smaller than `float64`, the output
    data-type is `float64`. Otherwise, the output data-type is the
    same as that of the input. If out is specified, that array is
    returned instead.

#### SEE ALSO
[`nanmean`](dask.array.nanmean.md#dask.array.nanmean)

[`nanmedian`](dask.array.nanmedian.md#dask.array.nanmedian)
: equivalent to `nanpercentile(..., 50)`

[`percentile`](dask.array.percentile.md#dask.array.percentile), [`median`](dask.array.median.md#dask.array.median), [`mean`](dask.array.mean.md#dask.array.mean)

[`nanquantile`](dask.array.nanquantile.md#dask.array.nanquantile)
: equivalent to nanpercentile, except q in range [0, 1].

### Notes

The behavior of numpy.nanpercentile with percentage q is that of
numpy.quantile with argument `q/100` (ignoring nan values).
For more information, please see numpy.quantile.

### References

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[10., 7., 4.], [3., 2., 1.]])
>>> a[0][1] = np.nan
>>> a
array([[10.,  nan,   4.],
      [ 3.,   2.,   1.]])
>>> np.percentile(a, 50)
np.float64(nan)
>>> np.nanpercentile(a, 50)
3.0
>>> np.nanpercentile(a, 50, axis=0)
array([6.5, 2. , 2.5])
>>> np.nanpercentile(a, 50, axis=1, keepdims=True)
array([[7.],
       [2.]])
>>> m = np.nanpercentile(a, 50, axis=0)
>>> out = np.zeros_like(m)
>>> np.nanpercentile(a, 50, axis=0, out=out)
array([6.5, 2. , 2.5])
>>> m
array([6.5,  2. ,  2.5])
```

```pycon
>>> b = a.copy()
>>> np.nanpercentile(b, 50, axis=1, overwrite_input=True)
array([7., 2.])
>>> assert not np.all(a==b)
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanprod.html.md

# dask.array.nanprod

### dask.array.nanprod(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Return the product of array elements over a given axis treating Not a
Numbers (NaNs) as ones.

This docstring was copied from numpy.nanprod.

Some inconsistencies with the Dask version may exist.

One is returned for slices that are all-NaN or empty.

* **Parameters:**
  **a**
  : Array containing numbers whose product is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the product is computed. The default is to compute
    the product of the flattened array.

  **dtype**
  : The type of the returned array and of the accumulator in which the
    elements are summed.  By default, the dtype of a is used.  An
    exception is when a has an integer type with less precision than
    the platform (u)intp. In that case, the default will be either
    (u)int32 or (u)int64 depending on whether the platform is 32 or 64
    bits. For inexact inputs, dtype must be inexact.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`. If provided, it must have the same shape as the
    expected output, but the type will be cast if necessary. See
    [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details. The casting of NaN to integer
    can yield unexpected results.

  **keepdims**
  : If True, the axes which are reduced are left in the result as
    dimensions with size one. With this option, the result will
    broadcast correctly against the original arr.

  **initial**
  : The starting value for this product. See ~numpy.ufunc.reduce
    for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **where**
  : Elements to include in the product. See ~numpy.ufunc.reduce
    for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **nanprod**
  : A new array holding the result is returned unless out is
    specified, in which case it is returned.

#### SEE ALSO
[`numpy.prod`](https://numpy.org/doc/stable/reference/generated/numpy.prod.html#numpy.prod)
: Product across array propagating NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Show which elements are NaN.

### Examples

```pycon
>>> import numpy as np
>>> np.nanprod(1)
1
>>> np.nanprod([1])
1
>>> np.nanprod([1, np.nan])
1.0
>>> a = np.array([[1, 2], [3, np.nan]])
>>> np.nanprod(a)
6.0
>>> np.nanprod(a, axis=0)
array([3., 2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanquantile.html.md

# dask.array.nanquantile

### dask.array.nanquantile(a, q, axis=None, out=None, overwrite_input=False, method='linear', keepdims=False, , weights=None, interpolation=None)

Compute the qth quantile of the data along the specified axis,
while ignoring nan values.
Returns the qth quantile(s) of the array elements.

This docstring was copied from numpy.nanquantile.

Some inconsistencies with the Dask version may exist.

This works by automatically chunking the reduced axes to a single chunk
and then calling `numpy.nanquantile` function across the remaining dimensions

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array, containing
    nan values to be ignored

  **q**
  : Probability or sequence of probabilities for the quantiles to compute.
    Values must be between 0 and 1 inclusive.

  **axis**
  : Axis or axes along which the quantiles are computed. The
    default is to compute the quantile(s) along a flattened
    version of the array.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output,
    but the type (of the output) will be cast if necessary.

  **overwrite_input**
  : If True, then allow the input array a to be modified by intermediate
    calculations, to save memory. In this case, the contents of the input
    a after this function completes is undefined.

  **method**
  : This parameter specifies the method to use for estimating the
    quantile.  There are many different methods, some unique to NumPy.
    See the notes for explanation.  The options sorted by their R type
    as summarized in the H&F paper [[1]](#r34d560472f8b-1) are:
    1. ‘inverted_cdf’
    2. ‘averaged_inverted_cdf’
    3. ‘closest_observation’
    4. ‘interpolated_inverted_cdf’
    5. ‘hazen’
    6. ‘weibull’
    7. ‘linear’  (default)
    8. ‘median_unbiased’
    9. ‘normal_unbiased’
    <br/>
    The first three methods are discontinuous.  NumPy further defines the
    following discontinuous variations of the default ‘linear’ (7.) option:
    * ‘lower’
    * ‘higher’,
    * ‘midpoint’
    * ‘nearest’
    <br/>
    #### Versionchanged
    Changed in version 1.22.0: This argument was previously called “interpolation” and only
    offered the “linear” default and last four options.

  **keepdims**
  : If this is set to True, the axes which are reduced are left in
    the result as dimensions with size one. With this option, the
    result will broadcast correctly against the original array a.
    <br/>
    If this is anything but the default value it will be passed
    through (in the special case of an empty array) to the
    mean function of the underlying array.  If the array is
    a sub-class and mean does not have the kwarg keepdims this
    will raise a RuntimeError.

  **weights**
  : An array of weights associated with the values in a. Each value in
    a contributes to the quantile according to its associated weight.
    The weights array can either be 1-D (in which case its length must be
    the size of a along the given axis) or of the same shape as a.
    If weights=None, then all data in a are assumed to have a
    weight equal to one.
    Only method=”inverted_cdf” supports weights.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **quantile**
  : If q is a single probability and axis=None, then the result
    is a scalar. If multiple probability levels are given, first axis of
    the result corresponds to the quantiles. The other axes are
    the axes that remain after the reduction of a. If the input
    contains integers or floats smaller than `float64`, the output
    data-type is `float64`. Otherwise, the output data-type is the
    same as that of the input. If out is specified, that array is
    returned instead.

#### SEE ALSO
[`quantile`](dask.array.quantile.md#dask.array.quantile)

[`nanmean`](dask.array.nanmean.md#dask.array.nanmean), [`nanmedian`](dask.array.nanmedian.md#dask.array.nanmedian)

[`nanmedian`](dask.array.nanmedian.md#dask.array.nanmedian)
: equivalent to `nanquantile(..., 0.5)`

[`nanpercentile`](dask.array.nanpercentile.md#dask.array.nanpercentile)
: same as nanquantile, but with q in the range [0, 100].

### Notes

The behavior of numpy.nanquantile is the same as that of
numpy.quantile (ignoring nan values).
For more information, please see numpy.quantile.

### References

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[10., 7., 4.], [3., 2., 1.]])
>>> a[0][1] = np.nan
>>> a
array([[10.,  nan,   4.],
      [ 3.,   2.,   1.]])
>>> np.quantile(a, 0.5)
np.float64(nan)
>>> np.nanquantile(a, 0.5)
3.0
>>> np.nanquantile(a, 0.5, axis=0)
array([6.5, 2. , 2.5])
>>> np.nanquantile(a, 0.5, axis=1, keepdims=True)
array([[7.],
       [2.]])
>>> m = np.nanquantile(a, 0.5, axis=0)
>>> out = np.zeros_like(m)
>>> np.nanquantile(a, 0.5, axis=0, out=out)
array([6.5, 2. , 2.5])
>>> m
array([6.5,  2. ,  2.5])
>>> b = a.copy()
>>> np.nanquantile(b, 0.5, axis=1, overwrite_input=True)
array([7., 2.])
>>> assert not np.all(a==b)
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanstd.html.md

# dask.array.nanstd

### dask.array.nanstd(a, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Compute the standard deviation along the specified axis, while
ignoring NaNs.

This docstring was copied from numpy.nanstd.

Some inconsistencies with the Dask version may exist.

Returns the standard deviation, a measure of the spread of a
distribution, of the non-NaN array elements. The standard deviation is
computed for the flattened array by default, otherwise over the
specified axis.

For all-NaN slices or slices with zero degrees of freedom, NaN is
returned and a RuntimeWarning is raised.

* **Parameters:**
  **a**
  : Calculate the standard deviation of the non-NaN values.

  **axis**
  : Axis or axes along which the standard deviation is computed. The default is
    to compute the standard deviation of the flattened array.

  **dtype**
  : Type to use in computing the standard deviation. For arrays of
    integer type the default is float64, for arrays of float types it
    is the same as the array type.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape as the expected output but the type (of the
    calculated values) will be cast if necessary.

  **ddof**
  : Means Delta Degrees of Freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of non-NaN
    elements.  By default ddof is zero.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    <br/>
    If this value is anything but the default it is passed through
    as-is to the relevant functions of the sub-classes.  If these
    functions do not have a keepdims kwarg, a RuntimeError will
    be raised.

  **where**
  : Elements to include in the standard deviation.
    See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **mean**
  : Provide the mean to prevent its recalculation. The mean should have
    a shape as if it was calculated with `keepdims=True`.
    The axis for the calculation of the mean should be the same as used in
    the call to this std function.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **correction**
  : Array API compatible name for the `ddof` parameter. Only one of them
    can be provided at the same time.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **standard_deviation**
  : If out is None, return a new array containing the standard
    deviation, otherwise return a reference to the output array. If
    ddof is >= the number of non-NaN elements in a slice or the slice
    contains only NaNs, then the result for that slice is NaN.

#### SEE ALSO
[`var`](dask.array.var.md#dask.array.var), [`mean`](dask.array.mean.md#dask.array.mean), [`std`](dask.array.std.md#dask.array.std)

[`nanvar`](dask.array.nanvar.md#dask.array.nanvar), [`nanmean`](dask.array.nanmean.md#dask.array.nanmean)

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

The standard deviation is the square root of the average of the squared
deviations from the mean: `std = sqrt(mean(abs(x - x.mean())**2))`.

The average squared deviation is normally calculated as
`x.sum() / N`, where `N = len(x)`.  If, however, ddof is
specified, the divisor `N - ddof` is used instead. In standard
statistical practice, `ddof=1` provides an unbiased estimator of the
variance of the infinite population. `ddof=0` provides a maximum
likelihood estimate of the variance for normally distributed variables.
The standard deviation computed in this function is the square root of
the estimated variance, so even with `ddof=1`, it will not be an
unbiased estimate of the standard deviation per se.

Note that, for complex numbers, std takes the absolute value before
squaring, so that the result is always real and nonnegative.

For floating-point input, the *std* is computed using the same
precision the input has. Depending on the input data, this can cause
the results to be inaccurate, especially for float32 (see example
below).  Specifying a higher-accuracy accumulator using the dtype
keyword can alleviate this issue.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, np.nan], [3, 4]])
>>> np.nanstd(a)
1.247219128924647
>>> np.nanstd(a, axis=0)
array([1., 0.])
>>> np.nanstd(a, axis=1)
array([0.,  0.5]) # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.nansum.html.md

# dask.array.nansum

### dask.array.nansum(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Return the sum of array elements over a given axis treating Not a
Numbers (NaNs) as zero.

This docstring was copied from numpy.nansum.

Some inconsistencies with the Dask version may exist.

In NumPy versions <= 1.9.0 Nan is returned for slices that are all-NaN or
empty. In later versions zero is returned.

* **Parameters:**
  **a**
  : Array containing numbers whose sum is desired. If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the sum is computed. The default is to compute the
    sum of the flattened array.

  **dtype**
  : The type of the returned array and of the accumulator in which the
    elements are summed.  By default, the dtype of a is used.  An
    exception is when a has an integer type with less precision than
    the platform (u)intp. In that case, the default will be either
    (u)int32 or (u)int64 depending on whether the platform is 32 or 64
    bits. For inexact inputs, dtype must be inexact.

  **out**
  : Alternate output array in which to place the result.  The default
    is `None`. If provided, it must have the same shape as the
    expected output, but the type will be cast if necessary.  See
    [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details. The casting of NaN to integer
    can yield unexpected results.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.
    <br/>
    If the value is anything but the default, then
    keepdims will be passed through to the mean or sum methods
    of sub-classes of ndarray.  If the sub-classes methods
    does not implement keepdims any exceptions will be raised.

  **initial**
  : Starting value for the sum. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **where**
  : Elements to include in the sum. See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.
* **Returns:**
  **nansum**
  : A new array holding the result is returned unless out is
    specified, in which it is returned. The result has the same
    size as a, and the same shape as a if axis is not None
    or a is a 1-d array.

#### SEE ALSO
[`numpy.sum`](https://numpy.org/doc/stable/reference/generated/numpy.sum.html#numpy.sum)
: Sum across array propagating NaNs.

[`isnan`](dask.array.isnan.md#dask.array.isnan)
: Show which elements are NaN.

[`isfinite`](dask.array.isfinite.md#dask.array.isfinite)
: Show which elements are not NaN or +/-inf.

### Notes

If both positive and negative infinity are present, the sum will be Not
A Number (NaN).

### Examples

```pycon
>>> import numpy as np
>>> np.nansum(1)
1
>>> np.nansum([1])
1
>>> np.nansum([1, np.nan])
1.0
>>> a = np.array([[1, 1], [1, np.nan]])
>>> np.nansum(a)
3.0
>>> np.nansum(a, axis=0)
array([2.,  1.])
>>> np.nansum([1, np.nan, np.inf])
inf
>>> np.nansum([1, np.nan, -np.inf])
-inf
>>> with np.errstate(invalid="ignore"):
...     np.nansum([1, np.nan, np.inf, -np.inf]) # both +/- infinity present
np.float64(nan)
```

<!-- !! processed by numpydoc !! -->


# dask.array.nanvar.html.md

# dask.array.nanvar

### dask.array.nanvar(a, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Compute the variance along the specified axis, while ignoring NaNs.

This docstring was copied from numpy.nanvar.

Some inconsistencies with the Dask version may exist.

Returns the variance of the array elements, a measure of the spread of
a distribution.  The variance is computed for the flattened array by
default, otherwise over the specified axis.

For all-NaN slices or slices with zero degrees of freedom, NaN is
returned and a RuntimeWarning is raised.

* **Parameters:**
  **a**
  : Array containing numbers whose variance is desired.  If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the variance is computed.  The default is to compute
    the variance of the flattened array.

  **dtype**
  : Type to use in computing the variance.  For arrays of integer type
    the default is float64; for arrays of float types it is the same as
    the array type.

  **out**
  : Alternate output array in which to place the result.  It must have
    the same shape as the expected output, but the type is cast if
    necessary.

  **ddof**
  : “Delta Degrees of Freedom”: the divisor used in the calculation is
    `N - ddof`, where `N` represents the number of non-NaN
    elements. By default ddof is zero.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the original a.

  **where**
  : Elements to include in the variance. See ~numpy.ufunc.reduce for
    details.
    <br/>
    #### Versionadded
    Added in version 1.22.0.

  **mean**
  : Provide the mean to prevent its recalculation. The mean should have
    a shape as if it was calculated with `keepdims=True`.
    The axis for the calculation of the mean should be the same as used in
    the call to this var function.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **correction**
  : Array API compatible name for the `ddof` parameter. Only one of them
    can be provided at the same time.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **variance**
  : If out is None, return a new array containing the variance,
    otherwise return a reference to the output array. If ddof is >= the
    number of non-NaN elements in a slice or the slice contains only
    NaNs, then the result for that slice is NaN.

#### SEE ALSO
[`std`](dask.array.std.md#dask.array.std)
: Standard deviation

[`mean`](dask.array.mean.md#dask.array.mean)
: Average

[`var`](dask.array.var.md#dask.array.var)
: Variance while not ignoring NaNs

[`nanstd`](dask.array.nanstd.md#dask.array.nanstd), [`nanmean`](dask.array.nanmean.md#dask.array.nanmean)

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

The variance is the average of the squared deviations from the mean,
i.e.,  `var = mean(abs(x - x.mean())**2)`.

The mean is normally calculated as `x.sum() / N`, where `N = len(x)`.
If, however, ddof is specified, the divisor `N - ddof` is used
instead.  In standard statistical practice, `ddof=1` provides an
unbiased estimator of the variance of a hypothetical infinite
population.  `ddof=0` provides a maximum likelihood estimate of the
variance for normally distributed variables.

Note that for complex numbers, the absolute value is taken before
squaring, so that the result is always real and nonnegative.

For floating-point input, the variance is computed using the same
precision the input has.  Depending on the input data, this can cause
the results to be inaccurate, especially for float32 (see example
below).  Specifying a higher-accuracy accumulator using the `dtype`
keyword can alleviate this issue.

For this function to work on sub-classes of ndarray, they must define
sum with the kwarg keepdims

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, np.nan], [3, 4]])
>>> np.nanvar(a)
1.5555555555555554
>>> np.nanvar(a, axis=0)
array([1.,  0.])
>>> np.nanvar(a, axis=1)
array([0.,  0.25])  # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.negative.html.md

# dask.array.negative

### dask.array.negative(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'negative'>*

This docstring was copied from numpy.negative.

Some inconsistencies with the Dask version may exist.

Numerical negation, element-wise.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Returned array or scalar: y = -x.
    This is a scalar if x is a scalar.

### Examples

```pycon
>>> import numpy as np
>>> np.negative([1.,-1.])
array([-1.,  1.])
```

The unary `-` operator can be used as a shorthand for `np.negative` on
ndarrays.

```pycon
>>> x1 = np.array(([1., -1.]))
>>> -x1
array([-1.,  1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nextafter.html.md

# dask.array.nextafter

### dask.array.nextafter(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'nextafter'>*

This docstring was copied from numpy.nextafter.

Some inconsistencies with the Dask version may exist.

Return the next floating-point value after x1 towards x2, element-wise.

* **Parameters:**
  **x1**
  : Values to find the next representable value of.

  **x2**
  : The direction where to look for the next representable value of x1.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : The next representable values of x1 in the direction of x2.
    This is a scalar if both x1 and x2 are scalars.

### Examples

```pycon
>>> import numpy as np
>>> eps = np.finfo(np.float64).eps
>>> np.nextafter(1, 2) == eps + 1
True
>>> np.nextafter([1, 2], [2, 1]) == [eps + 1, 2 - eps]
array([ True,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.nonzero.html.md

# dask.array.nonzero

### dask.array.nonzero(a)

Return the indices of the elements that are non-zero.

This docstring was copied from numpy.nonzero.

Some inconsistencies with the Dask version may exist.

Returns a tuple of arrays, one for each dimension of a,
containing the indices of the non-zero elements in that
dimension. The values in a are always tested and returned in
row-major, C-style order.

To group the indices by element, rather than dimension, use argwhere,
which returns a row for each non-zero element.

* **Parameters:**
  **a**
  : Input array.
* **Returns:**
  **tuple_of_arrays**
  : Indices of elements that are non-zero.

#### SEE ALSO
[`flatnonzero`](dask.array.flatnonzero.md#dask.array.flatnonzero)
: Return indices that are non-zero in the flattened version of the input array.

`ndarray.nonzero`
: Equivalent ndarray method.

[`count_nonzero`](dask.array.count_nonzero.md#dask.array.count_nonzero)
: Counts the number of non-zero elements in the input array.

### Notes

While the nonzero values can be obtained with `a[nonzero(a)]`, it is
recommended to use `x[x.astype(np.bool)]` or `x[x != 0]` instead, which
will correctly handle 0-d arrays.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([[3, 0, 0], [0, 4, 0], [5, 6, 0]])
>>> x
array([[3, 0, 0],
       [0, 4, 0],
       [5, 6, 0]])
>>> np.nonzero(x)
(array([0, 1, 2, 2]), array([0, 1, 0, 1]))
```

```pycon
>>> x[np.nonzero(x)]
array([3, 4, 5, 6])
>>> np.transpose(np.nonzero(x))
array([[0, 0],
       [1, 1],
       [2, 0],
       [2, 1]])
```

A common use for `nonzero` is to find the indices of an array, where
a condition is True.  Given an array a, the condition a > 3 is a
boolean array and since False is interpreted as 0, np.nonzero(a > 3)
yields the indices of the a where the condition is true.

```pycon
>>> a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>> a > 3
array([[False, False, False],
       [ True,  True,  True],
       [ True,  True,  True]])
>>> np.nonzero(a > 3)
(array([1, 1, 1, 2, 2, 2]), array([0, 1, 2, 0, 1, 2]))
```

Using this result to index a is equivalent to using the mask directly:

```pycon
>>> a[np.nonzero(a > 3)]
array([4, 5, 6, 7, 8, 9])
>>> a[a > 3]  # prefer this spelling
array([4, 5, 6, 7, 8, 9])
```

`nonzero` can also be called as a method of the array.

```pycon
>>> (a > 3).nonzero()
(array([1, 1, 1, 2, 2, 2]), array([0, 1, 2, 0, 1, 2]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.not_equal.html.md

# dask.array.not_equal

### dask.array.not_equal(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'not_equal'>*

This docstring was copied from numpy.not_equal.

Some inconsistencies with the Dask version may exist.

Return (x1 != x2) element-wise.

* **Parameters:**
  **x1, x2**
  : Input arrays.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array, element-wise comparison of x1 and x2.
    Typically of type bool, unless `dtype=np.object_` is passed.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`equal`](dask.array.equal.md#dask.array.equal), [`greater`](dask.array.greater.md#dask.array.greater), [`greater_equal`](dask.array.greater_equal.md#dask.array.greater_equal), [`less`](dask.array.less.md#dask.array.less), [`less_equal`](dask.array.less_equal.md#dask.array.less_equal)

### Examples

```pycon
>>> import numpy as np
>>> np.not_equal([1.,2.], [1., 3.])
array([False,  True])
>>> np.not_equal([1, 2], [[1, 3],[1, 4]])
array([[False,  True],
       [False,  True]])
```

The `!=` operator can be used as a shorthand for `np.not_equal` on
ndarrays.

```pycon
>>> a = np.array([1., 2.])
>>> b = np.array([1., 3.])
>>> a != b
array([False,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.array.notnull.html.md

# dask.array.notnull

### dask.array.notnull(values)

pandas.notnull for dask arrays

<!-- !! processed by numpydoc !! -->


# dask.array.ones.html.md

# dask.array.ones

### dask.array.ones(\*args, \*\*kwargs)

> Blocked variant of ones_like

> Follows the signature of ones_like exactly except that it also features
> optional keyword arguments `chunks: int, tuple, or dict` and `name: str`.

> Original signature follows below.

Return an array of ones with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if a is Fortran contiguous,
    ‘C’ otherwise. ‘K’ means match the layout of a as closely
    as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of a, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of ones with the same shape and type as a.

#### SEE ALSO
[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`full_like`](dask.array.full_like.md#dask.array.full_like)
: Return a new array with shape of input filled with value.

[`ones`](#dask.array.ones)
: Return a new array setting values to one.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6)
>>> x = x.reshape((2, 3))
>>> x
array([[0, 1, 2],
       [3, 4, 5]])
>>> np.ones_like(x)
array([[1, 1, 1],
       [1, 1, 1]])
```

```pycon
>>> y = np.arange(3, dtype=np.float64)
>>> y
array([0., 1., 2.])
>>> np.ones_like(y)
array([1.,  1.,  1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ones_like.html.md

# dask.array.ones_like

### dask.array.ones_like(a, dtype=None, order='C', chunks=None, name=None, shape=None)

Return an array of ones with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Whether to store multidimensional data in C- or Fortran-contiguous
    (row- or column-wise) order in memory.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if `len(array) % chunks != 0`.

  **name**
  : An optional keyname for the array. Defaults to hashing the input
    keyword arguments.

  **shape**
  : Overrides the shape of the result.
* **Returns:**
  **out**
  : Array of ones with the same shape and type as a.

#### SEE ALSO
[`zeros_like`](dask.array.zeros_like.md#dask.array.zeros_like)
: Return an array of zeros with shape and type of input.

[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`zeros`](dask.array.zeros.md#dask.array.zeros)
: Return a new array setting values to zero.

[`ones`](dask.array.ones.md#dask.array.ones)
: Return a new array setting values to one.

[`empty`](dask.array.empty.md#dask.array.empty)
: Return a new uninitialized array.

<!-- !! processed by numpydoc !! -->


# dask.array.outer.html.md

# dask.array.outer

### dask.array.outer(a, b)

Compute the outer product of two vectors.

This docstring was copied from numpy.outer.

Some inconsistencies with the Dask version may exist.

Given two vectors a and b of length `M` and `N`, respectively,
the outer product [[1]](#rc80873b5d19f-1) is:

```default
[[a_0*b_0  a_0*b_1 ... a_0*b_{N-1} ]
 [a_1*b_0    .
 [ ...          .
 [a_{M-1}*b_0            a_{M-1}*b_{N-1} ]]
```

* **Parameters:**
  **a**
  : First input vector.  Input is flattened if
    not already 1-dimensional.

  **b**
  : Second input vector.  Input is flattened if
    not already 1-dimensional.

  **out**
  : A location where the result is stored
* **Returns:**
  **out**
  : `out[i, j] = a[i] * b[j]`

#### SEE ALSO
`inner`

[`einsum`](dask.array.einsum.md#dask.array.einsum)
: `einsum('i,j->ij', a.ravel(), b.ravel())` is the equivalent.

`ufunc.outer`
: A generalization to dimensions other than 1D and other operations. `np.multiply.outer(a.ravel(), b.ravel())` is the equivalent.

`linalg.outer`
: An Array API compatible variation of `np.outer`, which accepts 1-dimensional inputs only.

[`tensordot`](dask.array.tensordot.md#dask.array.tensordot)
: `np.tensordot(a.ravel(), b.ravel(), axes=((), ()))` is the equivalent.

### References

### Examples

Make a (*very* coarse) grid for computing a Mandelbrot set:

```pycon
>>> import numpy as np
>>> rl = np.outer(np.ones((5,)), np.linspace(-2, 2, 5))
>>> rl
array([[-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.],
       [-2., -1.,  0.,  1.,  2.]])
>>> im = np.outer(1j*np.linspace(2, -2, 5), np.ones((5,)))
>>> im
array([[0.+2.j, 0.+2.j, 0.+2.j, 0.+2.j, 0.+2.j],
       [0.+1.j, 0.+1.j, 0.+1.j, 0.+1.j, 0.+1.j],
       [0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j],
       [0.-1.j, 0.-1.j, 0.-1.j, 0.-1.j, 0.-1.j],
       [0.-2.j, 0.-2.j, 0.-2.j, 0.-2.j, 0.-2.j]])
>>> grid = rl + im
>>> grid
array([[-2.+2.j, -1.+2.j,  0.+2.j,  1.+2.j,  2.+2.j],
       [-2.+1.j, -1.+1.j,  0.+1.j,  1.+1.j,  2.+1.j],
       [-2.+0.j, -1.+0.j,  0.+0.j,  1.+0.j,  2.+0.j],
       [-2.-1.j, -1.-1.j,  0.-1.j,  1.-1.j,  2.-1.j],
       [-2.-2.j, -1.-2.j,  0.-2.j,  1.-2.j,  2.-2.j]])
```

An example using a “vector” of letters:

```pycon
>>> x = np.array(['a', 'b', 'c'], dtype=np.object_)
>>> np.outer(x, [1, 2, 3])
array([['a', 'aa', 'aaa'],
       ['b', 'bb', 'bbb'],
       ['c', 'cc', 'ccc']], dtype=object)
```

<!-- !! processed by numpydoc !! -->


# dask.array.overlap.map_overlap.html.md

# dask.array.overlap.map_overlap

### dask.array.overlap.map_overlap(func, \*args, depth=None, boundary=None, trim=True, align_arrays=True, allow_rechunk=True, \*\*kwargs)

Map a function over blocks of arrays with some overlap

We share neighboring zones between blocks of the array, map a
function, and then trim away the neighboring strips. If depth is
larger than any chunk along a particular axis, then the array is
rechunked.

Note that this function will attempt to automatically determine the output
array type before computing it, please refer to the `meta` keyword argument
in `map_blocks` if you expect that the function will not succeed when
operating on 0-d arrays.

* **Parameters:**
  **func: function**
  : The function to apply to each extended block.
    If multiple arrays are provided, then the function should expect to
    receive chunks of each array in the same order.

  **args**

  **depth: int, tuple, dict or list, keyword only**
  : The number of elements that each block should share with its neighbors
    If a tuple or dict then this can be different per axis.
    If a list then each element of that list must be an int, tuple or dict
    defining depth for the corresponding array in args.
    Asymmetric depths may be specified using a dict value of (-/+) tuples.
    Note that asymmetric depths are currently only supported when
    `boundary` is ‘none’.
    The default value is 0.

  **boundary: str, tuple, dict or list, keyword only**
  : How to handle the boundaries.
    Values include ‘reflect’, ‘periodic’, ‘nearest’, ‘none’,
    or any constant value like 0 or np.nan.
    If a list then each element must be a str, tuple or dict defining the
    boundary for the corresponding array in args.

  **trim: bool, keyword only**
  : Whether or not to trim `depth` elements from each block after
    calling the map function.
    Set this to False if your mapping function already does this for you

  **align_arrays: bool, keyword only**
  : Whether or not to align chunks along equally sized dimensions when
    multiple arrays are provided.  This allows for larger chunks in some
    arrays to be broken into smaller ones that match chunk sizes in other
    arrays such that they are compatible for block function mapping. If
    this is false, then an error will be thrown if arrays do not already
    have the same number of blocks in each dimension.

  **allow_rechunk: bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.

  **\*\*kwargs:**
  : Other keyword arguments valid in `map_blocks`

### Examples

```pycon
>>> import numpy as np
>>> import dask.array as da
```

```pycon
>>> x = np.array([1, 1, 2, 3, 3, 3, 2, 1, 1])
>>> x = da.from_array(x, chunks=5)
>>> def derivative(x):
...     return x - np.roll(x, 1)
```

```pycon
>>> y = x.map_overlap(derivative, depth=1, boundary=0)
>>> y.compute()
array([ 1,  0,  1,  1,  0,  0, -1, -1,  0])
```

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> d.map_overlap(lambda x: x + x.size, depth=1, boundary='reflect').compute()
array([[16, 17, 18, 19],
       [20, 21, 22, 23],
       [24, 25, 26, 27],
       [28, 29, 30, 31]])
```

```pycon
>>> func = lambda x: x + x.size
>>> depth = {0: 1, 1: 1}
>>> boundary = {0: 'reflect', 1: 'none'}
>>> d.map_overlap(func, depth, boundary).compute()
array([[12,  13,  14,  15],
       [16,  17,  18,  19],
       [20,  21,  22,  23],
       [24,  25,  26,  27]])
```

The `da.map_overlap` function can also accept multiple arrays.

```pycon
>>> func = lambda x, y: x + y
>>> x = da.arange(8).reshape(2, 4).rechunk((1, 2))
>>> y = da.arange(4).rechunk(2)
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect').compute()
array([[ 0,  2,  4,  6],
       [ 4,  6,  8,  10]])
```

When multiple arrays are given, they do not need to have the
same number of dimensions but they must broadcast together.
Arrays are aligned block by block (just as in `da.map_blocks`)
so the blocks must have a common chunk size.  This common chunking
is determined automatically as long as `align_arrays` is True.

```pycon
>>> x = da.arange(8, chunks=4)
>>> y = da.arange(8, chunks=2)
>>> r = da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=True)
>>> len(r.to_delayed())
4
```

```pycon
>>> da.map_overlap(func, x, y, depth=1, boundary='reflect', align_arrays=False).compute()
Traceback (most recent call last):
    ...
ValueError: Shapes do not align {'.0': {2, 4}}
```

Note also that this function is equivalent to `map_blocks`
by default.  A non-zero `depth` must be defined for any
overlap to appear in the arrays provided to `func`.

```pycon
>>> func = lambda x: x.sum()
>>> x = da.ones(10, dtype='int')
>>> block_args = dict(chunks=(), drop_axis=0)
>>> da.map_blocks(func, x, **block_args).compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, boundary='reflect').compute()
np.int64(10)
>>> da.map_overlap(func, x, **block_args, depth=1, boundary='reflect').compute()
np.int64(12)
```

For functions that may not handle 0-d arrays, it’s also possible to specify
`meta` with an empty array matching the type of the expected result. In
the example below, `func` will result in an `IndexError` when computing
`meta`:

```pycon
>>> x = np.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=np.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=numpy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

Similarly, it’s possible to specify a non-NumPy array to `meta`:

```pycon
>>> import cupy
>>> x = cupy.arange(16).reshape((4, 4))
>>> d = da.from_array(x, chunks=(2, 2))
>>> y = d.map_overlap(lambda x: x + x[2], depth=1, boundary='reflect', meta=cupy.array(()))
>>> y
dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=cupy.ndarray>
>>> y.compute()
array([[ 4,  6,  8, 10],
       [ 8, 10, 12, 14],
       [20, 22, 24, 26],
       [24, 26, 28, 30]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.overlap.overlap.html.md

# dask.array.overlap.overlap

### dask.array.overlap.overlap(x, depth, boundary, , allow_rechunk=True)

Share boundaries between neighboring blocks

* **Parameters:**
  **x: da.Array**
  : A dask array

  **depth: dict**
  : The size of the shared boundary per axis

  **boundary: dict**
  : The boundary condition on each axis. Options are ‘reflect’, ‘periodic’,
    ‘nearest’, ‘none’, or an array value.  Such a value will fill the
    boundary with that value.

  **allow_rechunk: bool, keyword only**
  : Allows rechunking, otherwise chunk sizes need to match and core
    dimensions are to consist only of one chunk.

  **The depth input informs how many cells to overlap between neighboring**

  **blocks \`\`{0: 2, 2: 5}\`\` means share two cells in 0 axis, 5 cells in 2 axis.**

  **Axes missing from this input will not be overlapped.**

  **Any axis containing chunks smaller than depth will be rechunked if**

  **possible, provided the keyword \`\`allow_rechunk\`\` is True (recommended).**

### Examples

```pycon
>>> import numpy as np
>>> import dask.array as da
```

```pycon
>>> x = np.arange(64).reshape((8, 8))
>>> d = da.from_array(x, chunks=(4, 4))
>>> d.chunks
((4, 4), (4, 4))
```

```pycon
>>> g = da.overlap.overlap(d, depth={0: 2, 1: 1},
...                       boundary={0: 100, 1: 'reflect'})
>>> g.chunks
((8, 8), (6, 6))
```

```pycon
>>> np.array(g)
array([[100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [  0,   0,   1,   2,   3,   4,   3,   4,   5,   6,   7,   7],
       [  8,   8,   9,  10,  11,  12,  11,  12,  13,  14,  15,  15],
       [ 16,  16,  17,  18,  19,  20,  19,  20,  21,  22,  23,  23],
       [ 24,  24,  25,  26,  27,  28,  27,  28,  29,  30,  31,  31],
       [ 32,  32,  33,  34,  35,  36,  35,  36,  37,  38,  39,  39],
       [ 40,  40,  41,  42,  43,  44,  43,  44,  45,  46,  47,  47],
       [ 16,  16,  17,  18,  19,  20,  19,  20,  21,  22,  23,  23],
       [ 24,  24,  25,  26,  27,  28,  27,  28,  29,  30,  31,  31],
       [ 32,  32,  33,  34,  35,  36,  35,  36,  37,  38,  39,  39],
       [ 40,  40,  41,  42,  43,  44,  43,  44,  45,  46,  47,  47],
       [ 48,  48,  49,  50,  51,  52,  51,  52,  53,  54,  55,  55],
       [ 56,  56,  57,  58,  59,  60,  59,  60,  61,  62,  63,  63],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.overlap.trim_internal.html.md

# dask.array.overlap.trim_internal

### dask.array.overlap.trim_internal(x, axes, boundary=None)

Trim sides from each block

This couples well with the overlap operation, which may leave excess data on
each block

#### SEE ALSO
`dask.array.chunk.trim`

[`dask.array.map_blocks`](dask.array.map_blocks.md#dask.array.map_blocks)

<!-- !! processed by numpydoc !! -->


# dask.array.overlap.trim_overlap.html.md

# dask.array.overlap.trim_overlap

### dask.array.overlap.trim_overlap(x, depth, boundary=None)

Trim sides from each block.

This couples well with the `map_overlap` operation which may leave
excess data on each block.

#### SEE ALSO
[`dask.array.overlap.map_overlap`](dask.array.overlap.map_overlap.md#dask.array.overlap.map_overlap)

<!-- !! processed by numpydoc !! -->


# dask.array.pad.html.md

# dask.array.pad

### dask.array.pad(array, pad_width, mode='constant', \*\*kwargs)

Pad an array.

This docstring was copied from numpy.pad.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **array**
  : The array to pad.

  **pad_width**
  : Number of values padded to the edges of each axis.
    `((before_1, after_1), ... (before_N, after_N))` unique pad widths
    for each axis.
    `(before, after)` or `((before, after),)` yields same before
    and after pad for each axis.
    `(pad,)` or `int` is a shortcut for before = after = pad width
    for all axes.
    If a `dict`, each key is an axis and its corresponding value is an `int` or
    `int` pair describing the padding `(before, after)` or `pad` width for
    that axis.

  **mode**
  : One of the following string values or a user supplied function.
    <br/>
    ‘constant’ (default)
    : Pads with a constant value.
    <br/>
    ‘edge’
    : Pads with the edge values of array.
    <br/>
    ‘linear_ramp’
    : Pads with the linear ramp between end_value and the
      array edge value.
    <br/>
    ‘maximum’
    : Pads with the maximum value of all or part of the
      vector along each axis.
    <br/>
    ‘mean’
    : Pads with the mean value of all or part of the
      vector along each axis.
    <br/>
    ‘median’
    : Pads with the median value of all or part of the
      vector along each axis.
    <br/>
    ‘minimum’
    : Pads with the minimum value of all or part of the
      vector along each axis.
    <br/>
    ‘reflect’
    : Pads with the reflection of the vector mirrored on
      the first and last values of the vector along each
      axis.
    <br/>
    ‘symmetric’
    : Pads with the reflection of the vector mirrored
      along the edge of the array.
    <br/>
    ‘wrap’
    : Pads with the wrap of the vector along the axis.
      The first values are used to pad the end and the
      end values are used to pad the beginning.
    <br/>
    ‘empty’
    : Pads with undefined values.
    <br/>
    <function>
    : Padding function, see Notes.

  **stat_length**
  : Used in ‘maximum’, ‘mean’, ‘median’, and ‘minimum’.  Number of
    values at edge of each axis used to calculate the statistic value.
    <br/>
    `((before_1, after_1), ... (before_N, after_N))` unique statistic
    lengths for each axis.
    <br/>
    `(before, after)` or `((before, after),)` yields same before
    and after statistic lengths for each axis.
    <br/>
    `(stat_length,)` or `int` is a shortcut for
    `before = after = statistic` length for all axes.
    <br/>
    Default is `None`, to use the entire axis.

  **constant_values**
  : Used in ‘constant’.  The values to set the padded values for each
    axis.
    <br/>
    `((before_1, after_1), ... (before_N, after_N))` unique pad constants
    for each axis.
    <br/>
    `(before, after)` or `((before, after),)` yields same before
    and after constants for each axis.
    <br/>
    `(constant,)` or `constant` is a shortcut for
    `before = after = constant` for all axes.
    <br/>
    Default is 0.

  **end_values**
  : Used in ‘linear_ramp’.  The values used for the ending value of the
    linear_ramp and that will form the edge of the padded array.
    <br/>
    `((before_1, after_1), ... (before_N, after_N))` unique end values
    for each axis.
    <br/>
    `(before, after)` or `((before, after),)` yields same before
    and after end values for each axis.
    <br/>
    `(constant,)` or `constant` is a shortcut for
    `before = after = constant` for all axes.
    <br/>
    Default is 0.

  **reflect_type**
  : Used in ‘reflect’, and ‘symmetric’.  The ‘even’ style is the
    default with an unaltered reflection around the edge value.  For
    the ‘odd’ style, the extended part of the array is created by
    subtracting the reflected values from two times the edge value.
* **Returns:**
  **pad**
  : Padded array of rank equal to array with shape increased
    according to pad_width.

### Notes

For an array with rank greater than 1, some of the padding of later
axes is calculated from padding of previous axes.  This is easiest to
think about with a rank 2 array where the corners of the padded array
are calculated by using padded values from the first axis.

The padding function, if used, should modify a rank 1 array in-place. It
has the following signature:

```default
padding_func(vector, iaxis_pad_width, iaxis, kwargs)
```

where

vector
: A rank 1 array already padded with zeros.  Padded values are
  vector[:iaxis_pad_width[0]] and vector[-iaxis_pad_width[1]:].

iaxis_pad_width
: A 2-tuple of ints, iaxis_pad_width[0] represents the number of
  values padded at the beginning of vector where
  iaxis_pad_width[1] represents the number of values padded at
  the end of vector.

iaxis
: The axis currently being calculated.

kwargs
: Any keyword arguments the function requires.

### Examples

```pycon
>>> import numpy as np
>>> a = [1, 2, 3, 4, 5]
>>> np.pad(a, (2, 3), 'constant', constant_values=(4, 6))
array([4, 4, 1, ..., 6, 6, 6])
```

```pycon
>>> np.pad(a, (2, 3), 'edge')
array([1, 1, 1, ..., 5, 5, 5])
```

```pycon
>>> np.pad(a, (2, 3), 'linear_ramp', end_values=(5, -4))
array([ 5,  3,  1,  2,  3,  4,  5,  2, -1, -4])
```

```pycon
>>> np.pad(a, (2,), 'maximum')
array([5, 5, 1, 2, 3, 4, 5, 5, 5])
```

```pycon
>>> np.pad(a, (2,), 'mean')
array([3, 3, 1, 2, 3, 4, 5, 3, 3])
```

```pycon
>>> np.pad(a, (2,), 'median')
array([3, 3, 1, 2, 3, 4, 5, 3, 3])
```

```pycon
>>> a = [[1, 2], [3, 4]]
>>> np.pad(a, ((3, 2), (2, 3)), 'minimum')
array([[1, 1, 1, 2, 1, 1, 1],
       [1, 1, 1, 2, 1, 1, 1],
       [1, 1, 1, 2, 1, 1, 1],
       [1, 1, 1, 2, 1, 1, 1],
       [3, 3, 3, 4, 3, 3, 3],
       [1, 1, 1, 2, 1, 1, 1],
       [1, 1, 1, 2, 1, 1, 1]])
```

```pycon
>>> a = [1, 2, 3, 4, 5]
>>> np.pad(a, (2, 3), 'reflect')
array([3, 2, 1, 2, 3, 4, 5, 4, 3, 2])
```

```pycon
>>> np.pad(a, (2, 3), 'reflect', reflect_type='odd')
array([-1,  0,  1,  2,  3,  4,  5,  6,  7,  8])
```

```pycon
>>> np.pad(a, (2, 3), 'symmetric')
array([2, 1, 1, 2, 3, 4, 5, 5, 4, 3])
```

```pycon
>>> np.pad(a, (2, 3), 'symmetric', reflect_type='odd')
array([0, 1, 1, 2, 3, 4, 5, 5, 6, 7])
```

```pycon
>>> np.pad(a, (2, 3), 'wrap')
array([4, 5, 1, 2, 3, 4, 5, 1, 2, 3])
```

```pycon
>>> def pad_with(vector, pad_width, iaxis, kwargs):
...     pad_value = kwargs.get('padder', 10)
...     vector[:pad_width[0]] = pad_value
...     vector[-pad_width[1]:] = pad_value
>>> a = np.arange(6)
>>> a = a.reshape((2, 3))
>>> np.pad(a, 2, pad_with)
array([[10, 10, 10, 10, 10, 10, 10],
       [10, 10, 10, 10, 10, 10, 10],
       [10, 10,  0,  1,  2, 10, 10],
       [10, 10,  3,  4,  5, 10, 10],
       [10, 10, 10, 10, 10, 10, 10],
       [10, 10, 10, 10, 10, 10, 10]])
>>> np.pad(a, 2, pad_with, padder=100)
array([[100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100],
       [100, 100,   0,   1,   2, 100, 100],
       [100, 100,   3,   4,   5, 100, 100],
       [100, 100, 100, 100, 100, 100, 100],
       [100, 100, 100, 100, 100, 100, 100]])
```

```pycon
>>> a = np.arange(1, 7).reshape(2, 3)
>>> np.pad(a, {1: (1, 2)})
array([[0, 1, 2, 3, 0, 0],
       [0, 4, 5, 6, 0, 0]])
>>> np.pad(a, {-1: 2})
array([[0, 0, 1, 2, 3, 0, 0],
       [0, 0, 4, 5, 6, 0, 0]])
>>> np.pad(a, {0: (3, 0)})
array([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       [1, 2, 3],
       [4, 5, 6]])
>>> np.pad(a, {0: (3, 0), 1: 2})
array([[0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 1, 2, 3, 0, 0],
       [0, 0, 4, 5, 6, 0, 0]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.percentile.html.md

# dask.array.percentile

### dask.array.percentile(a, q, method='linear', internal_method='default', \*\*kwargs)

Approximate percentile of 1-D array

* **Parameters:**
  **a**

  **q**
  : Percentile or sequence of percentiles to compute, which must be between
    0 and 100 inclusive.

  **method**
  : The interpolation method to use when the desired percentile lies
    between two data points `i < j`.
    - ‘linear’: `i + (j - i) * fraction`, where `fraction`
      is the fractional part of the index surrounded by `i`
      and `j`.
    - ‘lower’: `i`.
    - ‘higher’: `j`.
    - ‘nearest’: `i` or `j`, whichever is nearest.
    - ‘midpoint’: `(i + j) / 2`.
    <br/>
    #### Versionchanged
    Changed in version 2022.1.0: This argument was previously called “interpolation”

  **internal_method**
  : What internal method to use. By default will use dask’s internal custom
    algorithm (`'dask'`).  If set to `'tdigest'` will use tdigest for
    floats and ints if method is linear and fallback to `'dask'` otherwise.
    <br/>
    #### Versionchanged
    Changed in version 2022.1.0: This argument was previously called “method”.

  **interpolation**
  : Deprecated name for the method keyword argument.
    <br/>
    #### Deprecated
    Deprecated since version 2022.1.0.

#### SEE ALSO
[`numpy.percentile`](https://numpy.org/doc/stable/reference/generated/numpy.percentile.html#numpy.percentile)
: Numpy’s equivalent Percentile function

<!-- !! processed by numpydoc !! -->


# dask.array.piecewise.html.md

# dask.array.piecewise

### dask.array.piecewise(x, condlist, funclist, \*args, \*\*kw)

Evaluate a piecewise-defined function.

This docstring was copied from numpy.piecewise.

Some inconsistencies with the Dask version may exist.

Given a set of conditions and corresponding functions, evaluate each
function on the input data wherever its condition is true.

* **Parameters:**
  **x**
  : The input domain.

  **condlist**
  : Each boolean array corresponds to a function in funclist.  Wherever
    condlist[i] is True, funclist[i](x) is used as the output value.
    <br/>
    Each boolean array in condlist selects a piece of x,
    and should therefore be of the same shape as x.
    <br/>
    The length of condlist must correspond to that of funclist.
    If one extra function is given, i.e. if
    `len(funclist) == len(condlist) + 1`, then that extra function
    is the default value, used wherever all conditions are false.

  **funclist**
  : Each function is evaluated over x wherever its corresponding
    condition is True.  It should take a 1d array as input and give a 1d
    array or a scalar value as output.  If, instead of a callable,
    a scalar is provided then a constant function (`lambda x: scalar`) is
    assumed.

  **args**
  : Any further arguments given to piecewise are passed to the functions
    upon execution, i.e., if called `piecewise(..., ..., 1, 'a')`, then
    each function is called as `f(x, 1, 'a')`.

  **kw**
  : Keyword arguments used in calling piecewise are passed to the
    functions upon execution, i.e., if called
    `piecewise(..., ..., alpha=1)`, then each function is called as
    `f(x, alpha=1)`.
* **Returns:**
  **out**
  : The output is the same shape and type as x and is found by
    calling the functions in funclist on the appropriate portions of x,
    as defined by the boolean arrays in condlist.  Portions not covered
    by any condition have a default value of 0.

#### SEE ALSO
[`choose`](dask.array.choose.md#dask.array.choose), [`select`](dask.array.select.md#dask.array.select), [`where`](dask.array.where.md#dask.array.where)

### Notes

This is similar to choose or select, except that functions are
evaluated on elements of x that satisfy the corresponding condition from
condlist.

The result is:

```default
      |--
      |funclist[0](x[condlist[0]])
out = |funclist[1](x[condlist[1]])
      |...
      |funclist[n2](x[condlist[n2]])
      |--
```

### Examples

```pycon
>>> import numpy as np
```

Define the signum function, which is -1 for `x < 0` and +1 for `x >= 0`.

```pycon
>>> x = np.linspace(-2.5, 2.5, 6)
>>> np.piecewise(x, [x < 0, x >= 0], [-1, 1])
array([-1., -1., -1.,  1.,  1.,  1.])
```

Define the absolute value, which is `-x` for `x <0` and `x` for
`x >= 0`.

```pycon
>>> np.piecewise(x, [x < 0, x >= 0], [lambda x: -x, lambda x: x])
array([2.5,  1.5,  0.5,  0.5,  1.5,  2.5])
```

Apply the same function to a scalar value.

```pycon
>>> y = -2
>>> np.piecewise(y, [y < 0, y >= 0], [lambda x: -x, lambda x: x])
array(2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.positive.html.md

# dask.array.positive

### dask.array.positive(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'positive'>*

This docstring was copied from numpy.positive.

Some inconsistencies with the Dask version may exist.

Numerical positive, element-wise.

* **Parameters:**
  **x**
  : Input array.
* **Returns:**
  **y**
  : Returned array or scalar: y = +x.
    This is a scalar if x is a scalar.

### Notes

Equivalent to x.copy(), but only defined for types that support
arithmetic.

### Examples

```pycon
>>> import numpy as np
```

```pycon
>>> x1 = np.array(([1., -1.]))
>>> np.positive(x1)
array([ 1., -1.])
```

The unary `+` operator can be used as a shorthand for `np.positive` on
ndarrays.

```pycon
>>> x1 = np.array(([1., -1.]))
>>> +x1
array([ 1., -1.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.power.html.md

# dask.array.power

### dask.array.power(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'power'>*

This docstring was copied from numpy.power.

Some inconsistencies with the Dask version may exist.

First array elements raised to powers from second array, element-wise.

Raise each base in x1 to the positionally-corresponding power in
x2.  x1 and x2 must be broadcastable to the same shape.

An integer type raised to a negative integer power will raise a
`ValueError`.

Negative values raised to a non-integral value will return `nan`.
To get complex results, cast the input to complex, or specify the
`dtype` to be `complex` (see the example below).

* **Parameters:**
  **x1**
  : The bases.

  **x2**
  : The exponents.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The bases in x1 raised to the exponents in x2.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`float_power`](dask.array.float_power.md#dask.array.float_power)
: power function that promotes integers to float

### Examples

```pycon
>>> import numpy as np
```

Cube each element in an array.

```pycon
>>> x1 = np.arange(6)
>>> x1
[0, 1, 2, 3, 4, 5]
>>> np.power(x1, 3)
array([  0,   1,   8,  27,  64, 125])
```

Raise the bases to different exponents.

```pycon
>>> x2 = [1.0, 2.0, 3.0, 3.0, 2.0, 1.0]
>>> np.power(x1, x2)
array([  0.,   1.,   8.,  27.,  16.,   5.])
```

The effect of broadcasting.

```pycon
>>> x2 = np.array([[1, 2, 3, 3, 2, 1], [1, 2, 3, 3, 2, 1]])
>>> x2
array([[1, 2, 3, 3, 2, 1],
       [1, 2, 3, 3, 2, 1]])
>>> np.power(x1, x2)
array([[ 0,  1,  8, 27, 16,  5],
       [ 0,  1,  8, 27, 16,  5]])
```

The `**` operator can be used as a shorthand for `np.power` on
ndarrays.

```pycon
>>> x2 = np.array([1, 2, 3, 3, 2, 1])
>>> x1 = np.arange(6)
>>> x1 ** x2
array([ 0,  1,  8, 27, 16,  5])
```

Negative values raised to a non-integral value will result in `nan`
(and a warning will be generated).

```pycon
>>> x3 = np.array([-1.0, -4.0])
>>> with np.errstate(invalid='ignore'):
...     p = np.power(x3, 1.5)
...
>>> p
array([nan, nan])
```

To get complex results, give the argument `dtype=np.complex128`.

```pycon
>>> np.power(x3, 1.5, dtype=np.complex128)
array([-1.83697020e-16-1.j, -1.46957616e-15-8.j])
```

<!-- !! processed by numpydoc !! -->


# dask.array.prod.html.md

# dask.array.prod

### dask.array.prod(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Return the product of array elements over a given axis.

This docstring was copied from numpy.prod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Axis or axes along which a product is performed.  The default,
    axis=None, will calculate the product of all the elements in the
    input array. If axis is negative it counts from the last to the
    first axis.
    <br/>
    If axis is a tuple of ints, a product is performed on all of the
    axes specified in the tuple instead of a single axis or all the
    axes as before.

  **dtype**
  : The type of the returned array, as well as of the accumulator in
    which the elements are multiplied.  The dtype of a is used by
    default unless a has an integer dtype of less precision than the
    default platform integer.  In that case, if a is signed then the
    platform integer is used while if a is unsigned then an unsigned
    integer of the same precision as the platform integer is used.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape as the expected output, but the type of the output
    values will be cast if necessary.

  **keepdims**
  : If this is set to True, the axes which are reduced are left in the
    result as dimensions with size one. With this option, the result
    will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the prod method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **initial**
  : The starting value for this product. See ~numpy.ufunc.reduce
    for details.

  **where**
  : Elements to include in the product. See ~numpy.ufunc.reduce
    for details.
* **Returns:**
  **product_along_axis**
  : An array shaped as a but with the specified axis removed.
    Returns a reference to out if specified.

#### SEE ALSO
`ndarray.prod`
: equivalent method

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

Arithmetic is modular when using integer types, and no error is
raised on overflow.  That means that, on a 32-bit platform:

```pycon
>>> x = np.array([536870910, 536870910, 536870910, 536870910])
>>> np.prod(x)
16 # may vary
```

The product of an empty array is the neutral element 1:

```pycon
>>> np.prod([])
1.0
```

### Examples

By default, calculate the product of all elements:

```pycon
>>> import numpy as np
>>> np.prod([1.,2.])
2.0
```

Even when the input array is two-dimensional:

```pycon
>>> a = np.array([[1., 2.], [3., 4.]])
>>> np.prod(a)
24.0
```

But we can also specify the axis over which to multiply:

```pycon
>>> np.prod(a, axis=1)
array([  2.,  12.])
>>> np.prod(a, axis=0)
array([3., 8.])
```

Or select specific elements to include:

```pycon
>>> np.prod([1., np.nan, 3.], where=[True, False, True])
3.0
```

If the type of x is unsigned, then the output type is
the unsigned platform integer:

```pycon
>>> x = np.array([1, 2, 3], dtype=np.uint8)
>>> np.prod(x).dtype == np.uint
True
```

If x is of a signed integer type, then the output type
is the default platform integer:

```pycon
>>> x = np.array([1, 2, 3], dtype=np.int8)
>>> np.prod(x).dtype == int
True
```

You can also start the product with a value other than one:

```pycon
>>> np.prod([1, 2], initial=5)
10
```

<!-- !! processed by numpydoc !! -->


# dask.array.ptp.html.md

# dask.array.ptp

### dask.array.ptp(a, axis=None)

Range of values (maximum - minimum) along an axis.

This docstring was copied from numpy.ptp.

Some inconsistencies with the Dask version may exist.

The name of the function comes from the acronym for ‘peak to peak’.

#### WARNING
ptp preserves the data type of the array. This means the
return value for an input of signed integers with n bits
(e.g. numpy.int8, numpy.int16, etc) is also a signed integer
with n bits.  In that case, peak-to-peak values greater than
`2**(n-1)-1` will be returned as negative values. An example
with a work-around is shown below.

* **Parameters:**
  **a**
  : Input values.

  **axis**
  : Axis along which to find the peaks.  By default, flatten the
    array.  axis may be negative, in
    which case it counts from the last to the first axis.
    If this is a tuple of ints, a reduction is performed on multiple
    axes, instead of a single axis or all the axes as before.

  **out**
  : Alternative output array in which to place the result. It must
    have the same shape and buffer length as the expected output,
    but the type of the output values will be cast if necessary.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the ptp method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.
* **Returns:**
  **ptp**
  : The range of a given array - scalar if array is one-dimensional
    or a new array holding the result along the given axis

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([[4, 9, 2, 10],
...               [6, 9, 7, 12]])
```

```pycon
>>> np.ptp(x, axis=1)
array([8, 6])
```

```pycon
>>> np.ptp(x, axis=0)
array([2, 0, 5, 2])
```

```pycon
>>> np.ptp(x)
10
```

This example shows that a negative value can be returned when
the input is an array of signed integers.

```pycon
>>> y = np.array([[1, 127],
...               [0, 127],
...               [-1, 127],
...               [-2, 127]], dtype=np.int8)
>>> np.ptp(y, axis=1)
array([ 126,  127, -128, -127], dtype=int8)
```

A work-around is to use the view() method to view the result as
unsigned integers with the same bit width:

```pycon
>>> np.ptp(y, axis=1).view(np.uint8)
array([126, 127, 128, 129], dtype=uint8)
```

<!-- !! processed by numpydoc !! -->


# dask.array.push.html.md

# dask.array.push

### dask.array.push(array, n, axis)

Dask-version of bottleneck.push

#### NOTE
Requires bottleneck to be installed.

<!-- !! processed by numpydoc !! -->


# dask.array.quantile.html.md

# dask.array.quantile

### dask.array.quantile(a, q, axis=None, out=None, overwrite_input=False, method='linear', keepdims=False, , weights=None, interpolation=None)

Compute the q-th quantile of the data along the specified axis.

This docstring was copied from numpy.quantile.

Some inconsistencies with the Dask version may exist.

This works by automatically chunking the reduced axes to a single chunk if necessary
and then calling `numpy.quantile` function across the remaining dimensions

* **Parameters:**
  **a**
  : Input array or object that can be converted to an array.

  **q**
  : Probability or sequence of probabilities of the quantiles to compute.
    Values must be between 0 and 1 inclusive.

  **axis**
  : Axis or axes along which the quantiles are computed. The default is
    to compute the quantile(s) along a flattened version of the array.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape and buffer length as the expected output, but the
    type (of the output) will be cast if necessary.

  **overwrite_input**
  : If True, then allow the input array a to be modified by
    intermediate calculations, to save memory. In this case, the
    contents of the input a after this function completes is
    undefined.

  **method**
  : This parameter specifies the method to use for estimating the
    quantile.  There are many different methods, some unique to NumPy.
    The recommended options, numbered as they appear in [[1]](#r48a33b30c54b-1), are:
    1. ‘inverted_cdf’
    2. ‘averaged_inverted_cdf’
    3. ‘closest_observation’
    4. ‘interpolated_inverted_cdf’
    5. ‘hazen’
    6. ‘weibull’
    7. ‘linear’  (default)
    8. ‘median_unbiased’
    9. ‘normal_unbiased’
    <br/>
    The first three methods are discontinuous. For backward compatibility
    with previous versions of NumPy, the following discontinuous variations
    of the default ‘linear’ (7.) option are available:
    * ‘lower’
    * ‘higher’,
    * ‘midpoint’
    * ‘nearest’
    <br/>
    See Notes for details.
    <br/>
    #### Versionchanged
    Changed in version 1.22.0: This argument was previously called “interpolation” and only
    offered the “linear” default and last four options.

  **keepdims**
  : If this is set to True, the axes which are reduced are left in
    the result as dimensions with size one. With this option, the
    result will broadcast correctly against the original array a.

  **weights**
  : An array of weights associated with the values in a. Each value in
    a contributes to the quantile according to its associated weight.
    The weights array can either be 1-D (in which case its length must be
    the size of a along the given axis) or of the same shape as a.
    If weights=None, then all data in a are assumed to have a
    weight equal to one.
    Only method=”inverted_cdf” supports weights.
    See the notes for more details.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **quantile**
  : If q is a single probability and axis=None, then the result
    is a scalar. If multiple probability levels are given, first axis
    of the result corresponds to the quantiles. The other axes are
    the axes that remain after the reduction of a. If the input
    contains integers or floats smaller than `float64`, the output
    data-type is `float64`. Otherwise, the output data-type is the
    same as that of the input. If out is specified, that array is
    returned instead.

#### SEE ALSO
[`mean`](dask.array.mean.md#dask.array.mean)

[`percentile`](dask.array.percentile.md#dask.array.percentile)
: equivalent to quantile, but with q in the range [0, 100].

[`median`](dask.array.median.md#dask.array.median)
: equivalent to `quantile(..., 0.5)`

[`nanquantile`](dask.array.nanquantile.md#dask.array.nanquantile)

### Notes

Given a sample a from an underlying distribution, quantile provides a
nonparametric estimate of the inverse cumulative distribution function.

By default, this is done by interpolating between adjacent elements in
`y`, a sorted copy of a:

```default
(1-g)*y[j] + g*y[j+1]
```

where the index `j` and coefficient `g` are the integral and
fractional components of `q * (n-1)`, and `n` is the number of
elements in the sample.

This is a special case of Equation 1 of H&F [[1]](#r48a33b30c54b-1). More generally,

- `j = (q*n + m - 1) // 1`, and
- `g = (q*n + m - 1) % 1`,

where `m` may be defined according to several different conventions.
The preferred convention may be selected using the `method` parameter:

| `method`                    |   number in H&F | `m`         |
|-----------------------------|-----------------|-------------|
| `interpolated_inverted_cdf` |               4 | `0`         |
| `hazen`                     |               5 | `1/2`       |
| `weibull`                   |               6 | `q`         |
| `linear` (default)          |               7 | `1 - q`     |
| `median_unbiased`           |               8 | `q/3 + 1/3` |
| `normal_unbiased`           |               9 | `q/4 + 3/8` |

Note that indices `j` and `j + 1` are clipped to the range `0` to
`n - 1` when the results of the formula would be outside the allowed
range of non-negative indices. The `- 1` in the formulas for `j` and
`g` accounts for Python’s 0-based indexing.

The table above includes only the estimators from H&F that are continuous
functions of probability q (estimators 4-9). NumPy also provides the
three discontinuous estimators from H&F (estimators 1-3), where `j` is
defined as above, `m` is defined as follows, and `g` is a function
of the real-valued `index = q*n + m - 1` and `j`.

1. `inverted_cdf`: `m = 0` and `g = int(index - j > 0)`
2. `averaged_inverted_cdf`: `m = 0` and
   `g = (1 + int(index - j > 0)) / 2`
3. `closest_observation`: `m = -1/2` and
   `g = 1 - int((index == j) & (j%2 == 1))`

For backward compatibility with previous versions of NumPy, quantile
provides four additional discontinuous estimators. Like
`method='linear'`, all have `m = 1 - q` so that `j = q*(n-1) // 1`,
but `g` is defined as follows.

- `lower`: `g = 0`
- `midpoint`: `g = 0.5`
- `higher`: `g = 1`
- `nearest`: `g = (q*(n-1) % 1) > 0.5`

**Weighted quantiles:**
More formally, the quantile at probability level $q$ of a cumulative
distribution function $F(y)=P(Y \leq y)$ with probability measure
$P$ is defined as any number $x$ that fulfills the
*coverage conditions*

$$
P(Y < x) \leq q \quad\text{and}\quad P(Y \leq x) \geq q

$$

with random variable $Y\sim P$.
Sample quantiles, the result of quantile, provide nonparametric
estimation of the underlying population counterparts, represented by the
unknown $F$, given a data vector a of length `n`.

Some of the estimators above arise when one considers $F$ as the
empirical distribution function of the data, i.e.
$F(y) = \frac{1}{n} \sum_i 1_{a_i \leq y}$.
Then, different methods correspond to different choices of $x$ that
fulfill the above coverage conditions. Methods that follow this approach
are `inverted_cdf` and `averaged_inverted_cdf`.

For weighted quantiles, the coverage conditions still hold. The
empirical cumulative distribution is simply replaced by its weighted
version, i.e.
$P(Y \leq t) = \frac{1}{\sum_i w_i} \sum_i w_i 1_{x_i \leq t}$.
Only `method="inverted_cdf"` supports weights.

### References

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[10, 7, 4], [3, 2, 1]])
>>> a
array([[10,  7,  4],
       [ 3,  2,  1]])
>>> np.quantile(a, 0.5)
3.5
>>> np.quantile(a, 0.5, axis=0)
array([6.5, 4.5, 2.5])
>>> np.quantile(a, 0.5, axis=1)
array([7.,  2.])
>>> np.quantile(a, 0.5, axis=1, keepdims=True)
array([[7.],
       [2.]])
>>> m = np.quantile(a, 0.5, axis=0)
>>> out = np.zeros_like(m)
>>> np.quantile(a, 0.5, axis=0, out=out)
array([6.5, 4.5, 2.5])
>>> m
array([6.5, 4.5, 2.5])
>>> b = a.copy()
>>> np.quantile(b, 0.5, axis=1, overwrite_input=True)
array([7.,  2.])
>>> assert not np.all(a == b)
```

See also numpy.percentile for a visualization of most methods.

<!-- !! processed by numpydoc !! -->


# dask.array.rad2deg.html.md

# dask.array.rad2deg

### dask.array.rad2deg(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'rad2deg'>*

This docstring was copied from numpy.rad2deg.

Some inconsistencies with the Dask version may exist.

Convert angles from radians to degrees.

* **Parameters:**
  **x**
  : Angle in radians.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding angle in degrees.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`deg2rad`](dask.array.deg2rad.md#dask.array.deg2rad)
: Convert angles from degrees to radians.

`unwrap`
: Remove large jumps in angle by wrapping.

### Notes

rad2deg(x) is `180 * x / pi`.

### Examples

```pycon
>>> import numpy as np
>>> np.rad2deg(np.pi/2)
90.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.radians.html.md

# dask.array.radians

### dask.array.radians(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'radians'>*

This docstring was copied from numpy.radians.

Some inconsistencies with the Dask version may exist.

Convert angles from degrees to radians.

* **Parameters:**
  **x**
  : Input array in degrees.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding radian values.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`deg2rad`](dask.array.deg2rad.md#dask.array.deg2rad)
: equivalent function

### Examples

```pycon
>>> import numpy as np
```

Convert a degree array to radians

```pycon
>>> deg = np.arange(12.) * 30.
>>> np.radians(deg)
array([ 0.        ,  0.52359878,  1.04719755,  1.57079633,  2.0943951 ,
        2.61799388,  3.14159265,  3.66519143,  4.1887902 ,  4.71238898,
        5.23598776,  5.75958653])
```

```pycon
>>> out = np.zeros((deg.shape))
>>> ret = np.radians(deg, out)
>>> ret is out
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.beta.html.md

# dask.array.random.beta

### dask.array.random.beta(\*args, \*\*kwargs)

Draw samples from a Beta distribution.

This docstring was copied from numpy.random.mtrand.RandomState.beta.

Some inconsistencies with the Dask version may exist.

The Beta distribution is a special case of the Dirichlet distribution,
and is related to the Gamma distribution.  It has the probability
distribution function

$$
f(x; a,b) = \frac{1}{B(\alpha, \beta)} x^{\alpha - 1}
(1 - x)^{\beta - 1},

$$

where the normalization, B, is the beta function,

$$
B(\alpha, \beta) = \int_0^1 t^{\alpha - 1}
(1 - t)^{\beta - 1} dt.

$$

It is often seen in Bayesian inference and order statistics.

#### NOTE
New code should use the ~numpy.random.Generator.beta
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **a**
  : Alpha, positive (>0).

  **b**
  : Beta, positive (>0).

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `a` and `b` are both scalars.
    Otherwise, `np.broadcast(a, b).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized beta distribution.

#### SEE ALSO
`random.Generator.beta`
: which should be used for new code.

<!-- !! processed by numpydoc !! -->


# dask.array.random.binomial.html.md

# dask.array.random.binomial

### dask.array.random.binomial(\*args, \*\*kwargs)

Draw samples from a binomial distribution.

This docstring was copied from numpy.random.mtrand.RandomState.binomial.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a binomial distribution with specified
parameters, n trials and p probability of success where
n an integer >= 0 and p is in the interval [0,1]. (n may be
input as a float, but it is truncated to an integer in use)

#### NOTE
New code should use the ~numpy.random.Generator.binomial
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **n**
  : Parameter of the distribution, >= 0. Floats are also accepted,
    but they will be truncated to integers.

  **p**
  : Parameter of the distribution, >= 0 and <=1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `n` and `p` are both scalars.
    Otherwise, `np.broadcast(n, p).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized binomial distribution, where
    each sample is equal to the number of successes over the n trials.

#### SEE ALSO
[`scipy.stats.binom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom.html#scipy.stats.binom)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.binomial`
: which should be used for new code.

### Notes

The probability mass function (PMF) for the binomial distribution is

$$
P(N) = \binom{n}{N}p^N(1-p)^{n-N},

$$

where $n$ is the number of trials, $p$ is the probability
of success, and $N$ is the number of successes.

When estimating the standard error of a proportion in a population by
using a random sample, the normal distribution works well unless the
product p\*n <=5, where p = population proportion estimate, and n =
number of samples, in which case the binomial distribution is used
instead. For example, a sample of 15 people shows 4 who are left
handed, and 11 who are right handed. Then p = 4/15 = 27%. 0.27\*15 = 4,
so the binomial distribution should be used in this case.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> n, p = 10, .5  # number of trials, probability of each trial
>>> s = np.random.binomial(n, p, 1000)
# result of flipping a coin 10 times, tested 1000 times.
```

A real world example. A company drills 9 wild-cat oil exploration
wells, each with an estimated probability of success of 0.1. All nine
wells fail. What is the probability of that happening?

Let’s do 20,000 trials of the model, and count the number that
generate zero positive results.

```pycon
>>> sum(np.random.binomial(9, 0.1, 20000) == 0)/20000.
# answer = 0.38885, or 38%.
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.chisquare.html.md

# dask.array.random.chisquare

### dask.array.random.chisquare(\*args, \*\*kwargs)

Draw samples from a chi-square distribution.

This docstring was copied from numpy.random.mtrand.RandomState.chisquare.

Some inconsistencies with the Dask version may exist.

When df independent random variables, each with standard normal
distributions (mean 0, variance 1), are squared and summed, the
resulting distribution is chi-square (see Notes).  This distribution
is often used in hypothesis testing.

#### NOTE
New code should use the ~numpy.random.Generator.chisquare
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **df**
  : Number of degrees of freedom, must be > 0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `df` is a scalar.  Otherwise,
    `np.array(df).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized chi-square distribution.
* **Raises:**
  ValueError
  : When df <= 0 or when an inappropriate size (e.g. `size=-1`)
    is given.

#### SEE ALSO
`random.Generator.chisquare`
: which should be used for new code.

### Notes

The variable obtained by summing the squares of df independent,
standard normally distributed random variables:

$$
Q = \sum_{i=1}^{\mathtt{df}} X^2_i

$$

is chi-square distributed, denoted

$$
Q \sim \chi^2_k.

$$

The probability density function of the chi-squared distribution is

$$
p(x) = \frac{(1/2)^{k/2}}{\Gamma(k/2)}
x^{k/2 - 1} e^{-x/2},

$$

where $\Gamma$ is the gamma function,

$$
\Gamma(x) = \int_0^{-\infty} t^{x - 1} e^{-t} dt.

$$

### References

### Examples

```pycon
>>> np.random.chisquare(2,4)
array([ 1.89920014,  9.00867716,  3.13710533,  5.62318272]) # random
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.choice.html.md

# dask.array.random.choice

### dask.array.random.choice(\*args, \*\*kwargs)

Generates a random sample from a given 1-D array

This docstring was copied from numpy.random.mtrand.RandomState.choice.

Some inconsistencies with the Dask version may exist.

#### NOTE
New code should use the ~numpy.random.Generator.choice
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

#### WARNING
This function uses the C-long dtype, which is 32bit on windows
and otherwise 64bit on 64bit platforms (and 32bit on 32bit ones).
Since NumPy 2.0, NumPy’s default integer is 32bit on 32bit platforms
and 64bit on 64bit platforms.

* **Parameters:**
  **a**
  : If an ndarray, a random sample is generated from its elements.
    If an int, the random sample is generated as if it were `np.arange(a)`

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.

  **replace**
  : Whether the sample is with or without replacement. Default is True,
    meaning that a value of `a` can be selected multiple times.

  **p**
  : The probabilities associated with each entry in a.
    If not given, the sample assumes a uniform distribution over all
    entries in `a`.
* **Returns:**
  **samples**
  : The generated random samples
* **Raises:**
  ValueError
  : If a is an int and less than zero, if a or p are not 1-dimensional,
    if a is an array-like of size 0, if p is not a vector of
    probabilities, if a and p have different lengths, or if
    replace=False and the sample size is greater than the population
    size

#### SEE ALSO
[`randint`](dask.array.random.randint.md#dask.array.random.randint), `shuffle`, [`permutation`](dask.array.random.permutation.md#dask.array.random.permutation)

`random.Generator.choice`
: which should be used in new code

### Notes

Setting user-specified probabilities through `p` uses a more general but less
efficient sampler than the default. The general sampler produces a different sample
than the optimized sampler even if each element of `p` is 1 / len(a).

Sampling random rows from a 2-D array is not possible with this function,
but is possible with Generator.choice through its `axis` keyword.

### Examples

Generate a uniform random sample from np.arange(5) of size 3:

```pycon
>>> np.random.choice(5, 3)
array([0, 3, 4]) # random
>>> #This is equivalent to np.random.randint(0,5,3)
```

Generate a non-uniform random sample from np.arange(5) of size 3:

```pycon
>>> np.random.choice(5, 3, p=[0.1, 0, 0.3, 0.6, 0])
array([3, 3, 0]) # random
```

Generate a uniform random sample from np.arange(5) of size 3 without
replacement:

```pycon
>>> np.random.choice(5, 3, replace=False)
array([3,1,0]) # random
>>> #This is equivalent to np.random.permutation(np.arange(5))[:3]
```

Generate a non-uniform random sample from np.arange(5) of size
3 without replacement:

```pycon
>>> np.random.choice(5, 3, replace=False, p=[0.1, 0, 0.3, 0.6, 0])
array([2, 3, 0]) # random
```

Any of the above can be repeated with an arbitrary array-like
instead of just integers. For instance:

```pycon
>>> aa_milne_arr = ['pooh', 'rabbit', 'piglet', 'Christopher']
>>> np.random.choice(aa_milne_arr, 5, p=[0.5, 0.1, 0.1, 0.3])
array(['pooh', 'pooh', 'pooh', 'Christopher', 'piglet'], # random
      dtype='<U11')
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.default_rng.html.md

# dask.array.random.default_rng

### dask.array.random.default_rng(seed=None)

Construct a new Generator with the default BitGenerator (PCG64).

* **Parameters:**
  **seed**
  : A seed to initialize the BitGenerator. If None, then fresh,
    unpredictable entropy will be pulled from the OS. If an `int` or
    `array_like[ints]` is passed, then it will be passed to
    SeedSequence to derive the initial BitGenerator state. One may
    also pass in a SeedSequence instance.
    Additionally, when passed a BitGenerator, it will be wrapped by
    Generator. If passed a Generator, it will be returned unaltered.
* **Returns:**
  Generator
  : The initialized generator object.

#### SEE ALSO
`np.random.default_rng`

### Notes

If `seed` is not a BitGenerator or a Generator, a new
BitGenerator is instantiated. This function does not manage a default
global instance.

### Examples

`default_rng` is the recommended constructor for the random number
class `Generator`. Here are several ways we can construct a random
number generator using `default_rng` and the `Generator` class.

Here we use `default_rng` to generate a random float:

```pycon
>>> import dask.array as da
>>> rng = da.random.default_rng(12345)
>>> print(rng)
Generator(PCG64)
>>> rfloat = rng.random().compute()
>>> rfloat
array(0.86999885)
>>> type(rfloat)
<class 'numpy.ndarray'>
```

Here we use `default_rng` to generate 3 random integers between 0
(inclusive) and 10 (exclusive):

```pycon
>>> import dask.array as da
>>> rng = da.random.default_rng(12345)
>>> rints = rng.integers(low=0, high=10, size=3).compute()
>>> rints
array([2, 8, 7])
>>> type(rints[0])
<class 'numpy.int64'>
```

Here we specify a seed so that we have reproducible results:

```pycon
>>> import dask.array as da
>>> rng = da.random.default_rng(seed=42)
>>> print(rng)
Generator(PCG64)
>>> arr1 = rng.random((3, 3)).compute()
>>> arr1
array([[0.91674416, 0.91098667, 0.8765925 ],
       [0.30931841, 0.95465607, 0.17509458],
       [0.99662814, 0.75203348, 0.15038118]])
```

If we exit and restart our Python interpreter, we’ll see that we
generate the same random numbers again:

```pycon
>>> import dask.array as da
>>> rng = da.random.default_rng(seed=42)
>>> arr2 = rng.random((3, 3)).compute()
>>> arr2
array([[0.91674416, 0.91098667, 0.8765925 ],
       [0.30931841, 0.95465607, 0.17509458],
       [0.99662814, 0.75203348, 0.15038118]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.exponential.html.md

# dask.array.random.exponential

### dask.array.random.exponential(\*args, \*\*kwargs)

Draw samples from an exponential distribution.

This docstring was copied from numpy.random.mtrand.RandomState.exponential.

Some inconsistencies with the Dask version may exist.

Its probability density function is

$$
f(x; \frac{1}{\beta}) = \frac{1}{\beta} \exp(-\frac{x}{\beta}),

$$

for `x > 0` and 0 elsewhere. $\beta$ is the scale parameter,
which is the inverse of the rate parameter $\lambda = 1/\beta$.
The rate parameter is an alternative, widely used parameterization
of the exponential distribution [[3]](#rafc1e1899521-3).

The exponential distribution is a continuous analogue of the
geometric distribution.  It describes many common situations, such as
the size of raindrops measured over many rainstorms [[1]](#rafc1e1899521-1), or the time
between page requests to Wikipedia [[2]](#rafc1e1899521-2).

#### NOTE
New code should use the ~numpy.random.Generator.exponential
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **scale**
  : The scale parameter, $\beta = 1/\lambda$. Must be
    non-negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `scale` is a scalar.  Otherwise,
    `np.array(scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized exponential distribution.

#### SEE ALSO
`random.Generator.exponential`
: which should be used for new code.

### References

### Examples

A real world example: Assume a company has 10000 customer support
agents and the average time between customer calls is 4 minutes.

```pycon
>>> n = 10000
>>> time_between_calls = np.random.default_rng().exponential(scale=4, size=n)
```

What is the probability that a customer will call in the next
4 to 5 minutes?

```pycon
>>> x = ((time_between_calls < 5).sum())/n
>>> y = ((time_between_calls < 4).sum())/n
>>> x-y
0.08 # may vary
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.f.html.md

# dask.array.random.f

### dask.array.random.f(\*args, \*\*kwargs)

Draw samples from an F distribution.

This docstring was copied from numpy.random.mtrand.RandomState.f.

Some inconsistencies with the Dask version may exist.

Samples are drawn from an F distribution with specified parameters,
dfnum (degrees of freedom in numerator) and dfden (degrees of
freedom in denominator), where both parameters must be greater than
zero.

The random variate of the F distribution (also known as the
Fisher distribution) is a continuous probability distribution
that arises in ANOVA tests, and is the ratio of two chi-square
variates.

#### NOTE
New code should use the ~numpy.random.Generator.f
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **dfnum**
  : Degrees of freedom in numerator, must be > 0.

  **dfden**
  : Degrees of freedom in denominator, must be > 0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `dfnum` and `dfden` are both scalars.
    Otherwise, `np.broadcast(dfnum, dfden).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Fisher distribution.

#### SEE ALSO
[`scipy.stats.f`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f.html#scipy.stats.f)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.f`
: which should be used for new code.

### Notes

The F statistic is used to compare in-group variances to between-group
variances. Calculating the distribution depends on the sampling, and
so it is a function of the respective degrees of freedom in the
problem.  The variable dfnum is the number of samples minus one, the
between-groups degrees of freedom, while dfden is the within-groups
degrees of freedom, the sum of the number of samples in each group
minus the number of groups.

### References

### Examples

An example from Glantz[1], pp 47-40:

Two groups, children of diabetics (25 people) and children from people
without diabetes (25 controls). Fasting blood glucose was measured,
case group had a mean value of 86.1, controls had a mean value of
82.2. Standard deviations were 2.09 and 2.49 respectively. Are these
data consistent with the null hypothesis that the parents diabetic
status does not affect their children’s blood glucose levels?
Calculating the F statistic from the data gives a value of 36.01.

Draw samples from the distribution:

```pycon
>>> dfnum = 1. # between group degrees of freedom
>>> dfden = 48. # within groups degrees of freedom
>>> s = np.random.f(dfnum, dfden, 1000)
```

The lower bound for the top 1% of the samples is :

```pycon
>>> np.sort(s)[-10]
7.61988120985 # random
```

So there is about a 1% chance that the F statistic will exceed 7.62,
the measured value is 36, so the null hypothesis is rejected at the 1%
level.

<!-- !! processed by numpydoc !! -->


# dask.array.random.gamma.html.md

# dask.array.random.gamma

### dask.array.random.gamma(\*args, \*\*kwargs)

Draw samples from a Gamma distribution.

This docstring was copied from numpy.random.mtrand.RandomState.gamma.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a Gamma distribution with specified parameters,
shape (sometimes designated “k”) and scale (sometimes designated
“theta”), where both parameters are > 0.

#### NOTE
New code should use the ~numpy.random.Generator.gamma
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **shape**
  : The shape of the gamma distribution. Must be non-negative.

  **scale**
  : The scale of the gamma distribution. Must be non-negative.
    Default is equal to 1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `shape` and `scale` are both scalars.
    Otherwise, `np.broadcast(shape, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized gamma distribution.

#### SEE ALSO
[`scipy.stats.gamma`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gamma.html#scipy.stats.gamma)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.gamma`
: which should be used for new code.

### Notes

The probability density for the Gamma distribution is

$$
p(x) = x^{k-1}\frac{e^{-x/\theta}}{\theta^k\Gamma(k)},

$$

where $k$ is the shape and $\theta$ the scale,
and $\Gamma$ is the Gamma function.

The Gamma distribution is often used to model the times to failure of
electronic components, and arises naturally in processes for which the
waiting times between Poisson distributed events are relevant.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> shape, scale = 2., 2.  # mean=4, std=2*sqrt(2)
>>> s = np.random.gamma(shape, scale, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> import scipy.special as sps
>>> count, bins, ignored = plt.hist(s, 50, density=True)
>>> y = bins**(shape-1)*(np.exp(-bins/scale) /
...                      (sps.gamma(shape)*scale**shape))
>>> plt.plot(bins, y, linewidth=2, color='r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.geometric.html.md

# dask.array.random.geometric

### dask.array.random.geometric(\*args, \*\*kwargs)

Draw samples from the geometric distribution.

This docstring was copied from numpy.random.mtrand.RandomState.geometric.

Some inconsistencies with the Dask version may exist.

Bernoulli trials are experiments with one of two outcomes:
success or failure (an example of such an experiment is flipping
a coin).  The geometric distribution models the number of trials
that must be run in order to achieve success.  It is therefore
supported on the positive integers, `k = 1, 2, ...`.

The probability mass function of the geometric distribution is

$$
f(k) = (1 - p)^{k - 1} p

$$

where p is the probability of success of an individual trial.

#### NOTE
New code should use the ~numpy.random.Generator.geometric
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **p**
  : The probability of success of an individual trial.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `p` is a scalar.  Otherwise,
    `np.array(p).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized geometric distribution.

#### SEE ALSO
`random.Generator.geometric`
: which should be used for new code.

### Examples

Draw ten thousand values from the geometric distribution,
with the probability of an individual success equal to 0.35:

```pycon
>>> z = np.random.geometric(p=0.35, size=10000)
```

How many trials succeeded after a single run?

```pycon
>>> (z == 1).sum() / 10000.
0.34889999999999999 #random
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.gumbel.html.md

# dask.array.random.gumbel

### dask.array.random.gumbel(\*args, \*\*kwargs)

Draw samples from a Gumbel distribution.

This docstring was copied from numpy.random.mtrand.RandomState.gumbel.

Some inconsistencies with the Dask version may exist.

Draw samples from a Gumbel distribution with specified location and
scale.  For more information on the Gumbel distribution, see
Notes and References below.

#### NOTE
New code should use the ~numpy.random.Generator.gumbel
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **loc**
  : The location of the mode of the distribution. Default is 0.

  **scale**
  : The scale parameter of the distribution. Default is 1. Must be non-
    negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `loc` and `scale` are both scalars.
    Otherwise, `np.broadcast(loc, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Gumbel distribution.

#### SEE ALSO
[`scipy.stats.gumbel_l`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gumbel_l.html#scipy.stats.gumbel_l)

[`scipy.stats.gumbel_r`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gumbel_r.html#scipy.stats.gumbel_r)

[`scipy.stats.genextreme`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.genextreme.html#scipy.stats.genextreme)

[`weibull`](dask.array.random.weibull.md#dask.array.random.weibull)

`random.Generator.gumbel`
: which should be used for new code.

### Notes

The Gumbel (or Smallest Extreme Value (SEV) or the Smallest Extreme
Value Type I) distribution is one of a class of Generalized Extreme
Value (GEV) distributions used in modeling extreme value problems.
The Gumbel is a special case of the Extreme Value Type I distribution
for maximums from distributions with “exponential-like” tails.

The probability density for the Gumbel distribution is

$$
p(x) = \frac{e^{-(x - \mu)/ \beta}}{\beta} e^{ -e^{-(x - \mu)/
\beta}},

$$

where $\mu$ is the mode, a location parameter, and
$\beta$ is the scale parameter.

The Gumbel (named for German mathematician Emil Julius Gumbel) was used
very early in the hydrology literature, for modeling the occurrence of
flood events. It is also used for modeling maximum wind speed and
rainfall rates.  It is a “fat-tailed” distribution - the probability of
an event in the tail of the distribution is larger than if one used a
Gaussian, hence the surprisingly frequent occurrence of 100-year
floods. Floods were initially modeled as a Gaussian process, which
underestimated the frequency of extreme events.

It is one of a class of extreme value distributions, the Generalized
Extreme Value (GEV) distributions, which also includes the Weibull and
Frechet.

The function has a mean of $\mu + 0.57721\beta$ and a variance
of $\frac{\pi^2}{6}\beta^2$.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> mu, beta = 0, 0.1 # location and scale
>>> s = np.random.gumbel(mu, beta, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 30, density=True)
>>> plt.plot(bins, (1/beta)*np.exp(-(bins - mu)/beta)
...          * np.exp( -np.exp( -(bins - mu) /beta) ),
...          linewidth=2, color='r')
>>> plt.show()
```

Show how an extreme value distribution can arise from a Gaussian process
and compare to a Gaussian:

```pycon
>>> means = []
>>> maxima = []
>>> for i in range(0,1000) :
...    a = np.random.normal(mu, beta, 1000)
...    means.append(a.mean())
...    maxima.append(a.max())
>>> count, bins, ignored = plt.hist(maxima, 30, density=True)
>>> beta = np.std(maxima) * np.sqrt(6) / np.pi
>>> mu = np.mean(maxima) - 0.57721*beta
>>> plt.plot(bins, (1/beta)*np.exp(-(bins - mu)/beta)
...          * np.exp(-np.exp(-(bins - mu)/beta)),
...          linewidth=2, color='r')
>>> plt.plot(bins, 1/(beta * np.sqrt(2 * np.pi))
...          * np.exp(-(bins - mu)**2 / (2 * beta**2)),
...          linewidth=2, color='g')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.hypergeometric.html.md

# dask.array.random.hypergeometric

### dask.array.random.hypergeometric(\*args, \*\*kwargs)

Draw samples from a Hypergeometric distribution.

This docstring was copied from numpy.random.mtrand.RandomState.hypergeometric.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a hypergeometric distribution with specified
parameters, ngood (ways to make a good selection), nbad (ways to make
a bad selection), and nsample (number of items sampled, which is less
than or equal to the sum `ngood + nbad`).

#### NOTE
New code should use the
~numpy.random.Generator.hypergeometric
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **ngood**
  : Number of ways to make a good selection.  Must be nonnegative.

  **nbad**
  : Number of ways to make a bad selection.  Must be nonnegative.

  **nsample**
  : Number of items sampled.  Must be at least 1 and at most
    `ngood + nbad`.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if ngood, nbad, and nsample
    are all scalars.  Otherwise, `np.broadcast(ngood, nbad, nsample).size`
    samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized hypergeometric distribution. Each
    sample is the number of good items within a randomly selected subset of
    size nsample taken from a set of ngood good items and nbad bad items.

#### SEE ALSO
[`scipy.stats.hypergeom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.hypergeom.html#scipy.stats.hypergeom)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.hypergeometric`
: which should be used for new code.

### Notes

The probability mass function (PMF) for the Hypergeometric distribution is

$$
P(x) = \frac{\binom{g}{x}\binom{b}{n-x}}{\binom{g+b}{n}},

$$

where $0 \le x \le n$ and $n-b \le x \le g$

for P(x) the probability of `x` good results in the drawn sample,
g = ngood, b = nbad, and n = nsample.

Consider an urn with black and white marbles in it, ngood of them
are black and nbad are white. If you draw nsample balls without
replacement, then the hypergeometric distribution describes the
distribution of black balls in the drawn sample.

Note that this distribution is very similar to the binomial
distribution, except that in this case, samples are drawn without
replacement, whereas in the Binomial case samples are drawn with
replacement (or the sample space is infinite). As the sample space
becomes large, this distribution approaches the binomial.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> ngood, nbad, nsamp = 100, 2, 10
# number of good, number of bad, and number of samples
>>> s = np.random.hypergeometric(ngood, nbad, nsamp, 1000)
>>> from matplotlib.pyplot import hist
>>> hist(s)
#   note that it is very unlikely to grab both bad items
```

Suppose you have an urn with 15 white and 15 black marbles.
If you pull 15 marbles at random, how likely is it that
12 or more of them are one color?

```pycon
>>> s = np.random.hypergeometric(15, 15, 15, 100000)
>>> sum(s>=12)/100000. + sum(s<=3)/100000.
#   answer = 0.003 ... pretty unlikely!
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.laplace.html.md

# dask.array.random.laplace

### dask.array.random.laplace(\*args, \*\*kwargs)

Draw samples from the Laplace or double exponential distribution with
specified location (or mean) and scale (decay).

This docstring was copied from numpy.random.mtrand.RandomState.laplace.

Some inconsistencies with the Dask version may exist.

The Laplace distribution is similar to the Gaussian/normal distribution,
but is sharper at the peak and has fatter tails. It represents the
difference between two independent, identically distributed exponential
random variables.

#### NOTE
New code should use the ~numpy.random.Generator.laplace
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **loc**
  : The position, $\mu$, of the distribution peak. Default is 0.

  **scale**
  : $\lambda$, the exponential decay. Default is 1. Must be non-
    negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `loc` and `scale` are both scalars.
    Otherwise, `np.broadcast(loc, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Laplace distribution.

#### SEE ALSO
`random.Generator.laplace`
: which should be used for new code.

### Notes

It has the probability density function

$$
f(x; \mu, \lambda) = \frac{1}{2\lambda}
\exp\left(-\frac{|x - \mu|}{\lambda}\right).

$$

The first law of Laplace, from 1774, states that the frequency
of an error can be expressed as an exponential function of the
absolute magnitude of the error, which leads to the Laplace
distribution. For many problems in economics and health
sciences, this distribution seems to model the data better
than the standard Gaussian distribution.

### References

### Examples

Draw samples from the distribution

```pycon
>>> loc, scale = 0., 1.
>>> s = np.random.laplace(loc, scale, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 30, density=True)
>>> x = np.arange(-8., 8., .01)
>>> pdf = np.exp(-abs(x-loc)/scale)/(2.*scale)
>>> plt.plot(x, pdf)
```

Plot Gaussian for comparison:

```pycon
>>> g = (1/(scale * np.sqrt(2 * np.pi)) *
...      np.exp(-(x - loc)**2 / (2 * scale**2)))
>>> plt.plot(x,g)
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.logistic.html.md

# dask.array.random.logistic

### dask.array.random.logistic(\*args, \*\*kwargs)

Draw samples from a logistic distribution.

This docstring was copied from numpy.random.mtrand.RandomState.logistic.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a logistic distribution with specified
parameters, loc (location or mean, also median), and scale (>0).

#### NOTE
New code should use the ~numpy.random.Generator.logistic
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **loc**
  : Parameter of the distribution. Default is 0.

  **scale**
  : Parameter of the distribution. Must be non-negative.
    Default is 1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `loc` and `scale` are both scalars.
    Otherwise, `np.broadcast(loc, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized logistic distribution.

#### SEE ALSO
[`scipy.stats.logistic`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.logistic.html#scipy.stats.logistic)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.logistic`
: which should be used for new code.

### Notes

The probability density for the Logistic distribution is

$$
P(x) = P(x) = \frac{e^{-(x-\mu)/s}}{s(1+e^{-(x-\mu)/s})^2},

$$

where $\mu$ = location and $s$ = scale.

The Logistic distribution is used in Extreme Value problems where it
can act as a mixture of Gumbel distributions, in Epidemiology, and by
the World Chess Federation (FIDE) where it is used in the Elo ranking
system, assuming the performance of each player is a logistically
distributed random variable.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> loc, scale = 10, 1
>>> s = np.random.logistic(loc, scale, 10000)
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, bins=50)
```

#   plot against distribution

```pycon
>>> def logist(x, loc, scale):
...     return np.exp((loc-x)/scale)/(scale*(1+np.exp((loc-x)/scale))**2)
>>> lgst_val = logist(bins, loc, scale)
>>> plt.plot(bins, lgst_val * count.max() / lgst_val.max())
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.lognormal.html.md

# dask.array.random.lognormal

### dask.array.random.lognormal(\*args, \*\*kwargs)

Draw samples from a log-normal distribution.

This docstring was copied from numpy.random.mtrand.RandomState.lognormal.

Some inconsistencies with the Dask version may exist.

Draw samples from a log-normal distribution with specified mean,
standard deviation, and array shape.  Note that the mean and standard
deviation are not the values for the distribution itself, but of the
underlying normal distribution it is derived from.

#### NOTE
New code should use the ~numpy.random.Generator.lognormal
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **mean**
  : Mean value of the underlying normal distribution. Default is 0.

  **sigma**
  : Standard deviation of the underlying normal distribution. Must be
    non-negative. Default is 1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `mean` and `sigma` are both scalars.
    Otherwise, `np.broadcast(mean, sigma).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized log-normal distribution.

#### SEE ALSO
[`scipy.stats.lognorm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.lognorm.html#scipy.stats.lognorm)
: probability density function, distribution, cumulative density function, etc.

`random.Generator.lognormal`
: which should be used for new code.

### Notes

A variable x has a log-normal distribution if log(x) is normally
distributed.  The probability density function for the log-normal
distribution is:

$$
p(x) = \frac{1}{\sigma x \sqrt{2\pi}}
e^{(-\frac{(ln(x)-\mu)^2}{2\sigma^2})}

$$

where $\mu$ is the mean and $\sigma$ is the standard
deviation of the normally distributed logarithm of the variable.
A log-normal distribution results if a random variable is the *product*
of a large number of independent, identically-distributed variables in
the same way that a normal distribution results if the variable is the
*sum* of a large number of independent, identically-distributed
variables.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> mu, sigma = 3., 1. # mean and standard deviation
>>> s = np.random.lognormal(mu, sigma, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 100, density=True, align='mid')
```

```pycon
>>> x = np.linspace(min(bins), max(bins), 10000)
>>> pdf = (np.exp(-(np.log(x) - mu)**2 / (2 * sigma**2))
...        / (x * sigma * np.sqrt(2 * np.pi)))
```

```pycon
>>> plt.plot(x, pdf, linewidth=2, color='r')
>>> plt.axis('tight')
>>> plt.show()
```

Demonstrate that taking the products of random samples from a uniform
distribution can be fit well by a log-normal probability density
function.

```pycon
>>> # Generate a thousand samples: each is the product of 100 random
>>> # values, drawn from a normal distribution.
>>> b = []
>>> for i in range(1000):
...    a = 10. + np.random.standard_normal(100)
...    b.append(np.prod(a))
```

```pycon
>>> b = np.array(b) / np.min(b) # scale values to be positive
>>> count, bins, ignored = plt.hist(b, 100, density=True, align='mid')
>>> sigma = np.std(np.log(b))
>>> mu = np.mean(np.log(b))
```

```pycon
>>> x = np.linspace(min(bins), max(bins), 10000)
>>> pdf = (np.exp(-(np.log(x) - mu)**2 / (2 * sigma**2))
...        / (x * sigma * np.sqrt(2 * np.pi)))
```

```pycon
>>> plt.plot(x, pdf, color='r', linewidth=2)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.logseries.html.md

# dask.array.random.logseries

### dask.array.random.logseries(\*args, \*\*kwargs)

Draw samples from a logarithmic series distribution.

This docstring was copied from numpy.random.mtrand.RandomState.logseries.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a log series distribution with specified
shape parameter, 0 <= `p` < 1.

#### NOTE
New code should use the ~numpy.random.Generator.logseries
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **p**
  : Shape parameter for the distribution.  Must be in the range [0, 1).

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `p` is a scalar.  Otherwise,
    `np.array(p).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized logarithmic series distribution.

#### SEE ALSO
[`scipy.stats.logser`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.logser.html#scipy.stats.logser)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.logseries`
: which should be used for new code.

### Notes

The probability density for the Log Series distribution is

$$
P(k) = \frac{-p^k}{k \ln(1-p)},

$$

where p = probability.

The log series distribution is frequently used to represent species
richness and occurrence, first proposed by Fisher, Corbet, and
Williams in 1943 [2].  It may also be used to model the numbers of
occupants seen in cars [3].

### References

### Examples

Draw samples from the distribution:

```pycon
>>> a = .6
>>> s = np.random.logseries(a, 10000)
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s)
```

#   plot against distribution

```pycon
>>> def logseries(k, p):
...     return -p**k/(k*np.log(1-p))
>>> plt.plot(bins, logseries(bins, a)*count.max()/
...          logseries(bins, a).max(), 'r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.multinomial.html.md

# dask.array.random.multinomial

### dask.array.random.multinomial(\*args, \*\*kwargs)

Draw samples from a multinomial distribution.

This docstring was copied from numpy.random.mtrand.RandomState.multinomial.

Some inconsistencies with the Dask version may exist.

The multinomial distribution is a multivariate generalization of the
binomial distribution.  Take an experiment with one of `p`
possible outcomes.  An example of such an experiment is throwing a dice,
where the outcome can be 1 through 6.  Each sample drawn from the
distribution represents n such experiments.  Its values,
`X_i = [X_0, X_1, ..., X_p]`, represent the number of times the
outcome was `i`.

#### NOTE
New code should use the ~numpy.random.Generator.multinomial
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

#### WARNING
This function defaults to the C-long dtype, which is 32bit on windows
and otherwise 64bit on 64bit platforms (and 32bit on 32bit ones).
Since NumPy 2.0, NumPy’s default integer is 32bit on 32bit platforms
and 64bit on 64bit platforms.

* **Parameters:**
  **n**
  : Number of experiments.

  **pvals**
  : Probabilities of each of the `p` different outcomes.  These
    must sum to 1 (however, the last element is always assumed to
    account for the remaining probability, as long as
    `sum(pvals[:-1]) <= 1)`.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : The drawn samples, of shape *size*, if that was provided.  If not,
    the shape is `(N,)`.
    <br/>
    In other words, each entry `out[i,j,...,:]` is an N-dimensional
    value drawn from the distribution.

#### SEE ALSO
`random.Generator.multinomial`
: which should be used for new code.

### Examples

Throw a dice 20 times:

```pycon
>>> np.random.multinomial(20, [1/6.]*6, size=1)
array([[4, 1, 7, 5, 2, 1]]) # random
```

It landed 4 times on 1, once on 2, etc.

Now, throw the dice 20 times, and 20 times again:

```pycon
>>> np.random.multinomial(20, [1/6.]*6, size=2)
array([[3, 4, 3, 3, 4, 3], # random
       [2, 4, 3, 4, 0, 7]])
```

For the first run, we threw 3 times 1, 4 times 2, etc.  For the second,
we threw 2 times 1, 4 times 2, etc.

A loaded die is more likely to land on number 6:

```pycon
>>> np.random.multinomial(100, [1/7.]*5 + [2/7.])
array([11, 16, 14, 17, 16, 26]) # random
```

The probability inputs should be normalized. As an implementation
detail, the value of the last entry is ignored and assumed to take
up any leftover probability mass, but this should not be relied on.
A biased coin which has twice as much weight on one side as on the
other should be sampled like so:

```pycon
>>> np.random.multinomial(100, [1.0 / 3, 2.0 / 3])  # RIGHT
array([38, 62]) # random
```

not like:

```pycon
>>> np.random.multinomial(100, [1.0, 2.0])  # WRONG
Traceback (most recent call last):
ValueError: pvals < 0, pvals > 1 or pvals contains NaNs
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.negative_binomial.html.md

# dask.array.random.negative_binomial

### dask.array.random.negative_binomial(\*args, \*\*kwargs)

Draw samples from a negative binomial distribution.

This docstring was copied from numpy.random.mtrand.RandomState.negative_binomial.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a negative binomial distribution with specified
parameters, n successes and p probability of success where n
is > 0 and p is in the interval [0, 1].

#### NOTE
New code should use the
~numpy.random.Generator.negative_binomial
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **n**
  : Parameter of the distribution, > 0.

  **p**
  : Parameter of the distribution, >= 0 and <=1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `n` and `p` are both scalars.
    Otherwise, `np.broadcast(n, p).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized negative binomial distribution,
    where each sample is equal to N, the number of failures that
    occurred before a total of n successes was reached.

  #### WARNING
  This function returns the C-long dtype, which is 32bit on windows
  and otherwise 64bit on 64bit platforms (and 32bit on 32bit ones).
  Since NumPy 2.0, NumPy’s default integer is 32bit on 32bit platforms
  and 64bit on 64bit platforms.

#### SEE ALSO
`random.Generator.negative_binomial`
: which should be used for new code.

### Notes

The probability mass function of the negative binomial distribution is

$$
P(N;n,p) = \frac{\Gamma(N+n)}{N!\Gamma(n)}p^{n}(1-p)^{N},

$$

where $n$ is the number of successes, $p$ is the
probability of success, $N+n$ is the number of trials, and
$\Gamma$ is the gamma function. When $n$ is an integer,
$\frac{\Gamma(N+n)}{N!\Gamma(n)} = \binom{N+n-1}{N}$, which is
the more common form of this term in the pmf. The negative
binomial distribution gives the probability of N failures given n
successes, with a success on the last trial.

If one throws a die repeatedly until the third time a “1” appears,
then the probability distribution of the number of non-“1”s that
appear before the third “1” is a negative binomial distribution.

### References

### Examples

Draw samples from the distribution:

A real world example. A company drills wild-cat oil
exploration wells, each with an estimated probability of
success of 0.1.  What is the probability of having one success
for each successive well, that is what is the probability of a
single success after drilling 5 wells, after 6 wells, etc.?

```pycon
>>> s = np.random.negative_binomial(1, 0.1, 100000)
>>> for i in range(1, 11):
...    probability = sum(s<i) / 100000.
...    print(i, "wells drilled, probability of one success =", probability)
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.noncentral_chisquare.html.md

# dask.array.random.noncentral_chisquare

### dask.array.random.noncentral_chisquare(\*args, \*\*kwargs)

Draw samples from a noncentral chi-square distribution.

This docstring was copied from numpy.random.mtrand.RandomState.noncentral_chisquare.

Some inconsistencies with the Dask version may exist.

The noncentral $\chi^2$ distribution is a generalization of
the $\chi^2$ distribution.

#### NOTE
New code should use the
~numpy.random.Generator.noncentral_chisquare
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **df**
  : Degrees of freedom, must be > 0.

  **nonc**
  : Non-centrality, must be non-negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `df` and `nonc` are both scalars.
    Otherwise, `np.broadcast(df, nonc).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized noncentral chi-square distribution.

#### SEE ALSO
`random.Generator.noncentral_chisquare`
: which should be used for new code.

### Notes

The probability density function for the noncentral Chi-square
distribution is

$$
P(x;df,nonc) = \sum^{\infty}_{i=0}
\frac{e^{-nonc/2}(nonc/2)^{i}}{i!}
P_{Y_{df+2i}}(x),

$$

where $Y_{q}$ is the Chi-square with q degrees of freedom.

### References

### Examples

Draw values from the distribution and plot the histogram

```pycon
>>> import matplotlib.pyplot as plt
>>> values = plt.hist(np.random.noncentral_chisquare(3, 20, 100000),
...                   bins=200, density=True)
>>> plt.show()
```

Draw values from a noncentral chisquare with very small noncentrality,
and compare to a chisquare.

```pycon
>>> plt.figure()
>>> values = plt.hist(np.random.noncentral_chisquare(3, .0000001, 100000),
...                   bins=np.arange(0., 25, .1), density=True)
>>> values2 = plt.hist(np.random.chisquare(3, 100000),
...                    bins=np.arange(0., 25, .1), density=True)
>>> plt.plot(values[1][0:-1], values[0]-values2[0], 'ob')
>>> plt.show()
```

Demonstrate how large values of non-centrality lead to a more symmetric
distribution.

```pycon
>>> plt.figure()
>>> values = plt.hist(np.random.noncentral_chisquare(3, 20, 100000),
...                   bins=200, density=True)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.noncentral_f.html.md

# dask.array.random.noncentral_f

### dask.array.random.noncentral_f(\*args, \*\*kwargs)

Draw samples from the noncentral F distribution.

This docstring was copied from numpy.random.mtrand.RandomState.noncentral_f.

Some inconsistencies with the Dask version may exist.

Samples are drawn from an F distribution with specified parameters,
dfnum (degrees of freedom in numerator) and dfden (degrees of
freedom in denominator), where both parameters > 1.
nonc is the non-centrality parameter.

#### NOTE
New code should use the
~numpy.random.Generator.noncentral_f
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **dfnum**
  : Numerator degrees of freedom, must be > 0.

  **dfden**
  : Denominator degrees of freedom, must be > 0.

  **nonc**
  : Non-centrality parameter, the sum of the squares of the numerator
    means, must be >= 0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `dfnum`, `dfden`, and `nonc`
    are all scalars.  Otherwise, `np.broadcast(dfnum, dfden, nonc).size`
    samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized noncentral Fisher distribution.

#### SEE ALSO
`random.Generator.noncentral_f`
: which should be used for new code.

### Notes

When calculating the power of an experiment (power = probability of
rejecting the null hypothesis when a specific alternative is true) the
non-central F statistic becomes important.  When the null hypothesis is
true, the F statistic follows a central F distribution. When the null
hypothesis is not true, then it follows a non-central F statistic.

### References

### Examples

In a study, testing for a specific alternative to the null hypothesis
requires use of the Noncentral F distribution. We need to calculate the
area in the tail of the distribution that exceeds the value of the F
distribution for the null hypothesis.  We’ll plot the two probability
distributions for comparison.

```pycon
>>> dfnum = 3 # between group deg of freedom
>>> dfden = 20 # within groups degrees of freedom
>>> nonc = 3.0
>>> nc_vals = np.random.noncentral_f(dfnum, dfden, nonc, 1000000)
>>> NF = np.histogram(nc_vals, bins=50, density=True)
>>> c_vals = np.random.f(dfnum, dfden, 1000000)
>>> F = np.histogram(c_vals, bins=50, density=True)
>>> import matplotlib.pyplot as plt
>>> plt.plot(F[1][1:], F[0])
>>> plt.plot(NF[1][1:], NF[0])
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.normal.html.md

# dask.array.random.normal

### dask.array.random.normal(\*args, \*\*kwargs)

Draw random samples from a normal (Gaussian) distribution.

This docstring was copied from numpy.random.mtrand.RandomState.normal.

Some inconsistencies with the Dask version may exist.

The probability density function of the normal distribution, first
derived by De Moivre and 200 years later by both Gauss and Laplace
independently [[2]](#r06ed1c8cfe9e-2), is often called the bell curve because of
its characteristic shape (see the example below).

The normal distributions occurs often in nature.  For example, it
describes the commonly occurring distribution of samples influenced
by a large number of tiny, random disturbances, each with its own
unique distribution [[2]](#r06ed1c8cfe9e-2).

#### NOTE
New code should use the ~numpy.random.Generator.normal
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **loc**
  : Mean (“centre”) of the distribution.

  **scale**
  : Standard deviation (spread or “width”) of the distribution. Must be
    non-negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `loc` and `scale` are both scalars.
    Otherwise, `np.broadcast(loc, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized normal distribution.

#### SEE ALSO
[`scipy.stats.norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html#scipy.stats.norm)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.normal`
: which should be used for new code.

### Notes

The probability density for the Gaussian distribution is

$$
p(x) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}
e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} },

$$

where $\mu$ is the mean and $\sigma$ the standard
deviation. The square of the standard deviation, $\sigma^2$,
is called the variance.

The function has its peak at the mean, and its “spread” increases with
the standard deviation (the function reaches 0.607 times its maximum at
$x + \sigma$ and $x - \sigma$ [[2]](#r06ed1c8cfe9e-2)).  This implies that
normal is more likely to return samples lying close to the mean, rather
than those far away.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> mu, sigma = 0, 0.1 # mean and standard deviation
>>> s = np.random.normal(mu, sigma, 1000)
```

Verify the mean and the standard deviation:

```pycon
>>> abs(mu - np.mean(s))
0.0  # may vary
```

```pycon
>>> abs(sigma - np.std(s, ddof=1))
0.0  # may vary
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 30, density=True)
>>> plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
...                np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
...          linewidth=2, color='r')
>>> plt.show()
```

Two-by-four array of samples from the normal distribution with
mean 3 and standard deviation 2.5:

```pycon
>>> np.random.normal(3, 2.5, size=(2, 4))
array([[-4.49401501,  4.00950034, -1.81814867,  7.29718677],   # random
       [ 0.39924804,  4.68456316,  4.99394529,  4.84057254]])  # random
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.pareto.html.md

# dask.array.random.pareto

### dask.array.random.pareto(\*args, \*\*kwargs)

Draw samples from a Pareto II or Lomax distribution with
specified shape.

This docstring was copied from numpy.random.mtrand.RandomState.pareto.

Some inconsistencies with the Dask version may exist.

The Lomax or Pareto II distribution is a shifted Pareto
distribution. The classical Pareto distribution can be
obtained from the Lomax distribution by adding 1 and
multiplying by the scale parameter `m` (see Notes).  The
smallest value of the Lomax distribution is zero while for the
classical Pareto distribution it is `mu`, where the standard
Pareto distribution has location `mu = 1`.  Lomax can also
be considered as a simplified version of the Generalized
Pareto distribution (available in SciPy), with the scale set
to one and the location set to zero.

The Pareto distribution must be greater than zero, and is
unbounded above.  It is also known as the “80-20 rule”.  In
this distribution, 80 percent of the weights are in the lowest
20 percent of the range, while the other 20 percent fill the
remaining 80 percent of the range.

#### NOTE
New code should use the ~numpy.random.Generator.pareto
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **a**
  : Shape of the distribution. Must be positive.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `a` is a scalar.  Otherwise,
    `np.array(a).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Pareto distribution.

#### SEE ALSO
[`scipy.stats.lomax`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.lomax.html#scipy.stats.lomax)
: probability density function, distribution or cumulative density function, etc.

[`scipy.stats.genpareto`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.genpareto.html#scipy.stats.genpareto)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.pareto`
: which should be used for new code.

### Notes

The probability density for the Pareto II distribution is

$$
p(x) = \frac{a}{(x+1)^{a+1}} , x \ge 0

$$

where $a > 0$ is the shape.

The Pareto II distribution is a shifted and scaled version of the
Pareto I distribution, which can be found in scipy.stats.pareto.

The Pareto distribution, named after the Italian economist
Vilfredo Pareto, is a power law probability distribution
useful in many real world problems.  Outside the field of
economics it is generally referred to as the Bradford
distribution. Pareto developed the distribution to describe
the distribution of wealth in an economy.  It has also found
use in insurance, web page access statistics, oil field sizes,
and many other problems, including the download frequency for
projects in Sourceforge [[1]](#r8b836986745a-1).  It is one of the so-called
“fat-tailed” distributions.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> a, m = 3., 2.  # shape and mode
>>> s = (np.random.pareto(a, 1000) + 1) * m
```

Display the histogram of the samples, along with the probability
density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, _ = plt.hist(s, 100, density=True)
>>> fit = a*m**a / bins**(a+1)
>>> plt.plot(bins, max(count)*fit/max(fit), linewidth=2, color='r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.permutation.html.md

# dask.array.random.permutation

### dask.array.random.permutation(\*args, \*\*kwargs)

Randomly permute a sequence, or return a permuted range.

This docstring was copied from numpy.random.mtrand.RandomState.permutation.

Some inconsistencies with the Dask version may exist.

If x is a multi-dimensional array, it is only shuffled along its
first index.

#### NOTE
New code should use the
~numpy.random.Generator.permutation
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **x**
  : If x is an integer, randomly permute `np.arange(x)`.
    If x is an array, make a copy and shuffle the elements
    randomly.
* **Returns:**
  **out**
  : Permuted sequence or array range.

#### SEE ALSO
`random.Generator.permutation`
: which should be used for new code.

### Examples

```pycon
>>> np.random.permutation(10)
array([1, 7, 4, 3, 0, 9, 2, 5, 8, 6]) # random
```

```pycon
>>> np.random.permutation([1, 4, 9, 12, 15])
array([15,  1,  9,  4, 12]) # random
```

```pycon
>>> arr = np.arange(9).reshape((3, 3))
>>> np.random.permutation(arr)
array([[6, 7, 8], # random
       [0, 1, 2],
       [3, 4, 5]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.poisson.html.md

# dask.array.random.poisson

### dask.array.random.poisson(\*args, \*\*kwargs)

Draw samples from a Poisson distribution.

This docstring was copied from numpy.random.mtrand.RandomState.poisson.

Some inconsistencies with the Dask version may exist.

The Poisson distribution is the limit of the binomial distribution
for large N.

#### NOTE
New code should use the ~numpy.random.Generator.poisson
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **lam**
  : Expected number of events occurring in a fixed-time interval,
    must be >= 0. A sequence must be broadcastable over the requested
    size.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `lam` is a scalar. Otherwise,
    `np.array(lam).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Poisson distribution.

#### SEE ALSO
`random.Generator.poisson`
: which should be used for new code.

### Notes

The probability mass function (PMF) of Poisson distribution is

$$
f(k; \lambda)=\frac{\lambda^k e^{-\lambda}}{k!}

$$

For events with an expected separation $\lambda$ the Poisson
distribution $f(k; \lambda)$ describes the probability of
$k$ events occurring within the observed
interval $\lambda$.

Because the output is limited to the range of the C int64 type, a
ValueError is raised when lam is within 10 sigma of the maximum
representable value.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> import numpy as np
>>> s = np.random.poisson(5, 10000)
```

Display histogram of the sample:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 14, density=True)
>>> plt.show()
```

Draw each 100 values for lambda 100 and 500:

```pycon
>>> s = np.random.poisson(lam=(100., 500.), size=(100, 2))
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.power.html.md

# dask.array.random.power

### dask.array.random.power(\*args, \*\*kwargs)

Draws samples in [0, 1] from a power distribution with positive
exponent a - 1.

This docstring was copied from numpy.random.mtrand.RandomState.power.

Some inconsistencies with the Dask version may exist.

Also known as the power function distribution.

#### NOTE
New code should use the ~numpy.random.Generator.power
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **a**
  : Parameter of the distribution. Must be non-negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `a` is a scalar.  Otherwise,
    `np.array(a).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized power distribution.
* **Raises:**
  ValueError
  : If a <= 0.

#### SEE ALSO
`random.Generator.power`
: which should be used for new code.

### Notes

The probability density function is

$$
P(x; a) = ax^{a-1}, 0 \le x \le 1, a>0.

$$

The power function distribution is just the inverse of the Pareto
distribution. It may also be seen as a special case of the Beta
distribution.

It is used, for example, in modeling the over-reporting of insurance
claims.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> a = 5. # shape
>>> samples = 1000
>>> s = np.random.power(a, samples)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, bins=30)
>>> x = np.linspace(0, 1, 100)
>>> y = a*x**(a-1.)
>>> normed_y = samples*np.diff(bins)[0]*y
>>> plt.plot(x, normed_y)
>>> plt.show()
```

Compare the power function distribution to the inverse of the Pareto.

```pycon
>>> from scipy import stats
>>> rvs = np.random.power(5, 1000000)
>>> rvsp = np.random.pareto(5, 1000000)
>>> xx = np.linspace(0,1,100)
>>> powpdf = stats.powerlaw.pdf(xx,5)
```

```pycon
>>> plt.figure()
>>> plt.hist(rvs, bins=50, density=True)
>>> plt.plot(xx,powpdf,'r-')
>>> plt.title('np.random.power(5)')
```

```pycon
>>> plt.figure()
>>> plt.hist(1./(1.+rvsp), bins=50, density=True)
>>> plt.plot(xx,powpdf,'r-')
>>> plt.title('inverse of 1 + np.random.pareto(5)')
```

```pycon
>>> plt.figure()
>>> plt.hist(1./(1.+rvsp), bins=50, density=True)
>>> plt.plot(xx,powpdf,'r-')
>>> plt.title('inverse of stats.pareto(5)')
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.randint.html.md

# dask.array.random.randint

### dask.array.random.randint(\*args, \*\*kwargs)

Return random integers from low (inclusive) to high (exclusive).

This docstring was copied from numpy.random.mtrand.RandomState.randint.

Some inconsistencies with the Dask version may exist.

Return random integers from the “discrete uniform” distribution of
the specified dtype in the “half-open” interval [low, high). If
high is None (the default), then results are from [0, low).

#### NOTE
New code should use the ~numpy.random.Generator.integers
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **low**
  : Lowest (signed) integers to be drawn from the distribution (unless
    `high=None`, in which case this parameter is one above the
    *highest* such integer).

  **high**
  : If provided, one above the largest (signed) integer to be drawn
    from the distribution (see above for behavior if `high=None`).
    If array-like, must contain integer values

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.

  **dtype**
  : Desired dtype of the result. Byteorder must be native.
    The default value is long.
    <br/>
    #### WARNING
    This function defaults to the C-long dtype, which is 32bit on windows
    and otherwise 64bit on 64bit platforms (and 32bit on 32bit ones).
    Since NumPy 2.0, NumPy’s default integer is 32bit on 32bit platforms
    and 64bit on 64bit platforms.  Which corresponds to np.intp.
    (dtype=int is not the same as in most NumPy functions.)
* **Returns:**
  **out**
  : size-shaped array of random integers from the appropriate
    distribution, or a single such random int if size not provided.

#### SEE ALSO
[`random_integers`](dask.array.random.random_integers.md#dask.array.random.random_integers)
: similar to randint, only for the closed interval [low, high], and 1 is the lowest value if high is omitted.

`random.Generator.integers`
: which should be used for new code.

### Examples

```pycon
>>> np.random.randint(2, size=10)
array([1, 0, 0, 0, 1, 1, 0, 0, 1, 0]) # random
>>> np.random.randint(1, size=10)
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
```

Generate a 2 x 4 array of ints between 0 and 4, inclusive:

```pycon
>>> np.random.randint(5, size=(2, 4))
array([[4, 0, 2, 1], # random
       [3, 2, 2, 0]])
```

Generate a 1 x 3 array with 3 different upper bounds

```pycon
>>> np.random.randint(1, [3, 5, 10])
array([2, 2, 9]) # random
```

Generate a 1 by 3 array with 3 different lower bounds

```pycon
>>> np.random.randint([1, 5, 7], 10)
array([9, 8, 7]) # random
```

Generate a 2 by 4 array using broadcasting with dtype of uint8

```pycon
>>> np.random.randint([1, 3, 5, 7], [[10], [20]], dtype=np.uint8)
array([[ 8,  6,  9,  7], # random
       [ 1, 16,  9, 12]], dtype=uint8)
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.random.html.md

# dask.array.random.random

### dask.array.random.random(\*args, \*\*kwargs)

Return random floats in the half-open interval [0.0, 1.0).

This docstring was copied from numpy.random.mtrand.RandomState.random_sample.

Some inconsistencies with the Dask version may exist.

Results are from the “continuous uniform” distribution over the
stated interval.  To sample $Unif[a, b), b > a$ multiply
the output of random_sample by (b-a) and add a:

```default
(b - a) * random_sample() + a
```

#### NOTE
New code should use the ~numpy.random.Generator.random
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : Array of random floats of shape size (unless `size=None`, in which
    case a single float is returned).

#### SEE ALSO
`random.Generator.random`
: which should be used for new code.

### Examples

```pycon
>>> np.random.random_sample()
0.47108547995356098 # random
>>> type(np.random.random_sample())
<class 'float'>
>>> np.random.random_sample((5,))
array([ 0.30220482,  0.86820401,  0.1654503 ,  0.11659149,  0.54323428]) # random
```

Three-by-two array of random numbers from [-5, 0):

```pycon
>>> 5 * np.random.random_sample((3, 2)) - 5
array([[-3.99149989, -0.52338984], # random
       [-2.99091858, -0.79479508],
       [-1.23204345, -1.75224494]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.random_integers.html.md

# dask.array.random.random_integers

### dask.array.random.random_integers(\*args, \*\*kwargs)

Random integers of type numpy.int_ between low and high, inclusive.

This docstring was copied from numpy.random.mtrand.RandomState.random_integers.

Some inconsistencies with the Dask version may exist.

Return random integers of type numpy.int_ from the “discrete uniform”
distribution in the closed interval [low, high].  If high is
None (the default), then results are from [1, low]. The numpy.int_
type translates to the C long integer type and its precision
is platform dependent.

This function has been deprecated. Use randint instead.

#### Deprecated
Deprecated since version 1.11.0.

* **Parameters:**
  **low**
  : Lowest (signed) integer to be drawn from the distribution (unless
    `high=None`, in which case this parameter is the *highest* such
    integer).

  **high**
  : If provided, the largest (signed) integer to be drawn from the
    distribution (see above for behavior if `high=None`).

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : size-shaped array of random integers from the appropriate
    distribution, or a single such random int if size not provided.

#### SEE ALSO
[`randint`](dask.array.random.randint.md#dask.array.random.randint)
: Similar to random_integers, only for the half-open interval [low, high), and 0 is the lowest value if high is omitted.

### Notes

To sample from N evenly spaced floating-point numbers between a and b,
use:

```default
a + (b - a) * (np.random.random_integers(N) - 1) / (N - 1.)
```

### Examples

```pycon
>>> np.random.random_integers(5)
4 # random
>>> type(np.random.random_integers(5))
<class 'numpy.int64'>
>>> np.random.random_integers(5, size=(3,2))
array([[5, 4], # random
       [3, 3],
       [4, 5]])
```

Choose five random numbers from the set of five evenly-spaced
numbers between 0 and 2.5, inclusive (*i.e.*, from the set
${0, 5/8, 10/8, 15/8, 20/8}$):

```pycon
>>> 2.5 * (np.random.random_integers(5, size=(5,)) - 1) / 4.
array([ 0.625,  1.25 ,  0.625,  0.625,  2.5  ]) # random
```

Roll two six sided dice 1000 times and sum the results:

```pycon
>>> d1 = np.random.random_integers(1, 6, 1000)
>>> d2 = np.random.random_integers(1, 6, 1000)
>>> dsums = d1 + d2
```

Display results as a histogram:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(dsums, 11, density=True)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.random_sample.html.md

# dask.array.random.random_sample

### dask.array.random.random_sample(\*args, \*\*kwargs)

Return random floats in the half-open interval [0.0, 1.0).

This docstring was copied from numpy.random.mtrand.RandomState.random_sample.

Some inconsistencies with the Dask version may exist.

Results are from the “continuous uniform” distribution over the
stated interval.  To sample $Unif[a, b), b > a$ multiply
the output of random_sample by (b-a) and add a:

```default
(b - a) * random_sample() + a
```

#### NOTE
New code should use the ~numpy.random.Generator.random
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : Array of random floats of shape size (unless `size=None`, in which
    case a single float is returned).

#### SEE ALSO
`random.Generator.random`
: which should be used for new code.

### Examples

```pycon
>>> np.random.random_sample()
0.47108547995356098 # random
>>> type(np.random.random_sample())
<class 'float'>
>>> np.random.random_sample((5,))
array([ 0.30220482,  0.86820401,  0.1654503 ,  0.11659149,  0.54323428]) # random
```

Three-by-two array of random numbers from [-5, 0):

```pycon
>>> 5 * np.random.random_sample((3, 2)) - 5
array([[-3.99149989, -0.52338984], # random
       [-2.99091858, -0.79479508],
       [-1.23204345, -1.75224494]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.rayleigh.html.md

# dask.array.random.rayleigh

### dask.array.random.rayleigh(\*args, \*\*kwargs)

Draw samples from a Rayleigh distribution.

This docstring was copied from numpy.random.mtrand.RandomState.rayleigh.

Some inconsistencies with the Dask version may exist.

The $\chi$ and Weibull distributions are generalizations of the
Rayleigh.

#### NOTE
New code should use the ~numpy.random.Generator.rayleigh
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **scale**
  : Scale, also equals the mode. Must be non-negative. Default is 1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `scale` is a scalar.  Otherwise,
    `np.array(scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Rayleigh distribution.

#### SEE ALSO
`random.Generator.rayleigh`
: which should be used for new code.

### Notes

The probability density function for the Rayleigh distribution is

$$
P(x;scale) = \frac{x}{scale^2}e^{\frac{-x^2}{2 \cdotp scale^2}}

$$

The Rayleigh distribution would arise, for example, if the East
and North components of the wind velocity had identical zero-mean
Gaussian distributions.  Then the wind speed would have a Rayleigh
distribution.

### References

### Examples

Draw values from the distribution and plot the histogram

```pycon
>>> from matplotlib.pyplot import hist
>>> values = hist(np.random.rayleigh(3, 100000), bins=200, density=True)
```

Wave heights tend to follow a Rayleigh distribution. If the mean wave
height is 1 meter, what fraction of waves are likely to be larger than 3
meters?

```pycon
>>> meanvalue = 1
>>> modevalue = np.sqrt(2 / np.pi) * meanvalue
>>> s = np.random.rayleigh(modevalue, 1000000)
```

The percentage of waves larger than 3 meters is:

```pycon
>>> 100.*sum(s>3)/1000000.
0.087300000000000003 # random
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.standard_cauchy.html.md

# dask.array.random.standard_cauchy

### dask.array.random.standard_cauchy(\*args, \*\*kwargs)

Draw samples from a standard Cauchy distribution with mode = 0.

This docstring was copied from numpy.random.mtrand.RandomState.standard_cauchy.

Some inconsistencies with the Dask version may exist.

Also known as the Lorentz distribution.

#### NOTE
New code should use the
~numpy.random.Generator.standard_cauchy
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **samples**
  : The drawn samples.

#### SEE ALSO
`random.Generator.standard_cauchy`
: which should be used for new code.

### Notes

The probability density function for the full Cauchy distribution is

$$
P(x; x_0, \gamma) = \frac{1}{\pi \gamma \bigl[ 1+
(\frac{x-x_0}{\gamma})^2 \bigr] }

$$

and the Standard Cauchy distribution just sets $x_0=0$ and
$\gamma=1$

The Cauchy distribution arises in the solution to the driven harmonic
oscillator problem, and also describes spectral line broadening. It
also describes the distribution of values at which a line tilted at
a random angle will cut the x axis.

When studying hypothesis tests that assume normality, seeing how the
tests perform on data from a Cauchy distribution is a good indicator of
their sensitivity to a heavy-tailed distribution, since the Cauchy looks
very much like a Gaussian distribution, but with heavier tails.

### References

### Examples

Draw samples and plot the distribution:

```pycon
>>> import matplotlib.pyplot as plt
>>> s = np.random.standard_cauchy(1000000)
>>> s = s[(s>-25) & (s<25)]  # truncate distribution so it plots well
>>> plt.hist(s, bins=100)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.standard_exponential.html.md

# dask.array.random.standard_exponential

### dask.array.random.standard_exponential(\*args, \*\*kwargs)

Draw samples from the standard exponential distribution.

This docstring was copied from numpy.random.mtrand.RandomState.standard_exponential.

Some inconsistencies with the Dask version may exist.

standard_exponential is identical to the exponential distribution
with a scale parameter of 1.

#### NOTE
New code should use the
~numpy.random.Generator.standard_exponential
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : Drawn samples.

#### SEE ALSO
`random.Generator.standard_exponential`
: which should be used for new code.

### Examples

Output a 3x8000 array:

```pycon
>>> n = np.random.standard_exponential((3, 8000))
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.standard_gamma.html.md

# dask.array.random.standard_gamma

### dask.array.random.standard_gamma(\*args, \*\*kwargs)

Draw samples from a standard Gamma distribution.

This docstring was copied from numpy.random.mtrand.RandomState.standard_gamma.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a Gamma distribution with specified parameters,
shape (sometimes designated “k”) and scale=1.

#### NOTE
New code should use the
~numpy.random.Generator.standard_gamma
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **shape**
  : Parameter, must be non-negative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `shape` is a scalar.  Otherwise,
    `np.array(shape).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized standard gamma distribution.

#### SEE ALSO
[`scipy.stats.gamma`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gamma.html#scipy.stats.gamma)
: probability density function, distribution or cumulative density function, etc.

`random.Generator.standard_gamma`
: which should be used for new code.

### Notes

The probability density for the Gamma distribution is

$$
p(x) = x^{k-1}\frac{e^{-x/\theta}}{\theta^k\Gamma(k)},

$$

where $k$ is the shape and $\theta$ the scale,
and $\Gamma$ is the Gamma function.

The Gamma distribution is often used to model the times to failure of
electronic components, and arises naturally in processes for which the
waiting times between Poisson distributed events are relevant.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> shape, scale = 2., 1. # mean and width
>>> s = np.random.standard_gamma(shape, 1000000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> import scipy.special as sps
>>> count, bins, ignored = plt.hist(s, 50, density=True)
>>> y = bins**(shape-1) * ((np.exp(-bins/scale))/
...                       (sps.gamma(shape) * scale**shape))
>>> plt.plot(bins, y, linewidth=2, color='r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.standard_normal.html.md

# dask.array.random.standard_normal

### dask.array.random.standard_normal(\*args, \*\*kwargs)

Draw samples from a standard Normal distribution (mean=0, stdev=1).

This docstring was copied from numpy.random.mtrand.RandomState.standard_normal.

Some inconsistencies with the Dask version may exist.

#### NOTE
New code should use the
~numpy.random.Generator.standard_normal
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  Default is None, in which case a
    single value is returned.
* **Returns:**
  **out**
  : A floating-point array of shape `size` of drawn samples, or a
    single sample if `size` was not specified.

#### SEE ALSO
[`normal`](dask.array.random.normal.md#dask.array.random.normal)
: Equivalent function with additional `loc` and `scale` arguments for setting the mean and standard deviation.

`random.Generator.standard_normal`
: which should be used for new code.

### Notes

For random samples from the normal distribution with mean `mu` and
standard deviation `sigma`, use one of:

```default
mu + sigma * np.random.standard_normal(size=...)
np.random.normal(mu, sigma, size=...)
```

### Examples

```pycon
>>> np.random.standard_normal()
2.1923875335537315 #random
```

```pycon
>>> s = np.random.standard_normal(8000)
>>> s
array([ 0.6888893 ,  0.78096262, -0.89086505, ...,  0.49876311,  # random
       -0.38672696, -0.4685006 ])                                # random
>>> s.shape
(8000,)
>>> s = np.random.standard_normal(size=(3, 4, 2))
>>> s.shape
(3, 4, 2)
```

Two-by-four array of samples from the normal distribution with
mean 3 and standard deviation 2.5:

```pycon
>>> 3 + 2.5 * np.random.standard_normal(size=(2, 4))
array([[-4.49401501,  4.00950034, -1.81814867,  7.29718677],   # random
       [ 0.39924804,  4.68456316,  4.99394529,  4.84057254]])  # random
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.standard_t.html.md

# dask.array.random.standard_t

### dask.array.random.standard_t(\*args, \*\*kwargs)

Draw samples from a standard Student’s t distribution with df degrees
of freedom.

This docstring was copied from numpy.random.mtrand.RandomState.standard_t.

Some inconsistencies with the Dask version may exist.

A special case of the hyperbolic distribution.  As df gets
large, the result resembles that of the standard normal
distribution (standard_normal).

#### NOTE
New code should use the ~numpy.random.Generator.standard_t
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **df**
  : Degrees of freedom, must be > 0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `df` is a scalar.  Otherwise,
    `np.array(df).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized standard Student’s t distribution.

#### SEE ALSO
`random.Generator.standard_t`
: which should be used for new code.

### Notes

The probability density function for the t distribution is

$$
P(x, df) = \frac{\Gamma(\frac{df+1}{2})}{\sqrt{\pi df}
\Gamma(\frac{df}{2})}\Bigl( 1+\frac{x^2}{df} \Bigr)^{-(df+1)/2}

$$

The t test is based on an assumption that the data come from a
Normal distribution. The t test provides a way to test whether
the sample mean (that is the mean calculated from the data) is
a good estimate of the true mean.

The derivation of the t-distribution was first published in
1908 by William Gosset while working for the Guinness Brewery
in Dublin. Due to proprietary issues, he had to publish under
a pseudonym, and so he used the name Student.

### References

### Examples

From Dalgaard page 83 [[1]](#rf4c55e723b8f-1), suppose the daily energy intake for 11
women in kilojoules (kJ) is:

```pycon
>>> intake = np.array([5260., 5470, 5640, 6180, 6390, 6515, 6805, 7515, \
...                    7515, 8230, 8770])
```

Does their energy intake deviate systematically from the recommended
value of 7725 kJ? Our null hypothesis will be the absence of deviation,
and the alternate hypothesis will be the presence of an effect that could be
either positive or negative, hence making our test 2-tailed.

Because we are estimating the mean and we have N=11 values in our sample,
we have N-1=10 degrees of freedom. We set our significance level to 95% and
compute the t statistic using the empirical mean and empirical standard
deviation of our intake. We use a ddof of 1 to base the computation of our
empirical standard deviation on an unbiased estimate of the variance (note:
the final estimate is not unbiased due to the concave nature of the square
root).

```pycon
>>> np.mean(intake)
6753.636363636364
>>> intake.std(ddof=1)
1142.1232221373727
>>> t = (np.mean(intake)-7725)/(intake.std(ddof=1)/np.sqrt(len(intake)))
>>> t
-2.8207540608310198
```

We draw 1000000 samples from Student’s t distribution with the adequate
degrees of freedom.

```pycon
>>> import matplotlib.pyplot as plt
>>> s = np.random.standard_t(10, size=1000000)
>>> h = plt.hist(s, bins=100, density=True)
```

Does our t statistic land in one of the two critical regions found at
both tails of the distribution?

```pycon
>>> np.sum(np.abs(t) < np.abs(s)) / float(len(s))
0.018318  #random < 0.05, statistic is in critical region
```

The probability value for this 2-tailed test is about 1.83%, which is
lower than the 5% pre-determined significance threshold.

Therefore, the probability of observing values as extreme as our intake
conditionally on the null hypothesis being true is too low, and we reject
the null hypothesis of no deviation.

<!-- !! processed by numpydoc !! -->


# dask.array.random.triangular.html.md

# dask.array.random.triangular

### dask.array.random.triangular(\*args, \*\*kwargs)

Draw samples from the triangular distribution over the
interval `[left, right]`.

This docstring was copied from numpy.random.mtrand.RandomState.triangular.

Some inconsistencies with the Dask version may exist.

The triangular distribution is a continuous probability
distribution with lower limit left, peak at mode, and upper
limit right. Unlike the other distributions, these parameters
directly define the shape of the pdf.

#### NOTE
New code should use the ~numpy.random.Generator.triangular
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **left**
  : Lower limit.

  **mode**
  : The value where the peak of the distribution occurs.
    The value must fulfill the condition `left <= mode <= right`.

  **right**
  : Upper limit, must be larger than left.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `left`, `mode`, and `right`
    are all scalars.  Otherwise, `np.broadcast(left, mode, right).size`
    samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized triangular distribution.

#### SEE ALSO
`random.Generator.triangular`
: which should be used for new code.

### Notes

The probability density function for the triangular distribution is

$$
P(x;l, m, r) = \begin{cases}
\frac{2(x-l)}{(r-l)(m-l)}& \text{for $l \leq x \leq m$},\\
\frac{2(r-x)}{(r-l)(r-m)}& \text{for $m \leq x \leq r$},\\
0& \text{otherwise}.
\end{cases}

$$

The triangular distribution is often used in ill-defined
problems where the underlying distribution is not known, but
some knowledge of the limits and mode exists. Often it is used
in simulations.

### References

### Examples

Draw values from the distribution and plot the histogram:

```pycon
>>> import matplotlib.pyplot as plt
>>> h = plt.hist(np.random.triangular(-3, 0, 8, 100000), bins=200,
...              density=True)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.uniform.html.md

# dask.array.random.uniform

### dask.array.random.uniform(\*args, \*\*kwargs)

Draw samples from a uniform distribution.

This docstring was copied from numpy.random.mtrand.RandomState.uniform.

Some inconsistencies with the Dask version may exist.

Samples are uniformly distributed over the half-open interval
`[low, high)` (includes low, but excludes high).  In other words,
any value within the given interval is equally likely to be drawn
by uniform.

#### NOTE
New code should use the ~numpy.random.Generator.uniform
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **low**
  : Lower boundary of the output interval.  All values generated will be
    greater than or equal to low.  The default value is 0.

  **high**
  : Upper boundary of the output interval.  All values generated will be
    less than or equal to high.  The high limit may be included in the
    returned array of floats due to floating-point rounding in the
    equation `low + (high-low) * random_sample()`.  The default value
    is 1.0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `low` and `high` are both scalars.
    Otherwise, `np.broadcast(low, high).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized uniform distribution.

#### SEE ALSO
[`randint`](dask.array.random.randint.md#dask.array.random.randint)
: Discrete uniform distribution, yielding integers.

[`random_integers`](dask.array.random.random_integers.md#dask.array.random.random_integers)
: Discrete uniform distribution over the closed interval `[low, high]`.

[`random_sample`](dask.array.random.random_sample.md#dask.array.random.random_sample)
: Floats uniformly distributed over `[0, 1)`.

[`random`](dask.array.random.random.md#dask.array.random.random)
: Alias for random_sample.

`rand`
: Convenience function that accepts dimensions as input, e.g., `rand(2,2)` would generate a 2-by-2 array of floats, uniformly distributed over `[0, 1)`.

`random.Generator.uniform`
: which should be used for new code.

### Notes

The probability density function of the uniform distribution is

$$
p(x) = \frac{1}{b - a}

$$

anywhere within the interval `[a, b)`, and zero elsewhere.

When `high` == `low`, values of `low` will be returned.
If `high` < `low`, the results are officially undefined
and may eventually raise an error, i.e. do not rely on this
function to behave when passed arguments satisfying that
inequality condition. The `high` limit may be included in the
returned array of floats due to floating-point rounding in the
equation `low + (high-low) * random_sample()`. For example:

```pycon
>>> x = np.float32(5*0.99999999)
>>> x
np.float32(5.0)
```

### Examples

Draw samples from the distribution:

```pycon
>>> s = np.random.uniform(-1,0,1000)
```

All values are within the given interval:

```pycon
>>> np.all(s >= -1)
True
>>> np.all(s < 0)
True
```

Display the histogram of the samples, along with the
probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> count, bins, ignored = plt.hist(s, 15, density=True)
>>> plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.vonmises.html.md

# dask.array.random.vonmises

### dask.array.random.vonmises(\*args, \*\*kwargs)

Draw samples from a von Mises distribution.

This docstring was copied from numpy.random.mtrand.RandomState.vonmises.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a von Mises distribution with specified mode
(mu) and concentration (kappa), on the interval [-pi, pi].

The von Mises distribution (also known as the circular normal
distribution) is a continuous probability distribution on the unit
circle.  It may be thought of as the circular analogue of the normal
distribution.

#### NOTE
New code should use the ~numpy.random.Generator.vonmises
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **mu**
  : Mode (“center”) of the distribution.

  **kappa**
  : Concentration of the distribution, has to be >=0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `mu` and `kappa` are both scalars.
    Otherwise, `np.broadcast(mu, kappa).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized von Mises distribution.

#### SEE ALSO
[`scipy.stats.vonmises`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.vonmises.html#scipy.stats.vonmises)
: probability density function, distribution, or cumulative density function, etc.

`random.Generator.vonmises`
: which should be used for new code.

### Notes

The probability density for the von Mises distribution is

$$
p(x) = \frac{e^{\kappa cos(x-\mu)}}{2\pi I_0(\kappa)},

$$

where $\mu$ is the mode and $\kappa$ the concentration,
and $I_0(\kappa)$ is the modified Bessel function of order 0.

The von Mises is named for Richard Edler von Mises, who was born in
Austria-Hungary, in what is now the Ukraine.  He fled to the United
States in 1939 and became a professor at Harvard.  He worked in
probability theory, aerodynamics, fluid mechanics, and philosophy of
science.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> mu, kappa = 0.0, 4.0 # mean and concentration
>>> s = np.random.vonmises(mu, kappa, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> from scipy.special import i0
>>> plt.hist(s, 50, density=True)
>>> x = np.linspace(-np.pi, np.pi, num=51)
>>> y = np.exp(kappa*np.cos(x-mu))/(2*np.pi*i0(kappa))
>>> plt.plot(x, y, linewidth=2, color='r')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.wald.html.md

# dask.array.random.wald

### dask.array.random.wald(\*args, \*\*kwargs)

Draw samples from a Wald, or inverse Gaussian, distribution.

This docstring was copied from numpy.random.mtrand.RandomState.wald.

Some inconsistencies with the Dask version may exist.

As the scale approaches infinity, the distribution becomes more like a
Gaussian. Some references claim that the Wald is an inverse Gaussian
with mean equal to 1, but this is by no means universal.

The inverse Gaussian distribution was first studied in relationship to
Brownian motion. In 1956 M.C.K. Tweedie used the name inverse Gaussian
because there is an inverse relationship between the time to cover a
unit distance and distance covered in unit time.

#### NOTE
New code should use the ~numpy.random.Generator.wald
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **mean**
  : Distribution mean, must be > 0.

  **scale**
  : Scale parameter, must be > 0.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `mean` and `scale` are both scalars.
    Otherwise, `np.broadcast(mean, scale).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Wald distribution.

#### SEE ALSO
`random.Generator.wald`
: which should be used for new code.

### Notes

The probability density function for the Wald distribution is

$$
P(x;mean,scale) = \sqrt{\frac{scale}{2\pi x^3}}e^
\frac{-scale(x-mean)^2}{2\cdotp mean^2x}

$$

As noted above the inverse Gaussian distribution first arise
from attempts to model Brownian motion. It is also a
competitor to the Weibull for use in reliability modeling and
modeling stock returns and interest rate processes.

### References

### Examples

Draw values from the distribution and plot the histogram:

```pycon
>>> import matplotlib.pyplot as plt
>>> h = plt.hist(np.random.wald(3, 2, 100000), bins=200, density=True)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.weibull.html.md

# dask.array.random.weibull

### dask.array.random.weibull(\*args, \*\*kwargs)

Draw samples from a Weibull distribution.

This docstring was copied from numpy.random.mtrand.RandomState.weibull.

Some inconsistencies with the Dask version may exist.

Draw samples from a 1-parameter Weibull distribution with the given
shape parameter a.

$$
X = (-ln(U))^{1/a}

$$

Here, U is drawn from the uniform distribution over (0,1].

The more common 2-parameter Weibull, including a scale parameter
$\lambda$ is just $X = \lambda(-ln(U))^{1/a}$.

#### NOTE
New code should use the ~numpy.random.Generator.weibull
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **a**
  : Shape parameter of the distribution.  Must be nonnegative.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `a` is a scalar.  Otherwise,
    `np.array(a).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Weibull distribution.

#### SEE ALSO
[`scipy.stats.weibull_max`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.weibull_max.html#scipy.stats.weibull_max)

[`scipy.stats.weibull_min`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.weibull_min.html#scipy.stats.weibull_min)

[`scipy.stats.genextreme`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.genextreme.html#scipy.stats.genextreme)

[`gumbel`](dask.array.random.gumbel.md#dask.array.random.gumbel)

`random.Generator.weibull`
: which should be used for new code.

### Notes

The Weibull (or Type III asymptotic extreme value distribution
for smallest values, SEV Type III, or Rosin-Rammler
distribution) is one of a class of Generalized Extreme Value
(GEV) distributions used in modeling extreme value problems.
This class includes the Gumbel and Frechet distributions.

The probability density for the Weibull distribution is

$$
p(x) = \frac{a}
{\lambda}(\frac{x}{\lambda})^{a-1}e^{-(x/\lambda)^a},

$$

where $a$ is the shape and $\lambda$ the scale.

The function has its peak (the mode) at
$\lambda(\frac{a-1}{a})^{1/a}$.

When `a = 1`, the Weibull distribution reduces to the exponential
distribution.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> a = 5. # shape
>>> s = np.random.weibull(a, 1000)
```

Display the histogram of the samples, along with
the probability density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> x = np.arange(1,100.)/50.
>>> def weib(x,n,a):
...     return (a / n) * (x / n)**(a - 1) * np.exp(-(x / n)**a)
```

```pycon
>>> count, bins, ignored = plt.hist(np.random.weibull(5.,1000))
>>> x = np.arange(1,100.)/50.
>>> scale = count.max()/weib(x, 1., 5.).max()
>>> plt.plot(x, weib(x, 1., 5.)*scale)
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.random.zipf.html.md

# dask.array.random.zipf

### dask.array.random.zipf(\*args, \*\*kwargs)

Draw samples from a Zipf distribution.

This docstring was copied from numpy.random.mtrand.RandomState.zipf.

Some inconsistencies with the Dask version may exist.

Samples are drawn from a Zipf distribution with specified parameter
a > 1.

The Zipf distribution (also known as the zeta distribution) is a
discrete probability distribution that satisfies Zipf’s law: the
frequency of an item is inversely proportional to its rank in a
frequency table.

#### NOTE
New code should use the ~numpy.random.Generator.zipf
method of a ~numpy.random.Generator instance instead;
please see the [Quick start](https://numpy.org/doc/stable/reference/random/index.html#random-quick-start).

* **Parameters:**
  **a**
  : Distribution parameter. Must be greater than 1.

  **size**
  : Output shape.  If the given shape is, e.g., `(m, n, k)`, then
    `m * n * k` samples are drawn.  If size is `None` (default),
    a single value is returned if `a` is a scalar. Otherwise,
    `np.array(a).size` samples are drawn.
* **Returns:**
  **out**
  : Drawn samples from the parameterized Zipf distribution.

#### SEE ALSO
[`scipy.stats.zipf`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.zipf.html#scipy.stats.zipf)
: probability density function, distribution, or cumulative density function, etc.

`random.Generator.zipf`
: which should be used for new code.

### Notes

The probability mass function (PMF) for the Zipf distribution is

$$
p(k) = \frac{k^{-a}}{\zeta(a)},

$$

for integers $k \geq 1$, where $\zeta$ is the Riemann Zeta
function.

It is named for the American linguist George Kingsley Zipf, who noted
that the frequency of any word in a sample of a language is inversely
proportional to its rank in the frequency table.

### References

### Examples

Draw samples from the distribution:

```pycon
>>> a = 4.0
>>> n = 20000
>>> s = np.random.zipf(a, n)
```

Display the histogram of the samples, along with
the expected histogram based on the probability
density function:

```pycon
>>> import matplotlib.pyplot as plt
>>> from scipy.special import zeta
```

bincount provides a fast histogram for small integers.

```pycon
>>> count = np.bincount(s)
>>> k = np.arange(1, s.max() + 1)
```

```pycon
>>> plt.bar(k, count[1:], alpha=0.5, label='sample count')
>>> plt.plot(k, n*(k**-a)/zeta(a), 'k.-', alpha=0.5,
...          label='expected count')
>>> plt.semilogy()
>>> plt.grid(alpha=0.4)
>>> plt.legend()
>>> plt.title(f'Zipf sample, a={a}, size={n}')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.ravel.html.md

# dask.array.ravel

### dask.array.ravel(array_like)

Return a contiguous flattened array.

This docstring was copied from numpy.ravel.

Some inconsistencies with the Dask version may exist.

A 1-D array, containing the elements of the input, is returned.  A copy is
made only if needed.

As of NumPy 1.10, the returned array will have the same type as the input
array. (for example, a masked array will be returned for a masked array
input)

* **Parameters:**
  **a**
  : Input array.  The elements in a are read in the order specified by
    order, and packed as a 1-D array.

  **order**
  : The elements of a are read using this index order. ‘C’ means
    to index the elements in row-major, C-style order,
    with the last axis index changing fastest, back to the first
    axis index changing slowest.  ‘F’ means to index the elements
    in column-major, Fortran-style order, with the
    first index changing fastest, and the last index changing
    slowest. Note that the ‘C’ and ‘F’ options take no account of
    the memory layout of the underlying array, and only refer to
    the order of axis indexing.  ‘A’ means to read the elements in
    Fortran-like index order if a is Fortran *contiguous* in
    memory, C-like order otherwise.  ‘K’ means to read the
    elements in the order they occur in memory, except for
    reversing the data when strides are negative.  By default, ‘C’
    index order is used.
* **Returns:**
  **y**
  : y is a contiguous 1-D array of the same subtype as a,
    with shape `(a.size,)`.
    Note that matrices are special cased for backward compatibility,
    if a is a matrix, then y is a 1-D ndarray.

#### SEE ALSO
`ndarray.flat`
: 1-D iterator over an array.

`ndarray.flatten`
: 1-D array copy of the elements of an array in row-major order.

`ndarray.reshape`
: Change the shape of an array without changing its data.

### Notes

In row-major, C-style order, in two dimensions, the row index
varies the slowest, and the column index the quickest.  This can
be generalized to multiple dimensions, where row-major order
implies that the index along the first axis varies slowest, and
the index along the last quickest.  The opposite holds for
column-major, Fortran-style index ordering.

When a view is desired in as many cases as possible, `arr.reshape(-1)`
may be preferable. However, `ravel` supports `K` in the optional
`order` argument while `reshape` does not.

### Examples

It is equivalent to `reshape(-1, order=order)`.

```pycon
>>> import numpy as np
>>> x = np.array([[1, 2, 3], [4, 5, 6]])
>>> np.ravel(x)
array([1, 2, 3, 4, 5, 6])
```

```pycon
>>> x.reshape(-1)
array([1, 2, 3, 4, 5, 6])
```

```pycon
>>> np.ravel(x, order='F')
array([1, 4, 2, 5, 3, 6])
```

When `order` is ‘A’, it will preserve the array’s ‘C’ or ‘F’ ordering:

```pycon
>>> np.ravel(x.T)
array([1, 4, 2, 5, 3, 6])
>>> np.ravel(x.T, order='A')
array([1, 2, 3, 4, 5, 6])
```

When `order` is ‘K’, it will preserve orderings that are neither ‘C’
nor ‘F’, but won’t reverse axes:

```pycon
>>> a = np.arange(3)[::-1]; a
array([2, 1, 0])
>>> a.ravel(order='C')
array([2, 1, 0])
>>> a.ravel(order='K')
array([2, 1, 0])
```

```pycon
>>> a = np.arange(12).reshape(2,3,2).swapaxes(1,2); a
array([[[ 0,  2,  4],
        [ 1,  3,  5]],
       [[ 6,  8, 10],
        [ 7,  9, 11]]])
>>> a.ravel(order='C')
array([ 0,  2,  4,  1,  3,  5,  6,  8, 10,  7,  9, 11])
>>> a.ravel(order='K')
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
```

<!-- !! processed by numpydoc !! -->


# dask.array.ravel_multi_index.html.md

# dask.array.ravel_multi_index

### dask.array.ravel_multi_index(multi_index, dims, mode='raise', order='C')

Converts a tuple of index arrays into an array of flat
indices, applying boundary modes to the multi-index.

* **Parameters:**
  **multi_index**
  : A tuple of integer arrays, one array for each dimension.

  **dims**
  : The shape of array into which the indices from `multi_index` apply.

  **mode**
  : Specifies how out-of-bounds indices are handled.  Can specify
    either one mode or a tuple of modes, one mode per index.
    * ‘raise’ – raise an error (default)
    * ‘wrap’ – wrap around
    * ‘clip’ – clip to the range
    <br/>
    In ‘clip’ mode, a negative index which would normally
    wrap will clip to 0 instead.

  **order**
  : Determines whether the multi-index should be viewed as
    indexing in row-major (C-style) or column-major
    (Fortran-style) order.
* **Returns:**
  **raveled_indices**
  : An array of indices into the flattened version of an array
    of dimensions `dims`.

#### SEE ALSO
[`unravel_index`](dask.array.unravel_index.md#dask.array.unravel_index)

### Examples

```pycon
>>> import numpy as np
>>> arr = np.array([[3,6,6],[4,5,1]])
>>> np.ravel_multi_index(arr, (7,6))
array([22, 41, 37])
>>> np.ravel_multi_index(arr, (7,6), order='F')
array([31, 41, 13])
>>> np.ravel_multi_index(arr, (4,6), mode='clip')
array([22, 23, 19])
>>> np.ravel_multi_index(arr, (4,4), mode=('clip','wrap'))
array([12, 13, 13])
```

```pycon
>>> np.ravel_multi_index((3,1,4,1), (6,7,8,9))
1621
```

<!-- !! processed by numpydoc !! -->


# dask.array.real.html.md

# dask.array.real

### dask.array.real(\*args, \*\*kwargs)

Return the real part of the complex argument.

This docstring was copied from numpy.real.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **val**
  : Input array.
* **Returns:**
  **out**
  : The real component of the complex argument. If val is real, the type
    of val is used for the output.  If val has complex elements, the
    returned type is float.

#### SEE ALSO
`real_if_close`, [`imag`](dask.array.imag.md#dask.array.imag), [`angle`](dask.array.angle.md#dask.array.angle)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1+2j, 3+4j, 5+6j])
>>> a.real
array([1.,  3.,  5.])
>>> a.real = 9
>>> a
array([9.+2.j,  9.+4.j,  9.+6.j])
>>> a.real = np.array([9, 8, 7])
>>> a
array([9.+2.j,  8.+4.j,  7.+6.j])
>>> np.real(1 + 1j)
1.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.rechunk.html.md

# dask.array.rechunk

### dask.array.rechunk(x, chunks='auto', threshold=None, block_size_limit=None, balance=False, method=None)

Convert blocks in dask array x for new chunks.

* **Parameters:**
  **x: dask array**
  : Array to be rechunked.

  **chunks:  int, tuple, dict or str, optional**
  : The new block dimensions to create. -1 indicates the full size of the
    corresponding dimension. Default is “auto” which automatically
    determines chunk sizes.

  **threshold: int, optional**
  : The graph growth factor under which we don’t bother introducing an
    intermediate step.

  **block_size_limit: int, optional**
  : The maximum block size (in bytes) we want to produce
    Defaults to the configuration value `array.chunk-size`

  **balance**
  : If True, try to make each chunk to be the same size.
    <br/>
    This means `balance=True` will remove any small leftover chunks, so
    using `x.rechunk(chunks=len(x) // N, balance=True)`
    will almost certainly result in `N` chunks.

  **method: {‘tasks’, ‘p2p’}, optional.**
  : Rechunking method to use.

### Examples

```pycon
>>> import dask.array as da
>>> x = da.ones((1000, 1000), chunks=(100, 100))
```

Specify uniform chunk sizes with a tuple

```pycon
>>> y = x.rechunk((1000, 10))
```

Or chunk only specific dimensions with a dictionary

```pycon
>>> y = x.rechunk({0: 1000})
```

Use the value `-1` to specify that you want a single chunk along a
dimension or the value `"auto"` to specify that dask can freely rechunk a
dimension to attain blocks of a uniform block size

```pycon
>>> y = x.rechunk({0: -1, 1: 'auto'}, block_size_limit=1e8)
```

If a chunk size does not divide the dimension then rechunk will leave any
unevenness to the last chunk.

```pycon
>>> x.rechunk(chunks=(400, -1)).chunks
((400, 400, 200), (1000,))
```

However if you want more balanced chunks, and don’t mind Dask choosing a
different chunksize for you then you can use the `balance=True` option.

```pycon
>>> x.rechunk(chunks=(400, -1), balance=True).chunks
((500, 500), (1000,))
```

<!-- !! processed by numpydoc !! -->


# dask.array.reciprocal.html.md

# dask.array.reciprocal

### dask.array.reciprocal(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'reciprocal'>*

This docstring was copied from numpy.reciprocal.

Some inconsistencies with the Dask version may exist.

Return the reciprocal of the argument, element-wise.

Calculates `1/x`.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : Return array.
    This is a scalar if x is a scalar.

### Notes

#### NOTE
This function is not designed to work with integers.

For integer arguments with absolute value larger than 1 the result is
always zero because of the way Python handles integer division.  For
integer zero the result is an overflow.

### Examples

```pycon
>>> import numpy as np
>>> np.reciprocal(2.)
0.5
>>> np.reciprocal([1, 2., 3.33])
array([ 1.       ,  0.5      ,  0.3003003])
```

<!-- !! processed by numpydoc !! -->


# dask.array.reduction.html.md

# dask.array.reduction

### dask.array.reduction(x, chunk, aggregate, axis=None, keepdims=False, dtype=None, split_every=None, combine=None, name=None, out=None, concatenate=True, output_size=1, meta=None, weights=None)

General version of reductions

* **Parameters:**
  **x: Array**
  : Data being reduced along one or more axes

  **chunk: callable(x_chunk, [weights_chunk=None], axis, keepdims)**
  : First function to be executed when resolving the dask graph.
    This function is applied in parallel to all original chunks of x.
    See below for function parameters.

  **combine: callable(x_chunk, axis, keepdims), optional**
  : Function used for intermediate recursive aggregation (see
    split_every below). If omitted, it defaults to aggregate.
    If the reduction can be performed in less than 3 steps, it will not
    be invoked at all.

  **aggregate: callable(x_chunk, axis, keepdims)**
  : Last function to be executed when resolving the dask graph,
    producing the final output. It is always invoked, even when the reduced
    Array counts a single chunk along the reduced axes.

  **axis: int or sequence of ints, optional**
  : Axis or axes to aggregate upon. If omitted, aggregate along all axes.

  **keepdims: boolean, optional**
  : Whether the reduction function should preserve the reduced axes,
    leaving them at size `output_size`, or remove them.

  **dtype: np.dtype**
  : data type of output. This argument was previously optional, but
    leaving as `None` will now raise an exception.

  **split_every: int >= 2 or dict(axis: int), optional**
  : Determines the depth of the recursive aggregation. If set to or more
    than the number of input chunks, the aggregation will be performed in
    two steps, one `chunk` function per input chunk and a single
    `aggregate` function at the end. If set to less than that, an
    intermediate `combine` function will be used, so that any one
    `combine` or `aggregate` function has no more than `split_every`
    inputs. The depth of the aggregation graph will be
    $\log_\text{split_every}(\text{input chunks along reduced axes})$.
    Setting to a low value can reduce cache size and network transfers, at
    the cost of more CPU and a larger dask graph.
    <br/>
    Omit to let dask heuristically decide a good default. A default can
    also be set globally with the `split_every` key in
    `dask.config`.

  **name: str, optional**
  : Prefix of the keys of the intermediate and output nodes. If omitted it
    defaults to the function names.

  **out: Array, optional**
  : Another dask array whose contents will be replaced. Omit to create a
    new one. Note that, unlike in numpy, this setting gives no performance
    benefits whatsoever, but can still be useful  if one needs to preserve
    the references to a previously existing Array.

  **concatenate: bool, optional**
  : If True (the default), the outputs of the `chunk`/`combine`
    functions are concatenated into a single np.array before being passed
    to the `combine`/`aggregate` functions. If False, the input of
    `combine` and `aggregate` will be either a list of the raw outputs
    of the previous step or a single output, and the function will have to
    concatenate it itself. It can be useful to set this to False if the
    chunk and/or combine steps do not produce np.arrays.

  **output_size: int >= 1, optional**
  : Size of the output of the `aggregate` function along the reduced
    axes. Ignored if keepdims is False.

  **weights**
  : Weights to be used in the reduction of x. Will be
    automatically broadcast to the shape of x, and so must have
    a compatible shape. For instance, if x has shape `(3, 4)`
    then acceptable shapes for weights are `(3, 4)`, `(4,)`,
    `(3, 1)`, `(1, 1)`, `(1)`, and `()`.
* **Returns:**
  dask array

  **Function Parameters**

  x_chunk: numpy.ndarray
  : Individual input chunk. For `chunk` functions, it is one of the
    original chunks of x. For `combine` and `aggregate` functions, it’s
    the concatenation of the outputs produced by the previous `chunk` or
    `combine` functions. If concatenate=False, it’s a list of the raw
    outputs from the previous functions.

  weights_chunk: numpy.ndarray, optional
  : Only applicable to the `chunk` function. Weights, with the
    same shape as x_chunk, to be applied during the reduction of
    the individual input chunk. If `weights` have not been
    provided then the function may omit this parameter. When
    weights_chunk is included then it must occur immediately
    after the x_chunk parameter, and must also have a default
    value for cases when `weights` are not provided.

  axis: tuple
  : Normalized list of axes to reduce upon, e.g. `(0, )`
    Scalar, negative, and None axes have been normalized away.
    Note that some numpy reduction functions cannot reduce along multiple
    axes at once and strictly require an int in input. Such functions have
    to be wrapped to cope.

  keepdims: bool
  : Whether the reduction function should preserve the reduced axes or
    remove them.

<!-- !! processed by numpydoc !! -->


# dask.array.register_chunk_type.html.md

# dask.array.register_chunk_type

### dask.array.register_chunk_type(type)

Register the given type as a valid chunk and downcast array type

* **Parameters:**
  **type**
  : Duck array type to be registered as a type Dask can safely wrap as a chunk and
    to which Dask does not defer in arithmetic operations and NumPy
    functions/ufuncs.

### Notes

A [`dask.array.Array`](dask.array.Array.md#dask.array.Array) can contain any sufficiently “NumPy-like” array in
its chunks. These are also referred to as “duck arrays” since they match the most
important parts of NumPy’s array API, and so, behave the same way when relying on
duck typing.

However, for multiple duck array types to interoperate properly, they need to
properly defer to each other in arithmetic operations and NumPy functions/ufuncs
according to a well-defined type casting hierarchy (
[see NEP 13](https://numpy.org/neps/nep-0013-ufunc-overrides.html#type-casting-hierarchy)
). In an effort to maintain this hierarchy, Dask defers to all other duck array
types except those in its internal registry. By default, this registry contains

* [`numpy.ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)
* [`numpy.ma.MaskedArray`](https://numpy.org/doc/stable/reference/maskedarray.baseclass.html#numpy.ma.MaskedArray)
* `cupy.ndarray`
* `sparse.SparseArray`
* [`scipy.sparse.spmatrix`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.spmatrix.html#scipy.sparse.spmatrix)

This function exists to append any other types to this registry. If a type is not
in this registry, and yet is a downcast type (it comes below
[`dask.array.Array`](dask.array.Array.md#dask.array.Array) in the type casting hierarchy), a `TypeError` will
be raised due to all operand types returning `NotImplemented`.

### Examples

Using a mock `FlaggedArray` class as an example chunk type unknown to Dask with
minimal duck array API:

```pycon
>>> import numpy.lib.mixins
>>> class FlaggedArray(numpy.lib.mixins.NDArrayOperatorsMixin):
...     def __init__(self, a, flag=False):
...         self.a = a
...         self.flag = flag
...     def __repr__(self):
...         return f"Flag: {self.flag}, Array: {self.a!r}"
...     def __array__(self):
...         return np.asarray(self.a)
...     def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
...         if method == '__call__':
...             downcast_inputs = []
...             flag = False
...             for input in inputs:
...                 if isinstance(input, self.__class__):
...                     flag = flag or input.flag
...                     downcast_inputs.append(input.a)
...                 elif isinstance(input, np.ndarray):
...                     downcast_inputs.append(input)
...                 else:
...                     return NotImplemented
...             return self.__class__(ufunc(*downcast_inputs, **kwargs), flag)
...         else:
...             return NotImplemented
...     @property
...     def shape(self):
...         return self.a.shape
...     @property
...     def ndim(self):
...         return self.a.ndim
...     @property
...     def dtype(self):
...         return self.a.dtype
...     def __getitem__(self, key):
...         return type(self)(self.a[key], self.flag)
...     def __setitem__(self, key, value):
...         self.a[key] = value
```

Before registering `FlaggedArray`, both types will attempt to defer to the
other:

```pycon
>>> import dask.array as da
>>> da.ones(5) - FlaggedArray(np.ones(5), True)
Traceback (most recent call last):
...
TypeError: operand type(s) all returned NotImplemented ...
```

However, once registered, Dask will be able to handle operations with this new
type:

```pycon
>>> da.register_chunk_type(FlaggedArray)
>>> x = da.ones(5) - FlaggedArray(np.ones(5), True)
>>> x
dask.array<sub, shape=(5,), dtype=float64, chunksize=(5,), chunktype=dask.FlaggedArray>
>>> x.compute()
Flag: True, Array: array([0., 0., 0., 0., 0.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.remainder.html.md

# dask.array.remainder

### dask.array.remainder(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'remainder'>*

This docstring was copied from numpy.remainder.

Some inconsistencies with the Dask version may exist.

Returns the element-wise remainder of division.

Computes the remainder complementary to the floor_divide function.  It is
equivalent to the Python modulus operator `x1 % x2` and has the same sign
as the divisor x2. The MATLAB function equivalent to `np.remainder`
is `mod`.

#### WARNING
This should not be confused with:

* Python’s math.remainder and C’s `remainder`, which
  compute the IEEE remainder, which are the complement to
  `round(x1 / x2)`.
* The MATLAB `rem` function and or the C `%` operator which is the
  complement to `int(x1 / x2)`.

* **Parameters:**
  **x1**
  : Dividend array.

  **x2**
  : Divisor array.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The element-wise remainder of the quotient `floor_divide(x1, x2)`.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`floor_divide`](dask.array.floor_divide.md#dask.array.floor_divide)
: Equivalent of Python `//` operator.

[`divmod`](dask.array.divmod.md#dask.array.divmod)
: Simultaneous floor division and remainder.

[`fmod`](dask.array.fmod.md#dask.array.fmod)
: Equivalent of the MATLAB `rem` function.

[`divide`](dask.array.divide.md#dask.array.divide), [`floor`](dask.array.floor.md#dask.array.floor)

### Notes

Returns 0 when x2 is 0 and both x1 and x2 are (arrays of)
integers.
`mod` is an alias of `remainder`.

### Examples

```pycon
>>> import numpy as np
>>> np.remainder([4, 7], [2, 3])
array([0, 1])
>>> np.remainder(np.arange(7), 5)
array([0, 1, 2, 3, 4, 0, 1])
```

The `%` operator can be used as a shorthand for `np.remainder` on
ndarrays.

```pycon
>>> x1 = np.arange(7)
>>> x1 % 5
array([0, 1, 2, 3, 4, 0, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.repeat.html.md

# dask.array.repeat

### dask.array.repeat(a, repeats, axis=None)

Repeat each element of an array after themselves

This docstring was copied from numpy.repeat.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array.

  **repeats**
  : The number of repetitions for each element.  repeats is broadcasted
    to fit the shape of the given axis.

  **axis**
  : The axis along which to repeat values.  By default, use the
    flattened input array, and return a flat output array.
* **Returns:**
  **repeated_array**
  : Output array which has the same shape as a, except along
    the given axis.

#### SEE ALSO
[`tile`](dask.array.tile.md#dask.array.tile)
: Tile an array.

[`unique`](dask.array.unique.md#dask.array.unique)
: Find the unique elements of an array.

### Examples

```pycon
>>> import numpy as np
>>> np.repeat(3, 4)
array([3, 3, 3, 3])
>>> x = np.array([[1,2],[3,4]])
>>> np.repeat(x, 2)
array([1, 1, 2, 2, 3, 3, 4, 4])
>>> np.repeat(x, 3, axis=1)
array([[1, 1, 1, 2, 2, 2],
       [3, 3, 3, 4, 4, 4]])
>>> np.repeat(x, [1, 2], axis=0)
array([[1, 2],
       [3, 4],
       [3, 4]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.reshape.html.md

# dask.array.reshape

### dask.array.reshape(x, shape, merge_chunks=True, limit=None)

Reshape array to new shape

* **Parameters:**
  **shape**
  : The new shape should be compatible with the original shape. If
    an integer, then the result will be a 1-D array of that length.
    One shape dimension can be -1. In this case, the value is
    inferred from the length of the array and remaining dimensions.

  **merge_chunks**
  : Whether to merge chunks using the logic in [`dask.array.rechunk()`](dask.array.rechunk.md#dask.array.rechunk)
    when communication is necessary given the input array chunking and
    the output shape. With `merge_chunks==False`, the input array will
    be rechunked to a chunksize of 1, which can create very many tasks.

  **limit: int (optional)**
  : The maximum block size to target in bytes. If no limit is provided,
    it defaults to using the `array.chunk-size` Dask config value.

#### SEE ALSO
[`dask.array.rechunk`](dask.array.rechunk.md#dask.array.rechunk)

[`numpy.reshape`](https://numpy.org/doc/stable/reference/generated/numpy.reshape.html#numpy.reshape)

### Notes

This is a parallelized version of the `np.reshape` function with the
following limitations:

1. It assumes that the array is stored in [row-major order](https://en.wikipedia.org/wiki/Row-_and_column-major_order)
2. It only allows for reshapings that collapse or merge dimensions like
   `(1, 2, 3, 4) -> (1, 6, 4)` or `(64,) -> (4, 4, 4)`

When communication is necessary this algorithm depends on the logic within
rechunk.  It endeavors to keep chunk sizes roughly the same when possible.

See [Reshaping](../array-chunks.md#array-chunks-reshaping) for a discussion the tradeoffs of
`merge_chunks`.

<!-- !! processed by numpydoc !! -->


# dask.array.reshape_blockwise.html.md

# dask.array.reshape_blockwise

### dask.array.reshape_blockwise(x: [Array](dask.array.Array.md#dask.array.Array), shape: [int](https://docs.python.org/3/library/functions.html#int) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[int](https://docs.python.org/3/library/functions.html#int), ...], chunks: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[int](https://docs.python.org/3/library/functions.html#int), ...], ...] | [None](https://docs.python.org/3/library/constants.html#None) = None) → [Array](dask.array.Array.md#dask.array.Array)

Blockwise-reshape into a new shape.

The regular reshape operation in Dask preserves C-ordering in the array
which requires a rechunking for most reshaping operations, making the
computation relatively expensive.

Blockwise-reshape reshapes every block into the new shape and concatenates
the results. This is a trivial blockwise computation but will return the
result in a different order than NumPy. This is a good solution for
subsequent operations that don’t rely on the order.

* **Parameters:**
  **x: Array**
  : The input array to reshape.

  **shape**
  : The new shape should be compatible with the original shape. If
    an integer, then the result will be a 1-D array of that length.
    One shape dimension can be -1. In this case, the value is
    inferred from the length of the array and remaining dimensions.

  **chunks: tuple of ints, default None**
  : The chunk sizes for every chunk in the output array. Dask will expand
    the chunks per dimension into the cross product of chunks for every
    chunk in the array.
    <br/>
    An error is raised if chunks is given and the number of dimensions
    decreases.
    <br/>
    #### NOTE
    This information is required if the number of dimensions is increased.
    Dask cannot infer the output chunks in this case. The keyword is ignored
    if the number of dimensions is reduced.

### Notes

This is a parallelized version of the `np.reshape` function with the
following limitations:

1. It does not return elements in the same order as NumPy would
2. It only allows for reshapings that collapse like
   `(1, 2, 3, 4) -> (1, 6, 4)`

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> x = da.from_array(np.arange(0, 27).reshape(3, 3, 3), chunks=(3, 2, (2, 1)))
>>> result = reshape_blockwise(x, (3, 9))
>>> result.chunks
((3,), (4, 2, 2, 1))
```

The resulting chunks are calculated automatically to match the new shape.

```pycon
>>> result.compute()
array([[ 0,  1,  3,  4,  2,  5,  6,  7,  8],
       [ 9, 10, 12, 13, 11, 14, 15, 16, 17],
       [18, 19, 21, 22, 20, 23, 24, 25, 26]])
```

```pycon
>>> result = reshape_blockwise(result, (3, 3, 3), chunks=x.chunks)
>>> result.chunks
((3,), (2, 1), (2, 1))
```

The resulting chunks are taken from the input. Chaining the reshape operation
together like this reverts the previous reshaping operation that reduces the
number of dimensions.

```pycon
>>> result.compute()
array([[[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8]],

       [[ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17]],

       [[18, 19, 20],
        [21, 22, 23],
        [24, 25, 26]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.result_type.html.md

# dask.array.result_type

### dask.array.result_type(\*arrays_and_dtypes)

This docstring was copied from numpy.result_type.

Some inconsistencies with the Dask version may exist.

Returns the type that results from applying the NumPy
[type promotion](https://numpy.org/doc/stable/reference/arrays.promotion.html#arrays-promotion) rules to the arguments.

* **Parameters:**
  **arrays_and_dtypes**
  : The operands of some operation whose result type is needed.
* **Returns:**
  **out**
  : The result type.

#### SEE ALSO
`dtype`, `promote_types`, `min_scalar_type`, `can_cast`

### Examples

```pycon
>>> import numpy as np
>>> np.result_type(3, np.arange(7, dtype=np.int8))
dtype('int8')
```

```pycon
>>> np.result_type(np.int32, np.complex64)
dtype('complex128')
```

```pycon
>>> np.result_type(3.0, -2)
dtype('float64')
```

<!-- !! processed by numpydoc !! -->


# dask.array.right_shift.html.md

# dask.array.right_shift

### dask.array.right_shift(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'right_shift'>*

This docstring was copied from numpy.right_shift.

Some inconsistencies with the Dask version may exist.

Shift the bits of an integer to the right.

Bits are shifted to the right x2.  Because the internal
representation of numbers is in binary format, this operation is
equivalent to dividing x1 by `2**x2`.

* **Parameters:**
  **x1**
  : Input values.

  **x2**
  : Number of bits to remove at the right of x1.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Return x1 with bits shifted x2 times to the right.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
[`left_shift`](dask.array.left_shift.md#dask.array.left_shift)
: Shift the bits of an integer to the left.

`binary_repr`
: Return the binary representation of the input number as a string.

### Examples

```pycon
>>> import numpy as np
>>> np.binary_repr(10)
'1010'
>>> np.right_shift(10, 1)
5
>>> np.binary_repr(5)
'101'
```

```pycon
>>> np.right_shift(10, [1,2,3])
array([5, 2, 1])
```

The `>>` operator can be used as a shorthand for `np.right_shift` on
ndarrays.

```pycon
>>> x1 = 10
>>> x2 = np.array([1,2,3])
>>> x1 >> x2
array([5, 2, 1])
```

<!-- !! processed by numpydoc !! -->


# dask.array.rint.html.md

# dask.array.rint

### dask.array.rint(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'rint'>*

This docstring was copied from numpy.rint.

Some inconsistencies with the Dask version may exist.

Round elements of the array to the nearest integer.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Output array is same shape and type as x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`fix`](dask.array.fix.md#dask.array.fix), [`ceil`](dask.array.ceil.md#dask.array.ceil), [`floor`](dask.array.floor.md#dask.array.floor), [`trunc`](dask.array.trunc.md#dask.array.trunc)

### Notes

For values exactly halfway between rounded decimal values, NumPy
rounds to the nearest even value. Thus 1.5 and 2.5 round to 2.0,
-0.5 and 0.5 round to 0.0, etc.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([-1.7, -1.5, -0.2, 0.2, 1.5, 1.7, 2.0])
>>> np.rint(a)
array([-2., -2., -0.,  0.,  2.,  2.,  2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.roll.html.md

# dask.array.roll

### dask.array.roll(array, shift, axis=None)

Roll array elements along a given axis.

This docstring was copied from numpy.roll.

Some inconsistencies with the Dask version may exist.

Elements that roll beyond the last position are re-introduced at
the first.

* **Parameters:**
  **a**
  : Input array.

  **shift**
  : The number of places by which elements are shifted.  If a tuple,
    then axis must be a tuple of the same size, and each of the
    given axes is shifted by the corresponding number.  If an int
    while axis is a tuple of ints, then the same value is used for
    all given axes.

  **axis**
  : Axis or axes along which elements are shifted.  By default, the
    array is flattened before shifting, after which the original
    shape is restored.
* **Returns:**
  **res**
  : Output array, with the same shape as a.

#### SEE ALSO
[`rollaxis`](dask.array.rollaxis.md#dask.array.rollaxis)
: Roll the specified axis backwards, until it lies in a given position.

### Notes

Supports rolling over multiple dimensions simultaneously.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(10)
>>> np.roll(x, 2)
array([8, 9, 0, 1, 2, 3, 4, 5, 6, 7])
>>> np.roll(x, -2)
array([2, 3, 4, 5, 6, 7, 8, 9, 0, 1])
```

```pycon
>>> x2 = np.reshape(x, (2, 5))
>>> x2
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
>>> np.roll(x2, 1)
array([[9, 0, 1, 2, 3],
       [4, 5, 6, 7, 8]])
>>> np.roll(x2, -1)
array([[1, 2, 3, 4, 5],
       [6, 7, 8, 9, 0]])
>>> np.roll(x2, 1, axis=0)
array([[5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4]])
>>> np.roll(x2, -1, axis=0)
array([[5, 6, 7, 8, 9],
       [0, 1, 2, 3, 4]])
>>> np.roll(x2, 1, axis=1)
array([[4, 0, 1, 2, 3],
       [9, 5, 6, 7, 8]])
>>> np.roll(x2, -1, axis=1)
array([[1, 2, 3, 4, 0],
       [6, 7, 8, 9, 5]])
>>> np.roll(x2, (1, 1), axis=(1, 0))
array([[9, 5, 6, 7, 8],
       [4, 0, 1, 2, 3]])
>>> np.roll(x2, (2, 1), axis=(1, 0))
array([[8, 9, 5, 6, 7],
       [3, 4, 0, 1, 2]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.rollaxis.html.md

# dask.array.rollaxis

### dask.array.rollaxis(a, axis, start=0)

<!-- !! processed by numpydoc !! -->


# dask.array.rot90.html.md

# dask.array.rot90

### dask.array.rot90(m, k=1, axes=(0, 1))

Rotate an array by 90 degrees in the plane specified by axes.

This docstring was copied from numpy.rot90.

Some inconsistencies with the Dask version may exist.

Rotation direction is from the first towards the second axis.
This means for a 2D array with the default k and axes, the
rotation will be counterclockwise.

* **Parameters:**
  **m**
  : Array of two or more dimensions.

  **k**
  : Number of times the array is rotated by 90 degrees.

  **axes**
  : The array is rotated in the plane defined by the axes.
    Axes must be different.
* **Returns:**
  **y**
  : A rotated view of m.

#### SEE ALSO
[`flip`](dask.array.flip.md#dask.array.flip)
: Reverse the order of elements in an array along the given axis.

[`fliplr`](dask.array.fliplr.md#dask.array.fliplr)
: Flip an array horizontally.

[`flipud`](dask.array.flipud.md#dask.array.flipud)
: Flip an array vertically.

### Notes

`rot90(m, k=1, axes=(1,0))`  is the reverse of
`rot90(m, k=1, axes=(0,1))`

`rot90(m, k=1, axes=(1,0))` is equivalent to
`rot90(m, k=-1, axes=(0,1))`

### Examples

```pycon
>>> import numpy as np
>>> m = np.array([[1,2],[3,4]], int)
>>> m
array([[1, 2],
       [3, 4]])
>>> np.rot90(m)
array([[2, 4],
       [1, 3]])
>>> np.rot90(m, 2)
array([[4, 3],
       [2, 1]])
>>> m = np.arange(8).reshape((2,2,2))
>>> np.rot90(m, 1, (1,2))
array([[[1, 3],
        [0, 2]],
       [[5, 7],
        [4, 6]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.round.html.md

# dask.array.round

### dask.array.round(a, decimals=0)

Evenly round to the given number of decimals.

This docstring was copied from numpy.round.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **decimals**
  : Number of decimal places to round to (default: 0).  If
    decimals is negative, it specifies the number of positions to
    the left of the decimal point.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape as the expected output, but the type of the output
    values will be cast if necessary. See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)
    for more details.
* **Returns:**
  **rounded_array**
  : An array of the same type as a, containing the rounded values.
    Unless out was specified, a new array is created.  A reference to
    the result is returned.
    <br/>
    The real and imaginary parts of complex numbers are rounded
    separately.  The result of rounding a float is a float.

#### SEE ALSO
`ndarray.round`
: equivalent method

[`around`](dask.array.around.md#dask.array.around)
: an alias for this function

[`ceil`](dask.array.ceil.md#dask.array.ceil), [`fix`](dask.array.fix.md#dask.array.fix), [`floor`](dask.array.floor.md#dask.array.floor), [`rint`](dask.array.rint.md#dask.array.rint), [`trunc`](dask.array.trunc.md#dask.array.trunc)

### Notes

For values exactly halfway between rounded decimal values, NumPy
rounds to the nearest even value. Thus 1.5 and 2.5 round to 2.0,
-0.5 and 0.5 round to 0.0, etc.

`np.round` uses a fast but sometimes inexact algorithm to round
floating-point datatypes. For positive decimals it is equivalent to
`np.true_divide(np.rint(a * 10**decimals), 10**decimals)`, which has
error due to the inexact representation of decimal fractions in the IEEE
floating point standard [[1]](#r0851155ae62b-1) and errors introduced when scaling by powers
of ten. For instance, note the extra “1” in the following:

```pycon
>>> np.round(56294995342131.5, 3)
56294995342131.51
```

If your goal is to print such values with a fixed number of decimals, it is
preferable to use numpy’s float printing routines to limit the number of
printed decimals:

```pycon
>>> np.format_float_positional(56294995342131.5, precision=3)
'56294995342131.5'
```

The float printing routines use an accurate but much more computationally
demanding algorithm to compute the number of digits after the decimal
point.

Alternatively, Python’s builtin round function uses a more accurate
but slower algorithm for 64-bit floating point values:

```pycon
>>> round(56294995342131.5, 3)
56294995342131.5
>>> np.round(16.055, 2), round(16.055, 2)  # equals 16.0549999999999997
(16.06, 16.05)
```

### References

### Examples

```pycon
>>> import numpy as np
>>> np.round([0.37, 1.64])
array([0., 2.])
>>> np.round([0.37, 1.64], decimals=1)
array([0.4, 1.6])
>>> np.round([.5, 1.5, 2.5, 3.5, 4.5]) # rounds to nearest even value
array([0., 2., 2., 4., 4.])
>>> np.round([1,2,3,11], decimals=1) # ndarray of ints is returned
array([ 1,  2,  3, 11])
>>> np.round([1,2,3,11], decimals=-1)
array([ 0,  0,  0, 10])
```

<!-- !! processed by numpydoc !! -->


# dask.array.searchsorted.html.md

# dask.array.searchsorted

### dask.array.searchsorted(a, v, side='left', sorter=None)

Find indices where elements should be inserted to maintain order.

This docstring was copied from numpy.searchsorted.

Some inconsistencies with the Dask version may exist.

Find the indices into a sorted array a such that, if the
corresponding elements in v were inserted before the indices, the
order of a would be preserved.

Assuming that a is sorted:

| side   | returned index i satisfies   |
|--------|------------------------------|
| left   | `a[i-1] < v <= a[i]`         |
| right  | `a[i-1] <= v < a[i]`         |
* **Parameters:**
  **a**
  : Input array. If sorter is None, then it must be sorted in
    ascending order, otherwise sorter must be an array of indices
    that sort it.

  **v**
  : Values to insert into a.

  **side**
  : If ‘left’, the index of the first suitable location found is given.
    If ‘right’, return the last such index.  If there is no suitable
    index, return either 0 or N (where N is the length of a).

  **sorter**
  : Optional array of integer indices that sort array a into ascending
    order. They are typically the result of argsort.
* **Returns:**
  **indices**
  : Array of insertion points with the same shape as v,
    or an integer if v is a scalar.

#### SEE ALSO
`sort`
: Return a sorted copy of an array.

[`histogram`](dask.array.histogram.md#dask.array.histogram)
: Produce histogram from 1-D data.

### Notes

Binary search is used to find the required insertion points.

As of NumPy 1.4.0 searchsorted works with real/complex arrays containing
nan values. The enhanced sort order is documented in sort.

This function uses the same algorithm as the builtin python
bisect.bisect_left (`side='left'`) and bisect.bisect_right
(`side='right'`) functions, which is also vectorized
in the v argument.

### Examples

```pycon
>>> import numpy as np
>>> np.searchsorted([11,12,13,14,15], 13)
2
>>> np.searchsorted([11,12,13,14,15], 13, side='right')
3
>>> np.searchsorted([11,12,13,14,15], [-10, 20, 12, 13])
array([0, 5, 1, 2])
```

When sorter is used, the returned indices refer to the sorted
array of a and not a itself:

```pycon
>>> a = np.array([40, 10, 20, 30])
>>> sorter = np.argsort(a)
>>> sorter
array([1, 2, 3, 0])  # Indices that would sort the array 'a'
>>> result = np.searchsorted(a, 25, sorter=sorter)
>>> result
2
>>> a[sorter[result]]
30  # The element at index 2 of the sorted array is 30.
```

<!-- !! processed by numpydoc !! -->


# dask.array.select.html.md

# dask.array.select

### dask.array.select(condlist, choicelist, default=0)

Return an array drawn from elements in choicelist, depending on conditions.

This docstring was copied from numpy.select.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **condlist**
  : The list of conditions which determine from which array in choicelist
    the output elements are taken. When multiple conditions are satisfied,
    the first one encountered in condlist is used.

  **choicelist**
  : The list of arrays from which the output elements are taken. It has
    to be of the same length as condlist.

  **default**
  : The element inserted in output when all conditions evaluate to False.
* **Returns:**
  **output**
  : The output at position m is the m-th element of the array in
    choicelist where the m-th element of the corresponding array in
    condlist is True.

#### SEE ALSO
[`where`](dask.array.where.md#dask.array.where)
: Return elements from one of two arrays depending on condition.

[`take`](dask.array.take.md#dask.array.take), [`choose`](dask.array.choose.md#dask.array.choose), [`compress`](dask.array.compress.md#dask.array.compress), [`diag`](dask.array.diag.md#dask.array.diag), [`diagonal`](dask.array.diagonal.md#dask.array.diagonal)

### Examples

```pycon
>>> import numpy as np
```

Beginning with an array of integers from 0 to 5 (inclusive),
elements less than `3` are negated, elements greater than `3`
are squared, and elements not meeting either of these conditions
(exactly `3`) are replaced with a default value of `42`.

```pycon
>>> x = np.arange(6)
>>> condlist = [x<3, x>3]
>>> choicelist = [-x, x**2]
>>> np.select(condlist, choicelist, 42)
array([ 0,  -1,  -2, 42, 16, 25])
```

When multiple conditions are satisfied, the first one encountered in
condlist is used.

```pycon
>>> condlist = [x<=4, x>3]
>>> choicelist = [x, x**2]
>>> np.select(condlist, choicelist, 55)
array([ 0,  1,  2,  3,  4, 25])
```

<!-- !! processed by numpydoc !! -->


# dask.array.shape.html.md

# dask.array.shape

### dask.array.shape(array)

Return the shape of an array.

This docstring was copied from numpy.shape.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array.
* **Returns:**
  **shape**
  : The elements of the shape tuple give the lengths of the
    corresponding array dimensions.

#### SEE ALSO
[`len`](https://docs.python.org/3/library/functions.html#len)
: `len(a)` is equivalent to `np.shape(a)[0]` for N-D arrays with `N>=1`.

`ndarray.shape`
: Equivalent array method.

### Examples

```pycon
>>> import numpy as np
>>> np.shape(np.eye(3))
(3, 3)
>>> np.shape([[1, 3]])
(1, 2)
>>> np.shape([0])
(1,)
>>> np.shape(0)
()
```

```pycon
>>> a = np.array([(1, 2), (3, 4), (5, 6)],
...              dtype=[('x', 'i4'), ('y', 'i4')])
>>> np.shape(a)
(3,)
>>> a.shape
(3,)
```

<!-- !! processed by numpydoc !! -->


# dask.array.shuffle.html.md

# dask.array.shuffle

### dask.array.shuffle(x, indexer: [list](https://docs.python.org/3/library/stdtypes.html#list)[[list](https://docs.python.org/3/library/stdtypes.html#list)[[int](https://docs.python.org/3/library/functions.html#int)]], axis: [int](https://docs.python.org/3/library/functions.html#int), chunks: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['auto'] = 'auto')

Reorders one dimensions of a Dask Array based on an indexer.

The indexer defines a list of positional groups that will end up in the same chunk
together. A single group is in at most one chunk on this dimension, but a chunk
might contain multiple groups to avoid fragmentation of the array.

The algorithm tries to balance the chunksizes as much as possible to ideally keep the
number of chunks consistent or at least manageable.

* **Parameters:**
  **x: dask array**
  : Array to be shuffled.

  **indexer:  list[list[int]]**
  : The indexer that determines which elements along the dimension will end up in the
    same chunk. Multiple groups can be in the same chunk to avoid fragmentation, but
    each group will end up in exactly one chunk.

  **axis: int**
  : The axis to shuffle along.

  **chunks: “auto”**
  : Hint on how to rechunk if single groups are becoming too large. The default is
    to split chunks along the other dimensions evenly to keep the chunksize
    consistent. The rechunking is done in a way that ensures that non all-to-all
    network communication is necessary, chunks are only split and not combined with
    other chunks.

### Examples

```pycon
>>> import dask.array as da
>>> import numpy as np
>>> arr = np.array([[1, 2, 3, 4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16]])
>>> x = da.from_array(arr, chunks=(2, 4))
```

Separate the elements in different groups.

```pycon
>>> y = x.shuffle([[6, 5, 2], [4, 1], [3, 0, 7]], axis=1)
```

The shuffle algorithm will combine the first 2 groups into a single chunk to keep
the number of chunks small.

The tolerance of increasing the chunk size is controlled by the configuration
“array.chunk-size-tolerance”. The default value is 1.25.

```pycon
>>> y.chunks
((2,), (5, 3))
```

The array was reordered along axis 1 according to the positional indexer that was given.

```pycon
>>> y.compute()
array([[ 7,  6,  3,  5,  2,  4,  1,  8],
       [15, 14, 11, 13, 10, 12,  9, 16]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.sign.html.md

# dask.array.sign

### dask.array.sign(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'sign'>*

This docstring was copied from numpy.sign.

Some inconsistencies with the Dask version may exist.

Returns an element-wise indication of the sign of a number.

The sign function returns `-1 if x < 0, 0 if x==0, 1 if x > 0`.  nan
is returned for nan inputs.

For complex inputs, the sign function returns `x / abs(x)`, the
generalization of the above (and `0 if x==0`).

#### Versionchanged
Changed in version 2.0.0: Definition of complex sign changed to follow the Array API standard.

* **Parameters:**
  **x**
  : Input values.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The sign of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`signbit`](dask.array.signbit.md#dask.array.signbit)

[`copysign`](dask.array.copysign.md#dask.array.copysign)

### Notes

There is more than one definition of sign in common use for complex
numbers.  The definition used here, $x/|x|$, is the more common
and useful one, but is different from the one used in numpy prior to
version 2.0, $x/\sqrt{x*x}$, which is equivalent to
`sign(x.real) + 0j if x.real != 0 else sign(x.imag) + 0j`.

### Examples

```pycon
>>> import numpy as np
>>> np.sign([-5., 4.5])
array([-1.,  1.])
>>> np.sign(0)
0
>>> np.sign([3-4j, 8j])
array([0.6-0.8j, 0. +1.j ])
```

<!-- !! processed by numpydoc !! -->


# dask.array.signbit.html.md

# dask.array.signbit

### dask.array.signbit(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'signbit'>*

This docstring was copied from numpy.signbit.

Some inconsistencies with the Dask version may exist.

Returns element-wise True where signbit is set (less than zero).

* **Parameters:**
  **x**
  : The input value(s).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **result**
  : Output array, or reference to out if that was supplied.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`sign`](dask.array.sign.md#dask.array.sign)

[`copysign`](dask.array.copysign.md#dask.array.copysign)

### Examples

```pycon
>>> import numpy as np
>>> np.signbit(-1.2)
True
>>> np.signbit(np.array([1, -2.3, 2.1]))
array([False,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.array.sin.html.md

# dask.array.sin

### dask.array.sin(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'sin'>*

This docstring was copied from numpy.sin.

Some inconsistencies with the Dask version may exist.

Trigonometric sine, element-wise.

* **Parameters:**
  **x**
  : Angle, in radians ($2 \pi$ rad equals 360 degrees).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The sine of each element of x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`arcsin`](dask.array.arcsin.md#dask.array.arcsin), [`sinh`](dask.array.sinh.md#dask.array.sinh), [`cos`](dask.array.cos.md#dask.array.cos)

### Notes

The sine is one of the fundamental functions of trigonometry (the
mathematical study of triangles).  Consider a circle of radius 1
centered on the origin.  A ray comes in from the $+x$ axis, makes
an angle at the origin (measured counter-clockwise from that axis), and
departs from the origin.  The $y$ coordinate of the outgoing
ray’s intersection with the unit circle is the sine of that angle.  It
ranges from -1 for $x=3\pi / 2$ to +1 for $\pi / 2.$  The
function has zeroes where the angle is a multiple of $\pi$.
Sines of angles between $\pi$ and $2\pi$ are negative.
The numerous properties of the sine and related functions are included
in any standard trigonometry text.

### Examples

```pycon
>>> import numpy as np
```

Print sine of one angle:

```pycon
>>> np.sin(np.pi/2.)
1.0
```

Print sines of an array of angles given in degrees:

```pycon
>>> np.sin(np.array((0., 30., 45., 60., 90.)) * np.pi / 180. )
array([ 0.        ,  0.5       ,  0.70710678,  0.8660254 ,  1.        ])
```

Plot the sine function:

```pycon
>>> import matplotlib.pylab as plt
>>> x = np.linspace(-np.pi, np.pi, 201)
>>> plt.plot(x, np.sin(x))
>>> plt.xlabel('Angle [rad]')
>>> plt.ylabel('sin(x)')
>>> plt.axis('tight')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.sinc.html.md

# dask.array.sinc

### dask.array.sinc(\*args, \*\*kwargs)

Return the normalized sinc function.

This docstring was copied from numpy.sinc.

Some inconsistencies with the Dask version may exist.

The sinc function is equal to $\sin(\pi x)/(\pi x)$ for any argument
$x\ne 0$. `sinc(0)` takes the limit value 1, making `sinc` not
only everywhere continuous but also infinitely differentiable.

#### NOTE
Note the normalization factor of `pi` used in the definition.
This is the most commonly used definition in signal processing.
Use `sinc(x / np.pi)` to obtain the unnormalized sinc function
$\sin(x)/x$ that is more common in mathematics.

* **Parameters:**
  **x**
  : Array (possibly multi-dimensional) of values for which to calculate
    `sinc(x)`.
* **Returns:**
  **out**
  : `sinc(x)`, which has the same shape as the input.

### Notes

The name sinc is short for “sine cardinal” or “sinus cardinalis”.

The sinc function is used in various signal processing applications,
including in anti-aliasing, in the construction of a Lanczos resampling
filter, and in interpolation.

For bandlimited interpolation of discrete-time signals, the ideal
interpolation kernel is proportional to the sinc function.

**Array API Standard Support**

sinc has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ⛔    |
| Dask      | ✅    | n/a   |

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> x = np.linspace(-4, 4, 41)
>>> np.sinc(x)
 array([-3.89804309e-17,  -4.92362781e-02,  -8.40918587e-02, # may vary
        -8.90384387e-02,  -5.84680802e-02,   3.89804309e-17,
        6.68206631e-02,   1.16434881e-01,   1.26137788e-01,
        8.50444803e-02,  -3.89804309e-17,  -1.03943254e-01,
        -1.89206682e-01,  -2.16236208e-01,  -1.55914881e-01,
        3.89804309e-17,   2.33872321e-01,   5.04551152e-01,
        7.56826729e-01,   9.35489284e-01,   1.00000000e+00,
        9.35489284e-01,   7.56826729e-01,   5.04551152e-01,
        2.33872321e-01,   3.89804309e-17,  -1.55914881e-01,
       -2.16236208e-01,  -1.89206682e-01,  -1.03943254e-01,
       -3.89804309e-17,   8.50444803e-02,   1.26137788e-01,
        1.16434881e-01,   6.68206631e-02,   3.89804309e-17,
        -5.84680802e-02,  -8.90384387e-02,  -8.40918587e-02,
        -4.92362781e-02,  -3.89804309e-17])
```

```pycon
>>> plt.plot(x, np.sinc(x))
[<matplotlib.lines.Line2D object at 0x...>]
>>> plt.title("Sinc Function")
Text(0.5, 1.0, 'Sinc Function')
>>> plt.ylabel("Amplitude")
Text(0, 0.5, 'Amplitude')
>>> plt.xlabel("X")
Text(0.5, 0, 'X')
>>> plt.show()
```

<!-- !! processed by numpydoc !! -->


# dask.array.sinh.html.md

# dask.array.sinh

### dask.array.sinh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'sinh'>*

This docstring was copied from numpy.sinh.

Some inconsistencies with the Dask version may exist.

Hyperbolic sine, element-wise.

Equivalent to `1/2 * (np.exp(x) - np.exp(-x))` or
`-1j * np.sin(1j*x)`.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding hyperbolic sine values.
    This is a scalar if x is a scalar.

### Notes

If out is provided, the function writes the result into it,
and returns a reference to out.  (See Examples)

### References

M. Abramowitz and I. A. Stegun, Handbook of Mathematical Functions.
New York, NY: Dover, 1972, pg. 83.

### Examples

```pycon
>>> import numpy as np
>>> np.sinh(0)
0.0
>>> np.sinh(np.pi*1j/2)
1j
>>> np.sinh(np.pi*1j) # (exact value is 0)
1.2246063538223773e-016j
>>> # Discrepancy due to vagaries of floating point arithmetic.
```

```pycon
>>> # Example of providing the optional output parameter
>>> out1 = np.array([0], dtype=np.float64)
>>> out2 = np.sinh([0.1], out1)
>>> out2 is out1
True
```

```pycon
>>> # Example of ValueError due to provision of shape mis-matched `out`
>>> np.sinh(np.zeros((3,3)),np.zeros((2,2)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (3,3) (2,2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.spacing.html.md

# dask.array.spacing

### dask.array.spacing(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'spacing'>*

This docstring was copied from numpy.spacing.

Some inconsistencies with the Dask version may exist.

Return the distance between x and the nearest adjacent number.

* **Parameters:**
  **x**
  : Values to find the spacing of.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : The spacing of values of x.
    This is a scalar if x is a scalar.

### Notes

It can be considered as a generalization of EPS:
`spacing(np.float64(1)) == np.finfo(np.float64).eps`, and there
should not be any representable number between `x + spacing(x)` and
x for any finite x.

Spacing of +- inf and NaN is NaN.

### Examples

```pycon
>>> import numpy as np
>>> np.spacing(1) == np.finfo(np.float64).eps
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.sqrt.html.md

# dask.array.sqrt

### dask.array.sqrt(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'sqrt'>*

This docstring was copied from numpy.sqrt.

Some inconsistencies with the Dask version may exist.

Return the non-negative square-root of an array, element-wise.

* **Parameters:**
  **x**
  : The values whose square-roots are required.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : An array of the same shape as x, containing the positive
    square-root of each element in x.  If any element in x is
    complex, a complex array is returned (and the square-roots of
    negative reals are calculated).  If all of the elements in x
    are real, so is y, with negative elements returning `nan`.
    If out was provided, y is a reference to it.
    This is a scalar if x is a scalar.

#### SEE ALSO
`emath.sqrt`
: A version which returns complex numbers when given negative reals. Note that 0.0 and -0.0 are handled differently for complex inputs.

### Notes

*sqrt* has–consistent with common convention–as its branch cut the
real “interval” [-inf, 0), and is continuous from above on it.
A branch cut is a curve in the complex plane across which a given
complex function fails to be continuous.

### Examples

```pycon
>>> import numpy as np
>>> np.sqrt([1,4,9])
array([ 1.,  2.,  3.])
```

```pycon
>>> np.sqrt([4, -1, -3+4J])
array([ 2.+0.j,  0.+1.j,  1.+2.j])
```

```pycon
>>> np.sqrt([4, -1, np.inf])
array([ 2., nan, inf])
```

<!-- !! processed by numpydoc !! -->


# dask.array.square.html.md

# dask.array.square

### dask.array.square(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'square'>*

This docstring was copied from numpy.square.

Some inconsistencies with the Dask version may exist.

Return the element-wise square of the input.

* **Parameters:**
  **x**
  : Input data.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **out**
  : Element-wise x\*x, of the same shape and dtype as x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`numpy.linalg.matrix_power`](https://numpy.org/doc/stable/reference/generated/numpy.linalg.matrix_power.html#numpy.linalg.matrix_power)

[`sqrt`](dask.array.sqrt.md#dask.array.sqrt)

[`power`](dask.array.power.md#dask.array.power)

### Examples

```pycon
>>> import numpy as np
>>> np.square([-1j, 1])
array([-1.-0.j,  1.+0.j])
```

<!-- !! processed by numpydoc !! -->


# dask.array.squeeze.html.md

# dask.array.squeeze

### dask.array.squeeze(a, axis=None)

Remove axes of length one from a.

This docstring was copied from numpy.squeeze.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input data.

  **axis**
  : Selects a subset of the entries of length one in the
    shape. If an axis is selected with shape entry greater than
    one, an error is raised.
* **Returns:**
  **squeezed**
  : The input array, but with all or a subset of the
    dimensions of length 1 removed. This is always a itself
    or a view into a. Note that if all axes are squeezed,
    the result is a 0d array and not a scalar.
* **Raises:**
  ValueError
  : If axis is not None, and an axis being squeezed is not of length 1

#### SEE ALSO
[`expand_dims`](dask.array.expand_dims.md#dask.array.expand_dims)
: The inverse operation, adding entries of length one

[`reshape`](dask.array.reshape.md#dask.array.reshape)
: Insert, remove, and combine dimensions, and resize existing ones

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([[[0], [1], [2]]])
>>> x.shape
(1, 3, 1)
>>> np.squeeze(x).shape
(3,)
>>> np.squeeze(x, axis=0).shape
(3, 1)
>>> np.squeeze(x, axis=1).shape
Traceback (most recent call last):
...
ValueError: cannot select an axis to squeeze out which has size
not equal to one
>>> np.squeeze(x, axis=2).shape
(1, 3)
>>> x = np.array([[1234]])
>>> x.shape
(1, 1)
>>> np.squeeze(x)
array(1234)  # 0d array
>>> np.squeeze(x).shape
()
>>> np.squeeze(x)[()]
1234
```

<!-- !! processed by numpydoc !! -->


# dask.array.stack.html.md

# dask.array.stack

### dask.array.stack(seq, axis=0, allow_unknown_chunksizes=False)

Stack arrays along a new axis

Given a sequence of dask arrays, form a new dask array by stacking them
along a new dimension (axis=0 by default)

* **Parameters:**
  **seq: list of dask.arrays**

  **axis: int**
  : Dimension along which to align all of the arrays

  **allow_unknown_chunksizes: bool**
  : Allow unknown chunksizes, such as come from converting from dask
    dataframes.  Dask.array is unable to verify that chunks line up.  If
    data comes from differently aligned sources then this can cause
    unexpected results.

#### SEE ALSO
[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)

### Examples

Create slices

```pycon
>>> import dask.array as da
>>> import numpy as np
```

```pycon
>>> data = [da.from_array(np.ones((4, 4)), chunks=(2, 2))
...         for i in range(3)]
```

```pycon
>>> x = da.stack(data, axis=0)
>>> x.shape
(3, 4, 4)
```

```pycon
>>> da.stack(data, axis=1).shape
(4, 3, 4)
```

```pycon
>>> da.stack(data, axis=-1).shape
(4, 4, 3)
```

Result is a new dask Array

<!-- !! processed by numpydoc !! -->


# dask.array.stats.chisquare.html.md

# dask.array.stats.chisquare

### dask.array.stats.chisquare(f_obs, f_exp=None, ddof=0, axis=0)

Calculate a one-way chi-square test.

Please see the docstring for [`scipy.stats.chisquare()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.html#scipy.stats.chisquare) for
complete information including notes, references, and examples.

Some inconsistencies with the Dask version may exist.

The chi-square test tests the null hypothesis that the categorical
data has the given frequencies.

* **Parameters:**
  **f_obs**
  : Observed frequencies in each category.

  **f_exp**
  : Expected frequencies in each category.  By default the categories are
    assumed to be equally likely.

  **ddof**
  : “Delta degrees of freedom”: adjustment to the degrees of freedom
    for the p-value.  The p-value is computed using a chi-squared
    distribution with `k - 1 - ddof` degrees of freedom, where k
    is the number of observed frequencies.  The default value of ddof
    is 0.

  **axis**
  : The axis of the broadcast result of f_obs and f_exp along which to
    apply the test.  If axis is None, all values in f_obs are treated
    as a single data set.  Default is 0.
* **Returns:**
  res: Delayed Power_divergenceResult
  : An object containing attributes:
    <br/>
    chisq
    : The chi-squared test statistic.  The value is a float if axis is
      None or f_obs and f_exp are 1-D.
    <br/>
    pvalue
    : The p-value of the test.  The value is a float if ddof and the
      return value chisq are scalars.

<!-- !! processed by numpydoc !! -->


# dask.array.stats.f_oneway.html.md

# dask.array.stats.f_oneway

### dask.array.stats.f_oneway(\*args)

Perform one-way ANOVA.

This docstring was copied from scipy.stats.f_oneway.

Some inconsistencies with the Dask version may exist.

The one-way ANOVA tests the null hypothesis that two or more groups have
the same population mean.  The test is applied to samples from two or
more groups, possibly with differing sizes.

* **Parameters:**
  **\*samples**
  : The sample measurements for each group.  There must be at least
    two arguments.  If the arrays are multidimensional, then all the
    dimensions of the array must be the same except for axis.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **equal_var**
  : If True (default), perform a standard one-way ANOVA test that
    assumes equal population variances [[2]](#r47f7c78b184f-2).
    If False, perform Welch’s ANOVA test, which does not assume
    equal population variances [[4]](#r47f7c78b184f-4).
    <br/>
    #### Versionadded
    Added in version 1.16.0.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **statistic**
  : The computed F statistic of the test.

  **pvalue**
  : The associated p-value from the F distribution.
* **Warns:**
  ~scipy.stats.ConstantInputWarning
  : Emitted if all values within each of the input arrays are identical.
    In this case the F statistic is either infinite or isn’t defined,
    so `np.inf` or `np.nan` is returned.

  RuntimeWarning
  : Emitted if the length of any input array is 0, or if all the input
    arrays have length 1.  `np.nan` is returned for the F statistic
    and the p-value in these cases.

### Notes

The ANOVA test has important assumptions that must be satisfied in order
for the associated p-value to be valid.

1. The samples are independent.
2. Each sample is from a normally distributed population.
3. The population standard deviations of the groups are all equal.  This
   property is known as homoscedasticity.

If these assumptions are not true for a given set of data, it may still
be possible to use the Kruskal-Wallis H-test (scipy.stats.kruskal) or
the Alexander-Govern test (scipy.stats.alexandergovern) although with
some loss of power.

The length of each group must be at least one, and there must be at
least one group with length greater than one.  If these conditions
are not satisfied, a warning is generated and (`np.nan`, `np.nan`)
is returned.

If all values in each group are identical, and there exist at least two
groups with different values, the function generates a warning and
returns (`np.inf`, 0).

If all values in all groups are the same, function generates a warning
and returns (`np.nan`, `np.nan`).

The algorithm is from Heiman [[2]](#r47f7c78b184f-2), pp.394-7.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

f_oneway has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ⛔    |
| JAX       | ✅    | ⛔    |
| Dask      | ✅    | n/a   |

f_oneway also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> import numpy as np
>>> from scipy.stats import f_oneway
```

Here are some data [[3]](#r47f7c78b184f-3) on a shell measurement (the length of the anterior
adductor muscle scar, standardized by dividing by length) in the mussel
Mytilus trossulus from five locations: Tillamook, Oregon; Newport, Oregon;
Petersburg, Alaska; Magadan, Russia; and Tvarminne, Finland, taken from a
much larger data set used in [[5]](#r47f7c78b184f-5).

```pycon
>>> tillamook = [0.0571, 0.0813, 0.0831, 0.0976, 0.0817, 0.0859, 0.0735,
...              0.0659, 0.0923, 0.0836]
>>> newport = [0.0873, 0.0662, 0.0672, 0.0819, 0.0749, 0.0649, 0.0835,
...            0.0725]
>>> petersburg = [0.0974, 0.1352, 0.0817, 0.1016, 0.0968, 0.1064, 0.105]
>>> magadan = [0.1033, 0.0915, 0.0781, 0.0685, 0.0677, 0.0697, 0.0764,
...            0.0689]
>>> tvarminne = [0.0703, 0.1026, 0.0956, 0.0973, 0.1039, 0.1045]
>>> f_oneway(tillamook, newport, petersburg, magadan, tvarminne)
F_onewayResult(statistic=7.121019471642447, pvalue=0.0002812242314534544)
```

f_oneway accepts multidimensional input arrays.  When the inputs
are multidimensional and axis is not given, the test is performed
along the first axis of the input arrays.  For the following data, the
test is performed three times, once for each column.

```pycon
>>> a = np.array([[9.87, 9.03, 6.81],
...               [7.18, 8.35, 7.00],
...               [8.39, 7.58, 7.68],
...               [7.45, 6.33, 9.35],
...               [6.41, 7.10, 9.33],
...               [8.00, 8.24, 8.44]])
>>> b = np.array([[6.35, 7.30, 7.16],
...               [6.65, 6.68, 7.63],
...               [5.72, 7.73, 6.72],
...               [7.01, 9.19, 7.41],
...               [7.75, 7.87, 8.30],
...               [6.90, 7.97, 6.97]])
>>> c = np.array([[3.31, 8.77, 1.01],
...               [8.25, 3.24, 3.62],
...               [6.32, 8.81, 5.19],
...               [7.48, 8.83, 8.91],
...               [8.59, 6.01, 6.07],
...               [3.07, 9.72, 7.48]])
>>> F = f_oneway(a, b, c)
>>> F.statistic
array([1.75676344, 0.03701228, 3.76439349])
>>> F.pvalue
array([0.20630784, 0.96375203, 0.04733157])
```

Welch ANOVA will be performed if equal_var is False.

<!-- !! processed by numpydoc !! -->


# dask.array.stats.kurtosis.html.md

# dask.array.stats.kurtosis

### dask.array.stats.kurtosis(a, axis=0, fisher=True, bias=True, nan_policy='propagate')

Compute the kurtosis (Fisher or Pearson) of a dataset.

This docstring was copied from scipy.stats.kurtosis.

Some inconsistencies with the Dask version may exist.

Kurtosis is the fourth central moment divided by the square of the
variance. If Fisher’s definition is used, then 3.0 is subtracted from
the result to give 0.0 for a normal distribution.

If bias is False then the kurtosis is calculated using k statistics to
eliminate bias coming from biased moment estimators

Use kurtosistest to see if result is close enough to normal.

* **Parameters:**
  **a**
  : Data for which the kurtosis is calculated.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **fisher**
  : If True, Fisher’s definition is used (normal ==> 0.0). If False,
    Pearson’s definition is used (normal ==> 3.0).

  **bias**
  : If False, then the calculations are corrected for statistical bias.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **kurtosis**
  : The kurtosis of values along an axis, returning NaN where all values
    are equal.

### Notes

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

kurtosis has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

kurtosis also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

In Fisher’s definition, the kurtosis of the normal distribution is zero.
In the following example, the kurtosis is close to zero, because it was
calculated from the dataset, not from the continuous distribution.

```pycon
>>> import numpy as np
>>> from scipy.stats import norm, kurtosis
>>> data = norm.rvs(size=1000, random_state=3)
>>> kurtosis(data)
-0.06928694200380558
```

The distribution with a higher kurtosis has a heavier tail.
The zero valued kurtosis of the normal distribution in Fisher’s definition
can serve as a reference point.

```pycon
>>> import matplotlib.pyplot as plt
>>> import scipy.stats as stats
>>> from scipy.stats import kurtosis
```

```pycon
>>> x = np.linspace(-5, 5, 100)
>>> ax = plt.subplot()
>>> distnames = ['laplace', 'norm', 'uniform']
```

```pycon
>>> for distname in distnames:
...     if distname == 'uniform':
...         dist = getattr(stats, distname)(loc=-2, scale=4)
...     else:
...         dist = getattr(stats, distname)
...     data = dist.rvs(size=1000)
...     kur = kurtosis(data, fisher=True)
...     y = dist.pdf(x)
...     ax.plot(x, y, label="{}, {}".format(distname, round(kur, 3)))
...     ax.legend()
```

The Laplace distribution has a heavier tail than the normal distribution.
The uniform distribution (which has negative kurtosis) has the thinnest
tail.

<!-- !! processed by numpydoc !! -->


# dask.array.stats.kurtosistest.html.md

# dask.array.stats.kurtosistest

### dask.array.stats.kurtosistest(a, axis=0, nan_policy='propagate')

Test whether a dataset has normal kurtosis.

This docstring was copied from scipy.stats.kurtosistest.

Some inconsistencies with the Dask version may exist.

This function tests the null hypothesis that the kurtosis
of the population from which the sample was drawn is that
of the normal distribution.

* **Parameters:**
  **a**
  : Array of the sample data. Must contain at least five observations.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **alternative**
  : Defines the alternative hypothesis.
    The following options are available (default is ‘two-sided’):
    * ‘two-sided’: the kurtosis of the distribution underlying the sample
      is different from that of the normal distribution
    * ‘less’: the kurtosis of the distribution underlying the sample
      is less than that of the normal distribution
    * ‘greater’: the kurtosis of the distribution underlying the sample
      is greater than that of the normal distribution
    <br/>
    #### Versionadded
    Added in version 1.7.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **statistic**
  : The computed z-score for this test.

  **pvalue**
  : The p-value for the hypothesis test.

#### SEE ALSO
[Kurtosis test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_kurtosistest.html#hypothesis-kurtosistest)
: Extended example

### Notes

Valid only for n>20. This function uses the method described in [[1]](#r0535d6059af7-1).

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

kurtosistest has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

kurtosistest also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> import numpy as np
>>> from scipy.stats import kurtosistest
>>> kurtosistest(list(range(20)))
KurtosistestResult(statistic=-1.7058104152122062, pvalue=0.08804338332528348)
>>> kurtosistest(list(range(20)), alternative='less')
KurtosistestResult(statistic=-1.7058104152122062, pvalue=0.04402169166264174)
>>> kurtosistest(list(range(20)), alternative='greater')
KurtosistestResult(statistic=-1.7058104152122062, pvalue=0.9559783083373583)
>>> rng = np.random.default_rng()
>>> s = rng.normal(0, 1, 1000)
>>> kurtosistest(s)
KurtosistestResult(statistic=-1.475047944490622, pvalue=0.14019965402996987)
```

For a more detailed example, see [Kurtosis test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_kurtosistest.html#hypothesis-kurtosistest).

<!-- !! processed by numpydoc !! -->


# dask.array.stats.moment.html.md

# dask.array.stats.moment

### dask.array.stats.moment(a, moment=1, axis=0, nan_policy='propagate')

Calculate the nth moment about the mean for a sample.

This docstring was copied from scipy.stats.moment.

Some inconsistencies with the Dask version may exist.

A moment is a specific quantitative measure of the shape of a set of
points. It is often used to calculate coefficients of skewness and kurtosis
due to its close relationship with them.

* **Parameters:**
  **a**
  : Input array.

  **order**
  : Order of central moment that is returned. Default is 1.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **center**
  : The point about which moments are taken. This can be the sample mean,
    the origin, or any other be point. If None (default) compute the
    center as the sample mean.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **n-th moment about the \`center\`**
  : The appropriate moment along the given axis or over all values if axis
    is None. The denominator for the moment calculation is the number of
    observations, no degrees of freedom correction is done.

#### SEE ALSO
[`kurtosis()`](dask.array.stats.kurtosis.md#dask.array.stats.kurtosis), [`skew()`](dask.array.stats.skew.md#dask.array.stats.skew), `describe()`

### Notes

The k-th moment of a data sample is:

$$
m_k = \frac{1}{n} \sum_{i = 1}^n (x_i - c)^k
$$

Where n is the number of samples, and c is the center around which the
moment is calculated. This function uses exponentiation by squares [[1]](#r330c2a4ee428-1) for
efficiency.

Note that, if a is an empty array (`a.size == 0`), array moment with
one element (moment.size == 1) is treated the same as scalar moment
(`np.isscalar(moment)`). This might produce arrays of unexpected shape.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

moment has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ⛔    | n/a   |

moment also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> from scipy.stats import moment
>>> moment([1, 2, 3, 4, 5], order=1)
0.0
>>> moment([1, 2, 3, 4, 5], order=2)
2.0
```

<!-- !! processed by numpydoc !! -->


# dask.array.stats.normaltest.html.md

# dask.array.stats.normaltest

### dask.array.stats.normaltest(a, axis=0, nan_policy='propagate')

Test whether a sample differs from a normal distribution.

This docstring was copied from scipy.stats.normaltest.

Some inconsistencies with the Dask version may exist.

This function tests the null hypothesis that a sample comes
from a normal distribution.  It is based on D’Agostino and
Pearson’s [[1]](#r32e5e6b3c76f-1), [[2]](#r32e5e6b3c76f-2) test that combines skew and kurtosis to
produce an omnibus test of normality.

* **Parameters:**
  **a**
  : The array containing the sample to be tested. Must contain
    at least eight observations.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **statistic**
  : `s^2 + k^2`, where `s` is the z-score returned by skewtest and
    `k` is the z-score returned by kurtosistest.

  **pvalue**
  : A 2-sided chi squared probability for the hypothesis test.

#### SEE ALSO
[Normal test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_normaltest.html#hypothesis-normaltest)
: Extended example

### Notes

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

normaltest has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

normaltest also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> import numpy as np
>>> from scipy import stats
>>> rng = np.random.default_rng()
>>> pts = 1000
>>> a = rng.normal(0, 1, size=pts)
>>> b = rng.normal(2, 1, size=pts)
>>> x = np.concatenate((a, b))
>>> res = stats.normaltest(x)
>>> res.statistic
53.619...  # random
>>> res.pvalue
2.273917413209226e-12  # random
```

For a more detailed example, see [Normal test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_normaltest.html#hypothesis-normaltest).

<!-- !! processed by numpydoc !! -->


# dask.array.stats.power_divergence.html.md

# dask.array.stats.power_divergence

### dask.array.stats.power_divergence(f_obs, f_exp=None, ddof=0, axis=0, lambda_=None)

Cressie-Read power divergence statistic and goodness of fit test.

This docstring was copied from scipy.stats.power_divergence.

Some inconsistencies with the Dask version may exist.

This function tests the null hypothesis that the categorical data
has the given frequencies, using the Cressie-Read power divergence
statistic.

* **Parameters:**
  **f_obs**
  : Observed frequencies in each category.

  **f_exp**
  : Expected frequencies in each category.  By default the categories are
    assumed to be equally likely.

  **ddof**
  : “Delta degrees of freedom”: adjustment to the degrees of freedom
    for the p-value.  The p-value is computed using a chi-squared
    distribution with `k - 1 - ddof` degrees of freedom, where k
    is the number of observed frequencies.  The default value of ddof
    is 0.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **lambda_**
  : The power in the Cressie-Read power divergence statistic.  The default
    is 1.  For convenience, lambda_ may be assigned one of the following
    strings, in which case the corresponding numerical value is used:
    * `"pearson"` (value 1)
      : Pearson’s chi-squared statistic. In this case, the function is
        equivalent to chisquare.
    * `"log-likelihood"` (value 0)
      : Log-likelihood ratio. Also known as the G-test [[3]](#r5ed189a69e5c-3).
    * `"freeman-tukey"` (value -1/2)
      : Freeman-Tukey statistic.
    * `"mod-log-likelihood"` (value -1)
      : Modified log-likelihood ratio.
    * `"neyman"` (value -2)
      : Neyman’s statistic.
    * `"cressie-read"` (value 2/3)
      : The power recommended in [[5]](#r5ed189a69e5c-5).

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  res: Power_divergenceResult
  : An object containing attributes:
    <br/>
    statistic
    : The Cressie-Read power divergence test statistic.  The value is
      a float if axis is None or if\` f_obs and f_exp are 1-D.
    <br/>
    pvalue
    : The p-value of the test.  The value is a float if ddof and the
      return value stat are scalars.

#### SEE ALSO
[`chisquare()`](dask.array.stats.chisquare.md#dask.array.stats.chisquare)

### Notes

This test is invalid when the observed or expected frequencies in each
category are too small.  A typical rule is that all of the observed
and expected frequencies should be at least 5.

Also, the sum of the observed and expected frequencies must be the same
for the test to be valid; power_divergence raises an error if the sums
do not agree within a relative tolerance of `eps**0.5`, where `eps`
is the precision of the input dtype.

When lambda_ is less than zero, the formula for the statistic involves
dividing by f_obs, so a warning or error may be generated if any value
in f_obs is 0.

Similarly, a warning or error may be generated if any value in f_exp is
zero when lambda_ >= 0.

The default degrees of freedom, k-1, are for the case when no parameters
of the distribution are estimated. If p parameters are estimated by
efficient maximum likelihood then the correct degrees of freedom are
k-1-p. If the parameters are estimated in a different way, then the
dof can be between k-1-p and k-1. However, it is also possible that
the asymptotic distribution is not a chisquare, in which case this
test is not appropriate.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

power_divergence has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

power_divergence also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

(See chisquare for more examples.)

When just f_obs is given, it is assumed that the expected frequencies
are uniform and given by the mean of the observed frequencies.  Here we
perform a G-test (i.e. use the log-likelihood ratio statistic):

```pycon
>>> import numpy as np
>>> from scipy.stats import power_divergence
>>> power_divergence([16, 18, 16, 14, 12, 12], lambda_='log-likelihood')
(2.006573162632538, 0.84823476779463769)
```

The expected frequencies can be given with the f_exp argument:

```pycon
>>> power_divergence([16, 18, 16, 14, 12, 12],
...                  f_exp=[16, 16, 16, 16, 16, 8],
...                  lambda_='log-likelihood')
(3.3281031458963746, 0.6495419288047497)
```

When f_obs is 2-D, by default the test is applied to each column.

```pycon
>>> obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
>>> obs.shape
(6, 2)
>>> power_divergence(obs, lambda_="log-likelihood")
(array([ 2.00657316,  6.77634498]), array([ 0.84823477,  0.23781225]))
```

By setting `axis=None`, the test is applied to all data in the array,
which is equivalent to applying the test to the flattened array.

```pycon
>>> power_divergence(obs, axis=None)
(23.31034482758621, 0.015975692534127565)
>>> power_divergence(obs.ravel())
(23.31034482758621, 0.015975692534127565)
```

ddof is the change to make to the default degrees of freedom.

```pycon
>>> power_divergence([16, 18, 16, 14, 12, 12], ddof=1)
(2.0, 0.73575888234288467)
```

The calculation of the p-values is done by broadcasting the
test statistic with ddof.

```pycon
>>> power_divergence([16, 18, 16, 14, 12, 12], ddof=[0,1,2])
(2.0, array([ 0.84914504,  0.73575888,  0.5724067 ]))
```

f_obs and f_exp are also broadcast.  In the following, f_obs has
shape (6,) and f_exp has shape (2, 6), so the result of broadcasting
f_obs and f_exp has shape (2, 6).  To compute the desired chi-squared
statistics, we must use `axis=1`:

```pycon
>>> power_divergence([16, 18, 16, 14, 12, 12],
...                  f_exp=[[16, 16, 16, 16, 16, 8],
...                         [8, 20, 20, 16, 12, 12]],
...                  axis=1)
(array([ 3.5 ,  9.25]), array([ 0.62338763,  0.09949846]))
```

<!-- !! processed by numpydoc !! -->


# dask.array.stats.skew.html.md

# dask.array.stats.skew

### dask.array.stats.skew(a, axis=0, bias=True, nan_policy='propagate')

Compute the sample skewness of a data set.

This docstring was copied from scipy.stats.skew.

Some inconsistencies with the Dask version may exist.

For normally distributed data, the skewness should be about zero. For
unimodal continuous distributions, a skewness value greater than zero means
that there is more weight in the right tail of the distribution. The
function skewtest can be used to determine if the skewness value
is close enough to zero, statistically speaking.

* **Parameters:**
  **a**
  : Input array.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **bias**
  : If False, then the calculations are corrected for statistical bias.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **skewness**
  : The skewness of values along an axis, returning NaN where all values
    are equal.

### Notes

The sample skewness is computed as the Fisher-Pearson coefficient
of skewness, i.e.

$$
g_1=\frac{m_3}{m_2^{3/2}}
$$

where

$$
m_i=\frac{1}{N}\sum_{n=1}^N(x[n]-\bar{x})^i
$$

is the biased sample $i\texttt{th}$ central moment, and
$\bar{x}$ is
the sample mean.  If `bias` is False, the calculations are
corrected for bias and the value computed is the adjusted
Fisher-Pearson standardized moment coefficient, i.e.

$$
G_1=\frac{k_3}{k_2^{3/2}}=
    \frac{\sqrt{N(N-1)}}{N-2}\frac{m_3}{m_2^{3/2}}.
$$

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

skew has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

skew also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> from scipy.stats import skew
>>> skew([1, 2, 3, 4, 5])
0.0
>>> skew([2, 8, 0, 4, 1, 9, 9, 0])
0.2650554122698573
```

<!-- !! processed by numpydoc !! -->


# dask.array.stats.skewtest.html.md

# dask.array.stats.skewtest

### dask.array.stats.skewtest(a, axis=0, nan_policy='propagate')

Test whether the skew is different from the normal distribution.

This docstring was copied from scipy.stats.skewtest.

Some inconsistencies with the Dask version may exist.

This function tests the null hypothesis that the skewness of
the population that the sample was drawn from is the same
as that of a corresponding normal distribution.

* **Parameters:**
  **a**
  : The data to be tested. Must contain at least eight observations.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **alternative**
  : Defines the alternative hypothesis. Default is ‘two-sided’.
    The following options are available:
    * ‘two-sided’: the skewness of the distribution underlying the sample
      is different from that of the normal distribution (i.e. 0)
    * ‘less’: the skewness of the distribution underlying the sample
      is less than that of the normal distribution
    * ‘greater’: the skewness of the distribution underlying the sample
      is greater than that of the normal distribution
    <br/>
    #### Versionadded
    Added in version 1.7.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **statistic**
  : The computed z-score for this test.

  **pvalue**
  : The p-value for the hypothesis test.

#### SEE ALSO
[Skewness test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_skewtest.html#hypothesis-skewtest)
: Extended example

### Notes

The sample size must be at least 8.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

skewtest has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ✅    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

skewtest also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.

See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> from scipy.stats import skewtest
>>> skewtest([1, 2, 3, 4, 5, 6, 7, 8])
SkewtestResult(statistic=1.0108048609177787, pvalue=0.3121098361421897)
>>> skewtest([2, 8, 0, 4, 1, 9, 9, 0])
SkewtestResult(statistic=0.44626385374196975, pvalue=0.6554066631275459)
>>> skewtest([1, 2, 3, 4, 5, 6, 7, 8000])
SkewtestResult(statistic=3.571773510360407, pvalue=0.0003545719905823133)
>>> skewtest([100, 100, 100, 100, 100, 100, 100, 101])
SkewtestResult(statistic=3.5717766638478072, pvalue=0.000354567720281634)
>>> skewtest([1, 2, 3, 4, 5, 6, 7, 8], alternative='less')
SkewtestResult(statistic=1.0108048609177787, pvalue=0.8439450819289052)
>>> skewtest([1, 2, 3, 4, 5, 6, 7, 8], alternative='greater')
SkewtestResult(statistic=1.0108048609177787, pvalue=0.15605491807109484)
```

For a more detailed example, see [Skewness test](https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_skewtest.html#hypothesis-skewtest).

<!-- !! processed by numpydoc !! -->


# dask.array.stats.ttest_1samp.html.md

# dask.array.stats.ttest_1samp

### dask.array.stats.ttest_1samp(a, popmean, axis=0, nan_policy='propagate')

Calculate the T-test for the mean of ONE group of scores.

This docstring was copied from scipy.stats.ttest_1samp.

Some inconsistencies with the Dask version may exist.

This is a test for the null hypothesis that the expected value
(mean) of a sample of independent observations a is equal to the given
population mean, popmean.

* **Parameters:**
  **a**
  : Sample observations.

  **popmean**
  : Expected value in null hypothesis. If array_like, then its length along
    axis must equal 1, and it must otherwise be broadcastable with a.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **alternative**
  : Defines the alternative hypothesis.
    The following options are available (default is ‘two-sided’):
    * ‘two-sided’: the mean of the underlying distribution of the sample
      is different than the given population mean (popmean)
    * ‘less’: the mean of the underlying distribution of the sample is
      less than the given population mean (popmean)
    * ‘greater’: the mean of the underlying distribution of the sample is
      greater than the given population mean (popmean)

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **result**
  : An object with the following attributes:
    <br/>
    statistic
    : The t-statistic.
    <br/>
    pvalue
    : The p-value associated with the given alternative.
    <br/>
    df
    : The number of degrees of freedom used in calculation of the
      t-statistic; this is one less than the size of the sample
      (`a.shape[axis]`).
      <br/>
      #### Versionadded
      Added in version 1.10.0.
    <br/>
    The object also has the following method:
    <br/>
    confidence_interval(confidence_level=0.95)
    : Computes a confidence interval around the population
      mean for the given confidence level.
      The confidence interval is returned in a `namedtuple` with
      fields low and high.
      <br/>
      #### Versionadded
      Added in version 1.10.0.

### Notes

The statistic is calculated as `(np.mean(a) - popmean)/se`, where
`se` is the standard error. Therefore, the statistic will be positive
when the sample mean is greater than the population mean and negative when
the sample mean is less than the population mean.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

ttest_1samp has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ⛔    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

ttest_1samp also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.
The `confidence_interval` method of the output object is incompatible with JAX JIT.
See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### Examples

Suppose we wish to test the null hypothesis that the mean of a population
is equal to 0.5. We choose a confidence level of 99%; that is, we will
reject the null hypothesis in favor of the alternative if the p-value is
less than 0.01.

When testing random variates from the standard uniform distribution, which
has a mean of 0.5, we expect the data to be consistent with the null
hypothesis most of the time.

```pycon
>>> import numpy as np
>>> from scipy import stats
>>> rng = np.random.default_rng()
>>> rvs = stats.uniform.rvs(size=50, random_state=rng)
>>> stats.ttest_1samp(rvs, popmean=0.5)
TtestResult(statistic=2.456308468440, pvalue=0.017628209047638, df=49)
```

As expected, the p-value of 0.017 is not below our threshold of 0.01, so
we cannot reject the null hypothesis.

When testing data from the standard *normal* distribution, which has a mean
of 0, we would expect the null hypothesis to be rejected.

```pycon
>>> rvs = stats.norm.rvs(size=50, random_state=rng)
>>> stats.ttest_1samp(rvs, popmean=0.5)
TtestResult(statistic=-7.433605518875, pvalue=1.416760157221e-09, df=49)
```

Indeed, the p-value is lower than our threshold of 0.01, so we reject the
null hypothesis in favor of the default “two-sided” alternative: the mean
of the population is *not* equal to 0.5.

However, suppose we were to test the null hypothesis against the
one-sided alternative that the mean of the population is *greater* than
0.5. Since the mean of the standard normal is less than 0.5, we would not
expect the null hypothesis to be rejected.

```pycon
>>> stats.ttest_1samp(rvs, popmean=0.5, alternative='greater')
TtestResult(statistic=-7.433605518875, pvalue=0.99999999929, df=49)
```

Unsurprisingly, with a p-value greater than our threshold, we would not
reject the null hypothesis.

Note that when working with a confidence level of 99%, a true null
hypothesis will be rejected approximately 1% of the time.

```pycon
>>> rvs = stats.uniform.rvs(size=(100, 50), random_state=rng)
>>> res = stats.ttest_1samp(rvs, popmean=0.5, axis=1)
>>> np.sum(res.pvalue < 0.01)
1
```

Indeed, even though all 100 samples above were drawn from the standard
uniform distribution, which *does* have a population mean of 0.5, we would
mistakenly reject the null hypothesis for one of them.

ttest_1samp can also compute a confidence interval around the population
mean.

```pycon
>>> rvs = stats.norm.rvs(size=50, random_state=rng)
>>> res = stats.ttest_1samp(rvs, popmean=0)
>>> ci = res.confidence_interval(confidence_level=0.95)
>>> ci
ConfidenceInterval(low=-0.3193887540880017, high=0.2898583388980972)
```

The bounds of the 95% confidence interval are the
minimum and maximum values of the parameter popmean for which the
p-value of the test would be 0.05.

```pycon
>>> res = stats.ttest_1samp(rvs, popmean=ci.low)
>>> np.testing.assert_allclose(res.pvalue, 0.05)
>>> res = stats.ttest_1samp(rvs, popmean=ci.high)
>>> np.testing.assert_allclose(res.pvalue, 0.05)
```

Under certain assumptions about the population from which a sample
is drawn, the confidence interval with confidence level 95% is expected
to contain the true population mean in 95% of sample replications.

```pycon
>>> rvs = stats.norm.rvs(size=(50, 1000), loc=1, random_state=rng)
>>> res = stats.ttest_1samp(rvs, popmean=0)
>>> ci = res.confidence_interval()
>>> contains_pop_mean = (ci.low < 1) & (ci.high > 1)
>>> contains_pop_mean.sum()
953
```

<!-- !! processed by numpydoc !! -->


# dask.array.stats.ttest_ind.html.md

# dask.array.stats.ttest_ind

### dask.array.stats.ttest_ind(a, b, axis=0, equal_var=True)

Calculate the T-test for the means of *two independent* samples of scores.

This docstring was copied from scipy.stats.ttest_ind.

Some inconsistencies with the Dask version may exist.

This is a test for the null hypothesis that 2 independent samples
have identical average (expected) values. This test assumes that the
populations have identical variances by default.

* **Parameters:**
  **a, b**
  : The arrays must have the same shape, except in the dimension
    corresponding to axis (the first, by default).

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **equal_var**
  : If True (default), perform a standard independent 2 sample test
    that assumes equal population variances [[1]](#rc139d793e52e-1).
    If False, perform Welch’s t-test, which does not assume equal
    population variance [[2]](#rc139d793e52e-2).
    <br/>
    #### Versionadded
    Added in version 0.11.0.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **alternative**
  : Defines the alternative hypothesis.
    The following options are available (default is ‘two-sided’):
    * ‘two-sided’: the means of the distributions underlying the samples
      are unequal.
    * ‘less’: the mean of the distribution underlying the first sample
      is less than the mean of the distribution underlying the second
      sample.
    * ‘greater’: the mean of the distribution underlying the first
      sample is greater than the mean of the distribution underlying
      the second sample.

  **trim**
  : If nonzero, performs a trimmed (Yuen’s) t-test.
    Defines the fraction of elements to be trimmed from each end of the
    input samples. If 0 (default), no elements will be trimmed from either
    side. The number of trimmed elements from each tail is the floor of the
    trim times the number of elements. Valid range is [0, .5).

  **method**
  : Defines the method used to compute the p-value. If method is an
    instance of PermutationMethod/MonteCarloMethod, the p-value is
    computed using
    scipy.stats.permutation_test/scipy.stats.monte_carlo_test with the
    provided configuration options and other appropriate settings.
    Otherwise, the p-value is computed by comparing the test statistic
    against a theoretical t-distribution.
    <br/>
    #### Versionadded
    Added in version 1.15.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **result**
  : An object with the following attributes:
    <br/>
    statistic
    : The t-statistic.
    <br/>
    pvalue
    : The p-value associated with the given alternative.
    <br/>
    df
    : The number of degrees of freedom used in calculation of the
      t-statistic.
      <br/>
      #### Versionadded
      Added in version 1.11.0.
    <br/>
    The object also has the following method:
    <br/>
    confidence_interval(confidence_level=0.95)
    : Computes a confidence interval around the difference in
      population means for the given confidence level.
      The confidence interval is returned in a `namedtuple` with
      fields `low` and `high`.
      <br/>
      #### Versionadded
      Added in version 1.11.0.

### Notes

Suppose we observe two independent samples, e.g. flower petal lengths, and
we are considering whether the two samples were drawn from the same
population (e.g. the same species of flower or two species with similar
petal characteristics) or two different populations.

The t-test quantifies the difference between the arithmetic means
of the two samples. The p-value quantifies the probability of observing
as or more extreme values assuming the null hypothesis, that the
samples are drawn from populations with the same population means, is true.
A p-value larger than a chosen threshold (e.g. 5% or 1%) indicates that
our observation is not so unlikely to have occurred by chance. Therefore,
we do not reject the null hypothesis of equal population means.
If the p-value is smaller than our threshold, then we have evidence
against the null hypothesis of equal population means.

By default, the p-value is determined by comparing the t-statistic of the
observed data against a theoretical t-distribution.

It is also possible to compute the test statistic using a permutation test by
passing `method=scipy.stats.PermutationMethod(n_resamples=permutations)`,
where `permutations` is the desired number of “permutations” to use in
forming the null distribution. When `1 < permutations < binom(n, k)`, where

* `k` is the number of observations in a,
* `n` is the total number of observations in a and b, and
* `binom(n, k)` is the binomial coefficient (`n` choose `k`),

the data are pooled (concatenated), randomly assigned to either group a
or b, and the t-statistic is calculated. This process is performed
repeatedly (`permutations` times), generating a distribution of the
t-statistic under the null hypothesis, and the t-statistic of the observed
data is compared to this distribution to determine the p-value.
Specifically, the p-value reported is the “achieved significance level”
(ASL) as defined in 4.4 of [[3]](#rc139d793e52e-3). Note that there are other ways of
estimating p-values using randomized permutation tests; for other
options, see the more general permutation_test.

When `permutations >= binom(n, k)`, an exact test is performed: the data
are partitioned between the groups in each distinct way exactly once.

The permutation test can be computationally expensive and not necessarily
more accurate than the analytical test, but it does not make strong
assumptions about the shape of the underlying distribution.

Use of trimming is commonly referred to as the trimmed t-test. At times
called Yuen’s t-test, this is an extension of Welch’s t-test, with the
difference being the use of winsorized means in calculation of the variance
and the trimmed sample size in calculation of the statistic. Trimming is
recommended if the underlying distribution is long-tailed or contaminated
with outliers [[4]](#rc139d793e52e-4).

The statistic is calculated as `(np.mean(a) - np.mean(b))/se`, where
`se` is the standard error. Therefore, the statistic will be positive
when the sample mean of a is greater than the sample mean of b and
negative when the sample mean of a is less than the sample mean of
b.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

ttest_ind has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ⛔    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

ttest_ind also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.
trim and method are incompatible with MArray input.The `confidence_interval` method of the output object is incompatible with JAX JIT.
See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

### Examples

```pycon
>>> import numpy as np
>>> from scipy import stats
>>> rng = np.random.default_rng()
```

Test with sample with identical means:

```pycon
>>> rvs1 = stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
>>> rvs2 = stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
>>> stats.ttest_ind(rvs1, rvs2)
TtestResult(statistic=-0.4390847099199348,
            pvalue=0.6606952038870015,
            df=998.0)
>>> stats.ttest_ind(rvs1, rvs2, equal_var=False)
TtestResult(statistic=-0.4390847099199348,
            pvalue=0.6606952553131064,
            df=997.4602304121448)
```

ttest_ind underestimates p for unequal variances:

```pycon
>>> rvs3 = stats.norm.rvs(loc=5, scale=20, size=500, random_state=rng)
>>> stats.ttest_ind(rvs1, rvs3)
TtestResult(statistic=-1.6370984482905417,
            pvalue=0.1019251574705033,
            df=998.0)
>>> stats.ttest_ind(rvs1, rvs3, equal_var=False)
TtestResult(statistic=-1.637098448290542,
            pvalue=0.10202110497954867,
            df=765.1098655246868)
```

When `n1 != n2`, the equal variance t-statistic is no longer equal to the
unequal variance t-statistic:

```pycon
>>> rvs4 = stats.norm.rvs(loc=5, scale=20, size=100, random_state=rng)
>>> stats.ttest_ind(rvs1, rvs4)
TtestResult(statistic=-1.9481646859513422,
            pvalue=0.05186270935842703,
            df=598.0)
>>> stats.ttest_ind(rvs1, rvs4, equal_var=False)
TtestResult(statistic=-1.3146566100751664,
            pvalue=0.1913495266513811,
            df=110.41349083985212)
```

T-test with different means, variance, and n:

```pycon
>>> rvs5 = stats.norm.rvs(loc=8, scale=20, size=100, random_state=rng)
>>> stats.ttest_ind(rvs1, rvs5)
TtestResult(statistic=-2.8415950600298774,
            pvalue=0.0046418707568707885,
            df=598.0)
>>> stats.ttest_ind(rvs1, rvs5, equal_var=False)
TtestResult(statistic=-1.8686598649188084,
            pvalue=0.06434714193919686,
            df=109.32167496550137)
```

Take these two samples, one of which has an extreme tail.

```pycon
>>> a = (56, 128.6, 12, 123.8, 64.34, 78, 763.3)
>>> b = (1.1, 2.9, 4.2)
```

Use the trim keyword to perform a trimmed (Yuen) t-test. For example,
using 20% trimming, `trim=.2`, the test will reduce the impact of one
(`np.floor(trim*len(a))`) element from each tail of sample a. It will
have no effect on sample b because `np.floor(trim*len(b))` is 0.

```pycon
>>> stats.ttest_ind(a, b, trim=.2)
TtestResult(statistic=3.4463884028073513,
            pvalue=0.01369338726499547,
            df=6.0)
```

<!-- !! processed by numpydoc !! -->


# dask.array.stats.ttest_rel.html.md

# dask.array.stats.ttest_rel

### dask.array.stats.ttest_rel(a, b, axis=0, nan_policy='propagate')

Calculate the t-test on TWO RELATED samples of scores, a and b.

This docstring was copied from scipy.stats.ttest_rel.

Some inconsistencies with the Dask version may exist.

This is a test for the null hypothesis that two related or
repeated samples have identical average (expected) values.

* **Parameters:**
  **a, b**
  : The arrays must have the same shape.

  **axis**
  : If an int, the axis of the input along which to compute the statistic.
    The statistic of each axis-slice (e.g. row) of the input will appear in a
    corresponding element of the output.
    If `None`, the input will be raveled before computing the statistic.

  **nan_policy**
  : Defines how to handle input NaNs.
    - `propagate`: if a NaN is present in the axis slice (e.g. row) along
      which the  statistic is computed, the corresponding entry of the output
      will be NaN.
    - `omit`: NaNs will be omitted when performing the calculation.
      If insufficient data remains in the axis slice along which the
      statistic is computed, the corresponding entry of the output will be
      NaN.
    - `raise`: if a NaN is present, a `ValueError` will be raised.

  **alternative**
  : Defines the alternative hypothesis.
    The following options are available (default is ‘two-sided’):
    * ‘two-sided’: the means of the distributions underlying the samples
      are unequal.
    * ‘less’: the mean of the distribution underlying the first sample
      is less than the mean of the distribution underlying the second
      sample.
    * ‘greater’: the mean of the distribution underlying the first
      sample is greater than the mean of the distribution underlying
      the second sample.
    <br/>
    #### Versionadded
    Added in version 1.6.0.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
* **Returns:**
  **result**
  : An object with the following attributes:
    <br/>
    statistic
    : The t-statistic.
    <br/>
    pvalue
    : The p-value associated with the given alternative.
    <br/>
    df
    : The number of degrees of freedom used in calculation of the
      t-statistic; this is one less than the size of the sample
      (`a.shape[axis]`).
      <br/>
      #### Versionadded
      Added in version 1.10.0.
    <br/>
    The object also has the following method:
    <br/>
    confidence_interval(confidence_level=0.95)
    : Computes a confidence interval around the difference in
      population means for the given confidence level.
      The confidence interval is returned in a `namedtuple` with
      fields low and high.
      <br/>
      #### Versionadded
      Added in version 1.10.0.

### Notes

Examples for use are scores of the same set of student in
different exams, or repeated sampling from the same units. The
test measures whether the average score differs significantly
across samples (e.g. exams). If we observe a large p-value, for
example greater than 0.05 or 0.1 then we cannot reject the null
hypothesis of identical average scores. If the p-value is smaller
than the threshold, e.g. 1%, 5% or 10%, then we reject the null
hypothesis of equal averages. Small p-values are associated with
large t-statistics.

The t-statistic is calculated as `np.mean(a - b)/se`, where `se` is the
standard error. Therefore, the t-statistic will be positive when the sample
mean of `a - b` is greater than zero and negative when the sample mean of
`a - b` is less than zero.

Beginning in SciPy 1.9, `np.matrix` inputs (not recommended for new
code) are converted to `np.ndarray` before the calculation is performed. In
this case, the output will be a scalar or `np.ndarray` of appropriate shape
rather than a 2D `np.matrix`. Similarly, while masked elements of masked
arrays are ignored, the output will be a scalar or `np.ndarray` rather than a
masked array with `mask=False`.

**Array API Standard Support**

ttest_rel has experimental support for Python Array API Standard compatible
backends in addition to NumPy. Please consider testing these features
by setting an environment variable `SCIPY_ARRAY_API=1` and providing
CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following
combinations of backend and device (or other capability) are supported.

| Library   | CPU   | GPU   |
|-----------|-------|-------|
| NumPy     | ✅    | n/a   |
| CuPy      | n/a   | ✅    |
| PyTorch   | ✅    | ⛔    |
| JAX       | ✅    | ✅    |
| Dask      | ✅    | n/a   |

ttest_rel also accepts [MArrays](https://mdhaber.github.io/marray/tutorial.html) backed by the backends indicated above; masked values will be treated as though they were not present.
The `confidence_interval` method of the output objectis incompatible with JAX JIT.
See [Support for the array API standard](https://docs.scipy.org/doc/scipy/dev/api-dev/array_api.html#dev-arrayapi) for more information.

### References

[https://en.wikipedia.org/wiki/T-test#Dependent_t-test_for_paired_samples](https://en.wikipedia.org/wiki/T-test#Dependent_t-test_for_paired_samples)

### Examples

```pycon
>>> import numpy as np
>>> from scipy import stats
>>> rng = np.random.default_rng()
```

```pycon
>>> rvs1 = stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
>>> rvs2 = (stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
...         + stats.norm.rvs(scale=0.2, size=500, random_state=rng))
>>> stats.ttest_rel(rvs1, rvs2)
TtestResult(statistic=-0.4549717054410304, pvalue=0.6493274702088672, df=499)
>>> rvs3 = (stats.norm.rvs(loc=8, scale=10, size=500, random_state=rng)
...         + stats.norm.rvs(scale=0.2, size=500, random_state=rng))
>>> stats.ttest_rel(rvs1, rvs3)
TtestResult(statistic=-5.879467544540889, pvalue=7.540777129099917e-09, df=499)
```

<!-- !! processed by numpydoc !! -->


# dask.array.std.html.md

# dask.array.std

### dask.array.std(a, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Compute the standard deviation along the specified axis.

This docstring was copied from numpy.std.

Some inconsistencies with the Dask version may exist.

Returns the standard deviation, a measure of the spread of a distribution,
of the array elements. The standard deviation is computed for the
flattened array by default, otherwise over the specified axis.

* **Parameters:**
  **a**
  : Calculate the standard deviation of these values.

  **axis**
  : Axis or axes along which the standard deviation is computed. The
    default is to compute the standard deviation of the flattened array.
    If this is a tuple of ints, a standard deviation is performed over
    multiple axes, instead of a single axis or all the axes as before.

  **dtype**
  : Type to use in computing the standard deviation. For arrays of
    integer type the default is float64, for arrays of float types it is
    the same as the array type.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape as the expected output but the type (of the calculated
    values) will be cast if necessary.
    See [Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type) for more details.

  **ddof**
  : Means Delta Degrees of Freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.
    By default ddof is zero. See Notes for details about use of ddof.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the std method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **where**
  : Elements to include in the standard deviation.
    See ~numpy.ufunc.reduce for details.
    <br/>
    #### Versionadded
    Added in version 1.20.0.

  **mean**
  : Provide the mean to prevent its recalculation. The mean should have
    a shape as if it was calculated with `keepdims=True`.
    The axis for the calculation of the mean should be the same as used in
    the call to this std function.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **correction**
  : Array API compatible name for the `ddof` parameter. Only one of them
    can be provided at the same time.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **standard_deviation**
  : If out is None, return a new array containing the standard deviation,
    otherwise return a reference to the output array.

#### SEE ALSO
[`var`](dask.array.var.md#dask.array.var), [`mean`](dask.array.mean.md#dask.array.mean), [`nanmean`](dask.array.nanmean.md#dask.array.nanmean), [`nanstd`](dask.array.nanstd.md#dask.array.nanstd), [`nanvar`](dask.array.nanvar.md#dask.array.nanvar)

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

There are several common variants of the array standard deviation
calculation. Assuming the input a is a one-dimensional NumPy array
and `mean` is either provided as an argument or computed as
`a.mean()`, NumPy computes the standard deviation of an array as:

```default
N = len(a)
d2 = abs(a - mean)**2  # abs is for complex `a`
var = d2.sum() / (N - ddof)  # note use of `ddof`
std = var**0.5
```

Different values of the argument ddof are useful in different
contexts. NumPy’s default `ddof=0` corresponds with the expression:

$$
\sqrt{\frac{\sum_i{|a_i - \bar{a}|^2 }}{N}}
$$

which is sometimes called the “population standard deviation” in the field
of statistics because it applies the definition of standard deviation to
a as if a were a complete population of possible observations.

Many other libraries define the standard deviation of an array
differently, e.g.:

$$
\sqrt{\frac{\sum_i{|a_i - \bar{a}|^2 }}{N - 1}}
$$

In statistics, the resulting quantity is sometimes called the “sample
standard deviation” because if a is a random sample from a larger
population, this calculation provides the square root of an unbiased
estimate of the variance of the population. The use of $N-1$ in the
denominator is often called “Bessel’s correction” because it corrects for
bias (toward lower values) in the variance estimate introduced when the
sample mean of a is used in place of the true mean of the population.
The resulting estimate of the standard deviation is still biased, but less
than it would have been without the correction. For this quantity, use
`ddof=1`.

Note that, for complex numbers, std takes the absolute
value before squaring, so that the result is always real and nonnegative.

For floating-point input, the standard deviation is computed using the same
precision the input has. Depending on the input data, this can cause
the results to be inaccurate, especially for float32 (see example below).
Specifying a higher-accuracy accumulator using the dtype keyword can
alleviate this issue.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, 4]])
>>> np.std(a)
1.1180339887498949 # may vary
>>> np.std(a, axis=0)
array([1.,  1.])
>>> np.std(a, axis=1)
array([0.5,  0.5])
```

In single precision, std() can be inaccurate:

```pycon
>>> a = np.zeros((2, 512*512), dtype=np.float32)
>>> a[0, :] = 1.0
>>> a[1, :] = 0.1
>>> np.std(a)
np.float32(0.45000005)
```

Computing the standard deviation in float64 is more accurate:

```pycon
>>> np.std(a, dtype=np.float64)
0.44999999925494177 # may vary
```

Specifying a where argument:

```pycon
>>> a = np.array([[14, 8, 11, 10], [7, 9, 10, 11], [10, 15, 5, 10]])
>>> np.std(a)
2.614064523559687 # may vary
>>> np.std(a, where=[[True], [True], [False]])
2.0
```

Using the mean keyword to save computation time:

```pycon
>>> import numpy as np
>>> from timeit import timeit
>>> a = np.array([[14, 8, 11, 10], [7, 9, 10, 11], [10, 15, 5, 10]])
>>> mean = np.mean(a, axis=1, keepdims=True)
>>>
>>> g = globals()
>>> n = 10000
>>> t1 = timeit("std = np.std(a, axis=1, mean=mean)", globals=g, number=n)
>>> t2 = timeit("std = np.std(a, axis=1)", globals=g, number=n)
>>> print(f'Percentage execution time saved {100*(t2-t1)/t2:.0f}%')

Percentage execution time saved 30%
```

<!-- !! processed by numpydoc !! -->


# dask.array.store.html.md

# dask.array.store

### dask.array.store(sources: [Array](dask.array.Array.md#dask.array.Array) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[Array](dask.array.Array.md#dask.array.Array)], targets: ArrayLike | [Delayed](../delayed-api.md#dask.delayed.Delayed) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[ArrayLike | [Delayed](../delayed-api.md#dask.delayed.Delayed)], lock: [bool](https://docs.python.org/3/library/functions.html#bool) | lock = True, regions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[slice](https://docs.python.org/3/library/functions.html#slice), ...] | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[slice](https://docs.python.org/3/library/functions.html#slice), ...]] | [None](https://docs.python.org/3/library/constants.html#None) = None, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True, return_stored: [bool](https://docs.python.org/3/library/functions.html#bool) = False, load_stored: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Store dask arrays in array-like objects, overwrite data in target

This stores dask arrays into object that supports numpy-style setitem
indexing.  It stores values chunk by chunk so that it does not have to
fill up memory.  For best performance you can align the block size of
the storage target with the block size of your array.

If your data fits in memory then you may prefer calling
`np.array(myarray)` instead.

* **Parameters:**
  **sources: Array or collection of Arrays**

  **targets: array-like or Delayed or collection of array-likes and/or Delayeds**
  : These should support setitem syntax `target[10:20] = ...`.
    If sources is a single item, targets must be a single item; if sources is a
    collection of arrays, targets must be a matching collection.

  **lock: boolean or threading.Lock, optional**
  : Whether or not to lock the data stores while storing.
    Pass True (lock each file individually), False (don’t lock) or a
    particular [`threading.Lock`](https://docs.python.org/3/library/threading.html#threading.Lock) object to be shared among all writes.

  **regions: tuple of slices or collection of tuples of slices, optional**
  : Each `region` tuple in `regions` should be such that
    `target[region].shape = source.shape`
    for the corresponding source and target in sources and targets,
    respectively. If this is a tuple, the contents will be assumed to be
    slices, so do not provide a tuple of tuples.

  **compute: boolean, optional**
  : If true compute immediately; return [`dask.delayed.Delayed`](../delayed-api.md#dask.delayed.Delayed) otherwise.

  **return_stored: boolean, optional**
  : Optionally return the stored result (default False).

  **load_stored: boolean, optional**
  : Optionally return the stored result, loaded in to memory (default None).
    If None, `load_stored` is True if `return_stored` is True and
    `compute` is False. *This is an advanced option.*
    When False, store will return the appropriate `target` for each chunk that is stored.
    Directly computing this result is not what you want.
    Instead, you can use the returned `target` to execute followup operations to the store.

  **kwargs:**
  : Parameters passed to compute/persist (only used if compute=True)
* **Returns:**
  If return_stored=True
  : tuple of Arrays

  If return_stored=False and compute=True
  : None

  If return_stored=False and compute=False
  : Delayed

### Examples

```pycon
>>> import h5py
>>> f = h5py.File('myfile.hdf5', mode='a')
>>> dset = f.create_dataset('/data', shape=x.shape,
...                                  chunks=x.chunks,
...                                  dtype='f8')
```

```pycon
>>> store(x, dset)
```

Alternatively store many arrays at the same time

```pycon
>>> store([x, y, z], [dset1, dset2, dset3])
```

<!-- !! processed by numpydoc !! -->


# dask.array.subtract.html.md

# dask.array.subtract

### dask.array.subtract(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'subtract'>*

This docstring was copied from numpy.subtract.

Some inconsistencies with the Dask version may exist.

Subtract arguments, element-wise.

* **Parameters:**
  **x1, x2**
  : The arrays to be subtracted from each other.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The difference of x1 and x2, element-wise.
    This is a scalar if both x1 and x2 are scalars.

### Notes

Equivalent to `x1 - x2` in terms of array broadcasting.

### Examples

```pycon
>>> import numpy as np
>>> np.subtract(1.0, 4.0)
-3.0
```

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> np.subtract(x1, x2)
array([[ 0.,  0.,  0.],
       [ 3.,  3.,  3.],
       [ 6.,  6.,  6.]])
```

The `-` operator can be used as a shorthand for `np.subtract` on
ndarrays.

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> x1 - x2
array([[0., 0., 0.],
       [3., 3., 3.],
       [6., 6., 6.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.sum.html.md

# dask.array.sum

### dask.array.sum(a, axis=None, dtype=None, keepdims=False, split_every=None, out=None)

Sum of array elements over a given axis.

This docstring was copied from numpy.sum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Elements to sum.

  **axis**
  : Axis or axes along which a sum is performed.  The default,
    axis=None, will sum all of the elements of the input array.  If
    axis is negative it counts from the last to the first axis. If
    axis is a tuple of ints, a sum is performed on all of the axes
    specified in the tuple instead of a single axis or all the axes as
    before.

  **dtype**
  : The type of the returned array and of the accumulator in which the
    elements are summed.  The dtype of a is used by default unless a
    has an integer dtype of less precision than the default platform
    integer.  In that case, if a is signed then the platform integer
    is used while if a is unsigned then an unsigned integer of the
    same precision as the platform integer is used.

  **out**
  : Alternative output array in which to place the result. It must have
    the same shape as the expected output, but the type of the output
    values will be cast if necessary.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the sum method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **initial**
  : Starting value for the sum. See ~numpy.ufunc.reduce for details.

  **where**
  : Elements to include in the sum. See ~numpy.ufunc.reduce for details.
* **Returns:**
  **sum_along_axis**
  : An array with the same shape as a, with the specified
    axis removed.   If a is a 0-d array, or if axis is None, a scalar
    is returned.  If an output array is specified, a reference to
    out is returned.

#### SEE ALSO
`ndarray.sum`
: Equivalent method.

[`add`](dask.array.add.md#dask.array.add)
: `numpy.add.reduce` equivalent function.

[`cumsum`](dask.array.cumsum.md#dask.array.cumsum)
: Cumulative sum of array elements.

`trapezoid`
: Integration of array values using composite trapezoidal rule.

[`mean`](dask.array.mean.md#dask.array.mean), [`average`](dask.array.average.md#dask.array.average)

### Notes

Arithmetic is modular when using integer types, and no error is
raised on overflow.

The sum of an empty array is the neutral element 0:

```pycon
>>> np.sum([])
0.0
```

For floating point numbers the numerical precision of sum (and
`np.add.reduce`) is in general limited by directly adding each number
individually to the result causing rounding errors in every step.
However, often numpy will use a  numerically better approach (partial
pairwise summation) leading to improved precision in many use-cases.
This improved precision is always provided when no `axis` is given.
When `axis` is given, it will depend on which axis is summed.
Technically, to provide the best speed possible, the improved precision
is only used when the summation is along the fast axis in memory.
Note that the exact precision may vary depending on other parameters.
In contrast to NumPy, Python’s `math.fsum` function uses a slower but
more precise approach to summation.
Especially when summing a large number of lower precision floating point
numbers, such as `float32`, numerical errors can become significant.
In such cases it can be advisable to use dtype=np.float64 to use a higher
precision for the output.

### Examples

```pycon
>>> import numpy as np
>>> np.sum([0.5, 1.5])
2.0
>>> np.sum([0.5, 0.7, 0.2, 1.5], dtype=np.int32)
np.int32(1)
>>> np.sum([[0, 1], [0, 5]])
6
>>> np.sum([[0, 1], [0, 5]], axis=0)
array([0, 6])
>>> np.sum([[0, 1], [0, 5]], axis=1)
array([1, 5])
>>> np.sum([[0, 1], [np.nan, 5]], where=[False, True], axis=1)
array([1., 5.])
```

If the accumulator is too small, overflow occurs:

```pycon
>>> np.ones(128, dtype=np.int8).sum(dtype=np.int8)
np.int8(-128)
```

You can also start the sum with a value other than zero:

```pycon
>>> np.sum([10], initial=5)
15
```

<!-- !! processed by numpydoc !! -->


# dask.array.swapaxes.html.md

# dask.array.swapaxes

### dask.array.swapaxes(a, axis1, axis2)

Interchange two axes of an array.

This docstring was copied from numpy.swapaxes.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **a**
  : Input array.

  **axis1**
  : First axis.

  **axis2**
  : Second axis.
* **Returns:**
  **a_swapped**
  : For NumPy >= 1.10.0, if a is an ndarray, then a view of a is
    returned; otherwise a new array is created. For earlier NumPy
    versions a view of a is returned only if the order of the
    axes is changed, otherwise the input array is returned.

### Examples

```pycon
>>> import numpy as np
>>> x = np.array([[1,2,3]])
>>> np.swapaxes(x,0,1)
array([[1],
       [2],
       [3]])
```

```pycon
>>> x = np.array([[[0,1],[2,3]],[[4,5],[6,7]]])
>>> x
array([[[0, 1],
        [2, 3]],
       [[4, 5],
        [6, 7]]])
```

```pycon
>>> np.swapaxes(x,0,2)
array([[[0, 4],
        [2, 6]],
       [[1, 5],
        [3, 7]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.take.html.md

# dask.array.take

### dask.array.take(a, indices, axis=0)

Take elements from an array along an axis.

This docstring was copied from numpy.take.

Some inconsistencies with the Dask version may exist.

When axis is not None, this function does the same thing as “fancy”
indexing (indexing arrays using arrays); however, it can be easier to use
if you need elements along a given axis. A call such as
`np.take(arr, indices, axis=3)` is equivalent to
`arr[:,:,:,indices,...]`.

Explained without fancy indexing, this is equivalent to the following use
of ndindex, which sets each of `ii`, `jj`, and `kk` to a tuple of
indices:

```default
Ni, Nk = a.shape[:axis], a.shape[axis+1:]
Nj = indices.shape
for ii in ndindex(Ni):
    for jj in ndindex(Nj):
        for kk in ndindex(Nk):
            out[ii + jj + kk] = a[ii + (indices[jj],) + kk]
```

* **Parameters:**
  **a**
  : The source array.

  **indices**
  : The indices of the values to extract.
    Also allow scalars for indices.

  **axis**
  : The axis over which to select values. By default, the flattened
    input array is used.

  **out**
  : If provided, the result will be placed in this array. It should
    be of the appropriate shape and dtype. Note that out is always
    buffered if mode=’raise’; use other modes for better performance.

  **mode**
  : Specifies how out-of-bounds indices will behave.
    * ‘raise’ – raise an error (default)
    * ‘wrap’ – wrap around
    * ‘clip’ – clip to the range
    <br/>
    ‘clip’ mode means that all indices that are too large are replaced
    by the index that addresses the last element along that axis. Note
    that this disables indexing with negative numbers.
* **Returns:**
  **out**
  : The returned array has the same type as a.

#### SEE ALSO
[`compress`](dask.array.compress.md#dask.array.compress)
: Take elements using a boolean mask

`ndarray.take`
: equivalent method

`take_along_axis`
: Take elements by matching the array and the index arrays

### Notes

By eliminating the inner loop in the description above, and using s_ to
build simple slice objects, take can be expressed  in terms of applying
fancy indexing to each 1-d slice:

```default
Ni, Nk = a.shape[:axis], a.shape[axis+1:]
for ii in ndindex(Ni):
    for kk in ndindex(Nk):
        out[ii + s_[...,] + kk] = a[ii + s_[:,] + kk][indices]
```

For this reason, it is equivalent to (but faster than) the following use
of apply_along_axis:

```default
out = np.apply_along_axis(lambda a_1d: a_1d[indices], axis, a)
```

### Examples

```pycon
>>> import numpy as np
>>> a = [4, 3, 5, 7, 6, 8]
>>> indices = [0, 1, 4]
>>> np.take(a, indices)
array([4, 3, 6])
```

In this example if a is an ndarray, “fancy” indexing can be used.

```pycon
>>> a = np.array(a)
>>> a[indices]
array([4, 3, 6])
```

If indices is not one dimensional, the output also has these dimensions.

```pycon
>>> np.take(a, [[0, 1], [2, 3]])
array([[4, 3],
       [5, 7]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.tan.html.md

# dask.array.tan

### dask.array.tan(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'tan'>*

This docstring was copied from numpy.tan.

Some inconsistencies with the Dask version may exist.

Compute tangent element-wise.

Equivalent to `np.sin(x)/np.cos(x)` element-wise.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding tangent values.
    This is a scalar if x is a scalar.

### Notes

If out is provided, the function writes the result into it,
and returns a reference to out.  (See Examples)

### References

M. Abramowitz and I. A. Stegun, Handbook of Mathematical Functions.
New York, NY: Dover, 1972.

### Examples

```pycon
>>> import numpy as np
>>> from math import pi
>>> np.tan(np.array([-pi,pi/2,pi]))
array([  1.22460635e-16,   1.63317787e+16,  -1.22460635e-16])
>>>
>>> # Example of providing the optional output parameter illustrating
>>> # that what is returned is a reference to said parameter
>>> out1 = np.array([0], dtype=np.float64)
>>> out2 = np.cos([0.1], out1)
>>> out2 is out1
True
>>>
>>> # Example of ValueError due to provision of shape mis-matched `out`
>>> np.cos(np.zeros((3,3)),np.zeros((2,2)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (3,3) (2,2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.tanh.html.md

# dask.array.tanh

### dask.array.tanh(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'tanh'>*

This docstring was copied from numpy.tanh.

Some inconsistencies with the Dask version may exist.

Hyperbolic tangent, element-wise.

Equivalent to `np.sinh(x)/np.cosh(x)` or `-1j * np.tan(1j*x)`.

* **Parameters:**
  **x**
  : Input array.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The corresponding hyperbolic tangent values.
    This is a scalar if x is a scalar.

### Notes

If out is provided, the function writes the result into it,
and returns a reference to out.  (See Examples)

### References

### Examples

```pycon
>>> import numpy as np
>>> np.tanh((0, np.pi*1j, np.pi*1j/2))
array([ 0. +0.00000000e+00j,  0. -1.22460635e-16j,  0. +1.63317787e+16j])
```

```pycon
>>> # Example of providing the optional output parameter illustrating
>>> # that what is returned is a reference to said parameter
>>> out1 = np.array([0], dtype=np.float64)
>>> out2 = np.tanh([0.1], out1)
>>> out2 is out1
True
```

```pycon
>>> # Example of ValueError due to provision of shape mis-matched `out`
>>> np.tanh(np.zeros((3,3)),np.zeros((2,2)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (3,3) (2,2)
```

<!-- !! processed by numpydoc !! -->


# dask.array.tensordot.html.md

# dask.array.tensordot

### dask.array.tensordot(lhs, rhs, axes=2)

Compute tensor dot product along specified axes.

This docstring was copied from numpy.tensordot.

Some inconsistencies with the Dask version may exist.

Given two tensors, a and b, and an array_like object containing
two array_like objects, `(a_axes, b_axes)`, sum the products of
a’s and b’s elements (components) over the axes specified by
`a_axes` and `b_axes`. The third argument can be a single non-negative
integer_like scalar, `N`; if it is such, then the last `N` dimensions
of a and the first `N` dimensions of b are summed over.

* **Parameters:**
  **a, b**
  : Tensors to “dot”.

  **axes**
  : * integer_like
      If an int N, sum over the last N axes of a and the first N axes
      of b in order. The sizes of the corresponding axes must match.
    * (2,) array_like
      Or, a list of axes to be summed over, first sequence applying to a,
      second to b. Both elements array_like must be of the same length.
      Each axis may appear at most once; repeated axes are not allowed.
      For example, `axes=([1, 1], [0, 0])` is invalid.

  **Returns**

  **——-**

  **output**
  : The tensor dot product of the input.

#### SEE ALSO
[`dot`](dask.array.dot.md#dask.array.dot), [`einsum`](dask.array.einsum.md#dask.array.einsum)

### Notes

Three common use cases are:
: * `axes = 0` : tensor product $a\otimes b$
  * `axes = 1` : tensor dot product $a\cdot b$
  * `axes = 2` : (default) tensor double contraction $a:b$

When axes is integer_like, the sequence of axes for evaluation
will be: from the -Nth axis to the -1th axis in a,
and from the 0th axis to (N-1)th axis in b.
For example, `axes = 2` is the equal to
`axes = [[-2, -1], [0, 1]]`.
When N-1 is smaller than 0, or when -N is larger than -1,
the element of a and b are defined as the axes.

When there is more than one axis to sum over - and they are not the last
(first) axes of a (b) - the argument axes should consist of
two sequences of the same length, with the first axis to sum over given
first in both sequences, the second axis second, and so forth.
The calculation can be referred to `numpy.einsum`.

For example, if `a.shape == (2, 3, 4)` and `b.shape == (3, 4, 5)`,
then `axes=([1, 2], [0, 1])` sums over the `(3, 4)` dimensions of
both arrays and produces an output of shape `(2, 5)`.

Each summation axis corresponds to a distinct contraction index; repeating
an axis (for example `axes=([1, 1], [0, 0])`) is invalid.

The shape of the result consists of the non-contracted axes of the
first tensor, followed by the non-contracted axes of the second.

### Examples

An example on integer_like:

```pycon
>>> a_0 = np.array([[1, 2], [3, 4]])
>>> b_0 = np.array([[5, 6], [7, 8]])
>>> c_0 = np.tensordot(a_0, b_0, axes=0)
>>> c_0.shape
(2, 2, 2, 2)
>>> c_0
array([[[[ 5,  6],
         [ 7,  8]],
        [[10, 12],
         [14, 16]]],
       [[[15, 18],
         [21, 24]],
        [[20, 24],
         [28, 32]]]])
```

An example on array_like:

```pycon
>>> a = np.arange(60.).reshape(3,4,5)
>>> b = np.arange(24.).reshape(4,3,2)
>>> c = np.tensordot(a,b, axes=([1,0],[0,1]))
>>> c.shape
(5, 2)
>>> c
array([[4400., 4730.],
       [4532., 4874.],
       [4664., 5018.],
       [4796., 5162.],
       [4928., 5306.]])
```

A slower but equivalent way of computing the same…

```pycon
>>> d = np.zeros((5,2))
>>> for i in range(5):
...   for j in range(2):
...     for k in range(3):
...       for n in range(4):
...         d[i,j] += a[k,n,i] * b[n,k,j]
>>> c == d
array([[ True,  True],
       [ True,  True],
       [ True,  True],
       [ True,  True],
       [ True,  True]])
```

An extended example taking advantage of the overloading of + and \*:

```pycon
>>> a = np.array(range(1, 9)).reshape((2, 2, 2))
>>> A = np.array(('a', 'b', 'c', 'd'), dtype=np.object_)
>>> A = A.reshape((2, 2))
>>> a; A
array([[[1, 2],
        [3, 4]],
       [[5, 6],
        [7, 8]]])
array([['a', 'b'],
       ['c', 'd']], dtype=object)
```

```pycon
>>> np.tensordot(a, A) # third argument default is 2 for double-contraction
array(['abbcccdddd', 'aaaaabbbbbbcccccccdddddddd'], dtype=object)
```

```pycon
>>> np.tensordot(a, A, 1)
array([[['acc', 'bdd'],
        ['aaacccc', 'bbbdddd']],
       [['aaaaacccccc', 'bbbbbdddddd'],
        ['aaaaaaacccccccc', 'bbbbbbbdddddddd']]], dtype=object)
```

```pycon
>>> np.tensordot(a, A, 0) # tensor product (result too long to incl.)
array([[[[['a', 'b'],
          ['c', 'd']],
          ...
```

```pycon
>>> np.tensordot(a, A, (0, 1))
array([[['abbbbb', 'cddddd'],
        ['aabbbbbb', 'ccdddddd']],
       [['aaabbbbbbb', 'cccddddddd'],
        ['aaaabbbbbbbb', 'ccccdddddddd']]], dtype=object)
```

```pycon
>>> np.tensordot(a, A, (2, 1))
array([[['abb', 'cdd'],
        ['aaabbbb', 'cccdddd']],
       [['aaaaabbbbbb', 'cccccdddddd'],
        ['aaaaaaabbbbbbbb', 'cccccccdddddddd']]], dtype=object)
```

```pycon
>>> np.tensordot(a, A, ((0, 1), (0, 1)))
array(['abbbcccccddddddd', 'aabbbbccccccdddddddd'], dtype=object)
```

```pycon
>>> np.tensordot(a, A, ((2, 1), (1, 0)))
array(['acccbbdddd', 'aaaaacccccccbbbbbbdddddddd'], dtype=object)
```

<!-- !! processed by numpydoc !! -->


# dask.array.tile.html.md

# dask.array.tile

### dask.array.tile(A, reps)

Construct an array by repeating A the number of times given by reps.

This docstring was copied from numpy.tile.

Some inconsistencies with the Dask version may exist.

If reps has length `d`, the result will have dimension of
`max(d, A.ndim)`.

If `A.ndim < d`, A is promoted to be d-dimensional by prepending new
axes. So a shape (3,) array is promoted to (1, 3) for 2-D replication,
or shape (1, 1, 3) for 3-D replication. If this is not the desired
behavior, promote A to d-dimensions manually before calling this
function.

If `A.ndim > d`, reps is promoted to A.ndim by prepending 1’s to it.
Thus for an A of shape (2, 3, 4, 5), a reps of (2, 2) is treated as
(1, 1, 2, 2).

Note : Although tile may be used for broadcasting, it is strongly
recommended to use numpy’s broadcasting operations and functions.

* **Parameters:**
  **A**
  : The input array.

  **reps**
  : The number of repetitions of A along each axis.
* **Returns:**
  **c**
  : The tiled output array.

#### SEE ALSO
[`repeat`](dask.array.repeat.md#dask.array.repeat)
: Repeat elements of an array.

[`broadcast_to`](dask.array.broadcast_to.md#dask.array.broadcast_to)
: Broadcast an array to a new shape

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([0, 1, 2])
>>> np.tile(a, 2)
array([0, 1, 2, 0, 1, 2])
>>> np.tile(a, (2, 2))
array([[0, 1, 2, 0, 1, 2],
       [0, 1, 2, 0, 1, 2]])
>>> np.tile(a, (2, 1, 2))
array([[[0, 1, 2, 0, 1, 2]],
       [[0, 1, 2, 0, 1, 2]]])
```

```pycon
>>> b = np.array([[1, 2], [3, 4]])
>>> np.tile(b, 2)
array([[1, 2, 1, 2],
       [3, 4, 3, 4]])
>>> np.tile(b, (2, 1))
array([[1, 2],
       [3, 4],
       [1, 2],
       [3, 4]])
```

```pycon
>>> c = np.array([1,2,3,4])
>>> np.tile(c,(4,1))
array([[1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.to_hdf5.html.md

# dask.array.to_hdf5

### dask.array.to_hdf5(filename, \*args, chunks=True, \*\*kwargs)

Store arrays in HDF5 file

This saves several dask arrays into several datapaths in an HDF5 file.
It creates the necessary datasets and handles clean file opening/closing.

* **Parameters:**
  **chunks: tuple or \`\`True\`\`**
  : Chunk shape, or `True` to pass the chunks from the dask array.
    Defaults to `True`.

#### SEE ALSO
`da.store`

`h5py.File.create_dataset`

### Examples

```pycon
>>> da.to_hdf5('myfile.hdf5', '/x', x)
```

or

```pycon
>>> da.to_hdf5('myfile.hdf5', {'/x': x, '/y': y})
```

Optionally provide arguments as though to `h5py.File.create_dataset`

```pycon
>>> da.to_hdf5('myfile.hdf5', '/x', x, compression='lzf', shuffle=True)
```

```pycon
>>> da.to_hdf5('myfile.hdf5', '/x', x, chunks=(10,20,30))
```

This can also be used as a method on a single Array

```pycon
>>> x.to_hdf5('myfile.hdf5', '/x')
```

<!-- !! processed by numpydoc !! -->


# dask.array.to_npy_stack.html.md

# dask.array.to_npy_stack

### dask.array.to_npy_stack(dirname, x, axis=0)

Write dask array to a stack of .npy files

This partitions the dask.array along one axis and stores each block along
that axis as a single .npy file in the specified directory

#### SEE ALSO
[`from_npy_stack`](dask.array.from_npy_stack.md#dask.array.from_npy_stack)

### Examples

```pycon
>>> x = da.ones((5, 10, 10), chunks=(2, 4, 4))
>>> da.to_npy_stack('data/', x, axis=0)
```

The `.npy` files store numpy arrays for `x[0:2], x[2:4], and x[4:5]`
respectively, as is specified by the chunk size along the zeroth axis:

```default
$ tree data/
data/
|-- 0.npy
|-- 1.npy
|-- 2.npy
|-- info
```

The `info` file stores the dtype, chunks, and axis information of the array.
You can load these stacks with the [`dask.array.from_npy_stack()`](dask.array.from_npy_stack.md#dask.array.from_npy_stack) function.

```pycon
>>> y = da.from_npy_stack('data/')
```

<!-- !! processed by numpydoc !! -->


# dask.array.to_tiledb.html.md

# dask.array.to_tiledb

### dask.array.to_tiledb(darray, uri, compute=True, return_stored=False, storage_options=None, key=None, \*\*kwargs)

Save array to the TileDB storage format

Save ‘array’ using the TileDB storage manager, to any TileDB-supported URI,
including local disk, S3, or HDFS.

See [https://docs.tiledb.io](https://docs.tiledb.io) for more information about TileDB.

* **Parameters:**
  **darray: dask.array**
  : A dask array to write.

  **uri:**
  : Any supported TileDB storage location.

  **storage_options: dict**
  : Dict containing any configuration options for the TileDB backend.
    see [https://docs.tiledb.io/en/stable/tutorials/config.html](https://docs.tiledb.io/en/stable/tutorials/config.html)

  **compute, return_stored: see \`\`store()\`\`**

  **key: str or None**
  : Encryption key
* **Returns:**
  None
  : Unless `return_stored` is set to `True` (`False` by default)

### Notes

TileDB only supports regularly-chunked arrays.
TileDB [tile extents](http://docs.dask.org/en/latest/array-chunks.html) correspond to form 2 of the dask
[chunk specification](https://docs.tiledb.io/en/stable/tutorials/tiling-dense.html), and the conversion is
done automatically for supported arrays.

### Examples

```pycon
>>> import dask.array as da, tempfile
>>> uri = tempfile.NamedTemporaryFile().name
>>> data = da.random.random(5,5)
>>> da.to_tiledb(data, uri)
>>> import tiledb
>>> tdb_ar = tiledb.open(uri)
>>> all(tdb_ar == data)
True
```

<!-- !! processed by numpydoc !! -->


# dask.array.to_zarr.html.md

# dask.array.to_zarr

### dask.array.to_zarr(arr, url, component=None, storage_options=None, region=None, compute=True, return_stored=False, mode='a', \*\*zarr_array_kwargs)

Save array to the zarr storage format

See [https://zarr.readthedocs.io](https://zarr.readthedocs.io) for details about the format.

* **Parameters:**
  **arr: dask.array**
  : Data to store

  **url: Zarr Array or str or MutableMapping**
  : Location of the data. A URL can include a protocol specifier like s3://
    for remote data. Can also be any MutableMapping instance, which should
    be serializable if used in multiple processes.

  **component: str or None**
  : If the location is a zarr group rather than an array, this is the
    subcomponent that should be created/over-written. If both component
    and ‘name’ in zarr_array_kwargs are specified, component takes
    precedence. This will change in a future version.

  **storage_options: dict**
  : Any additional parameters for the storage backend (ignored for local
    paths)

  **overwrite: bool**
  : If given array already exists, overwrite=False will cause an error,
    where overwrite=True will replace the existing data. Deprecated, please
    add to zarr_kwargs

  **region: tuple of slices or None**
  : The region of data that should be written if `url` is a zarr.Array.
    Not to be used with other types of `url`.

  **compute: bool**
  : See [`store()`](dask.array.store.md#dask.array.store) for more details.

  **return_stored: bool**
  : See [`store()`](dask.array.store.md#dask.array.store) for more details.

  **mode: Literal[“r+”, “a”, “w”, “w-“]**
  : Keyword argument mode passed to the storage backend when creating a zarr
    store from a URL string. Only used when `url` is a string (not when
    `url` is already a zarr.Array or MutableMapping instance).
    <br/>
    Common options include:
    : - `'r+'`: Read/write, must exist
      - `'a'`: Read/write, create if doesn’t exist (default)
      - `'w'`: Create, remove existing data if present
      - `'w-'`: Create, fail if exists

  **\*\*zarr_array_kwargs:**
  : #### Deprecated
    Deprecated since version 2025.12.0: Passing storage io-related arguments via `**kwargs` is deprecated.
    Please use the `mode` parameter instead when using `**kwargs` with the
    `mode` keys and corresponding values. `read_only` is not allowed anymore
    and will not have an effect.
    <br/>
    Keyword arguments passed to `Group.create_array()` (for zarr v3, where Group is a zarr group) or
    [`zarr.create()`](https://zarr.readthedocs.io/en/latest/api/zarr/functions/create/#zarr.create) (for zarr v2). This function automatically sets
    `shape`, `chunks`, and `dtype` based on the dask array, but these
    can be overridden.
    <br/>
    Common options include:
    - `compressor`: Compression algorithm (e.g., `zarr.Blosc()`)
    - `filters`: List of filters to apply
    - `fill_value`: Value to use for uninitialized portions
    - `order`: Memory layout (‘C’ or ‘F’)
    - `dimension_separator`: Separator for chunk keys (‘/’ or ‘.’)
    <br/>
    For the complete list of available arguments, see the zarr documentation:
    - zarr v3: [https://zarr.readthedocs.io/en/latest/api/zarr/group/#zarr.Group.create_array](https://zarr.readthedocs.io/en/latest/api/zarr/group/#zarr.Group.create_array)
    - zarr v2: [https://zarr.readthedocs.io/en/stable/api/zarr/create/#zarr.create](https://zarr.readthedocs.io/en/stable/api/zarr/create/#zarr.create)
* **Raises:**
  ValueError
  : If `arr` has unknown chunk sizes, which is not supported by Zarr.
    If `region` is specified and `url` is not a zarr.Array
    If `mode` is specified as r.

#### SEE ALSO
[`dask.array.store`](dask.array.store.md#dask.array.store)

[`dask.array.Array.compute_chunk_sizes`](dask.array.Array.compute_chunk_sizes.md#dask.array.Array.compute_chunk_sizes)

<!-- !! processed by numpydoc !! -->


# dask.array.topk.html.md

# dask.array.topk

### dask.array.topk(a, k, axis=-1, split_every=None)

Extract the k largest elements from a on the given axis,
and return them sorted from largest to smallest.
If k is negative, extract the -k smallest elements instead,
and return them sorted from smallest to largest.

This performs best when `k` is much smaller than the chunk size. All
results will be returned in a single chunk along the given axis.

* **Parameters:**
  **x: Array**
  : Data being sorted

  **k: int**

  **axis: int, optional**

  **split_every: int >=2, optional**
  : See `reduce()`. This parameter becomes very important when k is
    on the same order of magnitude of the chunk size or more, as it
    prevents getting the whole or a significant portion of the input array
    in memory all at once, with a negative impact on network transfer
    too when running on distributed.
* **Returns:**
  Selection of x with size abs(k) along the given axis.

### Examples

```pycon
>>> import dask.array as da
>>> x = np.array([5, 1, 3, 6])
>>> d = da.from_array(x, chunks=2)
>>> d.topk(2).compute()
array([6, 5])
>>> d.topk(-2).compute()
array([1, 3])
```

<!-- !! processed by numpydoc !! -->


# dask.array.trace.html.md

# dask.array.trace

### dask.array.trace(a, offset=0, axis1=0, axis2=1, dtype=None)

Return the sum along diagonals of the array.

This docstring was copied from numpy.trace.

Some inconsistencies with the Dask version may exist.

If a is 2-D, the sum along its diagonal with the given offset
is returned, i.e., the sum of elements `a[i,i+offset]` for all i.

If a has more than two dimensions, then the axes specified by axis1 and
axis2 are used to determine the 2-D sub-arrays whose traces are returned.
The shape of the resulting array is the same as that of a with axis1
and axis2 removed.

* **Parameters:**
  **a**
  : Input array, from which the diagonals are taken.

  **offset**
  : Offset of the diagonal from the main diagonal. Can be both positive
    and negative. Defaults to 0.

  **axis1, axis2**
  : Axes to be used as the first and second axis of the 2-D sub-arrays
    from which the diagonals should be taken. Defaults are the first two
    axes of a.

  **dtype**
  : Determines the data-type of the returned array and of the accumulator
    where the elements are summed. If dtype has the value None and a is
    of integer type of precision less than the default integer
    precision, then the default integer precision is used. Otherwise,
    the precision is the same as that of a.

  **out**
  : Array into which the output is placed. Its type is preserved and
    it must be of the right shape to hold the output.
* **Returns:**
  **sum_along_diagonals**
  : If a is 2-D, the sum along the diagonal is returned.  If a has
    larger dimensions, then an array of sums along diagonals is returned.

#### SEE ALSO
[`diag`](dask.array.diag.md#dask.array.diag), [`diagonal`](dask.array.diagonal.md#dask.array.diagonal), `diagflat`

### Examples

```pycon
>>> import numpy as np
>>> np.trace(np.eye(3))
3.0
>>> a = np.arange(8).reshape((2,2,2))
>>> np.trace(a)
array([6, 8])
```

```pycon
>>> a = np.arange(24).reshape((2,2,2,3))
>>> np.trace(a).shape
(2, 3)
```

<!-- !! processed by numpydoc !! -->


# dask.array.transpose.html.md

# dask.array.transpose

### dask.array.transpose(a, axes=None)

Returns an array with axes transposed.

This docstring was copied from numpy.transpose.

Some inconsistencies with the Dask version may exist.

For a 1-D array, this returns an unchanged view of the original array, as a
transposed vector is simply the same vector.
To convert a 1-D array into a 2-D column vector, an additional dimension
must be added, e.g., `np.atleast_2d(a).T` achieves this, as does
`a[:, np.newaxis]`.
For a 2-D array, this is the standard matrix transpose.
For an n-D array, if axes are given, their order indicates how the
axes are permuted (see Examples). If axes are not provided, then
`transpose(a).shape == a.shape[::-1]`.

* **Parameters:**
  **a**
  : Input array.

  **axes**
  : If specified, it must be a tuple or list which contains a permutation
    of [0, 1, …, N-1] where N is the number of axes of a. Negative
    indices can also be used to specify axes. The i-th axis of the returned
    array will correspond to the axis numbered `axes[i]` of the input.
    If not specified, defaults to `range(a.ndim)[::-1]`, which reverses
    the order of the axes.
* **Returns:**
  **p**
  : a with its axes permuted. A view is returned whenever possible.

#### SEE ALSO
`ndarray.transpose`
: Equivalent method.

[`moveaxis`](dask.array.moveaxis.md#dask.array.moveaxis)
: Move axes of an array to new positions.

`argsort`
: Return the indices that would sort an array.

### Notes

Use `transpose(a, argsort(axes))` to invert the transposition of tensors
when using the axes keyword argument.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, 4]])
>>> a
array([[1, 2],
       [3, 4]])
>>> np.transpose(a)
array([[1, 3],
       [2, 4]])
```

```pycon
>>> a = np.array([1, 2, 3, 4])
>>> a
array([1, 2, 3, 4])
>>> np.transpose(a)
array([1, 2, 3, 4])
```

```pycon
>>> a = np.ones((1, 2, 3))
>>> np.transpose(a, (1, 0, 2)).shape
(2, 1, 3)
```

```pycon
>>> a = np.ones((2, 3, 4, 5))
>>> np.transpose(a).shape
(5, 4, 3, 2)
```

```pycon
>>> a = np.arange(3*4*5).reshape((3, 4, 5))
>>> np.transpose(a, (-1, 0, -2)).shape
(5, 3, 4)
```

<!-- !! processed by numpydoc !! -->


# dask.array.tri.html.md

# dask.array.tri

### dask.array.tri(N, M=None, k=0, dtype=<class 'float'>, chunks='auto', \*, like=None)

An array with ones at and below the given diagonal and zeros elsewhere.

This docstring was copied from numpy.tri.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **N**
  : Number of rows in the array.

  **M**
  : Number of columns in the array.
    By default, M is taken equal to N.

  **k**
  : The sub-diagonal at and below which the array is filled.
    k = 0 is the main diagonal, while k < 0 is below it,
    and k > 0 is above.  The default is 0.

  **dtype**
  : Data type of the returned array.  The default is float.

  **like**
  : > Reference object to allow the creation of arrays which are not
    > NumPy arrays. If an array-like passed in as `like` supports
    > the `__array_function__` protocol, the result will be defined
    > by it. In this case, it ensures the creation of an array object
    > compatible with that passed in via this argument.
    <br/>
    #### Versionadded
    Added in version 1.20.0.
* **Returns:**
  **tri**
  : Array with its lower triangle filled with ones and zero elsewhere;
    in other words `T[i,j] == 1` for `j <= i + k`, 0 otherwise.

### Examples

```pycon
>>> import numpy as np
>>> np.tri(3, 5, 2, dtype=np.int_)
array([[1, 1, 1, 0, 0],
       [1, 1, 1, 1, 0],
       [1, 1, 1, 1, 1]])
```

```pycon
>>> np.tri(3, 5, -1)
array([[0.,  0.,  0.,  0.,  0.],
       [1.,  0.,  0.,  0.,  0.],
       [1.,  1.,  0.,  0.,  0.]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.tril.html.md

# dask.array.tril

### dask.array.tril(m, k=0)

Lower triangle of an array.

This docstring was copied from numpy.tril.

Some inconsistencies with the Dask version may exist.

Return a copy of an array with elements above the k-th diagonal zeroed.
For arrays with `ndim` exceeding 2, tril will apply to the final two
axes.

* **Parameters:**
  **m**
  : Input array.

  **k**
  : Diagonal above which to zero elements.  k = 0 (the default) is the
    main diagonal, k < 0 is below it and k > 0 is above.
* **Returns:**
  **tril**
  : Lower triangle of m, of same shape and data-type as m.

#### SEE ALSO
[`triu`](dask.array.triu.md#dask.array.triu)
: same thing, only for the upper triangle

### Examples

```pycon
>>> import numpy as np
>>> np.tril([[1,2,3],[4,5,6],[7,8,9],[10,11,12]], -1)
array([[ 0,  0,  0],
       [ 4,  0,  0],
       [ 7,  8,  0],
       [10, 11, 12]])
```

```pycon
>>> np.tril(np.arange(3*4*5).reshape(3, 4, 5))
array([[[ 0,  0,  0,  0,  0],
        [ 5,  6,  0,  0,  0],
        [10, 11, 12,  0,  0],
        [15, 16, 17, 18,  0]],
       [[20,  0,  0,  0,  0],
        [25, 26,  0,  0,  0],
        [30, 31, 32,  0,  0],
        [35, 36, 37, 38,  0]],
       [[40,  0,  0,  0,  0],
        [45, 46,  0,  0,  0],
        [50, 51, 52,  0,  0],
        [55, 56, 57, 58,  0]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.tril_indices.html.md

# dask.array.tril_indices

### dask.array.tril_indices(n, k=0, m=None, chunks='auto')

Return the indices for the lower-triangle of an (n, m) array.

This docstring was copied from numpy.tril_indices.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : The row dimension of the arrays for which the returned
    indices will be valid.

  **k**
  : Diagonal offset (see tril for details).

  **m**
  : The column dimension of the arrays for which the returned
    arrays will be valid.
    By default m is taken equal to n.
* **Returns:**
  **inds**
  : The row and column indices, respectively. The row indices are sorted
    in non-decreasing order, and the corresponding column indices are
    strictly increasing for each row.

#### SEE ALSO
[`triu_indices`](dask.array.triu_indices.md#dask.array.triu_indices)
: similar function, for upper-triangular.

`mask_indices`
: generic function accepting an arbitrary mask function.

[`tril`](dask.array.tril.md#dask.array.tril), [`triu`](dask.array.triu.md#dask.array.triu)

### Examples

```pycon
>>> import numpy as np
```

Compute two different sets of indices to access 4x4 arrays, one for the
lower triangular part starting at the main diagonal, and one starting two
diagonals further right:

```pycon
>>> il1 = np.tril_indices(4)
>>> il1
(array([0, 1, 1, 2, 2, 2, 3, 3, 3, 3]), array([0, 0, 1, 0, 1, 2, 0, 1, 2, 3]))
```

Note that row indices (first array) are non-decreasing, and the corresponding
column indices (second array) are strictly increasing for each row.
Here is how they can be used with a sample array:

```pycon
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
```

Both for indexing:

```pycon
>>> a[il1]
array([ 0,  4,  5, ..., 13, 14, 15])
```

And for assigning values:

```pycon
>>> a[il1] = -1
>>> a
array([[-1,  1,  2,  3],
       [-1, -1,  6,  7],
       [-1, -1, -1, 11],
       [-1, -1, -1, -1]])
```

These cover almost the whole array (two diagonals right of the main one):

```pycon
>>> il2 = np.tril_indices(4, 2)
>>> a[il2] = -10
>>> a
array([[-10, -10, -10,   3],
       [-10, -10, -10, -10],
       [-10, -10, -10, -10],
       [-10, -10, -10, -10]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.tril_indices_from.html.md

# dask.array.tril_indices_from

### dask.array.tril_indices_from(arr, k=0)

Return the indices for the lower-triangle of arr.

This docstring was copied from numpy.tril_indices_from.

Some inconsistencies with the Dask version may exist.

See tril_indices for full details.

* **Parameters:**
  **arr**
  : The indices will be valid for square arrays whose dimensions are
    the same as arr.

  **k**
  : Diagonal offset (see tril for details).

#### SEE ALSO
[`tril_indices`](dask.array.tril_indices.md#dask.array.tril_indices), [`tril`](dask.array.tril.md#dask.array.tril), [`triu_indices_from`](dask.array.triu_indices_from.md#dask.array.triu_indices_from)

### Examples

```pycon
>>> import numpy as np
```

Create a 4 by 4 array

```pycon
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
```

Pass the array to get the indices of the lower triangular elements.

```pycon
>>> trili = np.tril_indices_from(a)
>>> trili
(array([0, 1, 1, 2, 2, 2, 3, 3, 3, 3]), array([0, 0, 1, 0, 1, 2, 0, 1, 2, 3]))
```

```pycon
>>> a[trili]
array([ 0,  4,  5,  8,  9, 10, 12, 13, 14, 15])
```

This is syntactic sugar for tril_indices().

```pycon
>>> np.tril_indices(a.shape[0])
(array([0, 1, 1, 2, 2, 2, 3, 3, 3, 3]), array([0, 0, 1, 0, 1, 2, 0, 1, 2, 3]))
```

Use the k parameter to return the indices for the lower triangular array
up to the k-th diagonal.

```pycon
>>> trili1 = np.tril_indices_from(a, k=1)
>>> a[trili1]
array([ 0,  1,  4,  5,  6,  8,  9, 10, 11, 12, 13, 14, 15])
```

<!-- !! processed by numpydoc !! -->


# dask.array.triu.html.md

# dask.array.triu

### dask.array.triu(m, k=0)

Upper triangle of an array.

This docstring was copied from numpy.triu.

Some inconsistencies with the Dask version may exist.

Return a copy of an array with the elements below the k-th diagonal
zeroed. For arrays with `ndim` exceeding 2, triu will apply to the
final two axes.

Please refer to the documentation for tril for further details.

#### SEE ALSO
[`tril`](dask.array.tril.md#dask.array.tril)
: lower triangle of an array

### Examples

```pycon
>>> import numpy as np
>>> np.triu([[1,2,3],[4,5,6],[7,8,9],[10,11,12]], -1)
array([[ 1,  2,  3],
       [ 4,  5,  6],
       [ 0,  8,  9],
       [ 0,  0, 12]])
```

```pycon
>>> np.triu(np.arange(3*4*5).reshape(3, 4, 5))
array([[[ 0,  1,  2,  3,  4],
        [ 0,  6,  7,  8,  9],
        [ 0,  0, 12, 13, 14],
        [ 0,  0,  0, 18, 19]],
       [[20, 21, 22, 23, 24],
        [ 0, 26, 27, 28, 29],
        [ 0,  0, 32, 33, 34],
        [ 0,  0,  0, 38, 39]],
       [[40, 41, 42, 43, 44],
        [ 0, 46, 47, 48, 49],
        [ 0,  0, 52, 53, 54],
        [ 0,  0,  0, 58, 59]]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.triu_indices.html.md

# dask.array.triu_indices

### dask.array.triu_indices(n, k=0, m=None, chunks='auto')

Return the indices for the upper-triangle of an (n, m) array.

This docstring was copied from numpy.triu_indices.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : The size of the arrays for which the returned indices will
    be valid.

  **k**
  : Diagonal offset (see triu for details).

  **m**
  : The column dimension of the arrays for which the returned
    arrays will be valid.
    By default m is taken equal to n.
* **Returns:**
  **inds**
  : The row and column indices, respectively. The row indices are sorted
    in non-decreasing order, and the corresponding column indices are
    strictly increasing for each row.

#### SEE ALSO
[`tril_indices`](dask.array.tril_indices.md#dask.array.tril_indices)
: similar function, for lower-triangular.

`mask_indices`
: generic function accepting an arbitrary mask function.

[`triu`](dask.array.triu.md#dask.array.triu), [`tril`](dask.array.tril.md#dask.array.tril)

### Examples

```pycon
>>> import numpy as np
```

Compute two different sets of indices to access 4x4 arrays, one for the
upper triangular part starting at the main diagonal, and one starting two
diagonals further right:

```pycon
>>> iu1 = np.triu_indices(4)
>>> iu1
(array([0, 0, 0, 0, 1, 1, 1, 2, 2, 3]), array([0, 1, 2, 3, 1, 2, 3, 2, 3, 3]))
```

Note that row indices (first array) are non-decreasing, and the corresponding
column indices (second array) are strictly increasing for each row.

Here is how they can be used with a sample array:

```pycon
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
```

Both for indexing:

```pycon
>>> a[iu1]
array([ 0,  1,  2, ..., 10, 11, 15])
```

And for assigning values:

```pycon
>>> a[iu1] = -1
>>> a
array([[-1, -1, -1, -1],
       [ 4, -1, -1, -1],
       [ 8,  9, -1, -1],
       [12, 13, 14, -1]])
```

These cover only a small part of the whole array (two diagonals right
of the main one):

```pycon
>>> iu2 = np.triu_indices(4, 2)
>>> a[iu2] = -10
>>> a
array([[ -1,  -1, -10, -10],
       [  4,  -1,  -1, -10],
       [  8,   9,  -1,  -1],
       [ 12,  13,  14,  -1]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.triu_indices_from.html.md

# dask.array.triu_indices_from

### dask.array.triu_indices_from(arr, k=0)

Return the indices for the upper-triangle of arr.

This docstring was copied from numpy.triu_indices_from.

Some inconsistencies with the Dask version may exist.

See triu_indices for full details.

* **Parameters:**
  **arr**
  : The indices will be valid for square arrays.

  **k**
  : Diagonal offset (see triu for details).
* **Returns:**
  **triu_indices_from**
  : Indices for the upper-triangle of arr.

#### SEE ALSO
[`triu_indices`](dask.array.triu_indices.md#dask.array.triu_indices), [`triu`](dask.array.triu.md#dask.array.triu), [`tril_indices_from`](dask.array.tril_indices_from.md#dask.array.tril_indices_from)

### Examples

```pycon
>>> import numpy as np
```

Create a 4 by 4 array

```pycon
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])
```

Pass the array to get the indices of the upper triangular elements.

```pycon
>>> triui = np.triu_indices_from(a)
>>> triui
(array([0, 0, 0, 0, 1, 1, 1, 2, 2, 3]), array([0, 1, 2, 3, 1, 2, 3, 2, 3, 3]))
```

```pycon
>>> a[triui]
array([ 0,  1,  2,  3,  5,  6,  7, 10, 11, 15])
```

This is syntactic sugar for triu_indices().

```pycon
>>> np.triu_indices(a.shape[0])
(array([0, 0, 0, 0, 1, 1, 1, 2, 2, 3]), array([0, 1, 2, 3, 1, 2, 3, 2, 3, 3]))
```

Use the k parameter to return the indices for the upper triangular array
from the k-th diagonal.

```pycon
>>> triuim1 = np.triu_indices_from(a, k=1)
>>> a[triuim1]
array([ 1,  2,  3,  6,  7, 11])
```

<!-- !! processed by numpydoc !! -->


# dask.array.true_divide.html.md

# dask.array.true_divide

### dask.array.true_divide(x1, x2, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'divide'>*

This docstring was copied from numpy.divide.

Some inconsistencies with the Dask version may exist.

Divide arguments element-wise.

* **Parameters:**
  **x1**
  : Dividend array.

  **x2**
  : Divisor array.
    If `x1.shape != x2.shape`, they must be broadcastable to a common
    shape (which becomes the shape of the output).

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The quotient `x1/x2`, element-wise.
    This is a scalar if both x1 and x2 are scalars.

#### SEE ALSO
`seterr`
: Set whether to raise or warn on overflow, underflow and division by zero.

### Notes

Equivalent to `x1` / `x2` in terms of array-broadcasting.

The `true_divide(x1, x2)` function is an alias for
`divide(x1, x2)`.

### Examples

```pycon
>>> import numpy as np
>>> np.divide(2.0, 4.0)
0.5
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = np.arange(3.0)
>>> np.divide(x1, x2)
array([[nan, 1. , 1. ],
       [inf, 4. , 2.5],
       [inf, 7. , 4. ]])
```

The `/` operator can be used as a shorthand for `np.divide` on
ndarrays.

```pycon
>>> x1 = np.arange(9.0).reshape((3, 3))
>>> x2 = 2 * np.ones(3)
>>> x1 / x2
array([[0. , 0.5, 1. ],
       [1.5, 2. , 2.5],
       [3. , 3.5, 4. ]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.trunc.html.md

# dask.array.trunc

### dask.array.trunc(x, /, out=None, \*, where=True, casting='same_kind', order='K', dtype=None, subok=True) *= <ufunc 'trunc'>*

This docstring was copied from numpy.trunc.

Some inconsistencies with the Dask version may exist.

Return the truncated value of the input, element-wise.

The truncated value of the scalar x is the nearest integer i which
is closer to zero than x is. In short, the fractional part of the
signed number x is discarded.

* **Parameters:**
  **x**
  : Input data.

  **out**
  : A location into which the result is stored. If provided, it must have
    a shape that the inputs broadcast to. If not provided or None,
    a freshly-allocated array is returned. A tuple (possible only as a
    keyword argument) must have length equal to the number of outputs.

  **where**
  : This condition is broadcast over the input. At locations where the
    condition is True, the out array will be set to the ufunc result.
    Elsewhere, the out array will retain its original value.
    Note that if an uninitialized out array is created via the default
    `out=None`, locations within it where the condition is False will
    remain uninitialized.

  **\*\*kwargs**
  : For other keyword-only arguments, see the
    [ufunc docs](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs).
* **Returns:**
  **y**
  : The truncated value of each element in x.
    This is a scalar if x is a scalar.

#### SEE ALSO
[`ceil`](dask.array.ceil.md#dask.array.ceil), [`floor`](dask.array.floor.md#dask.array.floor), [`rint`](dask.array.rint.md#dask.array.rint), [`fix`](dask.array.fix.md#dask.array.fix)

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([-1.7, -1.5, -0.2, 0.2, 1.5, 1.7, 2.0])
>>> np.trunc(a)
array([-1., -1., -0.,  0.,  1.,  1.,  2.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.union1d.html.md

# dask.array.union1d

### dask.array.union1d(ar1, ar2)

Find the union of two arrays.

This docstring was copied from numpy.union1d.

Some inconsistencies with the Dask version may exist.

Return the unique, sorted array of values that are in either of the two
input arrays.

* **Parameters:**
  **ar1, ar2**
  : Input arrays. They are flattened if they are not already 1D.
* **Returns:**
  **union1d**
  : Unique, sorted union of the input arrays.

### Examples

```pycon
>>> import numpy as np
>>> np.union1d([-1, 0, 1], [-2, 0, 2])
array([-2, -1,  0,  1,  2])
```

To find the union of more than two arrays, use functools.reduce:

```pycon
>>> from functools import reduce
>>> reduce(np.union1d, ([1, 3, 4, 3], [3, 1, 2, 1], [6, 3, 4, 2]))
array([1, 2, 3, 4, 6])
```

<!-- !! processed by numpydoc !! -->


# dask.array.unique.html.md

# dask.array.unique

### dask.array.unique(ar, return_index=False, return_inverse=False, return_counts=False)

Find the unique elements of an array.

This docstring was copied from numpy.unique.

Some inconsistencies with the Dask version may exist.

Returns the sorted unique elements of an array. There are three optional
outputs in addition to the unique elements:

* the indices of the input array that give the unique values
* the indices of the unique array that reconstruct the input array
* the number of times each unique value comes up in the input array

* **Parameters:**
  **ar**
  : Input array. Unless axis is specified, this will be flattened if it
    is not already 1-D.

  **return_index**
  : If True, also return the indices of ar (along the specified axis,
    if provided, or in the flattened array) that result in the unique array.

  **return_inverse**
  : If True, also return the indices of the unique array (for the specified
    axis, if provided) that can be used to reconstruct ar.

  **return_counts**
  : If True, also return the number of times each unique item appears
    in ar.

  **axis**
  : The axis to operate on. If None, ar will be flattened. If an integer,
    the subarrays indexed by the given axis will be flattened and treated
    as the elements of a 1-D array with the dimension of the given axis,
    see the notes for more details.  Object arrays or structured arrays
    that contain objects are not supported if the axis kwarg is used. The
    default is None.

  **equal_nan**
  : If True, collapses multiple NaN values in the return array into one.
    <br/>
    #### Versionadded
    Added in version 1.24.

  **sorted**
  : If True, the unique elements are sorted. Elements may be sorted in
    practice even if `sorted=False`, but this could change without
    notice.
    <br/>
    #### Versionadded
    Added in version 2.3.
* **Returns:**
  **unique**
  : The sorted unique values.

  **unique_indices**
  : The indices of the first occurrences of the unique values in the
    original array. Only provided if return_index is True.

  **unique_inverse**
  : The indices to reconstruct the original array from the
    unique array. Only provided if return_inverse is True.

  **unique_counts**
  : The number of times each of the unique values comes up in the
    original array. Only provided if return_counts is True.

#### SEE ALSO
[`repeat`](dask.array.repeat.md#dask.array.repeat)
: Repeat elements of an array.

`sort`
: Return a sorted copy of an array.

### Notes

When an axis is specified the subarrays indexed by the axis are sorted.
This is done by making the specified axis the first dimension of the array
(move the axis to the first dimension to keep the order of the other axes)
and then flattening the subarrays in C order. The flattened subarrays are
then viewed as a structured type with each element given a label, with the
effect that we end up with a 1-D array of structured types that can be
treated in the same way as any other 1-D array. The result is that the
flattened subarrays are sorted in lexicographic order starting with the
first element.

#### Versionchanged
Changed in version 1.21: Like np.sort, NaN will sort to the end of the values.
For complex arrays all NaN values are considered equivalent
(no matter whether the NaN is in the real or imaginary part).
As the representant for the returned array the smallest one in the
lexicographical order is chosen - see np.sort for how the lexicographical
order is defined for complex arrays.

#### Versionchanged
Changed in version 2.0: For multi-dimensional inputs, `unique_inverse` is reshaped
such that the input can be reconstructed using
`np.take(unique, unique_inverse, axis=axis)`. The result is
now not 1-dimensional when `axis=None`.

Note that in NumPy 2.0.0 a higher dimensional array was returned also
when `axis` was not `None`.  This was reverted, but
`inverse.reshape(-1)` can be used to ensure compatibility with both
versions.

### Examples

```pycon
>>> import numpy as np
>>> np.unique([1, 1, 2, 2, 3, 3])
array([1, 2, 3])
>>> a = np.array([[1, 1], [2, 3]])
>>> np.unique(a)
array([1, 2, 3])
```

Return the unique rows of a 2D array

```pycon
>>> a = np.array([[1, 0, 0], [1, 0, 0], [2, 3, 4]])
>>> np.unique(a, axis=0)
array([[1, 0, 0], [2, 3, 4]])
```

Return the indices of the original array that give the unique values:

```pycon
>>> a = np.array(['a', 'b', 'b', 'c', 'a'])
>>> u, indices = np.unique(a, return_index=True)
>>> u
array(['a', 'b', 'c'], dtype='<U1')
>>> indices
array([0, 1, 3])
>>> a[indices]
array(['a', 'b', 'c'], dtype='<U1')
```

Reconstruct the input array from the unique values and inverse:

```pycon
>>> a = np.array([1, 2, 6, 4, 2, 3, 2])
>>> u, indices = np.unique(a, return_inverse=True)
>>> u
array([1, 2, 3, 4, 6])
>>> indices
array([0, 1, 4, 3, 1, 2, 1])
>>> u[indices]
array([1, 2, 6, 4, 2, 3, 2])
```

Reconstruct the input values from the unique values and counts:

```pycon
>>> a = np.array([1, 2, 6, 4, 2, 3, 2])
>>> values, counts = np.unique(a, return_counts=True)
>>> values
array([1, 2, 3, 4, 6])
>>> counts
array([1, 3, 1, 1, 1])
>>> np.repeat(values, counts)
array([1, 2, 2, 2, 3, 4, 6])    # original order not preserved
```

<!-- !! processed by numpydoc !! -->


# dask.array.unravel_index.html.md

# dask.array.unravel_index

### dask.array.unravel_index(indices, shape, order='C')

This docstring was copied from numpy.unravel_index.

Some inconsistencies with the Dask version may exist.

Converts a flat index or array of flat indices into a tuple
of coordinate arrays.

* **Parameters:**
  **indices**
  : An integer array whose elements are indices into the flattened
    version of an array of dimensions `shape`. Before version 1.6.0,
    this function accepted just one index value.

  **shape**
  : The shape of the array to use for unraveling `indices`.

  **order**
  : Determines whether the indices should be viewed as indexing in
    row-major (C-style) or column-major (Fortran-style) order.
* **Returns:**
  **unraveled_coords**
  : Each array in the tuple has the same shape as the `indices`
    array.

#### SEE ALSO
[`ravel_multi_index`](dask.array.ravel_multi_index.md#dask.array.ravel_multi_index)

### Examples

```pycon
>>> import numpy as np
>>> np.unravel_index([22, 41, 37], (7,6))
(array([3, 6, 6]), array([4, 5, 1]))
>>> np.unravel_index([31, 41, 13], (7,6), order='F')
(array([3, 6, 6]), array([4, 5, 1]))
```

```pycon
>>> np.unravel_index(1621, (6,7,8,9))
(3, 1, 4, 1)
```

<!-- !! processed by numpydoc !! -->


# dask.array.utils.meta_from_array.html.md

# dask.array.utils.meta_from_array

### dask.array.utils.meta_from_array(x, ndim=None, dtype=None)

Normalize an array to appropriate meta object

* **Parameters:**
  **x: array-like, callable**
  : Either an object that looks sufficiently like a Numpy array,
    or a callable that accepts shape and dtype keywords

  **ndim: int**
  : Number of dimensions of the array

  **dtype: Numpy dtype**
  : A valid input for `np.dtype`
* **Returns:**
  array-like with zero elements of the correct dtype

<!-- !! processed by numpydoc !! -->


# dask.array.var.html.md

# dask.array.var

### dask.array.var(a, axis=None, dtype=None, keepdims=False, ddof=0, split_every=None, out=None)

Compute the variance along the specified axis.

This docstring was copied from numpy.var.

Some inconsistencies with the Dask version may exist.

Returns the variance of the array elements, a measure of the spread of a
distribution.  The variance is computed for the flattened array by
default, otherwise over the specified axis.

* **Parameters:**
  **a**
  : Array containing numbers whose variance is desired.  If a is not an
    array, a conversion is attempted.

  **axis**
  : Axis or axes along which the variance is computed.  The default is to
    compute the variance of the flattened array.
    If this is a tuple of ints, a variance is performed over multiple axes,
    instead of a single axis or all the axes as before.

  **dtype**
  : Type to use in computing the variance.  For arrays of integer type
    the default is float64; for arrays of float types it is the same as
    the array type.

  **out**
  : Alternate output array in which to place the result.  It must have
    the same shape as the expected output, but the type is cast if
    necessary.

  **ddof**
  : “Delta Degrees of Freedom”: the divisor used in the calculation is
    `N - ddof`, where `N` represents the number of elements. By
    default ddof is zero. See notes for details about use of ddof.

  **keepdims**
  : If this is set to True, the axes which are reduced are left
    in the result as dimensions with size one. With this option,
    the result will broadcast correctly against the input array.
    <br/>
    If the default value is passed, then keepdims will not be
    passed through to the var method of sub-classes of
    ndarray, however any non-default value will be.  If the
    sub-class’ method does not implement keepdims any
    exceptions will be raised.

  **where**
  : Elements to include in the variance. See ~numpy.ufunc.reduce for
    details.
    <br/>
    #### Versionadded
    Added in version 1.20.0.

  **mean**
  : Provide the mean to prevent its recalculation. The mean should have
    a shape as if it was calculated with `keepdims=True`.
    The axis for the calculation of the mean should be the same as used in
    the call to this var function.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **correction**
  : Array API compatible name for the `ddof` parameter. Only one of them
    can be provided at the same time.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **variance**
  : If `out=None`, returns a new array containing the variance;
    otherwise, a reference to the output array is returned.

#### SEE ALSO
[`std`](dask.array.std.md#dask.array.std), [`mean`](dask.array.mean.md#dask.array.mean), [`nanmean`](dask.array.nanmean.md#dask.array.nanmean), [`nanstd`](dask.array.nanstd.md#dask.array.nanstd), [`nanvar`](dask.array.nanvar.md#dask.array.nanvar)

[Output type determination](https://numpy.org/doc/stable/user/basics.ufuncs.html#ufuncs-output-type)

### Notes

There are several common variants of the array variance calculation.
Assuming the input a is a one-dimensional NumPy array and `mean` is
either provided as an argument or computed as `a.mean()`, NumPy
computes the variance of an array as:

```default
N = len(a)
d2 = abs(a - mean)**2  # abs is for complex `a`
var = d2.sum() / (N - ddof)  # note use of `ddof`
```

Different values of the argument ddof are useful in different
contexts. NumPy’s default `ddof=0` corresponds with the expression:

$$
\frac{\sum_i{|a_i - \bar{a}|^2 }}{N}
$$

which is sometimes called the “population variance” in the field of
statistics because it applies the definition of variance to a as if a
were a complete population of possible observations.

Many other libraries define the variance of an array differently, e.g.:

$$
\frac{\sum_i{|a_i - \bar{a}|^2}}{N - 1}
$$

In statistics, the resulting quantity is sometimes called the “sample
variance” because if a is a random sample from a larger population,
this calculation provides an unbiased estimate of the variance of the
population.  The use of $N-1$ in the denominator is often called
“Bessel’s correction” because it corrects for bias (toward lower values)
in the variance estimate introduced when the sample mean of a is used
in place of the true mean of the population. For this quantity, use
`ddof=1`.

Note that for complex numbers, the absolute value is taken before
squaring, so that the result is always real and nonnegative.

For floating-point input, the variance is computed using the same
precision the input has.  Depending on the input data, this can cause
the results to be inaccurate, especially for float32 (see example
below).  Specifying a higher-accuracy accumulator using the `dtype`
keyword can alleviate this issue.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([[1, 2], [3, 4]])
>>> np.var(a)
1.25
>>> np.var(a, axis=0)
array([1.,  1.])
>>> np.var(a, axis=1)
array([0.25,  0.25])
```

In single precision, var() can be inaccurate:

```pycon
>>> a = np.zeros((2, 512*512), dtype=np.float32)
>>> a[0, :] = 1.0
>>> a[1, :] = 0.1
>>> np.var(a)
np.float32(0.20250003)
```

Computing the variance in float64 is more accurate:

```pycon
>>> np.var(a, dtype=np.float64)
0.20249999932944759 # may vary
>>> ((1-0.55)**2 + (0.1-0.55)**2)/2
0.2025
```

Specifying a where argument:

```pycon
>>> a = np.array([[14, 8, 11, 10], [7, 9, 10, 11], [10, 15, 5, 10]])
>>> np.var(a)
6.833333333333333 # may vary
>>> np.var(a, where=[[True], [True], [False]])
4.0
```

Using the mean keyword to save computation time:

```pycon
>>> import numpy as np
>>> from timeit import timeit
>>>
>>> a = np.array([[14, 8, 11, 10], [7, 9, 10, 11], [10, 15, 5, 10]])
>>> mean = np.mean(a, axis=1, keepdims=True)
>>>
>>> g = globals()
>>> n = 10000
>>> t1 = timeit("var = np.var(a, axis=1, mean=mean)", globals=g, number=n)
>>> t2 = timeit("var = np.var(a, axis=1)", globals=g, number=n)
>>> print(f'Percentage execution time saved {100*(t2-t1)/t2:.0f}%')

Percentage execution time saved 32%
```

<!-- !! processed by numpydoc !! -->


# dask.array.vdot.html.md

# dask.array.vdot

### dask.array.vdot(a, b,)

This docstring was copied from numpy.vdot.

Some inconsistencies with the Dask version may exist.

Return the dot product of two vectors.

The vdot function handles complex numbers differently than dot:
if the first argument is complex, it is replaced by its complex conjugate
in the dot product calculation. vdot also handles multidimensional
arrays differently than dot: it does not perform a matrix product, but
flattens the arguments to 1-D arrays before taking a vector dot product.

Consequently, when the arguments are 2-D arrays of the same shape, this
function effectively returns their
[Frobenius inner product](https://en.wikipedia.org/wiki/Frobenius_inner_product)
(also known as the *trace inner product* or the *standard inner product*
on a vector space of matrices).

* **Parameters:**
  **a**
  : If a is complex the complex conjugate is taken before calculation
    of the dot product.

  **b**
  : Second argument to the dot product.
* **Returns:**
  **output**
  : Dot product of a and b.  Can be an int, float, or
    complex depending on the types of a and b.

#### SEE ALSO
[`dot`](dask.array.dot.md#dask.array.dot)
: Return the dot product without using the complex conjugate of the first argument.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1+2j,3+4j])
>>> b = np.array([5+6j,7+8j])
>>> np.vdot(a, b)
(70-8j)
>>> np.vdot(b, a)
(70+8j)
```

Note that higher-dimensional arrays are flattened!

```pycon
>>> a = np.array([[1, 4], [5, 6]])
>>> b = np.array([[4, 1], [2, 2]])
>>> np.vdot(a, b)
30
>>> np.vdot(b, a)
30
>>> 1*4 + 4*1 + 5*2 + 6*2
30
```

<!-- !! processed by numpydoc !! -->


# dask.array.vstack.html.md

# dask.array.vstack

### dask.array.vstack(tup, allow_unknown_chunksizes=False)

Stack arrays in sequence vertically (row wise).

This docstring was copied from numpy.vstack.

Some inconsistencies with the Dask version may exist.

This is equivalent to concatenation along the first axis after 1-D arrays
of shape (N,) have been reshaped to (1,N). Rebuilds arrays divided by
vsplit.

This function makes most sense for arrays with up to 3 dimensions. For
instance, for pixel-data with a height (first axis), width (second axis),
and r/g/b channels (third axis). The functions concatenate, stack and
block provide more general stacking and concatenation operations.

* **Parameters:**
  **tup**
  : The arrays must have the same shape along all but the first axis.
    1-D arrays must have the same length. In the case of a single
    array_like input, it will be treated as a sequence of arrays; i.e.,
    each element along the zeroth axis is treated as a separate array.

  **dtype**
  : If provided, the destination array will have this dtype. Cannot be
    provided together with out.
    <br/>
    #### Versionadded
    Added in version 1.24.

  **casting**
  : Controls what kind of data casting may occur. Defaults to ‘same_kind’.
    <br/>
    #### Versionadded
    Added in version 1.24.
* **Returns:**
  **stacked**
  : The array formed by stacking the given arrays, will be at least 2-D.

#### SEE ALSO
[`concatenate`](dask.array.concatenate.md#dask.array.concatenate)
: Join a sequence of arrays along an existing axis.

[`stack`](dask.array.stack.md#dask.array.stack)
: Join a sequence of arrays along a new axis.

[`block`](dask.array.block.md#dask.array.block)
: Assemble an nd-array from nested lists of blocks.

[`hstack`](dask.array.hstack.md#dask.array.hstack)
: Stack arrays in sequence horizontally (column wise).

[`dstack`](dask.array.dstack.md#dask.array.dstack)
: Stack arrays in sequence depth wise (along third axis).

`column_stack`
: Stack 1-D arrays as columns into a 2-D array.

`vsplit`
: Split an array into multiple sub-arrays vertically (row-wise).

`unstack`
: Split an array into a tuple of sub-arrays along an axis.

### Examples

```pycon
>>> import numpy as np
>>> a = np.array([1, 2, 3])
>>> b = np.array([4, 5, 6])
>>> np.vstack((a,b))
array([[1, 2, 3],
       [4, 5, 6]])
```

```pycon
>>> a = np.array([[1], [2], [3]])
>>> b = np.array([[4], [5], [6]])
>>> np.vstack((a,b))
array([[1],
       [2],
       [3],
       [4],
       [5],
       [6]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.where.html.md

# dask.array.where

### dask.array.where(condition, /)

This docstring was copied from numpy.where.

Some inconsistencies with the Dask version may exist.

Return elements chosen from x or y depending on condition.

#### NOTE
When only condition is provided, this function is a shorthand for
`np.asarray(condition).nonzero()`. Using nonzero directly should be
preferred, as it behaves correctly for subclasses. The rest of this
documentation covers only the case where all three arguments are
provided.

* **Parameters:**
  **condition**
  : Where True, yield x, otherwise yield y.

  **x, y**
  : Values from which to choose. x, y and condition need to be
    broadcastable to some shape.
* **Returns:**
  **out**
  : An array with elements from x where condition is True, and elements
    from y elsewhere.

#### SEE ALSO
[`choose`](dask.array.choose.md#dask.array.choose)

[`nonzero`](dask.array.nonzero.md#dask.array.nonzero)
: The function that is called when x and y are omitted

### Notes

If all the arrays are 1-D, where is equivalent to:

```default
[xv if c else yv
 for c, xv, yv in zip(condition, x, y)]
```

### Examples

```pycon
>>> import numpy as np
>>> a = np.arange(10)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> np.where(a < 5, a, 10*a)
array([ 0,  1,  2,  3,  4, 50, 60, 70, 80, 90])
```

This can be used on multidimensional arrays too:

```pycon
>>> np.where([[True, False], [True, True]],
...          [[1, 2], [3, 4]],
...          [[9, 8], [7, 6]])
array([[1, 8],
       [3, 4]])
```

The shapes of x, y, and the condition are broadcast together:

```pycon
>>> x, y = np.ogrid[:3, :4]
>>> np.where(x < y, x, 10 + y)  # both x and 10+y are broadcast
array([[10,  0,  0,  0],
       [10, 11,  1,  1],
       [10, 11, 12,  2]])
```

```pycon
>>> a = np.array([[0, 1, 2],
...               [0, 2, 4],
...               [0, 3, 6]])
>>> np.where(a < 4, a, -1)  # -1 is broadcast
array([[ 0,  1,  2],
       [ 0,  2, -1],
       [ 0,  3, -1]])
```

<!-- !! processed by numpydoc !! -->


# dask.array.zeros.html.md

# dask.array.zeros

### dask.array.zeros(\*args, \*\*kwargs)

> Blocked variant of zeros_like

> Follows the signature of zeros_like exactly except that it also features
> optional keyword arguments `chunks: int, tuple, or dict` and `name: str`.

> Original signature follows below.

Return an array of zeros with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Overrides the memory layout of the result. ‘C’ means C-order,
    ‘F’ means F-order, ‘A’ means ‘F’ if a is Fortran contiguous,
    ‘C’ otherwise. ‘K’ means match the layout of a as closely
    as possible.

  **subok**
  : If True, then the newly created array will use the sub-class
    type of a, otherwise it will be a base-class array. Defaults
    to True.

  **shape**
  : Overrides the shape of the result. If order=’K’ and the number of
    dimensions is unchanged, will try to keep order, otherwise,
    order=’C’ is implied.

  **device**
  : The device on which to place the created array. Default: None.
    For Array-API interoperability only, so must be `"cpu"` if passed.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  **out**
  : Array of zeros with the same shape and type as a.

#### SEE ALSO
[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`full_like`](dask.array.full_like.md#dask.array.full_like)
: Return a new array with shape of input filled with value.

[`zeros`](#dask.array.zeros)
: Return a new array setting values to zero.

### Examples

```pycon
>>> import numpy as np
>>> x = np.arange(6)
>>> x = x.reshape((2, 3))
>>> x
array([[0, 1, 2],
       [3, 4, 5]])
>>> np.zeros_like(x)
array([[0, 0, 0],
       [0, 0, 0]])
```

```pycon
>>> y = np.arange(3, dtype=np.float64)
>>> y
array([0., 1., 2.])
>>> np.zeros_like(y)
array([0.,  0.,  0.])
```

<!-- !! processed by numpydoc !! -->


# dask.array.zeros_like.html.md

# dask.array.zeros_like

### dask.array.zeros_like(a, dtype=None, order='C', chunks=None, name=None, shape=None)

Return an array of zeros with the same shape and type as a given array.

* **Parameters:**
  **a**
  : The shape and data-type of a define these same attributes of
    the returned array.

  **dtype**
  : Overrides the data type of the result.

  **order**
  : Whether to store multidimensional data in C- or Fortran-contiguous
    (row- or column-wise) order in memory.

  **chunks**
  : The number of samples on each block. Note that the last block will have
    fewer samples if `len(array) % chunks != 0`.

  **name**
  : An optional keyname for the array. Defaults to hashing the input
    keyword arguments.

  **shape**
  : Overrides the shape of the result.
* **Returns:**
  **out**
  : Array of zeros with the same shape and type as a.

#### SEE ALSO
[`ones_like`](dask.array.ones_like.md#dask.array.ones_like)
: Return an array of ones with shape and type of input.

[`empty_like`](dask.array.empty_like.md#dask.array.empty_like)
: Return an empty array with shape and type of input.

[`zeros`](dask.array.zeros.md#dask.array.zeros)
: Return a new array setting values to zero.

[`ones`](dask.array.ones.md#dask.array.ones)
: Return a new array setting values to one.

[`empty`](dask.array.empty.md#dask.array.empty)
: Return a new uninitialized array.

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.accumulate.html.md

# dask.bag.Bag.accumulate

#### Bag.accumulate(binop, initial=<no_default>)

Repeatedly apply binary function to a sequence, accumulating results.

This assumes that the bag is ordered.  While this is typically the case
not all Dask.bag functions preserve this property.

### Examples

```pycon
>>> import dask.bag as db
>>> from operator import add
>>> b = db.from_sequence([1, 2, 3, 4, 5], npartitions=2)
>>> b.accumulate(add).compute()
[1, 3, 6, 10, 15]
```

Accumulate also takes an optional argument that will be used as the
first value.

```pycon
>>> b.accumulate(add, initial=-1).compute()
[-1, 0, 2, 5, 9, 14]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.all.html.md

# dask.bag.Bag.all

#### Bag.all(split_every=None)

Are all elements truthy?

### Examples

```pycon
>>> import dask.bag as db
>>> bool_bag = db.from_sequence([True, True, False])
>>> bool_bag.all().compute()
False
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.any.html.md

# dask.bag.Bag.any

#### Bag.any(split_every=None)

Are any of the elements truthy?

### Examples

```pycon
>>> import dask.bag as db
>>> bool_bag = db.from_sequence([True, True, False])
>>> bool_bag.any().compute()
True
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.compute.html.md

# dask.bag.Bag.compute

#### Bag.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.count.html.md

# dask.bag.Bag.count

#### Bag.count(split_every=None)

Count the number of elements.

### Examples

```pycon
>>> import dask.bag as db
>>> numbers = db.from_sequence([1, 2, 3])
>>> numbers.count().compute()
3
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.distinct.html.md

# dask.bag.Bag.distinct

#### Bag.distinct(key=None)

Distinct elements of collection

Unordered without repeats.

* **Parameters:**
  **key: {callable,str}**
  : Defines uniqueness of items in bag by calling `key` on each item.
    If a string is passed `key` is considered to be `lambda x: x[key]`.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(['Alice', 'Bob', 'Alice'])
>>> sorted(b.distinct())
['Alice', 'Bob']
>>> b = db.from_sequence([{'name': 'Alice'}, {'name': 'Bob'}, {'name': 'Alice'}])
>>> b.distinct(key=lambda x: x['name']).compute()
[{'name': 'Alice'}, {'name': 'Bob'}]
>>> b.distinct(key='name').compute()
[{'name': 'Alice'}, {'name': 'Bob'}]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.filter.html.md

# dask.bag.Bag.filter

#### Bag.filter(predicate)

Filter elements in collection by a predicate function.

```pycon
>>> def iseven(x):
...     return x % 2 == 0
```

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5))
>>> list(b.filter(iseven))
[0, 2, 4]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.flatten.html.md

# dask.bag.Bag.flatten

#### Bag.flatten()

Concatenate nested lists into one long list.

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([[1], [2, 3]])
>>> list(b)
[[1], [2, 3]]
```

```pycon
>>> list(b.flatten())
[1, 2, 3]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.fold.html.md

# dask.bag.Bag.fold

#### Bag.fold(binop, combine=None, initial=<no_default>, split_every=None, out_type=<class 'dask.bag.core.Item'>)

Parallelizable reduction

Fold is like the builtin function `reduce` except that it works in
parallel.  Fold takes two binary operator functions, one to reduce each
partition of our dataset and another to combine results between
partitions

1. `binop`: Binary operator to reduce within each partition
2. `combine`:  Binary operator to combine results from binop

Sequentially this would look like the following:

```pycon
>>> intermediates = [reduce(binop, part) for part in partitions]
>>> final = reduce(combine, intermediates)
```

If only one function is given then it is used for both functions
`binop` and `combine` as in the following example to compute the
sum:

```pycon
>>> def add(x, y):
...     return x + y
```

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5))
>>> b.fold(add).compute()
10
```

In full form we provide both binary operators as well as their default
arguments

```pycon
>>> b.fold(binop=add, combine=add, initial=0).compute()
10
```

More complex binary operators are also doable

```pycon
>>> def add_to_set(acc, x):
...     ''' Add new element x to set acc '''
...     return acc | set([x])
>>> b.fold(add_to_set, set.union, initial=set()).compute()
{0, 1, 2, 3, 4}
```

#### SEE ALSO
[`Bag.foldby`](dask.bag.Bag.foldby.md#dask.bag.Bag.foldby)

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.foldby.html.md

# dask.bag.Bag.foldby

#### Bag.foldby(key, binop, initial=<no_default>, combine=None, combine_initial=<no_default>, split_every=None)

Combined reduction and groupby.

Foldby provides a combined groupby and reduce for efficient parallel
split-apply-combine tasks.

The computation

```pycon
>>> b.foldby(key, binop, init)
```

is equivalent to the following:

```pycon
>>> def reduction(group):
...     return reduce(binop, group, init)
```

```pycon
>>> b.groupby(key).map(lambda (k, v): (k, reduction(v)))
```

But uses minimal communication and so is *much* faster.

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(10))
>>> iseven = lambda x: x % 2 == 0
>>> add = lambda x, y: x + y
>>> dict(b.foldby(iseven, add))
{True: 20, False: 25}
```

**Key Function**

The key function determines how to group the elements in your bag.
In the common case where your bag holds dictionaries then the key
function often gets out one of those elements.

```pycon
>>> def key(x):
...     return x['name']
```

This case is so common that it is special cased, and if you provide a
key that is not a callable function then dask.bag will turn it into one
automatically.  The following are equivalent:

```pycon
>>> b.foldby(lambda x: x['name'], ...)
>>> b.foldby('name', ...)
```

**Binops**

It can be tricky to construct the right binary operators to perform
analytic queries.  The `foldby` method accepts two binary operators,
`binop` and `combine`.  Binary operators two inputs and output must
have the same type.

Binop takes a running total and a new element and produces a new total:

```pycon
>>> def binop(total, x):
...     return total + x['amount']
```

Combine takes two totals and combines them:

```pycon
>>> def combine(total1, total2):
...     return total1 + total2
```

Each of these binary operators may have a default first value for
total, before any other value is seen.  For addition binary operators
like above this is often `0` or the identity element for your
operation.

**split_every**

Group partitions into groups of this size while performing reduction.
Defaults to 8.

```pycon
>>> b.foldby('name', binop, 0, combine, 0)
```

#### SEE ALSO
`toolz.reduceby`

`pyspark.combineByKey`

### Examples

We can compute the maximum of some `(key, value)` pairs, grouped
by the `key`. (You might be better off converting the `Bag` to
a `dask.dataframe` and using its groupby).

```pycon
>>> import random
>>> import dask.bag as db
```

```pycon
>>> tokens = list('abcdefg')
>>> values = range(10000)
>>> a = [(random.choice(tokens), random.choice(values))
...       for _ in range(100)]
>>> a[:2]
[('g', 676), ('a', 871)]
```

```pycon
>>> a = db.from_sequence(a)
```

```pycon
>>> def binop(t, x):
...     return max((t, x), key=lambda x: x[1])
```

```pycon
>>> a.foldby(lambda x: x[0], binop).compute()
[('g', ('g', 984)),
 ('a', ('a', 871)),
 ('b', ('b', 999)),
 ('c', ('c', 765)),
 ('f', ('f', 955)),
 ('e', ('e', 991)),
 ('d', ('d', 854))]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.frequencies.html.md

# dask.bag.Bag.frequencies

#### Bag.frequencies(split_every=None, sort=False)

Count number of occurrences of each distinct element.

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(['Alice', 'Bob', 'Alice'])
>>> dict(b.frequencies())
{'Alice': 2, 'Bob', 1}
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.groupby.html.md

# dask.bag.Bag.groupby

#### Bag.groupby(grouper, method=None, npartitions=None, blocksize=1048576, max_branch=None, shuffle=None)

Group collection by key function

This requires a full dataset read, serialization and shuffle.
This is expensive.  If possible you should use `foldby`.

* **Parameters:**
  **grouper: function**
  : Function on which to group elements

  **shuffle: str**
  : Either ‘disk’ for an on-disk shuffle or ‘tasks’ to use the task
    scheduling framework.  Use ‘disk’ if you are on a single machine
    and ‘tasks’ if you are on a distributed cluster.

  **npartitions: int**
  : If using the disk-based shuffle, the number of output partitions

  **blocksize: int**
  : If using the disk-based shuffle, the size of shuffle blocks (bytes)

  **max_branch: int**
  : If using the task-based shuffle, the amount of splitting each
    partition undergoes.  Increase this for fewer copies but more
    scheduler overhead.

#### SEE ALSO
[`Bag.foldby`](dask.bag.Bag.foldby.md#dask.bag.Bag.foldby)

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(10))
>>> iseven = lambda x: x % 2 == 0
>>> dict(b.groupby(iseven))
{True: [0, 2, 4, 6, 8], False: [1, 3, 5, 7, 9]}
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.html.md

# dask.bag.Bag

### *class* dask.bag.Bag(dsk: Graph, name: [str](https://docs.python.org/3/library/stdtypes.html#str), npartitions: [int](https://docs.python.org/3/library/functions.html#int))

Parallel collection of Python objects

### Examples

Create Bag from sequence

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5))
>>> list(b.filter(lambda x: x % 2 == 0).map(lambda x: x * 10))
[0, 20, 40]
```

Create Bag from filename or globstring of filenames

```pycon
>>> b = db.read_text('/path/to/mydata.*.json.gz').map(json.loads)
```

Create manually (expert use)

```pycon
>>> dsk = {('x', 0): (range, 5),
...        ('x', 1): (range, 5),
...        ('x', 2): (range, 5)}
>>> b = db.Bag(dsk, 'x', npartitions=3)
```

```pycon
>>> sorted(b.map(lambda x: x * 10))
[0, 0, 0, 10, 10, 10, 20, 20, 20, 30, 30, 30, 40, 40, 40]
```

```pycon
>>> int(b.fold(lambda x, y: x + y))
30
```

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(dsk: Graph, name: [str](https://docs.python.org/3/library/stdtypes.html#str), npartitions: [int](https://docs.python.org/3/library/functions.html#int))

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.bag.Bag.__init__)(dsk, name, npartitions)                                              |                                                                           |
|-----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|
| [`accumulate`](dask.bag.Bag.accumulate.md#dask.bag.Bag.accumulate)(binop[, initial])                      | Repeatedly apply binary function to a sequence, accumulating results.     |
| [`all`](dask.bag.Bag.all.md#dask.bag.Bag.all)([split_every])                                              | Are all elements truthy?                                                  |
| [`any`](dask.bag.Bag.any.md#dask.bag.Bag.any)([split_every])                                              | Are any of the elements truthy?                                           |
| [`compute`](dask.bag.Bag.compute.md#dask.bag.Bag.compute)(\*\*kwargs)                                     | Compute this dask collection                                              |
| [`count`](dask.bag.Bag.count.md#dask.bag.Bag.count)([split_every])                                        | Count the number of elements.                                             |
| [`distinct`](dask.bag.Bag.distinct.md#dask.bag.Bag.distinct)([key])                                       | Distinct elements of collection                                           |
| [`filter`](dask.bag.Bag.filter.md#dask.bag.Bag.filter)(predicate)                                         | Filter elements in collection by a predicate function.                    |
| [`flatten`](dask.bag.Bag.flatten.md#dask.bag.Bag.flatten)()                                               | Concatenate nested lists into one long list.                              |
| [`fold`](dask.bag.Bag.fold.md#dask.bag.Bag.fold)(binop[, combine, initial, split_every, ...])             | Parallelizable reduction                                                  |
| [`foldby`](dask.bag.Bag.foldby.md#dask.bag.Bag.foldby)(key, binop[, initial, combine, ...])               | Combined reduction and groupby.                                           |
| [`frequencies`](dask.bag.Bag.frequencies.md#dask.bag.Bag.frequencies)([split_every, sort])                | Count number of occurrences of each distinct element.                     |
| [`groupby`](dask.bag.Bag.groupby.md#dask.bag.Bag.groupby)(grouper[, method, npartitions, ...])            | Group collection by key function                                          |
| [`join`](dask.bag.Bag.join.md#dask.bag.Bag.join)(other, on_self[, on_other])                              | Joins collection with another collection.                                 |
| [`map`](dask.bag.Bag.map.md#dask.bag.Bag.map)(func, \*args, \*\*kwargs)                                   | Apply a function elementwise across one or more bags.                     |
| [`map_partitions`](dask.bag.Bag.map_partitions.md#dask.bag.Bag.map_partitions)(func, \*args, \*\*kwargs)  | Apply a function to every partition across one or more bags.              |
| [`max`](dask.bag.Bag.max.md#dask.bag.Bag.max)([split_every])                                              | Maximum element                                                           |
| [`mean`](dask.bag.Bag.mean.md#dask.bag.Bag.mean)()                                                        | Arithmetic mean                                                           |
| [`min`](dask.bag.Bag.min.md#dask.bag.Bag.min)([split_every])                                              | Minimum element                                                           |
| [`persist`](dask.bag.Bag.persist.md#dask.bag.Bag.persist)(\*\*kwargs)                                     | Persist this dask collection into memory                                  |
| [`pluck`](dask.bag.Bag.pluck.md#dask.bag.Bag.pluck)(key[, default])                                       | Select item from all tuples/dicts in collection.                          |
| [`product`](dask.bag.Bag.product.md#dask.bag.Bag.product)(other)                                          | Cartesian product between two bags.                                       |
| [`random_sample`](dask.bag.Bag.random_sample.md#dask.bag.Bag.random_sample)(prob[, random_state])         | Return elements from bag with probability of `prob`.                      |
| [`reduction`](dask.bag.Bag.reduction.md#dask.bag.Bag.reduction)(perpartition, aggregate[, ...])           | Reduce collection with reduction operators.                               |
| [`remove`](dask.bag.Bag.remove.md#dask.bag.Bag.remove)(predicate)                                         | Remove elements in collection that match predicate.                       |
| [`repartition`](dask.bag.Bag.repartition.md#dask.bag.Bag.repartition)([npartitions, partition_size])      | Repartition Bag across new divisions.                                     |
| [`starmap`](dask.bag.Bag.starmap.md#dask.bag.Bag.starmap)(func, \*\*kwargs)                               | Apply a function using argument tuples from the given bag.                |
| [`std`](dask.bag.Bag.std.md#dask.bag.Bag.std)([ddof])                                                     | Standard deviation                                                        |
| [`sum`](dask.bag.Bag.sum.md#dask.bag.Bag.sum)([split_every])                                              | Sum all elements                                                          |
| [`take`](dask.bag.Bag.take.md#dask.bag.Bag.take)(k[, npartitions, compute, warn])                         | Take the first k elements.                                                |
| [`to_avro`](dask.bag.Bag.to_avro.md#dask.bag.Bag.to_avro)(filename, schema[, name_function, ...])         | Write bag to set of avro files                                            |
| [`to_dataframe`](dask.bag.Bag.to_dataframe.md#dask.bag.Bag.to_dataframe)([meta, columns, optimize_graph]) | Create Dask Dataframe from a Dask Bag.                                    |
| [`to_delayed`](dask.bag.Bag.to_delayed.md#dask.bag.Bag.to_delayed)([optimize_graph])                      | Convert into a list of `dask.delayed` objects, one per partition.         |
| [`to_textfiles`](dask.bag.Bag.to_textfiles.md#dask.bag.Bag.to_textfiles)(path[, name_function, ...])      | Write dask Bag to disk, one filename per partition, one line per element. |
| [`topk`](dask.bag.Bag.topk.md#dask.bag.Bag.topk)(k[, key, split_every])                                   | K largest elements in collection                                          |
| `unzip`(n)                                                                                                | Transform a bag of tuples to `n` bags of their elements.                  |
| [`var`](dask.bag.Bag.var.md#dask.bag.Bag.var)([ddof])                                                     | Variance                                                                  |
| [`visualize`](dask.bag.Bag.visualize.md#dask.bag.Bag.visualize)([filename, format, optimize_graph])       | Render the computation of this object's task graph using graphviz.        |

### Attributes

| `str`   | String processing functions   |
|---------|-------------------------------|


# dask.bag.Bag.join.html.md

# dask.bag.Bag.join

#### Bag.join(other, on_self, on_other=None)

Joins collection with another collection.

Other collection must be one of the following:

1. An iterable.  We recommend tuples over lists for internal
   performance reasons.
2. A delayed object, pointing to a tuple.  This is recommended if the
   other collection is sizable and you’re using the distributed
   scheduler.  Dask is able to pass around data wrapped in delayed
   objects with greater sophistication.
3. A Bag with a single partition

You might also consider Dask Dataframe, whose join operations are much
more heavily optimized.

* **Parameters:**
  **other: Iterable, Delayed, Bag**
  : Other collection on which to join

  **on_self: callable**
  : Function to call on elements in this collection to determine a
    match

  **on_other: callable (defaults to on_self)**
  : Function to call on elements in the other collection to determine a
    match

### Examples

```pycon
>>> import dask.bag as db
>>> people = db.from_sequence(['Alice', 'Bob', 'Charlie'])
>>> fruit = ['Apple', 'Apricot', 'Banana']
>>> list(people.join(fruit, lambda x: x[0]))
[('Apple', 'Alice'), ('Apricot', 'Alice'), ('Banana', 'Bob')]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.map.html.md

# dask.bag.Bag.map

#### Bag.map(func, \*args, \*\*kwargs)

Apply a function elementwise across one or more bags.

Note that all `Bag` arguments must be partitioned identically.

* **Parameters:**
  **func**

  **\*args, \*\*kwargs**
  : Extra arguments and keyword arguments to pass to `func` *after*
    the calling bag instance. Non-Bag args/kwargs are broadcasted
    across all calls to `func`.

### Notes

For calls with multiple Bag arguments, corresponding partitions
should have the same length; if they do not, the call will error at
compute time.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5), npartitions=2)
>>> b2 = db.from_sequence(range(5, 10), npartitions=2)
```

Apply a function to all elements in a bag:

```pycon
>>> b.map(lambda x: x + 1).compute()
[1, 2, 3, 4, 5]
```

Apply a function with arguments from multiple bags:

```pycon
>>> from operator import add
>>> b.map(add, b2).compute()
[5, 7, 9, 11, 13]
```

Non-bag arguments are broadcast across all calls to the mapped
function:

```pycon
>>> b.map(add, 1).compute()
[1, 2, 3, 4, 5]
```

Keyword arguments are also supported, and have the same semantics as
regular arguments:

```pycon
>>> def myadd(x, y=0):
...     return x + y
>>> b.map(myadd, y=b2).compute()
[5, 7, 9, 11, 13]
>>> b.map(myadd, y=1).compute()
[1, 2, 3, 4, 5]
```

Both arguments and keyword arguments can also be instances of
`dask.bag.Item`. Here we’ll add the max value in the bag to each
element:

```pycon
>>> b.map(myadd, b.max()).compute()
[4, 5, 6, 7, 8]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.map_partitions.html.md

# dask.bag.Bag.map_partitions

#### Bag.map_partitions(func, \*args, \*\*kwargs)

Apply a function to every partition across one or more bags.

Note that all `Bag` arguments must be partitioned identically.

* **Parameters:**
  **func**
  : The function to be called on every partition.
    This function should expect an `Iterator` or `Iterable` for
    every partition and should return an `Iterator` or `Iterable`
    in return.

  **\*args, \*\*kwargs**
  : Arguments and keyword arguments to pass to `func`.
    Partitions from this bag will be the first argument, and these will
    be passed *after*.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(1, 101), npartitions=10)
>>> def div(nums, den=1):
...     return [num / den for num in nums]
```

Using a python object:

```pycon
>>> hi = b.max().compute()
>>> hi
100
>>> b.map_partitions(div, den=hi).take(5)
(0.01, 0.02, 0.03, 0.04, 0.05)
```

Using an `Item`:

```pycon
>>> b.map_partitions(div, den=b.max()).take(5)
(0.01, 0.02, 0.03, 0.04, 0.05)
```

Note that while both versions give the same output, the second forms a
single graph, and then computes everything at once, and in some cases
may be more efficient.

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.max.html.md

# dask.bag.Bag.max

#### Bag.max(split_every=None)

Maximum element

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.mean.html.md

# dask.bag.Bag.mean

#### Bag.mean()

Arithmetic mean

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.min.html.md

# dask.bag.Bag.min

#### Bag.min(split_every=None)

Minimum element

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.persist.html.md

# dask.bag.Bag.persist

#### Bag.persist(\*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.pluck.html.md

# dask.bag.Bag.pluck

#### Bag.pluck(key, default=<no_default>)

Select item from all tuples/dicts in collection.

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([{'name': 'Alice', 'credits': [1, 2, 3]},
...                       {'name': 'Bob',   'credits': [10, 20]}])
>>> list(b.pluck('name'))
['Alice', 'Bob']
>>> list(b.pluck('credits').pluck(0))
[1, 10]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.product.html.md

# dask.bag.Bag.product

#### Bag.product(other)

Cartesian product between two bags.

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.random_sample.html.md

# dask.bag.Bag.random_sample

#### Bag.random_sample(prob, random_state=None)

Return elements from bag with probability of `prob`.

* **Parameters:**
  **prob**
  : A float between 0 and 1, representing the probability that each
    element will be returned.

  **random_state**
  : If an integer, will be used to seed a new `random.Random` object.
    If provided, results in deterministic sampling.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(10))
>>> b.random_sample(0.5, 43).compute()
[0, 1, 3, 4, 7, 9]
>>> b.random_sample(0.5, 43).compute()
[0, 1, 3, 4, 7, 9]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.reduction.html.md

# dask.bag.Bag.reduction

#### Bag.reduction(perpartition, aggregate, split_every=None, out_type=<class 'dask.bag.core.Item'>, name=None)

Reduce collection with reduction operators.

* **Parameters:**
  **perpartition: function**
  : reduction to apply to each partition

  **aggregate: function**
  : reduction to apply to the results of all partitions

  **split_every: int (optional)**
  : Group partitions into groups of this size while performing reduction
    Defaults to 8

  **out_type: {Bag, Item}**
  : The out type of the result, Item if a single element, Bag if a list
    of elements.  Defaults to Item.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(10))
>>> b.reduction(sum, sum).compute()
45
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.remove.html.md

# dask.bag.Bag.remove

#### Bag.remove(predicate)

Remove elements in collection that match predicate.

```pycon
>>> def iseven(x):
...     return x % 2 == 0
```

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5))
>>> list(b.remove(iseven))
[1, 3]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.repartition.html.md

# dask.bag.Bag.repartition

#### Bag.repartition(npartitions=None, partition_size=None)

Repartition Bag across new divisions.

* **Parameters:**
  **npartitions**
  : Number of partitions of output.

  **partition_size**
  : Max number of bytes of memory for each partition. Use numbers or
    strings like 5MB.
    <br/>
    #### WARNING
    This keyword argument triggers computation to determine
    the memory size of each partition, which may be expensive.

### Notes

Exactly one of `npartitions` or `partition_size` should be specified.
A `ValueError` will be raised when that is not the case.

### Examples

```pycon
>>> b.repartition(5)  # set to have 5 partitions
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.starmap.html.md

# dask.bag.Bag.starmap

#### Bag.starmap(func, \*\*kwargs)

Apply a function using argument tuples from the given bag.

This is similar to `itertools.starmap`, except it also accepts
keyword arguments. In pseudocode, this is could be written as:

```pycon
>>> def starmap(func, bag, **kwargs):
...     return (func(*args, **kwargs) for args in bag)
```

* **Parameters:**
  **func**

  **\*\*kwargs**
  : Extra keyword arguments to pass to `func`. These can either be
    normal objects, `dask.bag.Item`, or `dask.delayed.Delayed`.

### Examples

```pycon
>>> import dask.bag as db
>>> data = [(1, 2), (3, 4), (5, 6), (7, 8), (9, 10)]
>>> b = db.from_sequence(data, npartitions=2)
```

Apply a function to each argument tuple:

```pycon
>>> from operator import add
>>> b.starmap(add).compute()
[3, 7, 11, 15, 19]
```

Apply a function to each argument tuple, with additional keyword
arguments:

```pycon
>>> def myadd(x, y, z=0):
...     return x + y + z
>>> b.starmap(myadd, z=10).compute()
[13, 17, 21, 25, 29]
```

Keyword arguments can also be instances of `dask.bag.Item` or
`dask.delayed.Delayed`:

```pycon
>>> max_second = b.pluck(1).max()
>>> max_second.compute()
10
>>> b.starmap(myadd, z=max_second).compute()
[13, 17, 21, 25, 29]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.std.html.md

# dask.bag.Bag.std

#### Bag.std(ddof=0)

Standard deviation

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.sum.html.md

# dask.bag.Bag.sum

#### Bag.sum(split_every=None)

Sum all elements

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.take.html.md

# dask.bag.Bag.take

#### Bag.take(k, npartitions=1, compute=True, warn=True)

Take the first k elements.

* **Parameters:**
  **k**
  : The number of elements to return

  **npartitions**
  : Elements are only taken from the first `npartitions`, with a
    default of 1. If there are fewer than `k` rows in the first
    `npartitions` a warning will be raised and any found rows
    returned. Pass -1 to use all partitions.

  **compute**
  : Whether to compute the result, default is True.

  **warn**
  : Whether to warn if the number of elements returned is less than
    requested, default is True.

  **>>> import dask.bag as db**

  **>>> b = db.from_sequence(range(1_000))**

  **>>> b.take(3)**

  **(0, 1, 2)**

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.to_avro.html.md

# dask.bag.Bag.to_avro

#### Bag.to_avro(filename, schema, name_function=None, storage_options=None, codec='null', sync_interval=16000, metadata=None, compute=True, \*\*kwargs)

Write bag to set of avro files

The schema is a complex dictionary describing the data, see
[https://avro.apache.org/docs/1.8.2/gettingstartedpython.html#Defining+a+schema](https://avro.apache.org/docs/1.8.2/gettingstartedpython.html#Defining+a+schema)
and [https://fastavro.readthedocs.io/en/latest/writer.html](https://fastavro.readthedocs.io/en/latest/writer.html) .
Its structure is as follows:

```default
{'name': 'Test',
 'namespace': 'Test',
 'doc': 'Descriptive text',
 'type': 'record',
 'fields': [
    {'name': 'a', 'type': 'int'},
 ]}
```

where the “name” field is required, but “namespace” and “doc” are optional
descriptors; “type” must always be “record”. The list of fields should
have an entry for every key of the input records, and the types are
like the primitive, complex or logical types of the Avro spec
( [https://avro.apache.org/docs/1.8.2/spec.html](https://avro.apache.org/docs/1.8.2/spec.html) ).

Results in one avro file per input partition.

* **Parameters:**
  **b: dask.bag.Bag**

  **filename: list of str or str**
  : Filenames to write to. If a list, number must match the number of
    partitions. If a string, must include a glob character “\*”, which will
    be expanded using name_function

  **schema: dict**
  : Avro schema dictionary, see above

  **name_function: None or callable**
  : Expands integers into strings, see
    `dask.bytes.utils.build_name_function`

  **storage_options: None or dict**
  : Extra key/value options to pass to the backend file-system

  **codec: ‘null’, ‘deflate’, or ‘snappy’**
  : Compression algorithm

  **sync_interval: int**
  : Number of records to include in each block within a file

  **metadata: None or dict**
  : Included in the file header

  **compute: bool**
  : If True, files are written immediately, and function blocks. If False,
    returns delayed objects, which can be computed by the user where
    convenient.

  **kwargs: passed to compute(), if compute=True**

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([{'name': 'Alice', 'value': 100},
...                       {'name': 'Bob', 'value': 200}])
>>> schema = {'name': 'People', 'doc': "Set of people's scores",
...           'type': 'record',
...           'fields': [
...               {'name': 'name', 'type': 'string'},
...               {'name': 'value', 'type': 'int'}]}
>>> b.to_avro('my-data.*.avro', schema)
['my-data.0.avro', 'my-data.1.avro']
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.to_dataframe.html.md

# dask.bag.Bag.to_dataframe

#### Bag.to_dataframe(meta=None, columns=None, optimize_graph=True)

Create Dask Dataframe from a Dask Bag.

Bag should contain tuples, dict records, or scalars.

Index will not be particularly meaningful.  Use `reindex` afterwards
if necessary.

* **Parameters:**
  **meta**
  : An empty `pd.DataFrame` that matches the dtypes and column names
    of the output. This metadata is necessary for many algorithms in
    dask dataframe to work.  For ease of use, some alternative inputs
    are also available. Instead of a `DataFrame`, a `dict` of
    `{name: dtype}` or iterable of `(name, dtype)` can be provided.
    If not provided or a list, a single element from the first
    partition will be computed, triggering a potentially expensive call
    to `compute`. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

  **columns**
  : Column names to use. If the passed data do not have names
    associated with them, this argument provides names for the columns.
    Otherwise this argument indicates the order of the columns in the
    result (any names not found in the data will become all-NA
    columns).  Note that if `meta` is provided, column names will be
    taken from there and this parameter is invalid.

  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    [`dask.dataframe.DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame).

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([{'name': 'Alice',   'balance': 100},
...                       {'name': 'Bob',     'balance': 200},
...                       {'name': 'Charlie', 'balance': 300}],
...                      npartitions=2)
>>> df = b.to_dataframe()
```

```pycon
>>> df.compute()
      name  balance
0    Alice      100
1      Bob      200
0  Charlie      300
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.to_delayed.html.md

# dask.bag.Bag.to_delayed

#### Bag.to_delayed(optimize_graph=True)

Convert into a list of `dask.delayed` objects, one per partition.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

#### SEE ALSO
[`dask.bag.from_delayed`](dask.bag.from_delayed.md#dask.bag.from_delayed)

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.to_textfiles.html.md

# dask.bag.Bag.to_textfiles

#### Bag.to_textfiles(path, name_function=None, compression='infer', encoding='utf-8', compute=True, storage_options=None, last_endline=False, \*\*kwargs)

Write dask Bag to disk, one filename per partition, one line per element.

**Paths**: This will create one file for each partition in your bag. You
can specify the filenames in a variety of ways.

Use a globstring

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz')
```

The \* will be replaced by the increasing sequence 1, 2, …

```default
/path/to/data/0.json.gz
/path/to/data/1.json.gz
```

Use a globstring and a `name_function=` keyword argument.  The
name_function function should expect an integer and produce a string.
Strings produced by name_function must preserve the order of their
respective partition indices.

```pycon
>>> from datetime import date, timedelta
>>> def name(i):
...     return str(date(2015, 1, 1) + i * timedelta(days=1))
```

```pycon
>>> name(0)
'2015-01-01'
>>> name(15)
'2015-01-16'
```

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz', name_function=name)
```

```default
/path/to/data/2015-01-01.json.gz
/path/to/data/2015-01-02.json.gz
...
```

You can also provide an explicit list of paths.

```pycon
>>> paths = ['/path/to/data/alice.json.gz', '/path/to/data/bob.json.gz', ...]
>>> b.to_textfiles(paths)
```

**Compression**: Filenames with extensions corresponding to known
compression algorithms (gz, bz2) will be compressed accordingly.

**Bag Contents**: The bag calling `to_textfiles` must be a bag of
text strings. For example, a bag of dictionaries could be written to
JSON text files by mapping `json.dumps` on to the bag first, and
then calling `to_textfiles` :

```pycon
>>> b_dict.map(json.dumps).to_textfiles("/path/to/data/*.json")
```

**Last endline**: By default the last line does not end with a newline
character. Pass `last_endline=True` to invert the default.

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.topk.html.md

# dask.bag.Bag.topk

#### Bag.topk(k, key=None, split_every=None)

K largest elements in collection

Optionally ordered by some key function

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence([10, 3, 5, 7, 11, 4])
>>> list(b.topk(2))
[11, 10]
```

```pycon
>>> list(b.topk(2, lambda x: -x))
[3, 4]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.var.html.md

# dask.bag.Bag.var

#### Bag.var(ddof=0)

Variance

<!-- !! processed by numpydoc !! -->


# dask.bag.Bag.visualize.html.md

# dask.bag.Bag.visualize

#### Bag.visualize(filename='mydask', format=None, optimize_graph=False, \*\*kwargs)

Render the computation of this object’s task graph using graphviz.

Requires `graphviz` to be installed.

* **Parameters:**
  **filename**
  : The name of the file to write to disk. If the provided filename
    doesn’t include an extension, ‘.png’ will be used by default.
    If filename is None, no file will be written, and we communicate
    with dot using only pipes.

  **format**
  : Format in which to write output file.  Default is ‘png’.

  **optimize_graph**
  : If True, the graph is optimized before rendering.  Otherwise,
    the graph is displayed as is. Default is False.

  **color: {None, ‘order’}, optional**
  : Options to color nodes.  Provide `cmap=` keyword for additional
    colormap

  **\*\*kwargs**
  : Additional keyword arguments to forward to `to_graphviz`.
* **Returns:**
  **result**
  : See dask.dot.dot_graph for more information.

#### SEE ALSO
[`dask.visualize`](../api.md#dask.visualize)

`dask.dot.dot_graph`

### Notes

For more information on optimization see here:

[https://docs.dask.org/en/latest/optimize.html](https://docs.dask.org/en/latest/optimize.html)

### Examples

```pycon
>>> x.visualize(filename='dask.pdf')
>>> x.visualize(filename='dask.pdf', color='order')
```

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.apply.html.md

# dask.bag.Item.apply

#### Item.apply(func)

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.compute.html.md

# dask.bag.Item.compute

#### Item.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.from_delayed.html.md

# dask.bag.Item.from_delayed

#### *static* Item.from_delayed(value)

Create bag item from a dask.delayed value.

See `dask.bag.from_delayed` for details

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.html.md

# dask.bag.Item

### *class* dask.bag.Item(dsk, key, layer=None)

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(dsk, key, layer=None)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.bag.Item.__init__)(dsk, key[, layer])                                              |                                                                    |
|-------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
| [`apply`](dask.bag.Item.apply.md#dask.bag.Item.apply)(func)                                           |                                                                    |
| [`compute`](dask.bag.Item.compute.md#dask.bag.Item.compute)(\*\*kwargs)                               | Compute this dask collection                                       |
| [`from_delayed`](dask.bag.Item.from_delayed.md#dask.bag.Item.from_delayed)(value)                     | Create bag item from a dask.delayed value.                         |
| [`persist`](dask.bag.Item.persist.md#dask.bag.Item.persist)(\*\*kwargs)                               | Persist this dask collection into memory                           |
| [`to_delayed`](dask.bag.Item.to_delayed.md#dask.bag.Item.to_delayed)([optimize_graph])                | Convert into a `dask.delayed` object.                              |
| [`visualize`](dask.bag.Item.visualize.md#dask.bag.Item.visualize)([filename, format, optimize_graph]) | Render the computation of this object's task graph using graphviz. |


# dask.bag.Item.persist.html.md

# dask.bag.Item.persist

#### Item.persist(\*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.to_delayed.html.md

# dask.bag.Item.to_delayed

#### Item.to_delayed(optimize_graph=True)

Convert into a `dask.delayed` object.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

<!-- !! processed by numpydoc !! -->


# dask.bag.Item.visualize.html.md

# dask.bag.Item.visualize

#### Item.visualize(filename='mydask', format=None, optimize_graph=False, \*\*kwargs)

Render the computation of this object’s task graph using graphviz.

Requires `graphviz` to be installed.

* **Parameters:**
  **filename**
  : The name of the file to write to disk. If the provided filename
    doesn’t include an extension, ‘.png’ will be used by default.
    If filename is None, no file will be written, and we communicate
    with dot using only pipes.

  **format**
  : Format in which to write output file.  Default is ‘png’.

  **optimize_graph**
  : If True, the graph is optimized before rendering.  Otherwise,
    the graph is displayed as is. Default is False.

  **color: {None, ‘order’}, optional**
  : Options to color nodes.  Provide `cmap=` keyword for additional
    colormap

  **\*\*kwargs**
  : Additional keyword arguments to forward to `to_graphviz`.
* **Returns:**
  **result**
  : See dask.dot.dot_graph for more information.

#### SEE ALSO
[`dask.visualize`](../api.md#dask.visualize)

`dask.dot.dot_graph`

### Notes

For more information on optimization see here:

[https://docs.dask.org/en/latest/optimize.html](https://docs.dask.org/en/latest/optimize.html)

### Examples

```pycon
>>> x.visualize(filename='dask.pdf')
>>> x.visualize(filename='dask.pdf', color='order')
```

<!-- !! processed by numpydoc !! -->


# dask.bag.concat.html.md

# dask.bag.concat

### dask.bag.concat(bags)

Concatenate many bags together, unioning all elements.

```pycon
>>> import dask.bag as db
>>> a = db.from_sequence([1, 2, 3])
>>> b = db.from_sequence([4, 5, 6])
>>> c = db.concat([a, b])
```

```pycon
>>> list(c)
[1, 2, 3, 4, 5, 6]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.from_delayed.html.md

# dask.bag.from_delayed

### dask.bag.from_delayed(values)

Create bag from many dask Delayed objects.

These objects will become the partitions of the resulting Bag.  They should
evaluate to a `list` or some other concrete sequence.

* **Parameters:**
  **values: list of delayed values**
  : An iterable of dask Delayed objects.  Each evaluating to a list.
* **Returns:**
  Bag

#### SEE ALSO
`dask.delayed`

### Examples

```pycon
>>> x, y, z = [delayed(load_sequence_from_file)(fn)
...             for fn in filenames]
>>> b = from_delayed([x, y, z])
```

<!-- !! processed by numpydoc !! -->


# dask.bag.from_sequence.html.md

# dask.bag.from_sequence

### dask.bag.from_sequence(seq, partition_size=None, npartitions=None)

Create a dask Bag from Python sequence.

This sequence should be relatively small in memory.  Dask Bag works
best when it handles loading your data itself.  Commonly we load a
sequence of filenames into a Bag and then use `.map` to open them.

* **Parameters:**
  **seq: Iterable**
  : A sequence of elements to put into the dask

  **partition_size: int (optional)**
  : The length of each partition

  **npartitions: int (optional)**
  : The number of desired partitions

  **It is best to provide either \`\`partition_size\`\` or \`\`npartitions\`\`**

  **(though not both.)**

#### SEE ALSO
[`read_text`](dask.bag.read_text.md#dask.bag.read_text)
: Create bag from text files

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(['Alice', 'Bob', 'Chuck'], partition_size=2)
```

<!-- !! processed by numpydoc !! -->


# dask.bag.from_url.html.md

# dask.bag.from_url

### dask.bag.from_url(urls)

Create a dask Bag from a url.

### Examples

```pycon
>>> a = from_url('http://raw.githubusercontent.com/dask/dask/main/README.rst')
>>> a.npartitions
1
```

```pycon
>>> a.take(8)
(b'Dask\n',
 b'====\n',
 b'\n',
 b'|Build Status| |Coverage| |Doc Status| |Discourse| |Version Status| |NumFOCUS|\n',
 b'\n',
 b'Dask is a flexible parallel computing library for analytics.  See\n',
 b'documentation_ for more information.\n',
 b'\n')
```

```pycon
>>> b = from_url(['http://github.com', 'http://google.com'])
>>> b.npartitions
2
```

<!-- !! processed by numpydoc !! -->


# dask.bag.map.html.md

# dask.bag.map

### dask.bag.map(func, \*args, \*\*kwargs)

Apply a function elementwise across one or more bags.

Note that all `Bag` arguments must be partitioned identically.

* **Parameters:**
  **func**

  **\*args, \*\*kwargs**
  : Arguments and keyword arguments to pass to `func`. Non-Bag args/kwargs
    are broadcasted across all calls to `func`.

### Notes

For calls with multiple Bag arguments, corresponding partitions should
have the same length; if they do not, the call will error at compute time.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(5), npartitions=2)
>>> b2 = db.from_sequence(range(5, 10), npartitions=2)
```

Apply a function to all elements in a bag:

```pycon
>>> db.map(lambda x: x + 1, b).compute()
[1, 2, 3, 4, 5]
```

Apply a function with arguments from multiple bags:

```pycon
>>> from operator import add
>>> db.map(add, b, b2).compute()
[5, 7, 9, 11, 13]
```

Non-bag arguments are broadcast across all calls to the mapped function:

```pycon
>>> db.map(add, b, 1).compute()
[1, 2, 3, 4, 5]
```

Keyword arguments are also supported, and have the same semantics as
regular arguments:

```pycon
>>> def myadd(x, y=0):
...     return x + y
>>> db.map(myadd, b, y=b2).compute()
[5, 7, 9, 11, 13]
>>> db.map(myadd, b, y=1).compute()
[1, 2, 3, 4, 5]
```

Both arguments and keyword arguments can also be instances of
`dask.bag.Item` or `dask.delayed.Delayed`. Here we’ll add the max value
in the bag to each element:

```pycon
>>> db.map(myadd, b, b.max()).compute()
[4, 5, 6, 7, 8]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.map_partitions.html.md

# dask.bag.map_partitions

### dask.bag.map_partitions(func, \*args, \*\*kwargs)

Apply a function to every partition across one or more bags.

Note that all `Bag` arguments must be partitioned identically.

* **Parameters:**
  **func**

  **\*args, \*\*kwargs**
  : Arguments and keyword arguments to pass to `func`.

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.from_sequence(range(1, 101), npartitions=10)
>>> def div(nums, den=1):
...     return [num / den for num in nums]
```

Using a python object:

```pycon
>>> hi = b.max().compute()
>>> hi
100
>>> b.map_partitions(div, den=hi).take(5)
(0.01, 0.02, 0.03, 0.04, 0.05)
```

Using an `Item`:

```pycon
>>> b.map_partitions(div, den=b.max()).take(5)
(0.01, 0.02, 0.03, 0.04, 0.05)
```

Note that while both versions give the same output, the second forms a
single graph, and then computes everything at once, and in some cases
may be more efficient.

<!-- !! processed by numpydoc !! -->


# dask.bag.random.choices.html.md

# dask.bag.random.choices

### dask.bag.random.choices(population, k=1, split_every=None)

Return a k sized list of elements chosen with replacement.

* **Parameters:**
  **population: Bag**
  : Elements to sample.

  **k: integer, optional**
  : Number of elements to sample.

  **split_every: int (optional)**
  : Group partitions into groups of this size while performing reduction.
    Defaults to 8.

### Examples

```pycon
>>> import dask.bag as db
>>> from dask.bag import random
>>> b = db.from_sequence(range(5), npartitions=2)
>>> list(random.choices(b, 3).compute())
[1, 1, 5]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.random.sample.html.md

# dask.bag.random.sample

### dask.bag.random.sample(population, k, split_every=None)

Chooses k unique random elements from a bag.

Returns a new bag containing elements from the population while
leaving the original population unchanged.

* **Parameters:**
  **population: Bag**
  : Elements to sample.

  **k: integer, optional**
  : Number of elements to sample.

  **split_every: int (optional)**
  : Group partitions into groups of this size while performing reduction.
    Defaults to 8.

### Examples

```pycon
>>> import dask.bag as db
>>> from dask.bag import random
>>> b = db.from_sequence(range(5), npartitions=2)
>>> list(random.sample(b, 3).compute())
[1, 3, 5]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.range.html.md

# dask.bag.range

### dask.bag.range(n, npartitions)

Numbers from zero to n

### Examples

```pycon
>>> import dask.bag as db
>>> b = db.range(5, npartitions=2)
>>> list(b)
[0, 1, 2, 3, 4]
```

<!-- !! processed by numpydoc !! -->


# dask.bag.read_avro.html.md

# dask.bag.read_avro

### dask.bag.read_avro(urlpath, blocksize=100000000, storage_options=None, compression=None)

Read set of avro files

Use this with arbitrary nested avro schemas. Please refer to the
fastavro documentation for its capabilities:
[https://github.com/fastavro/fastavro](https://github.com/fastavro/fastavro)

* **Parameters:**
  **urlpath: string or list**
  : Absolute or relative filepath, URL (may include protocols like
    `s3://`), or globstring pointing to data.

  **blocksize: int or None**
  : Size of chunks in bytes. If None, there will be no chunking and each
    file will become one partition.

  **storage_options: dict or None**
  : passed to backend file-system

  **compression: str or None**
  : Compression format of the targe(s), like ‘gzip’. Should only be used
    with blocksize=None.

<!-- !! processed by numpydoc !! -->


# dask.bag.read_text.html.md

# dask.bag.read_text

### dask.bag.read_text(urlpath, blocksize=None, compression='infer', encoding='utf-8', errors='strict', linedelimiter=None, collection=True, storage_options=None, files_per_partition=None, include_path=False)

Read lines from text files

* **Parameters:**
  **urlpath**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to read from alternative filesystems. To read from multiple files you
    can pass a globstring or a list of paths, with the caveat that they
    must all have the same protocol.

  **blocksize: None, int, or str**
  : Size (in bytes) to cut up larger files.  Streams by default.
    Can be `None` for streaming, an integer number of bytes, or a string
    like “128MiB”

  **compression: string**
  : Compression format like ‘gzip’ or ‘xz’.  Defaults to ‘infer’

  **encoding: string**

  **errors: string**

  **linedelimiter: string or None**

  **collection: bool, optional**
  : Return dask.bag if True, or list of delayed values if false

  **storage_options: dict**
  : Extra options that make sense to a particular storage connection, e.g.
    host, port, username, password, etc.

  **files_per_partition: None or int**
  : If set, group input files into partitions of the requested size,
    instead of one partition per file. Mutually exclusive with blocksize.

  **include_path: bool**
  : Whether or not to include the path in the bag.
    If true, elements are tuples of (line, path).
    Default is False.
* **Returns:**
  dask.bag.Bag or list
  : dask.bag.Bag if collection is True or list of Delayed lists otherwise.

#### SEE ALSO
[`from_sequence`](dask.bag.from_sequence.md#dask.bag.from_sequence)
: Build bag from Python sequence

### Examples

```pycon
>>> b = read_text('myfiles.1.txt')
>>> b = read_text('myfiles.*.txt')
>>> b = read_text('myfiles.*.txt.gz')
>>> b = read_text('s3://bucket/myfiles.*.txt')
>>> b = read_text('s3://key:secret@bucket/myfiles.*.txt')
>>> b = read_text('hdfs://namenode.example.com/myfiles.*.txt')
```

Parallelize a large file by providing the number of uncompressed bytes to
load into each partition.

```pycon
>>> b = read_text('largefile.txt', blocksize='10MB')
```

Get file paths of the bag by setting include_path=True

```pycon
>>> b = read_text('myfiles.*.txt', include_path=True)
>>> b.take(1)
(('first line of the first file', '/home/dask/myfiles.0.txt'),)
```

<!-- !! processed by numpydoc !! -->


# dask.bag.to_textfiles.html.md

# dask.bag.to_textfiles

### dask.bag.to_textfiles(b, path, name_function=None, compression='infer', encoding='utf-8', compute=True, storage_options=None, last_endline=False, \*\*kwargs)

Write dask Bag to disk, one filename per partition, one line per element.

**Paths**: This will create one file for each partition in your bag. You
can specify the filenames in a variety of ways.

Use a globstring

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz')
```

The \* will be replaced by the increasing sequence 1, 2, …

```default
/path/to/data/0.json.gz
/path/to/data/1.json.gz
```

Use a globstring and a `name_function=` keyword argument.  The
name_function function should expect an integer and produce a string.
Strings produced by name_function must preserve the order of their
respective partition indices.

```pycon
>>> from datetime import date, timedelta
>>> def name(i):
...     return str(date(2015, 1, 1) + i * timedelta(days=1))
```

```pycon
>>> name(0)
'2015-01-01'
>>> name(15)
'2015-01-16'
```

```pycon
>>> b.to_textfiles('/path/to/data/*.json.gz', name_function=name)
```

```default
/path/to/data/2015-01-01.json.gz
/path/to/data/2015-01-02.json.gz
...
```

You can also provide an explicit list of paths.

```pycon
>>> paths = ['/path/to/data/alice.json.gz', '/path/to/data/bob.json.gz', ...]
>>> b.to_textfiles(paths)
```

**Compression**: Filenames with extensions corresponding to known
compression algorithms (gz, bz2) will be compressed accordingly.

**Bag Contents**: The bag calling `to_textfiles` must be a bag of
text strings. For example, a bag of dictionaries could be written to
JSON text files by mapping `json.dumps` on to the bag first, and
then calling `to_textfiles` :

```pycon
>>> b_dict.map(json.dumps).to_textfiles("/path/to/data/*.json")
```

**Last endline**: By default the last line does not end with a newline
character. Pass `last_endline=True` to invert the default.

<!-- !! processed by numpydoc !! -->


# dask.bag.zip.html.md

# dask.bag.zip

### dask.bag.zip(\*bags)

Partition-wise bag zip

All passed bags must have the same number of partitions.

NOTE: corresponding partitions should have the same length; if they do not,
the “extra” elements from the longer partition(s) will be dropped.  If you
have this case chances are that what you really need is a data alignment
mechanism like pandas’s, and not a missing value filler like zip_longest.

### Examples

Correct usage:

```pycon
>>> import dask.bag as db
>>> evens = db.from_sequence(range(0, 10, 2), partition_size=4)
>>> odds = db.from_sequence(range(1, 10, 2), partition_size=4)
>>> pairs = db.zip(evens, odds)
>>> list(pairs)
[(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
```

Incorrect usage:

```pycon
>>> numbers = db.range(31, npartitions=1)
>>> fizz = numbers.filter(lambda n: n % 3 == 0)
>>> buzz = numbers.filter(lambda n: n % 5 == 0)
>>> fizzbuzz = db.zip(fizz, buzz)
>>> list(fizzbuzz)
[(0, 0), (3, 5), (6, 10), (9, 15), (12, 20), (15, 25), (18, 30)]
```

When what you really wanted was more along the lines of the following:

```pycon
>>> list(fizzbuzz)
(0, 0), (3, None), (None, 5), (6, None), (9, None), (None, 10),
(12, None), (15, 15), (18, None), (None, 20),
(21, None), (24, None), (None, 25), (27, None), (30, 30)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Aggregation.html.md

# dask.dataframe.Aggregation

### *class* dask.dataframe.Aggregation(name, chunk, agg, finalize=None)

User defined groupby-aggregation.

This class allows users to define their own custom aggregation in terms of
operations on Pandas dataframes in a map-reduce style. You need to specify
what operation to do on each chunk of data, how to combine those chunks of
data together, and then how to finalize the result.

See [Aggregate](../dataframe-groupby.md#dataframe-groupby-aggregate) for more.

* **Parameters:**
  **name**
  : the name of the aggregation. It should be unique, since intermediate
    result will be identified by this name.

  **chunk**
  : a function that will be called with the grouped column of each
    partition, takes a Pandas SeriesGroupBy in input.
    It can either return a single series or a tuple of series.
    The index has to be equal to the groups.

  **agg**
  : a function that will be called to aggregate the results of each chunk.
    Again the argument(s) will be a Pandas SeriesGroupBy. If `chunk`
    returned a tuple, `agg` will be called with all of them as
    individual positional arguments.

  **finalize**
  : an optional finalizer that will be called with the results from the
    aggregation.

### Examples

We could implement `sum` as follows:

```pycon
>>> custom_sum = dd.Aggregation(
...     name='custom_sum',
...     chunk=lambda s: s.sum(),
...     agg=lambda s0: s0.sum()
... )
>>> df.groupby('g').agg(custom_sum)
```

We can implement `mean` as follows:

```pycon
>>> custom_mean = dd.Aggregation(
...     name='custom_mean',
...     chunk=lambda s: (s.count(), s.sum()),
...     agg=lambda count, sum: (count.sum(), sum.sum()),
...     finalize=lambda count, sum: sum / count,
... )
>>> df.groupby('g').agg(custom_mean)
```

Though of course, both of these are built-in and so you don’t need to
implement them yourself.

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(name, chunk, agg, finalize=None)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.dataframe.Aggregation.__init__)(name, chunk, agg[, finalize])   |    |
|------------------------------------------------------------------------------------|----|


# dask.dataframe.DataFrame.abs.html.md

# dask.dataframe.DataFrame.abs

#### DataFrame.abs()

Return a Series/DataFrame with absolute numeric value of each element.

This docstring was copied from pandas.DataFrame.abs.

Some inconsistencies with the Dask version may exist.

This function only applies to elements that are all numeric.

* **Returns:**
  abs
  : Series/DataFrame containing the absolute value of each element.

#### SEE ALSO
[`numpy.absolute`](https://numpy.org/doc/stable/reference/generated/numpy.absolute.html#numpy.absolute)
: Calculate the absolute value element-wise.

### Notes

For `complex` inputs, `1.2 + 1j`, the absolute value is
$\sqrt{ a^2 + b^2 }$.

### Examples

Absolute numeric values in a Series.

```pycon
>>> s = pd.Series([-1.10, 2, -3.33, 4])
>>> s.abs()
0    1.10
1    2.00
2    3.33
3    4.00
dtype: float64
```

Absolute numeric values in a Series with complex numbers.

```pycon
>>> s = pd.Series([1.2 + 1j])
>>> s.abs()
0    1.56205
dtype: float64
```

Absolute numeric values in a Series with a Timedelta element.

```pycon
>>> s = pd.Series([pd.Timedelta("1 days")])
>>> s.abs()
0   1 days
dtype: timedelta64[us]
```

Select rows with data closest to certain value using argsort (from
[StackOverflow](https://stackoverflow.com/a/17758115)).

```pycon
>>> df = pd.DataFrame(
...     {"a": [4, 5, 6, 7], "b": [10, 20, 30, 40], "c": [100, 50, -30, -50]}
... )
>>> df
     a    b    c
0    4   10  100
1    5   20   50
2    6   30  -30
3    7   40  -50
>>> df.loc[(df.c - 43).abs().argsort()]
     a    b    c
1    5   20   50
0    4   10  100
2    6   30  -30
3    7   40  -50
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.add.html.md

# dask.dataframe.DataFrame.add

#### DataFrame.add(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.align.html.md

# dask.dataframe.DataFrame.align

#### DataFrame.align(other, join='outer', axis=None, fill_value=None)

Align two objects on their axes with the specified join method.

This docstring was copied from pandas.DataFrame.align.

Some inconsistencies with the Dask version may exist.

Join method is specified for each axis Index.

* **Parameters:**
  **other**
  : The object to align with.

  **join**
  : Type of alignment to be performed.
    * left: use only keys from left frame, preserve key order.
    * right: use only keys from right frame, preserve key order.
    * outer: use union of keys from both frames, sort keys lexicographically.
    * inner: use intersection of keys from both frames,
      preserve the order of the left keys.

  **axis**
  : Align on index (0), columns (1), or both (None).

  **level**
  : Broadcast across a level, matching Index values on the
    passed MultiIndex level.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **fill_value**
  : Value to use for missing values. Defaults to NaN, but can be any
    “compatible” value.
* **Returns:**
  tuple of (Series/DataFrame, type of other)
  : Aligned objects.

#### SEE ALSO
[`Series.align`](dask.dataframe.Series.align.md#dask.dataframe.Series.align)
: Align two objects on their axes with specified join method.

[`DataFrame.align`](#dask.dataframe.DataFrame.align)
: Align two objects on their axes with specified join method.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [[1, 2, 3, 4], [6, 7, 8, 9]], columns=["D", "B", "E", "A"], index=[1, 2]
... )
>>> other = pd.DataFrame(
...     [[10, 20, 30, 40], [60, 70, 80, 90], [600, 700, 800, 900]],
...     columns=["A", "B", "C", "D"],
...     index=[2, 3, 4],
... )
>>> df
   D  B  E  A
1  1  2  3  4
2  6  7  8  9
>>> other
    A    B    C    D
2   10   20   30   40
3   60   70   80   90
4  600  700  800  900
```

Align on columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=1)
>>> left
   A  B   C  D  E
1  4  2 NaN  1  3
2  9  7 NaN  6  8
>>> right
    A    B    C    D   E
2   10   20   30   40 NaN
3   60   70   80   90 NaN
4  600  700  800  900 NaN
```

We can also align on the index:

```pycon
>>> left, right = df.align(other, join="outer", axis=0)
>>> left
    D    B    E    A
1  1.0  2.0  3.0  4.0
2  6.0  7.0  8.0  9.0
3  NaN  NaN  NaN  NaN
4  NaN  NaN  NaN  NaN
>>> right
    A      B      C      D
1    NaN    NaN    NaN    NaN
2   10.0   20.0   30.0   40.0
3   60.0   70.0   80.0   90.0
4  600.0  700.0  800.0  900.0
```

Finally, the default axis=None will align on both index and columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=None)
>>> left
     A    B   C    D    E
1  4.0  2.0 NaN  1.0  3.0
2  9.0  7.0 NaN  6.0  8.0
3  NaN  NaN NaN  NaN  NaN
4  NaN  NaN NaN  NaN  NaN
>>> right
       A      B      C      D   E
1    NaN    NaN    NaN    NaN NaN
2   10.0   20.0   30.0   40.0 NaN
3   60.0   70.0   80.0   90.0 NaN
4  600.0  700.0  800.0  900.0 NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.all.html.md

# dask.dataframe.DataFrame.all

#### DataFrame.all(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether all elements are True, potentially over an axis.

This docstring was copied from pandas.DataFrame.all.

Some inconsistencies with the Dask version may exist.

Returns True unless there at least one element within a series or
along a Dataframe axis that is False or equivalent (e.g. zero or
empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be True, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`Series.all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)
: Return True if all elements are True.

[`DataFrame.any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)
: Return True if one (or more) elements are True.

### Examples

**Series**

```pycon
>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([], dtype="float64").all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True
```

**DataFrames**

Create a DataFrame from a dictionary.

```pycon
>>> df = pd.DataFrame({"col1": [True, True], "col2": [True, False]})
>>> df
   col1   col2
0  True   True
1  True  False
```

Default behaviour checks if values in each column all return True.

```pycon
>>> df.all()
col1     True
col2    False
dtype: bool
```

Specify `axis='columns'` to check if values in each row all return True.

```pycon
>>> df.all(axis="columns")
0     True
1    False
dtype: bool
```

Or `axis=None` for whether every value is True.

```pycon
>>> df.all(axis=None)
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.analyze.html.md

# dask.dataframe.DataFrame.analyze

#### DataFrame.analyze(filename: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, format: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Outputs statistics about every node in the expression.

analyze optimizes the expression and triggers a computation. It records statistics
like memory usage per partition to analyze how data flow through the graph.

#### WARNING
analyze adds plugins to the scheduler and the workers that have a non-trivial
cost. This method should not be used in production workflows.

* **Parameters:**
  **filename: str, None**
  : File to store the graph representation.

  **format: str, default is png**
  : File format for the graph representation.
* **Returns:**
  None, but writes a graph representation of the expression enriched with

  statistics to disk.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.any.html.md

# dask.dataframe.DataFrame.any

#### DataFrame.any(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether any element is True, potentially over an axis.

This docstring was copied from pandas.DataFrame.any.

Some inconsistencies with the Dask version may exist.

Returns False unless there is at least one element within a series or
along a Dataframe axis that is True or equivalent (e.g. non-zero or
non-empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be False, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`numpy.any`](https://numpy.org/doc/stable/reference/generated/numpy.any.html#numpy.any)
: Numpy version of this method.

[`Series.any`](dask.dataframe.Series.any.md#dask.dataframe.Series.any)
: Return whether any element is True.

[`Series.all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)
: Return whether all elements are True.

[`DataFrame.any`](#dask.dataframe.DataFrame.any)
: Return whether any element is True over requested axis.

[`DataFrame.all`](dask.dataframe.DataFrame.all.md#dask.dataframe.DataFrame.all)
: Return whether all elements are True over requested axis.

### Examples

**Series**

For Series input, the output is a scalar indicating whether any element
is True.

```pycon
>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([], dtype="float64").any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True
```

**DataFrame**

Whether each column contains at least one True element (the default).

```pycon
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
   A  B  C
0  1  0  0
1  2  2  0
```

```pycon
>>> df.any()
A     True
B     True
C    False
dtype: bool
```

Aggregating over the columns.

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
       A  B
0   True  1
1  False  2
```

```pycon
>>> df.any(axis="columns")
0    True
1    True
dtype: bool
```

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
       A  B
0   True  1
1  False  0
```

```pycon
>>> df.any(axis="columns")
0    True
1    False
dtype: bool
```

Aggregating over the entire DataFrame with `axis=None`.

```pycon
>>> df.any(axis=None)
True
```

any for an empty DataFrame is an empty Series.

```pycon
>>> pd.DataFrame([]).any()
Series([], dtype: bool)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.apply.html.md

# dask.dataframe.DataFrame.apply

#### DataFrame.apply(function, \*args, meta=<no_default>, axis=0, \*\*kwargs)

Parallel version of pandas.DataFrame.apply

This mimics the pandas version except for the following:

1. Only `axis=1` is supported (and must be specified explicitly).
2. The user should provide output metadata via the meta keyword.

* **Parameters:**
  **func**
  : Function to apply to each column/row

  **axis**
  : - 0 or ‘index’: apply function to each column (NOT SUPPORTED)
    - 1 or ‘columns’: apply function to each row

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

  **args**
  : Positional arguments to pass to function in addition to the array/series

  **Additional keyword arguments will be passed as keywords to the function**
* **Returns:**
  **applied**

#### SEE ALSO
[`DataFrame.map_partitions`](dask.dataframe.DataFrame.map_partitions.md#dask.dataframe.DataFrame.map_partitions)

### Examples

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 3, 4, 5],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

Apply a function to row-wise passing in extra arguments in `args` and
`kwargs`:

```pycon
>>> def myadd(row, a, b=1):
...     return row.sum() + a + b
>>> res = ddf.apply(myadd, axis=1, args=(2,), b=1.5)
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with name `'x'`, and dtype
`float64`:

```pycon
>>> res = ddf.apply(myadd, axis=1, args=(2,), b=1.5, meta=('x', 'f8'))
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ddf.apply(lambda row: row + 1, axis=1, meta=ddf)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.assign.html.md

# dask.dataframe.DataFrame.assign

#### DataFrame.assign(\*\*pairs)

Assign new columns to a DataFrame.

This docstring was copied from pandas.DataFrame.assign.

Some inconsistencies with the Dask version may exist.

Returns a new object with all original columns in addition to new ones.
Existing columns that are re-assigned will be overwritten.

* **Parameters:**
  **\*\*kwargs**
  : The column names are keywords. If the values are
    callable, they are computed on the DataFrame and
    assigned to the new columns. The callable must not
    change input DataFrame (though pandas doesn’t check it).
    If the values are not callable, (e.g. a Series, scalar, or array),
    they are simply assigned.
* **Returns:**
  DataFrame
  : A new DataFrame with the new columns in addition to
    all the existing columns.

#### SEE ALSO
[`DataFrame.loc`](dask.dataframe.DataFrame.loc.md#dask.dataframe.DataFrame.loc)
: Select a subset of a DataFrame by labels.

[`DataFrame.iloc`](dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)
: Select a subset of a DataFrame by positions.

### Notes

Assigning multiple columns within the same `assign` is possible.
Later items in ‘\*\*kwargs’ may refer to newly created or modified
columns in ‘df’; items are computed and assigned into ‘df’ in order.

### Examples

```pycon
>>> df = pd.DataFrame({"temp_c": [17.0, 25.0]}, index=["Portland", "Berkeley"])
>>> df
          temp_c
Portland    17.0
Berkeley    25.0
```

Where the value is a callable, evaluated on df:

```pycon
>>> df.assign(temp_f=lambda x: x.temp_c * 9 / 5 + 32)
          temp_c  temp_f
Portland    17.0    62.6
Berkeley    25.0    77.0
```

Alternatively, the same behavior can be achieved by directly
referencing an existing Series or sequence:

```pycon
>>> df.assign(temp_f=df["temp_c"] * 9 / 5 + 32)
          temp_c  temp_f
Portland    17.0    62.6
Berkeley    25.0    77.0
```

or by using `pandas.col()`:

```pycon
>>> df.assign(temp_f=pd.col("temp_c") * 9 / 5 + 32)
          temp_c  temp_f
Portland    17.0    62.6
Berkeley    25.0    77.0
```

You can create multiple columns within the same assign where one
of the columns depends on another one defined within the same assign:

```pycon
>>> df.assign(
...     temp_f=lambda x: x["temp_c"] * 9 / 5 + 32,
...     temp_k=lambda x: (x["temp_f"] + 459.67) * 5 / 9,
... )
          temp_c  temp_f  temp_k
Portland    17.0    62.6  290.15
Berkeley    25.0    77.0  298.15
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.astype.html.md

# dask.dataframe.DataFrame.astype

#### DataFrame.astype(dtypes)

Cast a pandas object to a specified dtype `dtype`.

This docstring was copied from pandas.DataFrame.astype.

Some inconsistencies with the Dask version may exist.

This method allows the conversion of the data types of pandas objects,
including DataFrames and Series, to the specified dtype. It supports casting
entire objects to a single data type or applying different data types to
individual columns using a mapping.

* **Parameters:**
  **dtype**
  : Use a str, numpy.dtype, pandas.ExtensionDtype or Python type to
    cast entire pandas object to the same type. Alternatively, use a
    mapping, e.g. {col: dtype, …}, where col is a column label and dtype is
    a numpy.dtype or Python type to cast one or more of the DataFrame’s
    columns to column-specific types.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **errors**
  : Control raising of exceptions on invalid data for provided dtype.
    - `raise` : allow exceptions to be raised
    - `ignore` : suppress exceptions. On error return original object.
* **Returns:**
  same type as caller
  : The pandas object casted to the specified `dtype`.

#### SEE ALSO
[`to_datetime`](dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)
: Convert argument to datetime.

[`to_timedelta`](dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)
: Convert argument to timedelta.

[`to_numeric`](dask.dataframe.to_numeric.md#dask.dataframe.to_numeric)
: Convert argument to a numeric type.

[`numpy.ndarray.astype`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html#numpy.ndarray.astype)
: Cast a numpy array to a specified type.

### Notes

#### Versionchanged
Changed in version 2.0.0: Using `astype` to convert from timezone-naive dtype to
timezone-aware dtype will raise an exception.
Use `Series.dt.tz_localize()` instead.

### Examples

Create a DataFrame:

```pycon
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object
```

Cast all columns to int32:

```pycon
>>> df.astype("int32").dtypes
col1    int32
col2    int32
dtype: object
```

Cast col1 to int32 using a dictionary:

```pycon
>>> df.astype({"col1": "int32"}).dtypes
col1    int32
col2    int64
dtype: object
```

Create a series:

```pycon
>>> ser = pd.Series([1, 2], dtype="int32")
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype("int64")
0    1
1    2
dtype: int64
```

Convert to categorical type:

```pycon
>>> ser.astype("category")
0    1
1    2
dtype: category
Categories (2, int32): [1, 2]
```

Convert to ordered categorical type with custom ordering:

```pycon
>>> from pandas.api.types import CategoricalDtype
>>> cat_dtype = CategoricalDtype(categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]
```

Create a series of dates:

```pycon
>>> ser_date = pd.Series(pd.date_range("20200101", periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[us]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.bfill.html.md

# dask.dataframe.DataFrame.bfill

#### DataFrame.bfill(axis=0, limit=None)

Fill NA/NaN values by using the next valid observation to fill the gap.

This docstring was copied from pandas.DataFrame.bfill.

Some inconsistencies with the Dask version may exist.

This method fills missing values in a backward direction along the
specified axis, propagating non-null values from later positions to
earlier positions containing NaN.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.ffill`](dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)
: Fill NA/NaN values by propagating the last valid observation to next valid.

### Examples

For Series:

```pycon
>>> s = pd.Series([1, None, None, 2])
>>> s.bfill()
0    1.0
1    2.0
2    2.0
3    2.0
dtype: float64
>>> s.bfill(limit=1)
0    1.0
1    NaN
2    2.0
3    2.0
dtype: float64
```

With DataFrame:

```pycon
>>> df = pd.DataFrame({"A": [1, None, None, 4], "B": [None, 5, None, 7]})
>>> df
      A     B
0   1.0   NaN
1   NaN   5.0
2   NaN   NaN
3   4.0   7.0
>>> df.bfill()
      A     B
0   1.0   5.0
1   4.0   5.0
2   4.0   7.0
3   4.0   7.0
>>> df.bfill(limit=1)
      A     B
0   1.0   5.0
1   NaN   5.0
2   4.0   7.0
3   4.0   7.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.categorize.html.md

# dask.dataframe.DataFrame.categorize

#### DataFrame.categorize(columns=None, index=None, split_every=None, \*\*kwargs)

Convert columns of the DataFrame to category dtype.

#### WARNING
This method eagerly computes the categories of the chosen columns.

* **Parameters:**
  **columns**
  : A list of column names to convert to categoricals. By default any
    column with an object dtype is converted to a categorical, and any
    unknown categoricals are made known.

  **index**
  : Whether to categorize the index. By default, object indices are
    converted to categorical, and unknown categorical indices are made
    known. Set True to always categorize the index, False to never.

  **split_every**
  : Group partitions into groups of this size while performing a
    tree-reduction. If set to False, no tree-reduction will be used.

  **kwargs**
  : Keyword arguments are passed on to compute.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.columns.html.md

# dask.dataframe.DataFrame.columns

#### *property* DataFrame.columns

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.compute.html.md

# dask.dataframe.DataFrame.compute

#### DataFrame.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.copy.html.md

# dask.dataframe.DataFrame.copy

#### DataFrame.copy(deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Make a copy of the dataframe

This is strictly a shallow copy of the underlying computational graph.
It does not affect the underlying data

* **Parameters:**
  **deep**
  : The deep value must be False and it is declared as a parameter just for
    compatibility with third-party libraries like cuDF and pandas

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.corr.html.md

# dask.dataframe.DataFrame.corr

#### DataFrame.corr(method='pearson', min_periods=None, numeric_only=False, split_every=False)

Compute pairwise correlation of columns, excluding NA/null values.

This docstring was copied from pandas.DataFrame.corr.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **method**
  : Method of correlation:
    * pearson : standard correlation coefficient
    * kendall : Kendall Tau correlation coefficient
    * spearman : Spearman rank correlation
    * callable: callable with input two 1d ndarrays
      : and returning a float. Note that the returned matrix from corr
        will have 1 along the diagonals and will be symmetric
        regardless of the callable’s behavior.

  **min_periods**
  : Minimum number of observations required per pair of columns
    to have a valid result. Currently only available for Pearson
    and Spearman correlation.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: The default value of `numeric_only` is now `False`.
* **Returns:**
  DataFrame
  : Correlation matrix.

#### SEE ALSO
`DataFrame.corrwith`
: Compute pairwise correlation with another DataFrame or Series.

[`Series.corr`](dask.dataframe.Series.corr.md#dask.dataframe.Series.corr)
: Compute the correlation between two Series.

### Notes

Pearson, Kendall and Spearman correlation are currently computed using pairwise complete observations.

* [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)
* [Kendall rank correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)
* [Spearman’s rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient)

### Examples

```pycon
>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> df = pd.DataFrame(
...     [(0.2, 0.3), (0.0, 0.6), (0.6, 0.0), (0.2, 0.1)],
...     columns=["dogs", "cats"],
... )
>>> df.corr(method=histogram_intersection)
      dogs  cats
dogs   1.0   0.3
cats   0.3   1.0
```

```pycon
>>> df = pd.DataFrame(
...     [(1, 1), (2, np.nan), (np.nan, 3), (4, 4)], columns=["dogs", "cats"]
... )
>>> df.corr(min_periods=3)
      dogs  cats
dogs   1.0   NaN
cats   NaN   1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.count.html.md

# dask.dataframe.DataFrame.count

#### DataFrame.count(axis=0, numeric_only=False, split_every=False)

Count non-NA cells for each column or row.

This docstring was copied from pandas.DataFrame.count.

Some inconsistencies with the Dask version may exist.

The values None, NaN, NaT, `pandas.NA` are considered NA.

* **Parameters:**
  **axis**
  : If 0 or ‘index’ counts are generated for each column.
    If 1 or ‘columns’ counts are generated for each row.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : For each column/row the number of non-NA/null entries.

#### SEE ALSO
[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Number of non-NA elements in a Series.

`DataFrame.value_counts`
: Count unique combinations of columns.

[`DataFrame.shape`](dask.dataframe.DataFrame.shape.md#dask.dataframe.DataFrame.shape)
: Number of DataFrame rows and columns (including NA elements).

[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Boolean same-sized DataFrame showing places of NA elements.

### Examples

Constructing DataFrame from a dictionary:

```pycon
>>> df = pd.DataFrame(
...     {
...         "Person": ["John", "Myla", "Lewis", "John", "Myla"],
...         "Age": [24.0, np.nan, 21.0, 33, 26],
...         "Single": [False, True, True, True, False],
...     }
... )
>>> df
   Person   Age  Single
0    John  24.0   False
1    Myla   NaN    True
2   Lewis  21.0    True
3    John  33.0    True
4    Myla  26.0   False
```

Notice the uncounted NA values:

```pycon
>>> df.count()
Person    5
Age       4
Single    5
dtype: int64
```

Counts for each **row**:

```pycon
>>> df.count(axis="columns")
0    3
1    2
2    3
3    3
4    3
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.cov.html.md

# dask.dataframe.DataFrame.cov

#### DataFrame.cov(min_periods=None, numeric_only=False, split_every=False)

Compute pairwise covariance of columns, excluding NA/null values.

This docstring was copied from pandas.DataFrame.cov.

Some inconsistencies with the Dask version may exist.

Compute the pairwise covariance among the series of a DataFrame.
The returned data frame is the [covariance matrix](https://en.wikipedia.org/wiki/Covariance_matrix) of the columns
of the DataFrame.

Both NA and null values are automatically excluded from the
calculation. (See the note below about bias from missing values.)
A threshold can be set for the minimum number of
observations for each value created. Comparisons with observations
below this threshold will be returned as `NaN`.

This method is generally used for the analysis of time series data to
understand the relationship between different measures
across time.

* **Parameters:**
  **min_periods**
  : Minimum number of observations required per pair of columns
    to have a valid result.

  **ddof**
  : Delta degrees of freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.
    This argument is applicable only when no `nan` is in the dataframe.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: The default value of `numeric_only` is now `False`.
* **Returns:**
  DataFrame
  : The covariance matrix of the series of the DataFrame.

#### SEE ALSO
[`Series.cov`](dask.dataframe.Series.cov.md#dask.dataframe.Series.cov)
: Compute covariance with another Series.

`core.window.ewm.ExponentialMovingWindow.cov`
: Exponential weighted sample covariance.

`core.window.expanding.Expanding.cov`
: Expanding sample covariance.

`core.window.rolling.Rolling.cov`
: Rolling sample covariance.

### Notes

Returns the covariance matrix of the DataFrame’s time series.
The covariance is normalized by N-ddof.

For DataFrames that have Series that are missing data (assuming that
data is [missing at random](https://en.wikipedia.org/wiki/Missing_data#Missing_at_random))
the returned covariance matrix will be an unbiased estimate
of the variance and covariance between the member Series.

However, for many applications this estimate may not be acceptable
because the estimate covariance matrix is not guaranteed to be positive
semi-definite. This could lead to estimate correlations having
absolute values which are greater than one, and/or a non-invertible
covariance matrix. See [Estimation of covariance matrices](https://en.wikipedia.org/w/index.php?title=Estimation_of_covariance_matrices) for more details.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [(1, 2), (0, 3), (2, 0), (1, 1)], columns=["dogs", "cats"]
... )
>>> df.cov()
          dogs      cats
dogs  0.666667 -1.000000
cats -1.000000  1.666667
```

```pycon
>>> np.random.seed(42)
>>> df = pd.DataFrame(
...     np.random.randn(1000, 5), columns=["a", "b", "c", "d", "e"]
... )
>>> df.cov()
          a         b         c         d         e
a  0.998438 -0.020161  0.059277 -0.008943  0.014144
b -0.020161  1.059352 -0.008543 -0.024738  0.009826
c  0.059277 -0.008543  1.010670 -0.001486 -0.000271
d -0.008943 -0.024738 -0.001486  0.921297 -0.013692
e  0.014144  0.009826 -0.000271 -0.013692  0.977795
```

**Minimum number of periods**

This method also supports an optional `min_periods` keyword
that specifies the required minimum number of non-NA observations for
each column pair in order to have a valid result:

```pycon
>>> np.random.seed(42)
>>> df = pd.DataFrame(np.random.randn(20, 3), columns=["a", "b", "c"])
>>> df.loc[df.index[:5], "a"] = np.nan
>>> df.loc[df.index[5:10], "b"] = np.nan
>>> df.cov(min_periods=12)
          a         b         c
a  0.316741       NaN -0.150812
b       NaN  1.248003  0.191417
c -0.150812  0.191417  0.895202
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.cummax.html.md

# dask.dataframe.DataFrame.cummax

#### DataFrame.cummax(axis=0, skipna=True)

Return cumulative maximum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummax.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
maximum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative maximum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.max`
: Similar functionality but ignores `NaN` values.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over DataFrame axis.

[`DataFrame.cummax`](#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummax()
0    2.0
1    NaN
2    5.0
3    5.0
4    5.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummax(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the maximum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummax()
     A    B
0  2.0  1.0
1  3.0  NaN
2  3.0  1.0
```

To iterate over columns and find the maximum in each row,
use `axis=1`

```pycon
>>> df.cummax(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.cummin.html.md

# dask.dataframe.DataFrame.cummin

#### DataFrame.cummin(axis=0, skipna=True)

Return cumulative minimum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummin.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
minimum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative minimum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.min`
: Similar functionality but ignores `NaN` values.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummin()
0    2.0
1    NaN
2    2.0
3   -1.0
4   -1.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummin(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the minimum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummin()
     A    B
0  2.0  1.0
1  2.0  NaN
2  1.0  0.0
```

To iterate over columns and find the minimum in each row,
use `axis=1`

```pycon
>>> df.cummin(axis=1)
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.cumprod.html.md

# dask.dataframe.DataFrame.cumprod

#### DataFrame.cumprod(axis=0, skipna=True, \*\*kwargs)

Return cumulative product over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumprod.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
product.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative product of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.prod`
: Similar functionality but ignores `NaN` values.

[`DataFrame.prod`](dask.dataframe.DataFrame.prod.md#dask.dataframe.DataFrame.prod)
: Return the product over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumprod()
0     2.0
1     NaN
2    10.0
3   -10.0
4    -0.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumprod(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the product
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumprod()
     A    B
0  2.0  1.0
1  6.0  NaN
2  6.0  0.0
```

To iterate over columns and find the product in each row,
use `axis=1`

```pycon
>>> df.cumprod(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.cumsum.html.md

# dask.dataframe.DataFrame.cumsum

#### DataFrame.cumsum(axis=0, skipna=True, \*\*kwargs)

Return cumulative sum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumsum.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
sum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative sum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.sum`
: Similar functionality but ignores `NaN` values.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumsum()
0    2.0
1    NaN
2    7.0
3    6.0
4    6.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumsum(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the sum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumsum()
     A    B
0  2.0  1.0
1  5.0  NaN
2  6.0  1.0
```

To iterate over columns and find the sum in each row,
use `axis=1`

```pycon
>>> df.cumsum(axis=1)
     A    B
0  2.0  3.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.describe.html.md

# dask.dataframe.DataFrame.describe

#### DataFrame.describe(split_every=False, percentiles=None, percentiles_method='default', include=None, exclude=None)

Generate descriptive statistics.

This docstring was copied from pandas.DataFrame.describe.

Some inconsistencies with the Dask version may exist.

Generate descriptive statistics.

Dask computes percentiles (used for the `25%`, `50%`, and
`75%` statistics) using an **approximate algorithm** by default.
Results may therefore differ slightly from pandas.  Use
`percentiles_method="dask"` for the built-in Dask algorithm or
`percentiles_method="tdigest"` for the t-digest algorithm.
See [`dask.dataframe.DataFrame.quantile()`](dask.dataframe.DataFrame.quantile.md#dask.dataframe.DataFrame.quantile) for details.

* **Parameters:**
  **split_every**
  : Number of partitions to aggregate at once. Defaults to `False`
    which uses a single-pass reduction over all partitions.

  **percentiles**
  : The percentiles to include in the output. All should fall
    between 0 and 1. By default, `[0.25, 0.5, 0.75]` is used.

  **percentiles_method**
  : Method for computing percentiles. `"default"` uses the internal
    Dask algorithm. `"tdigest"` uses the t-digest algorithm for
    floats and ints and falls back to `"dask"` otherwise.

  **Descriptive statistics include those that summarize the central**

  **tendency, dispersion and shape of a**

  **dataset’s distribution, excluding \`\`NaN\`\` values.**

  **Analyzes both numeric and object series, as well**

  **as \`\`DataFrame\`\` column sets of mixed data types. The output**

  **will vary depending on what is provided. Refer to the notes**

  **below for more detail.**
* **Returns:**
  Series or DataFrame
  : Summary statistics of the Series or Dataframe provided.

#### SEE ALSO
[`DataFrame.count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)
: Count number of non-NA/null observations.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Maximum of the values in the object.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Minimum of the values in the object.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Mean of the values.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Standard deviation of the observations.

[`DataFrame.select_dtypes`](dask.dataframe.DataFrame.select_dtypes.md#dask.dataframe.DataFrame.select_dtypes)
: Subset of a DataFrame including/excluding columns based on their dtype.

### Notes

For numeric data, the result’s index will include `count`,
`mean`, `std`, `min`, `max` as well as lower, `50` and
upper percentiles. By default the lower percentile is `25` and the
upper percentile is `75`. The `50` percentile is the
same as the median.

For object data (e.g. strings), the result’s index
will include `count`, `unique`, `top`, and `freq`. The `top`
is the most common value. The `freq` is the most common value’s
frequency.

If multiple object values have the highest count, then the
`count` and `top` results will be arbitrarily chosen from
among those with the highest count.

For mixed data types provided via a `DataFrame`, the default is to
return only an analysis of numeric columns. If the DataFrame consists
only of object and categorical data without any numeric columns, the
default is to return an analysis of both the object and categorical
columns. If `include='all'` is provided as an option, the result
will include a union of attributes of each type.

The include and exclude parameters can be used to limit
which columns in a `DataFrame` are analyzed for the output.
The parameters are ignored when analyzing a `Series`.

### Examples

Describing a numeric `Series`.

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
dtype: float64
```

Describing a categorical `Series`.

```pycon
>>> s = pd.Series(["a", "a", "b", "c"])
>>> s.describe()
count     4
unique    3
top       a
freq      2
dtype: object
```

Describing a timestamp `Series`.

```pycon
>>> s = pd.Series(
...     [
...         np.datetime64("2000-01-01"),
...         np.datetime64("2010-01-01"),
...         np.datetime64("2010-01-01"),
...     ]
... )
>>> s.describe()
count                      3
mean     2006-09-01 08:00:00
min      2000-01-01 00:00:00
25%      2004-12-31 12:00:00
50%      2010-01-01 00:00:00
75%      2010-01-01 00:00:00
max      2010-01-01 00:00:00
dtype: object
```

Describing a `DataFrame`. By default only numeric fields
are returned.

```pycon
>>> df = pd.DataFrame(
...     {
...         "categorical": pd.Categorical(["d", "e", "f"]),
...         "numeric": [1, 2, 3],
...         "object": ["a", "b", "c"],
...     }
... )
>>> df.describe()
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Describing all columns of a `DataFrame` regardless of data type.

```pycon
>>> df.describe(include="all")
       categorical  numeric object
count            3      3.0      3
unique           3      NaN      3
top              f      NaN      a
freq             1      NaN      1
mean           NaN      2.0    NaN
std            NaN      1.0    NaN
min            NaN      1.0    NaN
25%            NaN      1.5    NaN
50%            NaN      2.0    NaN
75%            NaN      2.5    NaN
max            NaN      3.0    NaN
```

Describing a column from a `DataFrame` by accessing it as
an attribute.

```pycon
>>> df.numeric.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
Name: numeric, dtype: float64
```

Including only numeric columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[np.number])
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Including only string columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[object])
       object
count       3
unique      3
top         a
freq        1
```

Including only categorical columns from a `DataFrame` description.

```pycon
>>> df.describe(include=["category"])
       categorical
count            3
unique           3
top              d
freq             1
```

Excluding numeric columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[np.number])
       categorical object
count            3      3
unique           3      3
top              f      a
freq             1      1
```

Excluding object columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[object])
       categorical  numeric
count            3      3.0
unique           3      NaN
top              f      NaN
freq             1      NaN
mean           NaN      2.0
std            NaN      1.0
min            NaN      1.0
25%            NaN      1.5
50%            NaN      2.0
75%            NaN      2.5
max            NaN      3.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.diff.html.md

# dask.dataframe.DataFrame.diff

#### DataFrame.diff(periods=1, axis=0)

First discrete difference of element.

This docstring was copied from pandas.DataFrame.diff.

Some inconsistencies with the Dask version may exist.

#### NOTE
Pandas currently uses an `object`-dtype column to represent
boolean data with missing values. This can cause issues for
boolean-specific operations, like `|`. To enable boolean-
specific operations, at the cost of metadata that doesn’t match
pandas, use `.astype(bool)` after the `shift`.

Calculates the difference of a DataFrame element compared with another
element in the DataFrame (default is element in previous row).

* **Parameters:**
  **periods**
  : Periods to shift for calculating difference, accepts negative
    values.

  **axis**
  : Take difference over rows (0) or columns (1).
* **Returns:**
  DataFrame
  : First differences of the Series.

#### SEE ALSO
`DataFrame.pct_change`
: Percent change over given number of periods.

`DataFrame.shift`
: Shift index by desired number of periods with an optional time freq.

[`Series.diff`](dask.dataframe.Series.diff.md#dask.dataframe.Series.diff)
: First discrete difference of object.

### Notes

For boolean dtypes, this uses `operator.xor()` rather than
`operator.sub()`.
The result is calculated according to current dtype in DataFrame,
however dtype of the result is always float64.

### Examples

Difference with previous row

```pycon
>>> df = pd.DataFrame(
...     {
...         "a": [1, 2, 3, 4, 5, 6],
...         "b": [1, 1, 2, 3, 5, 8],
...         "c": [1, 4, 9, 16, 25, 36],
...     }
... )
>>> df
   a  b   c
0  1  1   1
1  2  1   4
2  3  2   9
3  4  3  16
4  5  5  25
5  6  8  36
>>> df.diff()
     a    b     c
0  NaN  NaN   NaN
1  1.0  0.0   3.0
2  1.0  1.0   5.0
3  1.0  1.0   7.0
4  1.0  2.0   9.0
5  1.0  3.0  11.0
```

Difference with previous column

```pycon
>>> df.diff(axis=1)
    a  b   c
0 NaN  0   0
1 NaN -1   3
2 NaN -1   7
3 NaN -1  13
4 NaN  0  20
5 NaN  2  28
```

Difference with 3rd previous row

```pycon
>>> df.diff(periods=3)
     a    b     c
0  NaN  NaN   NaN
1  NaN  NaN   NaN
2  NaN  NaN   NaN
3  3.0  2.0  15.0
4  3.0  4.0  21.0
5  3.0  6.0  27.0
```

Difference with following row

```pycon
>>> df.diff(periods=-1)
     a    b     c
0 -1.0  0.0  -3.0
1 -1.0 -1.0  -5.0
2 -1.0 -1.0  -7.0
3 -1.0 -2.0  -9.0
4 -1.0 -3.0 -11.0
5  NaN  NaN   NaN
```

Overflow in input dtype

```pycon
>>> df = pd.DataFrame({"a": [1, 0]}, dtype=np.uint8)
>>> df.diff()
       a
0    NaN
1  255.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.div.html.md

# dask.dataframe.DataFrame.div

#### DataFrame.div(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.divide.html.md

# dask.dataframe.DataFrame.divide

#### DataFrame.divide(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.divisions.html.md

# dask.dataframe.DataFrame.divisions

#### *property* DataFrame.divisions

Tuple of `npartitions + 1` values, in ascending order, marking the
lower/upper bounds of each partition’s index. Divisions allow Dask
to know which partition will contain a given value, significantly
speeding up operations like loc, merge, and groupby by not
having to search the full dataset.

Example: for `divisions = (0, 10, 50, 100)`, there are three partitions,
where the index in each partition contains values [0, 10), [10, 50),
and [50, 100], respectively. Dask therefore knows `df.loc[45]`
will be in the second partition.

When every item in `divisions` is `None`, the divisions are unknown.
Most operations can still be performed, but some will be much slower,
and a few may fail.

It is not supported to set `divisions` directly. Instead, use `set_index`,
which sorts and splits the data as needed.
See [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions).

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.drop.html.md

# dask.dataframe.DataFrame.drop

#### DataFrame.drop(labels=None, axis=0, columns=None, errors='raise')

Drop specified labels from rows or columns.

This docstring was copied from pandas.DataFrame.drop.

Some inconsistencies with the Dask version may exist.

Remove rows or columns by specifying label names and corresponding
axis, or by directly specifying index or column names. When using a
multi-index, labels on different levels can be removed by specifying
the level. See the [user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#advanced-shown-levels)
for more information about the now unused levels.

* **Parameters:**
  **labels**
  : Index or column labels to drop. A tuple will be used as a single
    label and not treated as an iterable.

  **axis**
  : Whether to drop labels from the index (0 or ‘index’) or
    columns (1 or ‘columns’).

  **index**
  : Alternative to specifying axis (`labels, axis=0`
    is equivalent to `index=labels`).

  **columns**
  : Alternative to specifying axis (`labels, axis=1`
    is equivalent to `columns=labels`).

  **level**
  : For MultiIndex, level from which the labels will be removed.

  **inplace**
  : If False, return a copy. Otherwise, do operation
    in place and return None.

  **errors**
  : If ‘ignore’, suppress error and only existing labels are
    dropped.
* **Returns:**
  DataFrame or None
  : Returns DataFrame or None DataFrame with the specified
    index or column labels removed or None if inplace=True.
* **Raises:**
  KeyError
  : If any of the labels is not found in the selected axis.

#### SEE ALSO
[`DataFrame.loc`](dask.dataframe.DataFrame.loc.md#dask.dataframe.DataFrame.loc)
: Label-location based indexer for selection by label.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Return DataFrame with labels on given axis omitted where (all or any) data are missing.

[`DataFrame.drop_duplicates`](dask.dataframe.DataFrame.drop_duplicates.md#dask.dataframe.DataFrame.drop_duplicates)
: Return DataFrame with duplicate rows removed, optionally only considering certain columns.

`Series.drop`
: Return Series with specified index labels removed.

### Examples

```pycon
>>> df = pd.DataFrame(np.arange(12).reshape(3, 4), columns=["A", "B", "C", "D"])
>>> df
   A  B   C   D
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11
```

Drop columns

```pycon
>>> df.drop(["B", "C"], axis=1)
   A   D
0  0   3
1  4   7
2  8  11
```

```pycon
>>> df.drop(columns=["B", "C"])
   A   D
0  0   3
1  4   7
2  8  11
```

Drop a row by index

```pycon
>>> df.drop([0, 1])
   A  B   C   D
2  8  9  10  11
```

Drop columns and/or rows of MultiIndex DataFrame

```pycon
>>> midx = pd.MultiIndex(
...     levels=[["llama", "cow", "falcon"], ["speed", "weight", "length"]],
...     codes=[[0, 0, 0, 1, 1, 1, 2, 2, 2], [0, 1, 2, 0, 1, 2, 0, 1, 2]],
... )
>>> df = pd.DataFrame(
...     index=midx,
...     columns=["big", "small"],
...     data=[
...         [45, 30],
...         [200, 100],
...         [1.5, 1],
...         [30, 20],
...         [250, 150],
...         [1.5, 0.8],
...         [320, 250],
...         [1, 0.8],
...         [0.3, 0.2],
...     ],
... )
>>> df
                big     small
llama   speed   45.0    30.0
        weight  200.0   100.0
        length  1.5     1.0
cow     speed   30.0    20.0
        weight  250.0   150.0
        length  1.5     0.8
falcon  speed   320.0   250.0
        weight  1.0     0.8
        length  0.3     0.2
```

Drop a specific index combination from the MultiIndex
DataFrame, i.e., drop the combination `'falcon'` and
`'weight'`, which deletes only the corresponding row

```pycon
>>> df.drop(index=("falcon", "weight"))
                big     small
llama   speed   45.0    30.0
        weight  200.0   100.0
        length  1.5     1.0
cow     speed   30.0    20.0
        weight  250.0   150.0
        length  1.5     0.8
falcon  speed   320.0   250.0
        length  0.3     0.2
```

```pycon
>>> df.drop(index="cow", columns="small")
                big
llama   speed   45.0
        weight  200.0
        length  1.5
falcon  speed   320.0
        weight  1.0
        length  0.3
```

```pycon
>>> df.drop(index="length", level=1)
                big     small
llama   speed   45.0    30.0
        weight  200.0   100.0
cow     speed   30.0    20.0
        weight  250.0   150.0
falcon  speed   320.0   250.0
        weight  1.0     0.8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.drop_duplicates.html.md

# dask.dataframe.DataFrame.drop_duplicates

#### DataFrame.drop_duplicates(subset=None, split_every=None, split_out=True, shuffle_method=None, ignore_index=False, keep='first')

Return DataFrame with duplicate rows removed.

This docstring was copied from pandas.DataFrame.drop_duplicates.

Some inconsistencies with the Dask version may exist.

Known inconsistencies: 
: keep=False will raise a `NotImplementedError`

Considering certain columns is optional. Indexes, including time indexes
are ignored.

* **Parameters:**
  **subset**
  : Only consider certain columns for identifying duplicates, by
    default use all of the columns.

  **keep**
  : Determines which duplicates (if any) to keep.
    - ‘first’ : Drop duplicates except for the first occurrence.
    - ‘last’ : Drop duplicates except for the last occurrence.
    - `False` : Drop all duplicates.

  **inplace**
  : Whether to modify the DataFrame rather than creating a new one.

  **ignore_index**
  : If `True`, the resulting axis will be labeled 0, 1, …, n - 1.
* **Returns:**
  DataFrame or None
  : DataFrame with duplicates removed or None if `inplace=True`.

#### SEE ALSO
`DataFrame.value_counts`
: Count unique combinations of columns.

### Notes

This method requires columns specified by `subset` to be of hashable type.
Passing unhashable columns will raise a `TypeError`.

### Examples

Consider dataset containing ramen rating.

```pycon
>>> df = pd.DataFrame(
...     {
...         "brand": ["Yum Yum", "Yum Yum", "Indomie", "Indomie", "Indomie"],
...         "style": ["cup", "cup", "cup", "pack", "pack"],
...         "rating": [4, 4, 3.5, 15, 5],
...     }
... )
>>> df
    brand style  rating
0  Yum Yum   cup     4.0
1  Yum Yum   cup     4.0
2  Indomie   cup     3.5
3  Indomie  pack    15.0
4  Indomie  pack     5.0
```

By default, it removes duplicate rows based on all columns.

```pycon
>>> df.drop_duplicates()
    brand style  rating
0  Yum Yum   cup     4.0
2  Indomie   cup     3.5
3  Indomie  pack    15.0
4  Indomie  pack     5.0
```

To remove duplicates on specific column(s), use `subset`.

```pycon
>>> df.drop_duplicates(subset=["brand"])
    brand style  rating
0  Yum Yum   cup     4.0
2  Indomie   cup     3.5
```

To remove duplicates and keep last occurrences, use `keep`.

```pycon
>>> df.drop_duplicates(subset=["brand", "style"], keep="last")
    brand style  rating
1  Yum Yum   cup     4.0
2  Indomie   cup     3.5
4  Indomie  pack     5.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.dropna.html.md

# dask.dataframe.DataFrame.dropna

#### DataFrame.dropna(how=<no_default>, subset=None, thresh=<no_default>)

Remove missing values.

This docstring was copied from pandas.DataFrame.dropna.

Some inconsistencies with the Dask version may exist.

See the [User Guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data) for more on which values are
considered missing, and how to work with missing data.

* **Parameters:**
  **axis**
  : Determine if rows or columns which contain missing values are
    removed.
    * 0, or ‘index’ : Drop rows which contain missing values.
    * 1, or ‘columns’ : Drop columns which contain missing value.
    <br/>
    Only a single axis is allowed.

  **how**
  : Determine if row or column is removed from DataFrame, when we have
    at least one NA or all NA.
    * ‘any’ : If any NA values are present, drop that row or column.
    * ‘all’ : If all values are NA, drop that row or column.

  **thresh**
  : Require that many non-NA values. Cannot be combined with how.

  **subset**
  : Labels along other axis to consider, e.g. if you are dropping rows
    these would be a list of columns to include.

  **inplace**
  : Whether to modify the DataFrame rather than creating a new one.

  **ignore_index**
  : If `True`, the resulting axis will be labeled 0, 1, …, n - 1.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  DataFrame or None
  : DataFrame with NA entries dropped from it or None if `inplace=True`.

#### SEE ALSO
[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Indicate missing values.

`DataFrame.notna`
: Indicate existing (non-missing) values.

[`DataFrame.fillna`](dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)
: Replace missing values.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Drop missing values.

[`Index.dropna`](dask.dataframe.Index.dropna.md#dask.dataframe.Index.dropna)
: Drop missing indices.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "name": ["Alfred", "Batman", "Catwoman"],
...         "toy": [np.nan, "Batmobile", "Bullwhip"],
...         "born": [pd.NaT, pd.Timestamp("1940-04-25"), pd.NaT],
...     }
... )
>>> df
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

Drop the rows where at least one element is missing.

```pycon
>>> df.dropna()
     name        toy       born
1  Batman  Batmobile 1940-04-25
```

Drop the columns where at least one element is missing.

```pycon
>>> df.dropna(axis="columns")
       name
0    Alfred
1    Batman
2  Catwoman
```

Drop the rows where all elements are missing.

```pycon
>>> df.dropna(how="all")
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

Keep only the rows with at least 2 non-NA values.

```pycon
>>> df.dropna(thresh=2)
       name        toy       born
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

Define in which columns to look for missing values.

```pycon
>>> df.dropna(subset=["name", "toy"])
       name        toy       born
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.dtypes.html.md

# dask.dataframe.DataFrame.dtypes

#### *property* DataFrame.dtypes

Return data types

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.eq.html.md

# dask.dataframe.DataFrame.eq

#### DataFrame.eq(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.eval.html.md

# dask.dataframe.DataFrame.eval

#### DataFrame.eval(expr, \*\*kwargs)

Evaluate a string describing operations on DataFrame columns.

This docstring was copied from pandas.DataFrame.eval.

Some inconsistencies with the Dask version may exist.

#### WARNING
This method can run arbitrary code which can make you vulnerable to code
injection if you pass user input to this function.

Operates on columns only, not specific rows or elements.  This allows
eval to run arbitrary code, which can make you vulnerable to code
injection if you pass user input to this function.

* **Parameters:**
  **expr**
  : The expression string to evaluate.
    <br/>
    You can refer to variables
    in the environment by prefixing them with an ‘@’ character like
    `@a + b`.
    <br/>
    You can refer to column names that are not valid Python variable names
    by surrounding them in backticks. Thus, column names containing spaces
    or punctuation (besides underscores) or starting with digits must be
    surrounded by backticks. (For example, a column named “Area (cm^2)” would
    be referenced as ``Area (cm^2)``). Column names which are Python keywords
    (like “if”, “for”, “import”, etc) cannot be used.
    <br/>
    For example, if one of your columns is called `a a` and you want
    to sum it with `b`, your query should be ``a a` + b`.
    <br/>
    See the documentation for [`eval()`](#dask.dataframe.DataFrame.eval) for full details of
    supported operations and functions in the expression string.

  **inplace**
  : If the expression contains an assignment, whether to perform the
    operation inplace and mutate the existing DataFrame. Otherwise,
    a new DataFrame is returned.

  **\*\*kwargs**
  : See the documentation for [`eval()`](#dask.dataframe.DataFrame.eval) for complete details
    on the keyword arguments accepted by
    [`eval()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.eval.html#pandas.DataFrame.eval).
* **Returns:**
  ndarray, scalar, pandas object, or None
  : The result of the evaluation or None if `inplace=True`.

#### SEE ALSO
[`DataFrame.query`](dask.dataframe.DataFrame.query.md#dask.dataframe.DataFrame.query)
: Evaluates a boolean expression to query the columns of a frame.

[`DataFrame.assign`](dask.dataframe.DataFrame.assign.md#dask.dataframe.DataFrame.assign)
: Can evaluate an expression or function to create new values for a column.

[`eval`](#dask.dataframe.DataFrame.eval)
: Evaluate a Python expression as a string using various backends.

### Notes

For more details see the API documentation for [`eval()`](#dask.dataframe.DataFrame.eval).
For detailed examples see [enhancing performance with eval](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-eval).

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"A": range(1, 6), "B": range(10, 0, -2), "C&C": range(10, 5, -1)}
... )
>>> df
   A   B  C&C
0  1  10   10
1  2   8    9
2  3   6    8
3  4   4    7
4  5   2    6
>>> df.eval("A + B")
0    11
1    10
2     9
3     8
4     7
dtype: int64
```

Assignment is allowed though by default the original DataFrame is not
modified.

```pycon
>>> df.eval("D = A + B")
   A   B  C&C   D
0  1  10   10  11
1  2   8    9  10
2  3   6    8   9
3  4   4    7   8
4  5   2    6   7
>>> df
   A   B  C&C
0  1  10   10
1  2   8    9
2  3   6    8
3  4   4    7
4  5   2    6
```

Multiple columns can be assigned to using multi-line expressions:

```pycon
>>> df.eval(
...     '''
... D = A + B
... E = A - B
... '''
... )
   A   B  C&C   D  E
0  1  10   10  11 -9
1  2   8    9  10 -6
2  3   6    8   9 -3
3  4   4    7   8  0
4  5   2    6   7  3
```

For columns with spaces or other disallowed characters in their name, you can
use backtick quoting.

```pycon
>>> df.eval("B * `C&C`")
0    100
1     72
2     48
3     28
4     12
dtype: int64
```

Local variables shall be explicitly referenced using `@`
character in front of the name:

```pycon
>>> local_var = 2
>>> df.eval("@local_var * A")
0     2
1     4
2     6
3     8
4    10
Name: A, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.explain.html.md

# dask.dataframe.DataFrame.explain

#### DataFrame.explain(stage: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['logical', 'simplified-logical', 'tuned-logical', 'physical', 'simplified-physical', 'fused'] = 'fused', format: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None)

Create a graph representation of the Expression.

explain runs the optimizer and creates a graph of the optimized expression
with graphviz. No computation is triggered.

* **Parameters:**
  **stage: {“logical”, “simplified-logical”, “tuned-logical”, “physical”, “simplified-physical”, “fused”}**
  : The optimizer stage that is returned. Default is “fused”.
    - logical: outputs the expression as is
    - simplified-logical: simplifies the expression which includes predicate
      pushdown and column projection.
    - tuned-logical: applies additional optimizations like partition squashing
    - physical: outputs the physical expression; this expression can actually
      be computed
    - simplified-physical: runs another simplification after the physical
      plan is generated
    - fused: fuses the physical expression to reduce the nodes in the graph.
    <br/>
    #### WARNING
    The optimizer stages are subject to change.

  **format: str, default None**
  : The format of the output. Default is “png”.
* **Returns:**
  None, but opens a new window with the graph visualization and outputs

  a file with the graph representation.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.explode.html.md

# dask.dataframe.DataFrame.explode

#### DataFrame.explode(column)

Transform each element of a list-like to a row, replicating index values.

This docstring was copied from pandas.DataFrame.explode.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **column**
  : Column(s) to explode.
    For multiple columns, specify a non-empty list with each element
    be str or tuple, and all specified columns their list-like data
    on same row of the frame must have matching length.

  **ignore_index**
  : If True, the resulting index will be labeled 0, 1, …, n - 1.
* **Returns:**
  DataFrame
  : Exploded lists to rows of the subset columns;
    index will be duplicated for these rows.
* **Raises:**
  ValueError
  : * If columns of the frame are not unique.
    * If specified columns to explode is empty list.
    * If specified columns to explode have not matching count of
      elements rowwise in the frame.

#### SEE ALSO
`DataFrame.unstack`
: Pivot a level of the (necessarily hierarchical) index labels.

[`DataFrame.melt`](dask.dataframe.DataFrame.melt.md#dask.dataframe.DataFrame.melt)
: Unpivot a DataFrame from wide format to long format.

[`Series.explode`](dask.dataframe.Series.explode.md#dask.dataframe.Series.explode)
: Explode a DataFrame from list-like columns to long format.

### Notes

This routine will explode list-likes including lists, tuples, sets,
Series, and np.ndarray. The result dtype of the subset rows will
be object. Scalars will be returned unchanged, and empty list-likes will
result in a np.nan for that row. In addition, the ordering of rows in the
output will be non-deterministic when exploding sets.

Reference [the user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html#reshaping-explode) for more examples.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [[0, 1, 2], "foo", [], [3, 4]],
...         "B": 1,
...         "C": [["a", "b", "c"], np.nan, [], ["d", "e"]],
...     }
... )
>>> df
           A  B          C
0  [0, 1, 2]  1  [a, b, c]
1        foo  1        NaN
2         []  1         []
3     [3, 4]  1     [d, e]
```

Single-column explode.

```pycon
>>> df.explode("A")
     A  B          C
0    0  1  [a, b, c]
0    1  1  [a, b, c]
0    2  1  [a, b, c]
1  foo  1        NaN
2  NaN  1         []
3    3  1     [d, e]
3    4  1     [d, e]
```

Multi-column explode.

```pycon
>>> df.explode(list("AC"))
     A  B    C
0    0  1    a
0    1  1    b
0    2  1    c
1  foo  1  NaN
2  NaN  1  NaN
3    3  1    d
3    4  1    e
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.ffill.html.md

# dask.dataframe.DataFrame.ffill

#### DataFrame.ffill(axis=0, limit=None)

Fill NA/NaN values by propagating the last valid observation to next valid.

This docstring was copied from pandas.DataFrame.ffill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.bfill`](dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)
: Fill NA/NaN values by using the next valid observation to fill the gap.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

```pycon
>>> df.ffill()
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  3.0  4.0 NaN  1.0
3  3.0  3.0 NaN  4.0
```

```pycon
>>> ser = pd.Series([1, np.nan, 2, 3])
>>> ser.ffill()
0   1.0
1   1.0
2   2.0
3   3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.fillna.html.md

# dask.dataframe.DataFrame.fillna

#### DataFrame.fillna(value=None, axis=None)

Fill NA/NaN values with value.

This docstring was copied from pandas.DataFrame.fillna.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **value**
  : Value to use to fill holes (e.g. 0), alternately a
    dict/Series/DataFrame of values specifying which value to use for
    each index (for a Series) or column (for a DataFrame).  Values not
    in the dict/Series/DataFrame will not be filled. This value cannot
    be a list.

  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : This is the maximum number of entries along the entire axis
    where NaNs will be filled. Must be greater than 0 if not None.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`ffill`](dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)
: Fill values by propagating the last valid observation to next valid.

[`bfill`](dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)
: Fill values by using the next valid observation to fill the gap.

`interpolate`
: Fill NaN values using interpolation.

`reindex`
: Conform object to new index.

`asfreq`
: Convert TimeSeries to specified frequency.

### Notes

For non-object dtype, `value=None` will use the NA value of the dtype.
See more details in the [Filling missing data](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data-fillna)
section.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

Replace all NaN elements with 0s.

```pycon
>>> df.fillna(0)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  0.0
3  0.0  3.0  0.0  4.0
```

Replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1,
2, and 3 respectively.

```pycon
>>> values = {"A": 0, "B": 1, "C": 2, "D": 3}
>>> df.fillna(value=values)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  2.0  1.0
2  0.0  1.0  2.0  3.0
3  0.0  3.0  2.0  4.0
```

Only replace the first NaN element.

```pycon
>>> df.fillna(value=values, limit=1)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  NaN  1.0
2  NaN  1.0  NaN  3.0
3  NaN  3.0  NaN  4.0
```

When filling using a DataFrame, replacement happens along
the same column names and same indices

```pycon
>>> df2 = pd.DataFrame(np.zeros((4, 4)), columns=list("ABCE"))
>>> df.fillna(df2)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  NaN
3  0.0  3.0  0.0  4.0
```

Note that column D is not affected since it is not present in df2.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.floordiv.html.md

# dask.dataframe.DataFrame.floordiv

#### DataFrame.floordiv(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.from_dict.html.md

# dask.dataframe.DataFrame.from_dict

#### *classmethod* DataFrame.from_dict(data, , npartitions=1, orient='columns', dtype=None, columns=None)

Construct a Dask DataFrame from a Python Dictionary

#### SEE ALSO
`dask.dataframe.from_dict`

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.ge.html.md

# dask.dataframe.DataFrame.ge

#### DataFrame.ge(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.get_partition.html.md

# dask.dataframe.DataFrame.get_partition

#### DataFrame.get_partition(n)

Get a dask DataFrame/Series representing the nth partition.

* **Parameters:**
  **n**
  : The 0-indexed partition number to select.
* **Returns:**
  Dask DataFrame or Series
  : The same type as the original object.

#### SEE ALSO
[`DataFrame.partitions`](dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.groupby.html.md

# dask.dataframe.DataFrame.groupby

#### DataFrame.groupby(by, group_keys=True, sort=None, observed=None, dropna=None, \*\*kwargs)

Group DataFrame using a mapper or by a Series of columns.

This docstring was copied from pandas.DataFrame.groupby.

Some inconsistencies with the Dask version may exist.

A groupby operation involves some combination of splitting the
object, applying a function, and combining the results. This can be
used to group large amounts of data and compute operations on these
groups.

* **Parameters:**
  **by**
  : Used to determine the groups for the groupby.
    If `by` is a function, it’s called on each value of the object’s
    index. If a dict or Series is passed, the Series or dict VALUES
    will be used to determine the groups (the Series’ values are first
    aligned; see `.align()` method). If a list or ndarray of length
    equal to the number of rows is passed (see the [groupby user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#splitting-an-object-into-groups)),
    the values are used as-is to determine the groups. A label or list
    of labels may be passed to group by the columns in `self`.
    Notice that a tuple is interpreted as a (single) key.

  **level**
  : If the axis is a MultiIndex (hierarchical), group by a particular
    level or levels. Do not specify both `by` and `level`.

  **as_index**
  : Return object with group labels as the
    index. Only relevant for DataFrame input. as_index=False is
    effectively “SQL-style” grouped output. This argument has no effect
    on filtrations (see the [filtrations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).

  **sort**
  : Sort group keys. Get better performance by turning this off.
    Note this does not influence the order of observations within each
    group. Groupby preserves the order of rows within each group. If False,
    the groups will appear in the same order as they did in the original
    DataFrame.
    This argument has no effect on filtrations (see the [filtrations
    in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: Specifying `sort=False` with an ordered categorical grouper will no
    longer sort the values.

  **group_keys**
  : When calling apply and the `by` argument produces a like-indexed
    (i.e. [a transform](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#groupby-transform)) result, add group keys to
    index to identify pieces. By default group keys are not included
    when the result’s index (and column) labels match the inputs, and
    are included otherwise.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `True`.

  **observed**
  : This only applies if any of the groupers are Categoricals.
    If True: only show observed values for categorical groupers.
    If False: show all values for categorical groupers.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0: The default value is now `True`.

  **dropna**
  : If True, and if group keys contain NA values, NA values together
    with row/column will be dropped.
    If False, NA values will also be treated as the key in groups.
* **Returns:**
  pandas.api.typing.DataFrameGroupBy
  : Returns a groupby object that contains information about the groups.

#### SEE ALSO
[`resample`](dask.dataframe.DataFrame.resample.md#dask.dataframe.DataFrame.resample)
: Convenience method for frequency conversion and resampling of time series.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/groupby.html) for more
detailed usage and examples, including splitting an object into groups,
iterating through groups, selecting a group, aggregation, and more.

The implementation of groupby is hash-based, meaning in particular that
objects that compare as equal will be considered to be in the same group.
An exception to this is that pandas has special handling of NA values:
any NA values will be collapsed to a single group, regardless of how
they compare. See the user guide linked above for more details.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "Animal": ["Falcon", "Falcon", "Parrot", "Parrot"],
...         "Max Speed": [380.0, 370.0, 24.0, 26.0],
...     }
... )
>>> df
   Animal  Max Speed
0  Falcon      380.0
1  Falcon      370.0
2  Parrot       24.0
3  Parrot       26.0
>>> df.groupby(["Animal"]).mean()
        Max Speed
Animal
Falcon      375.0
Parrot       25.0
```

**Hierarchical Indexes**

We can groupby different levels of a hierarchical index
using the level parameter:

```pycon
>>> arrays = [
...     ["Falcon", "Falcon", "Parrot", "Parrot"],
...     ["Captive", "Wild", "Captive", "Wild"],
... ]
>>> index = pd.MultiIndex.from_arrays(arrays, names=("Animal", "Type"))
>>> df = pd.DataFrame({"Max Speed": [390.0, 350.0, 30.0, 20.0]}, index=index)
>>> df
                Max Speed
Animal Type
Falcon Captive      390.0
       Wild         350.0
Parrot Captive       30.0
       Wild          20.0
>>> df.groupby(level=0).mean()
        Max Speed
Animal
Falcon      370.0
Parrot       25.0
>>> df.groupby(level="Type").mean()
         Max Speed
Type
Captive      210.0
Wild         185.0
```

We can also choose to include NA in group keys or not by setting
dropna parameter, the default setting is True.

```pycon
>>> arr = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]]
>>> df = pd.DataFrame(arr, columns=["a", "b", "c"])
```

```pycon
>>> df.groupby(by=["b"]).sum()
    a   c
b
1.0 2   3
2.0 2   5
```

```pycon
>>> df.groupby(by=["b"], dropna=False).sum()
    a   c
b
1.0 2   3
2.0 2   5
NaN 1   4
```

```pycon
>>> arr = [["a", 12, 12], [None, 12.3, 33.0], ["b", 12.3, 123], ["a", 1, 1]]
>>> df = pd.DataFrame(arr, columns=["a", "b", "c"])
```

```pycon
>>> df.groupby(by="a").sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0
```

```pycon
>>> df.groupby(by="a", dropna=False).sum()
    b     c
a
a   13.0   13.0
b   12.3  123.0
NaN 12.3   33.0
```

When using `.apply()`, use `group_keys` to include or exclude the
group keys. The `group_keys` argument defaults to `True` (include).

```pycon
>>> df = pd.DataFrame(
...     {
...         "Animal": ["Falcon", "Falcon", "Parrot", "Parrot"],
...         "Max Speed": [380.0, 370.0, 24.0, 26.0],
...     }
... )
>>> df.groupby("Animal", group_keys=True)[["Max Speed"]].apply(lambda x: x)
          Max Speed
Animal
Falcon 0      380.0
       1      370.0
Parrot 2       24.0
       3       26.0
```

```pycon
>>> df.groupby("Animal", group_keys=False)[["Max Speed"]].apply(lambda x: x)
   Max Speed
0      380.0
1      370.0
2       24.0
3       26.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.gt.html.md

# dask.dataframe.DataFrame.gt

#### DataFrame.gt(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.head.html.md

# dask.dataframe.DataFrame.head

#### DataFrame.head(n: [int](https://docs.python.org/3/library/functions.html#int) = 5, npartitions=1, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

First n rows of the dataset

* **Parameters:**
  **n**
  : The number of rows to return. Default is 5.

  **npartitions**
  : Elements are only taken from the first `npartitions`, with a
    default of 1. If there are fewer than `n` rows in the first
    `npartitions` a warning will be raised and any found rows
    returned. Pass -1 to use all partitions.

  **compute**
  : Whether to compute the result, default is True.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.html.md

# dask.dataframe.DataFrame

### *class* dask.dataframe.DataFrame(expr)

DataFrame-like Expr Collection.

The constructor takes the expression that represents the query as input. The class
is not meant to be instantiated directly. Instead, use one of the IO connectors from
Dask.

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(expr)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.dataframe.DataFrame.__init__)(expr)                                                                                                    |                                                                                 |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|
| [`abs`](dask.dataframe.DataFrame.abs.md#dask.dataframe.DataFrame.abs)()                                                                                   | Return a Series/DataFrame with absolute numeric value of each element.          |
| [`add`](dask.dataframe.DataFrame.add.md#dask.dataframe.DataFrame.add)(other[, axis, level, fill_value])                                                   |                                                                                 |
| `add_prefix`(prefix[, axis])                                                                                                                              | Prefix labels with string prefix.                                               |
| `add_suffix`(suffix[, axis])                                                                                                                              | Suffix labels with string suffix.                                               |
| [`align`](dask.dataframe.DataFrame.align.md#dask.dataframe.DataFrame.align)(other[, join, axis, fill_value])                                              | Align two objects on their axes with the specified join method.                 |
| [`all`](dask.dataframe.DataFrame.all.md#dask.dataframe.DataFrame.all)([axis, skipna, split_every])                                                        | Return whether all elements are True, potentially over an axis.                 |
| [`analyze`](dask.dataframe.DataFrame.analyze.md#dask.dataframe.DataFrame.analyze)([filename, format])                                                     | Outputs statistics about every node in the expression.                          |
| [`any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)([axis, skipna, split_every])                                                        | Return whether any element is True, potentially over an axis.                   |
| [`apply`](dask.dataframe.DataFrame.apply.md#dask.dataframe.DataFrame.apply)(function, \*args[, meta, axis])                                               | Parallel version of pandas.DataFrame.apply                                      |
| [`assign`](dask.dataframe.DataFrame.assign.md#dask.dataframe.DataFrame.assign)(\*\*pairs)                                                                 | Assign new columns to a DataFrame.                                              |
| [`astype`](dask.dataframe.DataFrame.astype.md#dask.dataframe.DataFrame.astype)(dtypes)                                                                    | Cast a pandas object to a specified dtype `dtype`.                              |
| [`bfill`](dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)([axis, limit])                                                                | Fill NA/NaN values by using the next valid observation to fill the gap.         |
| [`categorize`](dask.dataframe.DataFrame.categorize.md#dask.dataframe.DataFrame.categorize)([columns, index, split_every])                                 | Convert columns of the DataFrame to category dtype.                             |
| `clear_divisions`()                                                                                                                                       | Forget division information.                                                    |
| `clip`([lower, upper, axis])                                                                                                                              | Trim values at input threshold(s).                                              |
| `combine`(other, func[, fill_value, overwrite])                                                                                                           | Perform column-wise combine with another DataFrame.                             |
| `combine_first`(other)                                                                                                                                    | Update null elements with value in the same location in other.                  |
| [`compute`](dask.dataframe.DataFrame.compute.md#dask.dataframe.DataFrame.compute)(\*\*kwargs)                                                             | Compute this dask collection                                                    |
| `compute_current_divisions`([col, set_divisions])                                                                                                         | Compute the current divisions of the DataFrame.                                 |
| [`copy`](dask.dataframe.DataFrame.copy.md#dask.dataframe.DataFrame.copy)([deep])                                                                          | Make a copy of the dataframe                                                    |
| [`corr`](dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)([method, min_periods, numeric_only, ...])                                        | Compute pairwise correlation of columns, excluding NA/null values.              |
| [`count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)([axis, numeric_only, split_every])                                            | Count non-NA cells for each column or row.                                      |
| [`cov`](dask.dataframe.DataFrame.cov.md#dask.dataframe.DataFrame.cov)([min_periods, numeric_only, split_every])                                           | Compute pairwise covariance of columns, excluding NA/null values.               |
| [`cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)([axis, skipna])                                                            | Return cumulative maximum over a DataFrame or Series axis.                      |
| [`cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)([axis, skipna])                                                            | Return cumulative minimum over a DataFrame or Series axis.                      |
| [`cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)([axis, skipna])                                                         | Return cumulative product over a DataFrame or Series axis.                      |
| [`cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)([axis, skipna])                                                            | Return cumulative sum over a DataFrame or Series axis.                          |
| [`describe`](dask.dataframe.DataFrame.describe.md#dask.dataframe.DataFrame.describe)([split_every, percentiles, ...])                                     | Generate descriptive statistics.                                                |
| [`diff`](dask.dataframe.DataFrame.diff.md#dask.dataframe.DataFrame.diff)([periods, axis])                                                                 | First discrete difference of element.                                           |
| [`div`](dask.dataframe.DataFrame.div.md#dask.dataframe.DataFrame.div)(other[, axis, level, fill_value])                                                   |                                                                                 |
| [`divide`](dask.dataframe.DataFrame.divide.md#dask.dataframe.DataFrame.divide)(other[, axis, level, fill_value])                                          |                                                                                 |
| `dot`(other[, meta])                                                                                                                                      | Compute the dot product between the Series and the columns of other.            |
| [`drop`](dask.dataframe.DataFrame.drop.md#dask.dataframe.DataFrame.drop)([labels, axis, columns, errors])                                                 | Drop specified labels from rows or columns.                                     |
| [`drop_duplicates`](dask.dataframe.DataFrame.drop_duplicates.md#dask.dataframe.DataFrame.drop_duplicates)([subset, split_every, ...])                     | Return DataFrame with duplicate rows removed.                                   |
| [`dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)([how, subset, thresh])                                                     | Remove missing values.                                                          |
| `enforce_runtime_divisions`()                                                                                                                             | Enforce the current divisions at runtime.                                       |
| [`eq`](dask.dataframe.DataFrame.eq.md#dask.dataframe.DataFrame.eq)(other[, level, axis])                                                                  |                                                                                 |
| [`eval`](dask.dataframe.DataFrame.eval.md#dask.dataframe.DataFrame.eval)(expr, \*\*kwargs)                                                                | Evaluate a string describing operations on DataFrame columns.                   |
| [`explain`](dask.dataframe.DataFrame.explain.md#dask.dataframe.DataFrame.explain)([stage, format])                                                        | Create a graph representation of the Expression.                                |
| [`explode`](dask.dataframe.DataFrame.explode.md#dask.dataframe.DataFrame.explode)(column)                                                                 | Transform each element of a list-like to a row, replicating index values.       |
| [`ffill`](dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)([axis, limit])                                                                | Fill NA/NaN values by propagating the last valid observation to next valid.     |
| [`fillna`](dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)([value, axis])                                                             | Fill NA/NaN values with value.                                                  |
| [`floordiv`](dask.dataframe.DataFrame.floordiv.md#dask.dataframe.DataFrame.floordiv)(other[, axis, level, fill_value])                                    |                                                                                 |
| [`from_dict`](dask.dataframe.DataFrame.from_dict.md#dask.dataframe.DataFrame.from_dict)(data, \*[, npartitions, orient, ...])                             | Construct a Dask DataFrame from a Python Dictionary                             |
| [`ge`](dask.dataframe.DataFrame.ge.md#dask.dataframe.DataFrame.ge)(other[, level, axis])                                                                  |                                                                                 |
| [`get_partition`](dask.dataframe.DataFrame.get_partition.md#dask.dataframe.DataFrame.get_partition)(n)                                                    | Get a dask DataFrame/Series representing the nth partition.                     |
| [`groupby`](dask.dataframe.DataFrame.groupby.md#dask.dataframe.DataFrame.groupby)(by[, group_keys, sort, observed, dropna])                               | Group DataFrame using a mapper or by a Series of columns.                       |
| [`gt`](dask.dataframe.DataFrame.gt.md#dask.dataframe.DataFrame.gt)(other[, level, axis])                                                                  |                                                                                 |
| [`head`](dask.dataframe.DataFrame.head.md#dask.dataframe.DataFrame.head)([n, npartitions, compute])                                                       | First n rows of the dataset                                                     |
| [`idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)([axis, skipna, numeric_only, split_every])                                 | Return index of first occurrence of maximum over requested axis.                |
| [`idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)([axis, skipna, numeric_only, split_every])                                 | Return index of first occurrence of minimum over requested axis.                |
| [`info`](dask.dataframe.DataFrame.info.md#dask.dataframe.DataFrame.info)([buf, verbose, memory_usage])                                                    | Concise summary of a Dask DataFrame                                             |
| [`isin`](dask.dataframe.DataFrame.isin.md#dask.dataframe.DataFrame.isin)(values)                                                                          | Whether each element in the DataFrame is contained in values.                   |
| [`isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)()                                                                                | Detect missing values.                                                          |
| [`isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)()                                                                          | DataFrame.isnull is an alias for DataFrame.isna.                                |
| [`items`](dask.dataframe.DataFrame.items.md#dask.dataframe.DataFrame.items)()                                                                             | Iterate over (column name, Series) pairs.                                       |
| [`iterrows`](dask.dataframe.DataFrame.iterrows.md#dask.dataframe.DataFrame.iterrows)()                                                                    | Iterate over DataFrame rows as (index, Series) pairs.                           |
| [`itertuples`](dask.dataframe.DataFrame.itertuples.md#dask.dataframe.DataFrame.itertuples)([index, name])                                                 | Iterate over DataFrame rows as namedtuples.                                     |
| [`join`](dask.dataframe.DataFrame.join.md#dask.dataframe.DataFrame.join)(other[, on, how, lsuffix, rsuffix, ...])                                         | Join columns of another DataFrame.                                              |
| `kurt`([axis, fisher, bias, nan_policy, ...])                                                                                                             | Return unbiased kurtosis over requested axis.                                   |
| `kurtosis`([axis, fisher, bias, nan_policy, ...])                                                                                                         | Return unbiased kurtosis over requested axis.                                   |
| [`le`](dask.dataframe.DataFrame.le.md#dask.dataframe.DataFrame.le)(other[, level, axis])                                                                  |                                                                                 |
| `lower_once`()                                                                                                                                            |                                                                                 |
| [`lt`](dask.dataframe.DataFrame.lt.md#dask.dataframe.DataFrame.lt)(other[, level, axis])                                                                  |                                                                                 |
| `map`(func[, na_action, meta])                                                                                                                            |                                                                                 |
| `map_overlap`(func, before, after, \*args[, ...])                                                                                                         | Apply a function to each partition, sharing rows with adjacent partitions.      |
| [`map_partitions`](dask.dataframe.DataFrame.map_partitions.md#dask.dataframe.DataFrame.map_partitions)(func, \*args[, meta, ...])                         | Apply a Python function to each partition                                       |
| [`mask`](dask.dataframe.DataFrame.mask.md#dask.dataframe.DataFrame.mask)(cond[, other])                                                                   | Replace values where the condition is True.                                     |
| [`max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)([axis, skipna, numeric_only, split_every])                                          | Return the maximum of the values over the requested axis.                       |
| [`mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)([axis, skipna, numeric_only, split_every])                                       | Return the mean of the values over the requested axis.                          |
| [`median`](dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)([axis, numeric_only])                                                      | Return the median of the values over the requested axis.                        |
| [`median_approximate`](dask.dataframe.DataFrame.median_approximate.md#dask.dataframe.DataFrame.median_approximate)([axis, method, numeric_only])          | Return the approximate median of the values over the requested axis.            |
| [`melt`](dask.dataframe.DataFrame.melt.md#dask.dataframe.DataFrame.melt)([id_vars, value_vars, var_name, ...])                                            | Unpivot DataFrame from wide to long format, optionally leaving identifiers set. |
| [`memory_usage`](dask.dataframe.DataFrame.memory_usage.md#dask.dataframe.DataFrame.memory_usage)([deep, index])                                           | Return the memory usage of each column in bytes.                                |
| [`memory_usage_per_partition`](dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition)([index, deep]) | Return the memory usage of each partition                                       |
| [`merge`](dask.dataframe.DataFrame.merge.md#dask.dataframe.DataFrame.merge)(right[, how, on, left_on, right_on, ...])                                     | Merge the DataFrame with another DataFrame                                      |
| [`min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)([axis, skipna, numeric_only, split_every])                                          | Return the minimum of the values over the requested axis.                       |
| [`mod`](dask.dataframe.DataFrame.mod.md#dask.dataframe.DataFrame.mod)(other[, axis, level, fill_value])                                                   |                                                                                 |
| [`mode`](dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)([dropna, split_every, numeric_only])                                             | Get the mode(s) of each element along the selected axis.                        |
| [`mul`](dask.dataframe.DataFrame.mul.md#dask.dataframe.DataFrame.mul)(other[, axis, level, fill_value])                                                   |                                                                                 |
| [`ne`](dask.dataframe.DataFrame.ne.md#dask.dataframe.DataFrame.ne)(other[, level, axis])                                                                  |                                                                                 |
| [`nlargest`](dask.dataframe.DataFrame.nlargest.md#dask.dataframe.DataFrame.nlargest)([n, columns, split_every])                                           | Return the first n rows ordered by columns in descending order.                 |
| `notnull`()                                                                                                                                               | DataFrame.notnull is an alias for DataFrame.notna.                              |
| [`nsmallest`](dask.dataframe.DataFrame.nsmallest.md#dask.dataframe.DataFrame.nsmallest)([n, columns, split_every])                                        | Return the first n rows ordered by columns in ascending order.                  |
| `nunique`([axis, dropna, split_every])                                                                                                                    | Count number of distinct elements in specified axis.                            |
| `nunique_approx`([split_every])                                                                                                                           | Approximate number of unique rows.                                              |
| `optimize`([fuse])                                                                                                                                        | Optimizes the DataFrame.                                                        |
| [`persist`](dask.dataframe.DataFrame.persist.md#dask.dataframe.DataFrame.persist)([fuse])                                                                 | Persist this dask collection into memory                                        |
| `pipe`(func, \*args, \*\*kwargs)                                                                                                                          | Apply chainable functions that expect Series or DataFrames.                     |
| [`pivot_table`](dask.dataframe.DataFrame.pivot_table.md#dask.dataframe.DataFrame.pivot_table)(index, columns, values[, aggfunc])                          | Create a spreadsheet-style pivot table as a DataFrame.                          |
| [`pop`](dask.dataframe.DataFrame.pop.md#dask.dataframe.DataFrame.pop)(item)                                                                               | Return item and drop it from DataFrame.                                         |
| [`pow`](dask.dataframe.DataFrame.pow.md#dask.dataframe.DataFrame.pow)(other[, axis, level, fill_value])                                                   |                                                                                 |
| `pprint`()                                                                                                                                                | Outputs a string representation of the DataFrame.                               |
| [`prod`](dask.dataframe.DataFrame.prod.md#dask.dataframe.DataFrame.prod)([axis, skipna, numeric_only, ...])                                               | Return the product of the values over the requested axis.                       |
| `product`([axis, skipna, numeric_only, ...])                                                                                                              | Return the product of the values over the requested axis.                       |
| [`quantile`](dask.dataframe.DataFrame.quantile.md#dask.dataframe.DataFrame.quantile)([q, axis, numeric_only, method])                                     | Approximate row-wise and precise column-wise quantiles of DataFrame             |
| [`query`](dask.dataframe.DataFrame.query.md#dask.dataframe.DataFrame.query)(expr, \*\*kwargs)                                                             | Filter dataframe with complex expression                                        |
| [`radd`](dask.dataframe.DataFrame.radd.md#dask.dataframe.DataFrame.radd)(other[, axis, level, fill_value])                                                |                                                                                 |
| [`random_split`](dask.dataframe.DataFrame.random_split.md#dask.dataframe.DataFrame.random_split)(frac[, random_state, shuffle])                           | Pseudorandomly split dataframe into different pieces row-wise                   |
| [`rdiv`](dask.dataframe.DataFrame.rdiv.md#dask.dataframe.DataFrame.rdiv)(other[, axis, level, fill_value])                                                |                                                                                 |
| `reduction`(chunk[, aggregate, combine, meta, ...])                                                                                                       | Generic row-wise reductions.                                                    |
| [`rename`](dask.dataframe.DataFrame.rename.md#dask.dataframe.DataFrame.rename)([index, columns])                                                          | Rename columns or index labels.                                                 |
| [`rename_axis`](dask.dataframe.DataFrame.rename_axis.md#dask.dataframe.DataFrame.rename_axis)([mapper, index, columns, axis])                             | Set the name of the axis for the index or columns.                              |
| [`repartition`](dask.dataframe.DataFrame.repartition.md#dask.dataframe.DataFrame.repartition)([divisions, npartitions, ...])                              | Repartition a collection                                                        |
| [`replace`](dask.dataframe.DataFrame.replace.md#dask.dataframe.DataFrame.replace)([to_replace, value, regex])                                             | Replace values given in to_replace with value.                                  |
| [`resample`](dask.dataframe.DataFrame.resample.md#dask.dataframe.DataFrame.resample)(rule[, closed, label])                                               | Resample time-series data.                                                      |
| [`reset_index`](dask.dataframe.DataFrame.reset_index.md#dask.dataframe.DataFrame.reset_index)([drop])                                                     | Reset the index to the default index.                                           |
| [`rfloordiv`](dask.dataframe.DataFrame.rfloordiv.md#dask.dataframe.DataFrame.rfloordiv)(other[, axis, level, fill_value])                                 |                                                                                 |
| [`rmod`](dask.dataframe.DataFrame.rmod.md#dask.dataframe.DataFrame.rmod)(other[, axis, level, fill_value])                                                |                                                                                 |
| [`rmul`](dask.dataframe.DataFrame.rmul.md#dask.dataframe.DataFrame.rmul)(other[, axis, level, fill_value])                                                |                                                                                 |
| [`rolling`](dask.dataframe.DataFrame.rolling.md#dask.dataframe.DataFrame.rolling)(window, \*\*kwargs)                                                     | Provides rolling transformations.                                               |
| [`round`](dask.dataframe.DataFrame.round.md#dask.dataframe.DataFrame.round)([decimals])                                                                   | Round numeric columns in a DataFrame to a variable number of decimal places.    |
| [`rpow`](dask.dataframe.DataFrame.rpow.md#dask.dataframe.DataFrame.rpow)(other[, axis, level, fill_value])                                                |                                                                                 |
| [`rsub`](dask.dataframe.DataFrame.rsub.md#dask.dataframe.DataFrame.rsub)(other[, axis, level, fill_value])                                                |                                                                                 |
| [`rtruediv`](dask.dataframe.DataFrame.rtruediv.md#dask.dataframe.DataFrame.rtruediv)(other[, axis, level, fill_value])                                    |                                                                                 |
| [`sample`](dask.dataframe.DataFrame.sample.md#dask.dataframe.DataFrame.sample)([n, frac, replace, random_state])                                          | Random sample of items                                                          |
| [`select_dtypes`](dask.dataframe.DataFrame.select_dtypes.md#dask.dataframe.DataFrame.select_dtypes)([include, exclude])                                   | Return a subset of the DataFrame's columns based on the column dtypes.          |
| [`sem`](dask.dataframe.DataFrame.sem.md#dask.dataframe.DataFrame.sem)([axis, skipna, ddof, split_every, ...])                                             | Return unbiased standard error of the mean over requested axis.                 |
| [`set_index`](dask.dataframe.DataFrame.set_index.md#dask.dataframe.DataFrame.set_index)(other[, drop, sorted, ...])                                       | Set the DataFrame index (row labels) using an existing column.                  |
| `shift`([periods, freq, axis])                                                                                                                            | Shift index by desired number of periods with an optional time freq.            |
| [`shuffle`](dask.dataframe.DataFrame.shuffle.md#dask.dataframe.DataFrame.shuffle)([on, ignore_index, npartitions, ...])                                   | Rearrange DataFrame into new partitions                                         |
| `simplify`()                                                                                                                                              |                                                                                 |
| `skew`([axis, bias, nan_policy, numeric_only])                                                                                                            | Return unbiased skew over requested axis.                                       |
| [`sort_values`](dask.dataframe.DataFrame.sort_values.md#dask.dataframe.DataFrame.sort_values)(by[, npartitions, ascending, ...])                          | Sort the dataset by a single column.                                            |
| [`squeeze`](dask.dataframe.DataFrame.squeeze.md#dask.dataframe.DataFrame.squeeze)([axis])                                                                 | Squeeze 1 dimensional axis objects into scalars.                                |
| [`std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)([axis, skipna, ddof, numeric_only, ...])                                            | Return sample standard deviation over requested axis.                           |
| [`sub`](dask.dataframe.DataFrame.sub.md#dask.dataframe.DataFrame.sub)(other[, axis, level, fill_value])                                                   |                                                                                 |
| [`sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)([axis, skipna, numeric_only, min_count, ...])                                       | Return the sum of the values over the requested axis.                           |
| [`tail`](dask.dataframe.DataFrame.tail.md#dask.dataframe.DataFrame.tail)([n, compute])                                                                    | Last n rows of the dataset                                                      |
| [`to_backend`](dask.dataframe.DataFrame.to_backend.md#dask.dataframe.DataFrame.to_backend)([backend])                                                     | Move to a new DataFrame backend                                                 |
| [`to_bag`](dask.dataframe.DataFrame.to_bag.md#dask.dataframe.DataFrame.to_bag)([index, format])                                                           | Create a Dask Bag from a Series                                                 |
| [`to_csv`](dask.dataframe.DataFrame.to_csv.md#dask.dataframe.DataFrame.to_csv)(filename, \*\*kwargs)                                                      | See dd.to_csv docstring for more information                                    |
| [`to_dask_array`](dask.dataframe.DataFrame.to_dask_array.md#dask.dataframe.DataFrame.to_dask_array)([lengths, meta, optimize])                            | Convert a dask DataFrame to a dask array.                                       |
| [`to_delayed`](dask.dataframe.DataFrame.to_delayed.md#dask.dataframe.DataFrame.to_delayed)([optimize_graph])                                              | Convert into a list of `dask.delayed` objects, one per partition.               |
| [`to_hdf`](dask.dataframe.DataFrame.to_hdf.md#dask.dataframe.DataFrame.to_hdf)(path_or_buf, key[, mode, append])                                          | See dd.to_hdf docstring for more information                                    |
| [`to_html`](dask.dataframe.DataFrame.to_html.md#dask.dataframe.DataFrame.to_html)([max_rows])                                                             | Render a DataFrame as an HTML table.                                            |
| [`to_json`](dask.dataframe.DataFrame.to_json.md#dask.dataframe.DataFrame.to_json)(filename, \*args, \*\*kwargs)                                           | See dd.to_json docstring for more information                                   |
| [`to_orc`](dask.dataframe.DataFrame.to_orc.md#dask.dataframe.DataFrame.to_orc)(path, \*args, \*\*kwargs)                                                  | See dd.to_orc docstring for more information                                    |
| [`to_parquet`](dask.dataframe.DataFrame.to_parquet.md#dask.dataframe.DataFrame.to_parquet)(path, \*\*kwargs)                                              |                                                                                 |
| [`to_records`](dask.dataframe.DataFrame.to_records.md#dask.dataframe.DataFrame.to_records)([index, lengths])                                              |                                                                                 |
| [`to_sql`](dask.dataframe.DataFrame.to_sql.md#dask.dataframe.DataFrame.to_sql)(name, uri[, schema, if_exists, ...])                                       |                                                                                 |
| [`to_string`](dask.dataframe.DataFrame.to_string.md#dask.dataframe.DataFrame.to_string)([max_rows])                                                       | Render a DataFrame to a console-friendly tabular output.                        |
| [`to_timestamp`](dask.dataframe.DataFrame.to_timestamp.md#dask.dataframe.DataFrame.to_timestamp)([freq, how])                                             | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.      |
| [`truediv`](dask.dataframe.DataFrame.truediv.md#dask.dataframe.DataFrame.truediv)(other[, axis, level, fill_value])                                       |                                                                                 |
| [`var`](dask.dataframe.DataFrame.var.md#dask.dataframe.DataFrame.var)([axis, skipna, ddof, numeric_only, ...])                                            | Return unbiased variance over requested axis.                                   |
| [`visualize`](dask.dataframe.DataFrame.visualize.md#dask.dataframe.DataFrame.visualize)([tasks])                                                          | Visualize the expression or task graph                                          |
| [`where`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)(cond[, other])                                                                | Replace values where the condition is False.                                    |

### Attributes

| `axes`                                                                                                    |                                                                                                                  |
|-----------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| [`columns`](dask.dataframe.DataFrame.columns.md#dask.dataframe.DataFrame.columns)                         |                                                                                                                  |
| `dask`                                                                                                    |                                                                                                                  |
| [`divisions`](dask.dataframe.DataFrame.divisions.md#dask.dataframe.DataFrame.divisions)                   | Tuple of `npartitions + 1` values, in ascending order, marking the lower/upper bounds of each partition's index. |
| [`dtypes`](dask.dataframe.DataFrame.dtypes.md#dask.dataframe.DataFrame.dtypes)                            | Return data types                                                                                                |
| `empty`                                                                                                   |                                                                                                                  |
| `expr`                                                                                                    |                                                                                                                  |
| [`iloc`](dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)                                  | Purely integer-location based indexing for selection by position.                                                |
| [`index`](dask.dataframe.DataFrame.index.md#dask.dataframe.DataFrame.index)                               | Return dask Index instance                                                                                       |
| [`known_divisions`](dask.dataframe.DataFrame.known_divisions.md#dask.dataframe.DataFrame.known_divisions) | Whether the divisions are known.                                                                                 |
| [`loc`](dask.dataframe.DataFrame.loc.md#dask.dataframe.DataFrame.loc)                                     | Purely label-location based indexer for selection by label.                                                      |
| `nbytes`                                                                                                  |                                                                                                                  |
| [`ndim`](dask.dataframe.DataFrame.ndim.md#dask.dataframe.DataFrame.ndim)                                  | Return dimensionality                                                                                            |
| [`npartitions`](dask.dataframe.DataFrame.npartitions.md#dask.dataframe.DataFrame.npartitions)             | Return number of partitions                                                                                      |
| [`partitions`](dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions)                | Slice dataframe by partitions                                                                                    |
| [`shape`](dask.dataframe.DataFrame.shape.md#dask.dataframe.DataFrame.shape)                               |                                                                                                                  |
| [`size`](dask.dataframe.DataFrame.size.md#dask.dataframe.DataFrame.size)                                  | Size of the Series or DataFrame as a Delayed object.                                                             |
| [`values`](dask.dataframe.DataFrame.values.md#dask.dataframe.DataFrame.values)                            | Return a dask.array of the values of this dataframe                                                              |


# dask.dataframe.DataFrame.idxmax.html.md

# dask.dataframe.DataFrame.idxmax

#### DataFrame.idxmax(axis=0, skipna=True, numeric_only=False, split_every=False)

Return index of first occurrence of maximum over requested axis.

This docstring was copied from pandas.DataFrame.idxmax.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of maxima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return index of the maximum element.

### Notes

This method is the DataFrame version of `ndarray.argmax`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the maximum value in each column.

```pycon
>>> df.idxmax()
consumption      Wheat Products
co2_emissions              Beef
dtype: str
```

To return the index for the maximum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmax(axis="columns")
Pork              co2_emissions
Wheat Products     consumption
Beef              co2_emissions
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.idxmin.html.md

# dask.dataframe.DataFrame.idxmin

#### DataFrame.idxmin(axis=0, skipna=True, numeric_only=False, split_every=False)

Return index of first occurrence of minimum over requested axis.

This docstring was copied from pandas.DataFrame.idxmin.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of minima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return index of the minimum element.

### Notes

This method is the DataFrame version of `ndarray.argmin`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the minimum value in each column.

```pycon
>>> df.idxmin()
consumption                Pork
co2_emissions    Wheat Products
dtype: str
```

To return the index for the minimum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmin(axis="columns")
Pork                consumption
Wheat Products    co2_emissions
Beef                consumption
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.iloc.html.md

# dask.dataframe.DataFrame.iloc

#### *property* DataFrame.iloc

Purely integer-location based indexing for selection by position.

Only indexing the column positions is supported. Trying to select
row positions will raise a ValueError.

See [Indexing into Dask DataFrames](../dataframe-indexing.md#dataframe-indexing) for more.

### Examples

```pycon
>>> df.iloc[:, [2, 0, 1]]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.index.html.md

# dask.dataframe.DataFrame.index

#### *property* DataFrame.index

Return dask Index instance

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.info.html.md

# dask.dataframe.DataFrame.info

#### DataFrame.info(buf=None, verbose=False, memory_usage=False)

Concise summary of a Dask DataFrame

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.isin.html.md

# dask.dataframe.DataFrame.isin

#### DataFrame.isin(values)

Whether each element in the DataFrame is contained in values.

This docstring was copied from pandas.DataFrame.isin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **values**
  : The result will only be true at a location if all the
    labels match. If values is a Series, that’s the index. If
    values is a dict, the keys must be the column names,
    which must match. If values is a DataFrame,
    then both the index and column labels must match.
* **Returns:**
  DataFrame
  : DataFrame of booleans showing whether each element in the DataFrame
    is contained in values.

#### SEE ALSO
[`DataFrame.eq`](dask.dataframe.DataFrame.eq.md#dask.dataframe.DataFrame.eq)
: Equality test for DataFrame.

[`Series.isin`](dask.dataframe.Series.isin.md#dask.dataframe.Series.isin)
: Equivalent method on Series.

[`Series.str.contains`](dask.dataframe.Series.str.contains.md#dask.dataframe.Series.str.contains)
: Test if pattern or regex is contained within a string of a Series or Index.

### Notes

`__iter__` is used (and not `__contains__`) to iterate over values
when checking if it contains the elements in DataFrame.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"num_legs": [2, 4], "num_wings": [2, 0]}, index=["falcon", "dog"]
... )
>>> df
        num_legs  num_wings
falcon         2          2
dog            4          0
```

When `values` is a list check whether every value in the DataFrame
is present in the list (which animals have 0 or 2 legs or wings)

```pycon
>>> df.isin([0, 2])
        num_legs  num_wings
falcon      True       True
dog        False       True
```

To check if `values` is *not* in the DataFrame, use the `~` operator:

```pycon
>>> ~df.isin([0, 2])
        num_legs  num_wings
falcon     False      False
dog         True      False
```

When `values` is a dict, we can pass values to check for each
column separately:

```pycon
>>> df.isin({"num_wings": [0, 3]})
        num_legs  num_wings
falcon     False      False
dog        False       True
```

When `values` is a Series or DataFrame the index and column must
match. Note that ‘falcon’ does not match based on the number of legs
in other.

```pycon
>>> other = pd.DataFrame(
...     {"num_legs": [8, 3], "num_wings": [0, 2]}, index=["spider", "falcon"]
... )
>>> df.isin(other)
        num_legs  num_wings
falcon     False       True
dog        False      False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.isna.html.md

# dask.dataframe.DataFrame.isna

#### DataFrame.isna()

Detect missing values.

This docstring was copied from pandas.DataFrame.isna.

Some inconsistencies with the Dask version may exist.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](#dask.dataframe.DataFrame.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.isnull.html.md

# dask.dataframe.DataFrame.isnull

#### DataFrame.isnull()

DataFrame.isnull is an alias for DataFrame.isna.

This docstring was copied from pandas.DataFrame.isnull.

Some inconsistencies with the Dask version may exist.

Detect missing values.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.items.html.md

# dask.dataframe.DataFrame.items

#### DataFrame.items()

Iterate over (column name, Series) pairs.

This docstring was copied from pandas.DataFrame.items.

Some inconsistencies with the Dask version may exist.

Iterates over the DataFrame columns, returning a tuple with
the column name and the content as a Series.

* **Yields:**
  **label**
  : The column names for the DataFrame being iterated over.

  **content**
  : The column entries belonging to each label, as a Series.

#### SEE ALSO
[`DataFrame.iterrows`](dask.dataframe.DataFrame.iterrows.md#dask.dataframe.DataFrame.iterrows)
: Iterate over DataFrame rows as (index, Series) pairs.

[`DataFrame.itertuples`](dask.dataframe.DataFrame.itertuples.md#dask.dataframe.DataFrame.itertuples)
: Iterate over DataFrame rows as namedtuples of the values.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "species": ["bear", "bear", "marsupial"],
...         "population": [1864, 22000, 80000],
...     },
...     index=["panda", "polar", "koala"],
... )
>>> df
        species   population
panda   bear      1864
polar   bear      22000
koala   marsupial 80000
>>> for label, content in df.items():
...     print(f"label: {label}")
...     print(f"content: {content}", sep="\n")
label: species
content:
panda         bear
polar         bear
koala    marsupial
Name: species, dtype: str
label: population
content:
panda     1864
polar    22000
koala    80000
Name: population, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.iterrows.html.md

# dask.dataframe.DataFrame.iterrows

#### DataFrame.iterrows()

Iterate over DataFrame rows as (index, Series) pairs.

This docstring was copied from pandas.DataFrame.iterrows.

Some inconsistencies with the Dask version may exist.

* **Yields:**
  **index**
  : The index of the row. A tuple for a MultiIndex.

  **data**
  : The data of the row as a Series.

#### SEE ALSO
[`DataFrame.itertuples`](dask.dataframe.DataFrame.itertuples.md#dask.dataframe.DataFrame.itertuples)
: Iterate over DataFrame rows as namedtuples of the values.

[`DataFrame.items`](dask.dataframe.DataFrame.items.md#dask.dataframe.DataFrame.items)
: Iterate over (column name, Series) pairs.

### Notes

1. Because `iterrows` returns a Series for each row,
   it does **not** preserve dtypes across the rows (dtypes are
   preserved across columns for DataFrames).

   To preserve dtypes while iterating over the rows, it is better
   to use [`itertuples()`](dask.dataframe.DataFrame.itertuples.md#dask.dataframe.DataFrame.itertuples) which returns namedtuples of the values
   and which is generally faster than `iterrows`.
2. You should **never modify** something you are iterating over.
   This is not guaranteed to work in all cases. Depending on the
   data types, the iterator returns a copy and not a view, and writing
   to it will have no effect.

### Examples

```pycon
>>> df = pd.DataFrame([[1, 1.5]], columns=["int", "float"])
>>> row = next(df.iterrows())[1]
>>> row
int      1.0
float    1.5
Name: 0, dtype: float64
>>> print(row["int"].dtype)
float64
>>> print(df["int"].dtype)
int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.itertuples.html.md

# dask.dataframe.DataFrame.itertuples

#### DataFrame.itertuples(index=True, name='Pandas')

Iterate over DataFrame rows as namedtuples.

This docstring was copied from pandas.DataFrame.itertuples.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **index**
  : If True, return the index as the first element of the tuple.

  **name**
  : The name of the returned namedtuples or None to return regular
    tuples.
* **Returns:**
  iterator
  : An object to iterate over namedtuples for each row in the
    DataFrame with the first field possibly being the index and
    following fields being the column values.

#### SEE ALSO
[`DataFrame.iterrows`](dask.dataframe.DataFrame.iterrows.md#dask.dataframe.DataFrame.iterrows)
: Iterate over DataFrame rows as (index, Series) pairs.

[`DataFrame.items`](dask.dataframe.DataFrame.items.md#dask.dataframe.DataFrame.items)
: Iterate over (column name, Series) pairs.

### Notes

The column names will be renamed to positional names if they are
invalid Python identifiers, repeated, or start with an underscore.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"num_legs": [4, 2], "num_wings": [0, 2]}, index=["dog", "hawk"]
... )
>>> df
      num_legs  num_wings
dog          4          0
hawk         2          2
>>> for row in df.itertuples():
...     print(row)
Pandas(Index='dog', num_legs=4, num_wings=0)
Pandas(Index='hawk', num_legs=2, num_wings=2)
```

By setting the index parameter to False we can remove the index
as the first element of the tuple:

```pycon
>>> for row in df.itertuples(index=False):
...     print(row)
Pandas(num_legs=4, num_wings=0)
Pandas(num_legs=2, num_wings=2)
```

With the name parameter set we set a custom name for the yielded
namedtuples:

```pycon
>>> for row in df.itertuples(name="Animal"):
...     print(row)
Animal(Index='dog', num_legs=4, num_wings=0)
Animal(Index='hawk', num_legs=2, num_wings=2)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.join.html.md

# dask.dataframe.DataFrame.join

#### DataFrame.join(other, on=None, how='left', lsuffix='', rsuffix='', shuffle_method=None, npartitions=None)

Join columns of another DataFrame.

This docstring was copied from pandas.DataFrame.join.

Some inconsistencies with the Dask version may exist.

Join columns with other DataFrame either on index or on a key
column. Efficiently join multiple DataFrame objects by index at once by
passing a list.

* **Parameters:**
  **other**
  : Index should be similar to one of the columns in this one. If a
    Series is passed, its name attribute must be set, and that will be
    used as the column name in the resulting joined DataFrame.

  **on**
  : Column or index level name(s) in the caller to join on the index
    in other, otherwise joins index-on-index. If multiple
    values given, the other DataFrame must have a MultiIndex. Can
    pass an array as the join key if it is not already contained in
    the calling DataFrame. Like an Excel VLOOKUP operation.

  **how**
  : default ‘left’
    How to handle the operation of the two objects.
    * left: use calling frame’s index (or column if on is specified)
    * right: use other’s index.
    * outer: form union of calling frame’s index (or column if on is
      specified) with other’s index, and sort it lexicographically.
    * inner: form intersection of calling frame’s index (or column if
      on is specified) with other’s index, preserving the order
      of the calling’s one.
    * cross: creates the cartesian product from both frames, preserves the order
      of the left keys.
    * left_anti: use set difference of calling frame’s index and other’s
      index.
    * right_anti: use set difference of other’s index and calling frame’s
      index.

  **lsuffix**
  : Suffix to use from left frame’s overlapping columns.

  **rsuffix**
  : Suffix to use from right frame’s overlapping columns.

  **sort**
  : Order result DataFrame lexicographically by the join key. If False,
    the order of the join key depends on the join type (how keyword).

  **validate**
  : If specified, checks if join is of specified type.
    * “one_to_one” or “1:1”: check if join keys are unique in both left
      and right datasets.
    * “one_to_many” or “1:m”: check if join keys are unique in left dataset.
    * “many_to_one” or “m:1”: check if join keys are unique in right dataset.
    * “many_to_many” or “m:m”: allowed, but does not result in checks.
* **Returns:**
  DataFrame
  : A dataframe containing columns from both the caller and other.

#### SEE ALSO
[`DataFrame.merge`](dask.dataframe.DataFrame.merge.md#dask.dataframe.DataFrame.merge)
: For column(s)-on-column(s) operations.

### Notes

Parameters on, lsuffix, and rsuffix are not supported when
passing a list of DataFrame objects.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "key": ["K0", "K1", "K2", "K3", "K4", "K5"],
...         "A": ["A0", "A1", "A2", "A3", "A4", "A5"],
...     }
... )
```

```pycon
>>> df
  key   A
0  K0  A0
1  K1  A1
2  K2  A2
3  K3  A3
4  K4  A4
5  K5  A5
```

```pycon
>>> other = pd.DataFrame({"key": ["K0", "K1", "K2"], "B": ["B0", "B1", "B2"]})
```

```pycon
>>> other
  key   B
0  K0  B0
1  K1  B1
2  K2  B2
```

Join DataFrames using their indexes.

```pycon
>>> df.join(other, lsuffix="_caller", rsuffix="_other")
  key_caller   A key_other    B
0         K0  A0        K0   B0
1         K1  A1        K1   B1
2         K2  A2        K2   B2
3         K3  A3       NaN  NaN
4         K4  A4       NaN  NaN
5         K5  A5       NaN  NaN
```

If we want to join using the key columns, we need to set key to be
the index in both df and other. The joined DataFrame will have
key as its index.

```pycon
>>> df.set_index("key").join(other.set_index("key"))
      A    B
key
K0   A0   B0
K1   A1   B1
K2   A2   B2
K3   A3  NaN
K4   A4  NaN
K5   A5  NaN
```

Another option to join using the key columns is to use the on
parameter. DataFrame.join always uses other’s index but we can use
any column in df. This method preserves the original DataFrame’s
index in the result.

```pycon
>>> df.join(other.set_index("key"), on="key")
  key   A    B
0  K0  A0   B0
1  K1  A1   B1
2  K2  A2   B2
3  K3  A3  NaN
4  K4  A4  NaN
5  K5  A5  NaN
```

Using non-unique key values shows how they are matched.

```pycon
>>> df = pd.DataFrame(
...     {
...         "key": ["K0", "K1", "K1", "K3", "K0", "K1"],
...         "A": ["A0", "A1", "A2", "A3", "A4", "A5"],
...     }
... )
```

```pycon
>>> df
  key   A
0  K0  A0
1  K1  A1
2  K1  A2
3  K3  A3
4  K0  A4
5  K1  A5
```

```pycon
>>> df.join(other.set_index("key"), on="key", validate="m:1")
  key   A    B
0  K0  A0   B0
1  K1  A1   B1
2  K1  A2   B1
3  K3  A3  NaN
4  K0  A4   B0
5  K1  A5   B1
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.known_divisions.html.md

# dask.dataframe.DataFrame.known_divisions

#### *property* DataFrame.known_divisions

Whether the divisions are known.

This check can be expensive if the division calculation is expensive.
DataFrame.set_index is a good example where the calculation needs an
inspection of the data.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.le.html.md

# dask.dataframe.DataFrame.le

#### DataFrame.le(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.loc.html.md

# dask.dataframe.DataFrame.loc

#### *property* DataFrame.loc

Purely label-location based indexer for selection by label.

```pycon
>>> df.loc["b"]
>>> df.loc["b":"d"]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.lt.html.md

# dask.dataframe.DataFrame.lt

#### DataFrame.lt(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.map_partitions.html.md

# dask.dataframe.DataFrame.map_partitions

#### DataFrame.map_partitions(func, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, parent_meta=None, required_columns=None, \*\*kwargs)

Apply a Python function to each partition

* **Parameters:**
  **func**
  : Function applied to each partition.

  **args, kwargs**
  : Arguments and keywords to pass to the function. Arguments and
    keywords may contain `FrameBase` or regular python objects.
    DataFrame-like args (both dask and pandas) must have the same
    number of partitions as `self` or comprise a single partition.
    Key-word arguments, Single-partition arguments, and general
    python-object arguments will be broadcasted to all partitions.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition, and will
    raise an error if this doesn’t work, but it won’t raise if dtypes
    don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **clear_divisions**
  : Whether divisions should be cleared. If True, transform_divisions
    will be ignored.

  **required_columns**
  : List of columns that `func` requires for execution. These columns
    must belong to the first DataFrame argument (in `args`). If None
    is specified (the default), the query optimizer will assume that
    all input columns are required.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

### Examples

Given a DataFrame, Series, or Index, such as:

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 3, 4, 5],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

One can use `map_partitions` to apply a function on each partition.
Extra arguments and keywords can optionally be provided, and will be
passed to the function after the partition.

Here we apply a function with arguments and keywords to a DataFrame,
resulting in a Series:

```pycon
>>> def myadd(df, a, b=1):
...     return df.x + df.y + a + b
>>> res = ddf.map_partitions(myadd, 1, b=2)
>>> res.dtype
dtype('float64')
```

Here we apply a function to a Series resulting in a Series:

```pycon
>>> res = ddf.x.map_partitions(lambda x: len(x)) # ddf.x is a Dask Series Structure
>>> res.dtype
dtype('int64')
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with no name, and dtype
`float64`:

```pycon
>>> res = ddf.map_partitions(myadd, 1, b=2, meta=(None, 'f8'))
```

Here we map a function that takes in a DataFrame, and returns a
DataFrame with a new column:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y))
>>> res.dtypes
x      int64
y    float64
z    float64
dtype: object
```

As before, the output metadata can also be specified manually. This
time we pass in a `dict`, as the output is a DataFrame:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y),
...                          meta={'x': 'i8', 'y': 'f8', 'z': 'f8'})
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ddf.map_partitions(lambda df: df.head(), meta=ddf)
```

Also note that the index and divisions are assumed to remain unchanged.
If the function you’re mapping changes the index/divisions, you’ll need
to pass `clear_divisions=True`.

```pycon
>>> ddf.map_partitions(func, clear_divisions=True)
```

Your map function gets information about where it is in the dataframe by
accepting a special `partition_info` keyword argument.

```pycon
>>> def func(partition, partition_info=None):
...     pass
```

This will receive the following information:

```pycon
>>> partition_info
{'number': 1, 'division': 3}
```

For each argument and keyword arguments that are dask dataframes you will
receive the number (n) which represents the nth partition of the dataframe
and the division (the first index value in the partition). If divisions
are not known (for instance if the index is not sorted) then you will get
None as the division.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.mask.html.md

# dask.dataframe.DataFrame.mask

#### DataFrame.mask(cond, other=nan)

Replace values where the condition is True.

This docstring was copied from pandas.DataFrame.mask.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **cond**
  : Where cond is False, keep the original value. Where
    True, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is True are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Return an object of same shape as caller.

[`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Return an object of same shape as caller.

### Notes

The mask method is an application of the if-then idiom. For each
element in the caller, if `cond` is `False` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with True.

The signature for [`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where) or
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `mask` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.max.html.md

# dask.dataframe.DataFrame.max

#### DataFrame.max(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the maximum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.max.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the maximum, use `idxmax`.
This is the equivalent of the `numpy.ndarray` method `argmax`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.max()
8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.mean.html.md

# dask.dataframe.DataFrame.mean

#### DataFrame.mean(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the mean of the values over the requested axis.

This docstring was copied from pandas.DataFrame.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.mean()
2.0
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.mean()
a   1.5
b   2.5
dtype: float64
```

Using axis=1

```pycon
>>> df.mean(axis=1)
tiger   1.5
zebra   2.5
dtype: float64
```

In this case, numeric_only should be set to True to avoid
getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.mean(numeric_only=True)
a   1.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.median.html.md

# dask.dataframe.DataFrame.median

#### DataFrame.median(axis=0, numeric_only=False)

Return the median of the values over the requested axis.

This docstring was copied from pandas.DataFrame.median.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.median()
2.0
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.median()
a   1.5
b   2.5
dtype: float64
```

Using axis=1

```pycon
>>> df.median(axis=1)
tiger   1.5
zebra   2.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.median(numeric_only=True)
a   1.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.median_approximate.html.md

# dask.dataframe.DataFrame.median_approximate

#### DataFrame.median_approximate(axis=0, method='default', numeric_only=False)

Return the approximate median of the values over the requested axis.

* **Parameters:**
  **axis**
  : 0 or `"index"` for row-wise, 1 or `"columns"` for column-wise

  **method**
  : What method to use. By default will use Dask’s internal custom
    algorithm (`"dask"`).  If set to `"tdigest"` will use tdigest
    for floats and ints and fallback to the `"dask"` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.melt.html.md

# dask.dataframe.DataFrame.melt

#### DataFrame.melt(id_vars=None, value_vars=None, var_name=None, value_name='value', col_level=None)

Unpivot DataFrame from wide to long format, optionally leaving identifiers set.

This docstring was copied from pandas.DataFrame.melt.

Some inconsistencies with the Dask version may exist.

This function is useful to massage a DataFrame into a format where one
or more columns are identifier variables (id_vars), while all other
columns, considered measured variables (value_vars), are “unpivoted” to
the row axis, leaving just two non-identifier columns, ‘variable’ and
‘value’.

* **Parameters:**
  **id_vars**
  : Column(s) to use as identifier variables.

  **value_vars**
  : Column(s) to unpivot. If not specified, uses all columns that
    are not set as id_vars.

  **var_name**
  : Name to use for the ‘variable’ column. If None it uses
    `frame.columns.name` or ‘variable’.

  **value_name**
  : Name to use for the ‘value’ column, can’t be an existing column label.

  **col_level**
  : If columns are a MultiIndex then use this level to melt.

  **ignore_index**
  : If True, original index is ignored. If False, original index is retained.
    Index labels will be repeated as necessary.
* **Returns:**
  DataFrame
  : Unpivoted DataFrame.

#### SEE ALSO
[`melt`](dask.dataframe.melt.md#dask.dataframe.melt)
: Identical method.

[`pivot_table`](dask.dataframe.pivot_table.md#dask.dataframe.pivot_table)
: Create a spreadsheet-style pivot table as a DataFrame.

`DataFrame.pivot`
: Return reshaped DataFrame organized by given index / column values.

[`DataFrame.explode`](dask.dataframe.DataFrame.explode.md#dask.dataframe.DataFrame.explode)
: Explode a DataFrame from list-like columns to long format.

### Notes

Reference [the user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html#reshaping-melt) for more examples.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": {0: "a", 1: "b", 2: "c"},
...         "B": {0: 1, 1: 3, 2: 5},
...         "C": {0: 2, 1: 4, 2: 6},
...     }
... )
>>> df
A  B  C
0  a  1  2
1  b  3  4
2  c  5  6
```

```pycon
>>> df.melt(id_vars=["A"], value_vars=["B"])
A variable  value
0  a        B      1
1  b        B      3
2  c        B      5
```

```pycon
>>> df.melt(id_vars=["A"], value_vars=["B", "C"])
A variable  value
0  a        B      1
1  b        B      3
2  c        B      5
3  a        C      2
4  b        C      4
5  c        C      6
```

The names of ‘variable’ and ‘value’ columns can be customized:

```pycon
>>> df.melt(
...     id_vars=["A"],
...     value_vars=["B"],
...     var_name="myVarname",
...     value_name="myValname",
... )
A myVarname  myValname
0  a         B          1
1  b         B          3
2  c         B          5
```

Original index values can be kept around:

```pycon
>>> df.melt(id_vars=["A"], value_vars=["B", "C"], ignore_index=False)
A variable  value
0  a        B      1
1  b        B      3
2  c        B      5
0  a        C      2
1  b        C      4
2  c        C      6
```

If you have multi-index columns:

```pycon
>>> df.columns = [list("ABC"), list("DEF")]
>>> df
A  B  C
D  E  F
0  a  1  2
1  b  3  4
2  c  5  6
```

```pycon
>>> df.melt(col_level=0, id_vars=["A"], value_vars=["B"])
A variable  value
0  a        B      1
1  b        B      3
2  c        B      5
```

```pycon
>>> df.melt(id_vars=[("A", "D")], value_vars=[("B", "E")])
(A, D) variable_0 variable_1  value
0      a          B          E      1
1      b          B          E      3
2      c          B          E      5
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.memory_usage.html.md

# dask.dataframe.DataFrame.memory_usage

#### DataFrame.memory_usage(deep=False, index=True)

Return the memory usage of each column in bytes.

This docstring was copied from pandas.DataFrame.memory_usage.

Some inconsistencies with the Dask version may exist.

The memory usage can optionally include the contribution of
the index and elements of object dtype.

This value is displayed in DataFrame.info by default. This can be
suppressed by setting `pandas.options.display.memory_usage` to False.

* **Parameters:**
  **index**
  : Specifies whether to include the memory usage of the DataFrame’s
    index in returned Series. If `index=True`, the memory usage of
    the index is the first item in the output.

  **deep**
  : If True, introspect the data deeply by interrogating
    object dtypes for system-level memory consumption, and include
    it in the returned values.
* **Returns:**
  Series
  : A Series whose index is the original column names and whose values
    is the memory usage of each column in bytes.

#### SEE ALSO
[`numpy.ndarray.nbytes`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.nbytes.html#numpy.ndarray.nbytes)
: Total bytes consumed by the elements of an ndarray.

[`Series.memory_usage`](dask.dataframe.Series.memory_usage.md#dask.dataframe.Series.memory_usage)
: Bytes consumed by a Series.

`Categorical`
: Memory-efficient array for string values with many repeated values.

[`DataFrame.info`](dask.dataframe.DataFrame.info.md#dask.dataframe.DataFrame.info)
: Concise summary of a DataFrame.

### Notes

See the [Frequently Asked Questions](https://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#df-memory-usage) for more
details.

### Examples

```pycon
>>> dtypes = ["int64", "float64", "complex128", "object", "bool"]
>>> data = dict([(t, np.ones(shape=5000, dtype=int).astype(t)) for t in dtypes])
>>> df = pd.DataFrame(data)
>>> df.head()
   int64  float64            complex128  object  bool
0      1      1.0              1.0+0.0j       1  True
1      1      1.0              1.0+0.0j       1  True
2      1      1.0              1.0+0.0j       1  True
3      1      1.0              1.0+0.0j       1  True
4      1      1.0              1.0+0.0j       1  True
```

```pycon
>>> df.memory_usage()
Index           132
int64         40000
float64       40000
complex128    80000
object        40000
bool           5000
dtype: int64
```

```pycon
>>> df.memory_usage(index=False)
int64         40000
float64       40000
complex128    80000
object        40000
bool           5000
dtype: int64
```

The memory footprint of object dtype columns is ignored by default:

```pycon
>>> df.memory_usage(deep=True)
Index            132
int64          40000
float64        40000
complex128     80000
object        180000
bool            5000
dtype: int64
```

Use a Categorical for efficient storage of an object-dtype column with
many repeated values.

```pycon
>>> df["object"].astype("category").memory_usage(deep=True)
5140
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.memory_usage_per_partition.html.md

# dask.dataframe.DataFrame.memory_usage_per_partition

#### DataFrame.memory_usage_per_partition(index: [bool](https://docs.python.org/3/library/functions.html#bool) = True, deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Return the memory usage of each partition

* **Parameters:**
  **index**
  : Specifies whether to include the memory usage of the index in
    returned Series.

  **deep**
  : If True, introspect the data deeply by interrogating
    `object` dtypes for system-level memory consumption, and include
    it in the returned values.
* **Returns:**
  Series
  : A Series whose index is the partition number and whose values
    are the memory usage of each partition in bytes.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.merge.html.md

# dask.dataframe.DataFrame.merge

#### DataFrame.merge(right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, suffixes=('_x', '_y'), indicator=False, shuffle_method=None, npartitions=None, broadcast=None)

Merge the DataFrame with another DataFrame

This will merge the two datasets, either on the indices, a certain column
in each dataset or the index in one dataset and the column in another.

* **Parameters:**
  **right: dask.dataframe.DataFrame**

  **how**
  : How to handle the operation of the two objects:
    - left: use calling frame’s index (or column if on is specified)
    - right: use other frame’s index
    - outer: form union of calling frame’s index (or column if on is
      specified) with other frame’s index, and sort it
      lexicographically
    - inner: form intersection of calling frame’s index (or column if
      on is specified) with other frame’s index, preserving the order
      of the calling’s one
    - leftsemi: Choose all rows in left where the join keys can be found
      in right. Won’t duplicate rows if the keys are duplicated in right.
      Drops all columns from right.

  **on**
  : Column or index level names to join on. These must be found in both
    DataFrames. If on is None and not merging on indexes then this
    defaults to the intersection of the columns in both DataFrames.

  **left_on**
  : Column to join on in the left DataFrame. Other than in pandas
    arrays and lists are only support if their length is 1.

  **right_on**
  : Column to join on in the right DataFrame. Other than in pandas
    arrays and lists are only support if their length is 1.

  **left_index**
  : Use the index from the left DataFrame as the join key.

  **right_index**
  : Use the index from the right DataFrame as the join key.

  **suffixes**
  : Suffix to apply to overlapping column names in the left and
    right side, respectively

  **indicator**
  : If True, adds a column to output DataFrame called “_merge” with
    information on the source of each row. If string, column with
    information on source of each row will be added to output DataFrame,
    and column will be named value of string. Information column is
    Categorical-type and takes on a value of “left_only” for observations
    whose merge key only appears in left DataFrame, “right_only” for
    observations whose merge key only appears in right DataFrame,
    and “both” if the observation’s merge key is found in both.

  **npartitions: int or None, optional**
  : The ideal number of output partitions. This is only utilised when
    performing a hash_join (merging on columns only). If `None` then
    `npartitions = max(lhs.npartitions, rhs.npartitions)`.
    Default is `None`.

  **shuffle_method: {‘disk’, ‘tasks’, ‘p2p’}, optional**
  : Either `'disk'` for single-node operation or `'tasks'` and
    `'p2p'`` for distributed operation.  Will be inferred by your
    current scheduler.

  **broadcast: boolean or float, optional**
  : Whether to use a broadcast-based join in lieu of a shuffle-based
    join for supported cases.  By default, a simple heuristic will be
    used to select the underlying algorithm. If a floating-point value
    is specified, that number will be used as the `broadcast_bias`
    within the simple heuristic (a large number makes Dask more likely
    to choose the `broadcast_join` code path). See `broadcast_join`
    for more information.

### Notes

There are three ways to join dataframes:

1. Joining on indices. In this case the divisions are
   aligned using the function `dask.dataframe.multi.align_partitions`.
   Afterwards, each partition is merged with the pandas merge function.
2. Joining one on index and one on column. In this case the divisions of
   dataframe merged by index ($d_i$) are used to divide the column
   merged dataframe ($d_c$) one using
   `dask.dataframe.multi.rearrange_by_divisions`. In this case the
   merged dataframe ($d_m$) has the exact same divisions
   as ($d_i$). This can lead to issues if you merge multiple rows from
   ($d_c$) to one row in ($d_i$).
3. Joining both on columns. In this case a hash join is performed using
   `dask.dataframe.multi.hash_join`.

In some cases, you may see a `MemoryError` if the `merge` operation requires
an internal `shuffle`, because shuffling places all rows that have the same
index in the same partition. To avoid this error, make sure all rows with the
same `on`-column value can fit on a single partition.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.min.html.md

# dask.dataframe.DataFrame.min

#### DataFrame.min(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the minimum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.min.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the minimum, use `idxmin`.
This is the equivalent of the `numpy.ndarray` method `argmin`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.min()
0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.mod.html.md

# dask.dataframe.DataFrame.mod

#### DataFrame.mod(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.mode.html.md

# dask.dataframe.DataFrame.mode

#### DataFrame.mode(dropna=True, split_every=False, numeric_only=False)

Get the mode(s) of each element along the selected axis.

This docstring was copied from pandas.DataFrame.mode.

Some inconsistencies with the Dask version may exist.

The mode of a set of values is the value that appears most often.
It can be multiple values.

* **Parameters:**
  **axis**
  : The axis to iterate over while searching for the mode:
    * 0 or ‘index’ : get mode of each column
    * 1 or ‘columns’ : get mode of each row.

  **numeric_only**
  : If True, only apply to numeric columns.

  **dropna**
  : Don’t consider counts of NaN/NaT.
* **Returns:**
  DataFrame
  : The modes of each column or row.

#### SEE ALSO
`Series.mode`
: Return the highest frequency value in a Series.

[`Series.value_counts`](dask.dataframe.Series.value_counts.md#dask.dataframe.Series.value_counts)
: Return the counts of values in a Series.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         ("bird", 2, 2),
...         ("mammal", 4, np.nan),
...         ("arthropod", 8, 0),
...         ("bird", 2, np.nan),
...     ],
...     index=("falcon", "horse", "spider", "ostrich"),
...     columns=("species", "legs", "wings"),
... )
>>> df
           species  legs  wings
falcon        bird     2    2.0
horse       mammal     4    NaN
spider   arthropod     8    0.0
ostrich       bird     2    NaN
```

By default, missing values are not considered, and the mode of wings
are both 0 and 2. Because the resulting DataFrame has two rows,
the second row of `species` and `legs` contains `NaN`.

```pycon
>>> df.mode()
  species  legs  wings
0    bird   2.0    0.0
1     NaN   NaN    2.0
```

Setting `dropna=False` `NaN` values are considered and they can be
the mode (like for wings).

```pycon
>>> df.mode(dropna=False)
  species  legs  wings
0    bird     2    NaN
```

Setting `numeric_only=True`, only the mode of numeric columns is
computed, and columns of other types are ignored.

```pycon
>>> df.mode(numeric_only=True)
   legs  wings
0   2.0    0.0
1   NaN    2.0
```

To compute the mode over columns and not rows, use the axis parameter:

```pycon
>>> df.mode(axis="columns", numeric_only=True)
           0    1
falcon   2.0  NaN
horse    4.0  NaN
spider   0.0  8.0
ostrich  2.0  NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.mul.html.md

# dask.dataframe.DataFrame.mul

#### DataFrame.mul(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.ndim.html.md

# dask.dataframe.DataFrame.ndim

#### *property* DataFrame.ndim

Return dimensionality

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.ne.html.md

# dask.dataframe.DataFrame.ne

#### DataFrame.ne(other, level=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.nlargest.html.md

# dask.dataframe.DataFrame.nlargest

#### DataFrame.nlargest(n=5, columns=None, split_every=None)

Return the first n rows ordered by columns in descending order.

This docstring was copied from pandas.DataFrame.nlargest.

Some inconsistencies with the Dask version may exist.

Return the first n rows with the largest values in columns, in
descending order. The columns that are not specified are returned as
well, but not used for ordering.

This method is equivalent to
`df.sort_values(columns, ascending=False).head(n)`, but more
performant.

* **Parameters:**
  **n**
  : Number of rows to return.

  **columns**
  : Column label(s) to order by.

  **keep**
  : Where there are duplicate values:
    - `first` : prioritize the first occurrence(s)
    - `last` : prioritize the last occurrence(s)
    - `all` : keep all the ties of the smallest item even if it means
      selecting more than `n` items.
* **Returns:**
  DataFrame
  : The first n rows ordered by the given columns in descending
    order.

#### SEE ALSO
[`DataFrame.nsmallest`](dask.dataframe.DataFrame.nsmallest.md#dask.dataframe.DataFrame.nsmallest)
: Return the first n rows ordered by columns in ascending order.

[`DataFrame.sort_values`](dask.dataframe.DataFrame.sort_values.md#dask.dataframe.DataFrame.sort_values)
: Sort DataFrame by the values.

[`DataFrame.head`](dask.dataframe.DataFrame.head.md#dask.dataframe.DataFrame.head)
: Return the first n rows without re-ordering.

### Notes

This function cannot be used with all column types. For example, when
specifying columns with object or category dtypes, `TypeError` is
raised.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "population": [
...             59000000,
...             65000000,
...             434000,
...             434000,
...             434000,
...             337000,
...             11300,
...             11300,
...             11300,
...         ],
...         "GDP": [1937894, 2583560, 12011, 4520, 12128, 17036, 182, 38, 311],
...         "alpha-2": ["IT", "FR", "MT", "MV", "BN", "IS", "NR", "TV", "AI"],
...     },
...     index=[
...         "Italy",
...         "France",
...         "Malta",
...         "Maldives",
...         "Brunei",
...         "Iceland",
...         "Nauru",
...         "Tuvalu",
...         "Anguilla",
...     ],
... )
>>> df
          population      GDP alpha-2
Italy       59000000  1937894      IT
France      65000000  2583560      FR
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
Iceland       337000    17036      IS
Nauru          11300      182      NR
Tuvalu         11300       38      TV
Anguilla       11300      311      AI
```

In the following example, we will use `nlargest` to select the three
rows having the largest values in column “population”.

```pycon
>>> df.nlargest(3, "population")
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Malta       434000    12011      MT
```

When using `keep='last'`, ties are resolved in reverse order:

```pycon
>>> df.nlargest(3, "population", keep="last")
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Brunei      434000    12128      BN
```

When using `keep='all'`, the number of element kept can go beyond `n`
if there are duplicate values for the smallest element, all the
ties are kept:

```pycon
>>> df.nlargest(3, "population", keep="all")
          population      GDP alpha-2
France      65000000  2583560      FR
Italy       59000000  1937894      IT
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
```

However, `nlargest` does not keep `n` distinct largest elements:

```pycon
>>> df.nlargest(5, "population", keep="all")
          population      GDP alpha-2
France      65000000  2583560      FR
Italy       59000000  1937894      IT
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
```

To order by the largest values in column “population” and then “GDP”,
we can specify multiple columns like in the next example.

```pycon
>>> df.nlargest(3, ["population", "GDP"])
        population      GDP alpha-2
France    65000000  2583560      FR
Italy     59000000  1937894      IT
Brunei      434000    12128      BN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.npartitions.html.md

# dask.dataframe.DataFrame.npartitions

#### *property* DataFrame.npartitions

Return number of partitions

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.nsmallest.html.md

# dask.dataframe.DataFrame.nsmallest

#### DataFrame.nsmallest(n=5, columns=None, split_every=None)

Return the first n rows ordered by columns in ascending order.

This docstring was copied from pandas.DataFrame.nsmallest.

Some inconsistencies with the Dask version may exist.

Return the first n rows with the smallest values in columns, in
ascending order. The columns that are not specified are returned as
well, but not used for ordering.

This method is equivalent to
`df.sort_values(columns, ascending=True).head(n)`, but more
performant.

* **Parameters:**
  **n**
  : Number of items to retrieve.

  **columns**
  : Column name or names to order by.

  **keep**
  : Where there are duplicate values:
    - `first` : take the first occurrence.
    - `last` : take the last occurrence.
    - `all` : keep all the ties of the largest item even if it means
      selecting more than `n` items.
* **Returns:**
  DataFrame
  : DataFrame with the first n rows ordered by columns in ascending order.

#### SEE ALSO
[`DataFrame.nlargest`](dask.dataframe.DataFrame.nlargest.md#dask.dataframe.DataFrame.nlargest)
: Return the first n rows ordered by columns in descending order.

[`DataFrame.sort_values`](dask.dataframe.DataFrame.sort_values.md#dask.dataframe.DataFrame.sort_values)
: Sort DataFrame by the values.

[`DataFrame.head`](dask.dataframe.DataFrame.head.md#dask.dataframe.DataFrame.head)
: Return the first n rows without re-ordering.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "population": [
...             59000000,
...             65000000,
...             434000,
...             434000,
...             434000,
...             337000,
...             337000,
...             11300,
...             11300,
...         ],
...         "GDP": [1937894, 2583560, 12011, 4520, 12128, 17036, 182, 38, 311],
...         "alpha-2": ["IT", "FR", "MT", "MV", "BN", "IS", "NR", "TV", "AI"],
...     },
...     index=[
...         "Italy",
...         "France",
...         "Malta",
...         "Maldives",
...         "Brunei",
...         "Iceland",
...         "Nauru",
...         "Tuvalu",
...         "Anguilla",
...     ],
... )
>>> df
          population      GDP alpha-2
Italy       59000000  1937894      IT
France      65000000  2583560      FR
Malta         434000    12011      MT
Maldives      434000     4520      MV
Brunei        434000    12128      BN
Iceland       337000    17036      IS
Nauru         337000      182      NR
Tuvalu         11300       38      TV
Anguilla       11300      311      AI
```

In the following example, we will use `nsmallest` to select the
three rows having the smallest values in column “population”.

```pycon
>>> df.nsmallest(3, "population")
          population    GDP alpha-2
Tuvalu         11300     38      TV
Anguilla       11300    311      AI
Iceland       337000  17036      IS
```

When using `keep='last'`, ties are resolved in reverse order:

```pycon
>>> df.nsmallest(3, "population", keep="last")
          population  GDP alpha-2
Anguilla       11300  311      AI
Tuvalu         11300   38      TV
Nauru         337000  182      NR
```

When using `keep='all'`, the number of element kept can go beyond `n`
if there are duplicate values for the largest element, all the
ties are kept.

```pycon
>>> df.nsmallest(3, "population", keep="all")
          population    GDP alpha-2
Tuvalu         11300     38      TV
Anguilla       11300    311      AI
Iceland       337000  17036      IS
Nauru         337000    182      NR
```

However, `nsmallest` does not keep `n` distinct
smallest elements:

```pycon
>>> df.nsmallest(4, "population", keep="all")
          population    GDP alpha-2
Tuvalu         11300     38      TV
Anguilla       11300    311      AI
Iceland       337000  17036      IS
Nauru         337000    182      NR
```

To order by the smallest values in column “population” and then “GDP”, we can
specify multiple columns like in the next example.

```pycon
>>> df.nsmallest(3, ["population", "GDP"])
          population  GDP alpha-2
Tuvalu         11300   38      TV
Anguilla       11300  311      AI
Nauru         337000  182      NR
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.partitions.html.md

# dask.dataframe.DataFrame.partitions

#### *property* DataFrame.partitions

Slice dataframe by partitions

This allows partitionwise slicing of a Dask Dataframe.  You can perform normal
Numpy-style slicing, but now rather than slice elements of the array you
slice along partitions so, for example, `df.partitions[:5]` produces a new
Dask Dataframe of the first five partitions. Valid indexers are integers, sequences
of integers, slices, or boolean masks.

* **Returns:**
  A Dask DataFrame

### Examples

```pycon
>>> df.partitions[0]
>>> df.partitions[:3]
>>> df.partitions[::10]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.persist.html.md

# dask.dataframe.DataFrame.persist

#### DataFrame.persist(fuse=True, \*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.pivot_table.html.md

# dask.dataframe.DataFrame.pivot_table

#### DataFrame.pivot_table(index, columns, values, aggfunc='mean')

Create a spreadsheet-style pivot table as a DataFrame. Target `columns`
must have category dtype to infer result’s `columns`.
`index`, `columns`, `values` and `aggfunc` must be all scalar.

* **Parameters:**
  **values**
  : column to aggregate

  **index**
  : column to be index

  **columns**
  : column to be columns

  **aggfunc**
* **Returns:**
  **table**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.pop.html.md

# dask.dataframe.DataFrame.pop

#### DataFrame.pop(item)

Return item and drop it from DataFrame. Raise KeyError if not found.

This docstring was copied from pandas.DataFrame.pop.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **item**
  : Label of column to be popped.
* **Returns:**
  Series
  : Series representing the item that is dropped.

#### SEE ALSO
[`DataFrame.drop`](dask.dataframe.DataFrame.drop.md#dask.dataframe.DataFrame.drop)
: Drop specified labels from rows or columns.

[`DataFrame.drop_duplicates`](dask.dataframe.DataFrame.drop_duplicates.md#dask.dataframe.DataFrame.drop_duplicates)
: Return DataFrame with duplicate rows removed.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         ("falcon", "bird", 389.0),
...         ("parrot", "bird", 24.0),
...         ("lion", "mammal", 80.5),
...         ("monkey", "mammal", np.nan),
...     ],
...     columns=("name", "class", "max_speed"),
... )
>>> df
     name   class  max_speed
0  falcon    bird      389.0
1  parrot    bird       24.0
2    lion  mammal       80.5
3  monkey  mammal        NaN
```

```pycon
>>> df.pop("class")
0      bird
1      bird
2    mammal
3    mammal
Name: class, dtype: str
```

```pycon
>>> df
     name  max_speed
0  falcon      389.0
1  parrot       24.0
2    lion       80.5
3  monkey        NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.pow.html.md

# dask.dataframe.DataFrame.pow

#### DataFrame.pow(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.prod.html.md

# dask.dataframe.DataFrame.prod

#### DataFrame.prod(axis=0, skipna=True, numeric_only=False, min_count=0, split_every=False, \*\*kwargs)

Return the product of the values over the requested axis.

This docstring was copied from pandas.DataFrame.prod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.prod with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **min_count**
  : The required number of valid values to perform the operation. If fewer than
    `min_count` non-NA values are present the result will be NA.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : The product of the values over the requested axis.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

By default, the product of an empty or all-NA Series is `1`

```pycon
>>> pd.Series([], dtype="float64").prod()
1.0
```

This can be controlled with the `min_count` parameter

```pycon
>>> pd.Series([], dtype="float64").prod(min_count=1)
nan
```

Thanks to the `skipna` parameter, `min_count` handles all-NA and
empty series identically.

```pycon
>>> pd.Series([np.nan]).prod()
1.0
```

```pycon
>>> pd.Series([np.nan]).prod(min_count=1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.quantile.html.md

# dask.dataframe.DataFrame.quantile

#### DataFrame.quantile(q=0.5, axis=0, numeric_only=False, method='default')

Approximate row-wise and precise column-wise quantiles of DataFrame

* **Parameters:**
  **q**
  : Iterable of numbers ranging from 0 to 1 for the desired quantiles

  **axis**
  : 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise

  **method**
  : What method to use. By default will use dask’s internal custom
    algorithm (`'dask'`).  If set to `'tdigest'` will use tdigest
    for floats and ints and fallback to the `'dask'` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.query.html.md

# dask.dataframe.DataFrame.query

#### DataFrame.query(expr, \*\*kwargs)

Filter dataframe with complex expression

Blocked version of pd.DataFrame.query

* **Parameters:**
  **expr: str**
  : The query string to evaluate.
    You can refer to column names that are not valid Python variable names
    by surrounding them in backticks.
    Dask does not fully support referring to variables using the ‘@’ character,
    use f-strings or the `local_dict` keyword argument instead.

#### SEE ALSO
[`pandas.DataFrame.query`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html#pandas.DataFrame.query)

[`pandas.eval`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.eval.html#pandas.eval)

### Examples

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 1, 2],
...                    'y': [1, 2, 3, 4],
...                    'z z': [4, 3, 2, 1]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

Refer to column names directly:

```pycon
>>> ddf.query('y > x').compute()
   x  y  z z
2  1  3    2
3  2  4    1
```

Refer to column name using backticks:

```pycon
>>> ddf.query('`z z` > x').compute()
   x  y  z z
0  1  1    4
1  2  2    3
2  1  3    2
```

Refer to variable name using f-strings:

```pycon
>>> value = 1
>>> ddf.query(f'x == {value}').compute()
   x  y  z z
0  1  1    4
2  1  3    2
```

Refer to variable name using `local_dict`:

```pycon
>>> ddf.query('x == @value', local_dict={"value": value}).compute()
   x  y  z z
0  1  1    4
2  1  3    2
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.radd.html.md

# dask.dataframe.DataFrame.radd

#### DataFrame.radd(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.random_split.html.md

# dask.dataframe.DataFrame.random_split

#### DataFrame.random_split(frac, random_state=None, shuffle=False)

Pseudorandomly split dataframe into different pieces row-wise

* **Parameters:**
  **frac**
  : List of floats that should sum to one.

  **random_state**
  : If int or None create a new RandomState with this as the seed.
    Otherwise draw from the passed RandomState.

  **shuffle**
  : If set to True, the dataframe is shuffled (within partition)
    before the split.

#### SEE ALSO
`dask.DataFrame.sample`

### Examples

50/50 split

```pycon
>>> a, b = df.random_split([0.5, 0.5])
```

80/10/10 split, consistent random_state

```pycon
>>> a, b, c = df.random_split([0.8, 0.1, 0.1], random_state=123)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rdiv.html.md

# dask.dataframe.DataFrame.rdiv

#### DataFrame.rdiv(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rename.html.md

# dask.dataframe.DataFrame.rename

#### DataFrame.rename(index=None, columns=None)

Rename columns or index labels.

This docstring was copied from pandas.DataFrame.rename.

Some inconsistencies with the Dask version may exist.

Function / dict values must be unique (1-to-1). Labels not contained in
a dict / Series will be left as-is. Extra labels listed don’t throw an
error.

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#basics-rename) for more.

* **Parameters:**
  **mapper**
  : Dict-like or function transformations to apply to
    that axis’ values. Use either `mapper` and `axis` to
    specify the axis to target with `mapper`, or `index` and
    `columns`.

  **index**
  : Alternative to specifying axis (`mapper, axis=0`
    is equivalent to `index=mapper`).

  **columns**
  : Alternative to specifying axis (`mapper, axis=1`
    is equivalent to `columns=mapper`).

  **axis**
  : Axis to target with `mapper`. Can be either the axis name
    (‘index’, ‘columns’) or number (0, 1). The default is ‘index’.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **inplace**
  : Whether to modify the DataFrame rather than creating a new one.
    If True then value of copy is ignored.

  **level**
  : In case of a MultiIndex, only rename labels in the specified
    level.

  **errors**
  : If ‘raise’, raise a KeyError when a dict-like mapper, index,
    or columns contains labels that are not present in the Index
    being transformed.
    If ‘ignore’, existing keys will be renamed and extra keys will be
    ignored.
* **Returns:**
  DataFrame or None
  : DataFrame with the renamed axis labels or None if `inplace=True`.
* **Raises:**
  KeyError
  : If any of the labels is not found in the selected axis and
    “errors=’raise’”.

#### SEE ALSO
[`DataFrame.rename_axis`](dask.dataframe.DataFrame.rename_axis.md#dask.dataframe.DataFrame.rename_axis)
: Set the name of the axis.

### Examples

`DataFrame.rename` supports two calling conventions

* `(index=index_mapper, columns=columns_mapper, ...)`
* `(mapper, axis={'index', 'columns'}, ...)`

We *highly* recommend using keyword arguments to clarify your
intent.

Rename columns using a mapping:

```pycon
>>> df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
>>> df.rename(columns={"A": "a", "B": "c"})
   a  c
0  1  4
1  2  5
2  3  6
```

Rename index using a mapping:

```pycon
>>> df.rename(index={0: "x", 1: "y", 2: "z"})
   A  B
x  1  4
y  2  5
z  3  6
```

Cast index labels to a different type:

```pycon
>>> df.index
RangeIndex(start=0, stop=3, step=1)
>>> df.rename(index=str).index
Index(['0', '1', '2'], dtype='str')
```

```pycon
>>> df.rename(columns={"A": "a", "B": "b", "C": "c"}, errors="raise")
Traceback (most recent call last):
KeyError: ['C'] not found in axis
```

Using axis-style parameters:

```pycon
>>> df.rename(str.lower, axis="columns")
   a  b
0  1  4
1  2  5
2  3  6
```

```pycon
>>> df.rename({1: 2, 2: 4}, axis="index")
   A  B
0  1  4
2  2  5
4  3  6
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rename_axis.html.md

# dask.dataframe.DataFrame.rename_axis

#### DataFrame.rename_axis(mapper=<no_default>, index=<no_default>, columns=<no_default>, axis=0)

Set the name of the axis for the index or columns.

This docstring was copied from pandas.DataFrame.rename_axis.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **mapper**
  : Value to set the axis name attribute.
    <br/>
    Use either `mapper` and `axis` to
    specify the axis to target with `mapper`, or `index`
    and/or `columns`.

  **index**
  : A scalar, list-like, dict-like or functions transformations to
    apply to that axis’ values.

  **columns**
  : A scalar, list-like, dict-like or functions transformations to
    apply to that axis’ values.

  **axis**
  : The axis to rename.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **inplace**
  : Modifies the object directly, instead of creating a new Series
    or DataFrame.
* **Returns:**
  DataFrame, or None
  : The same type as the caller or None if `inplace=True`.

#### SEE ALSO
[`Series.rename`](dask.dataframe.Series.rename.md#dask.dataframe.Series.rename)
: Alter Series index labels or name.

[`DataFrame.rename`](dask.dataframe.DataFrame.rename.md#dask.dataframe.DataFrame.rename)
: Alter DataFrame index labels or name.

[`Index.rename`](dask.dataframe.Index.rename.md#dask.dataframe.Index.rename)
: Set new names on index.

### Notes

`DataFrame.rename_axis` supports two calling conventions

* `(index=index_mapper, columns=columns_mapper, ...)`
* `(mapper, axis={'index', 'columns'}, ...)`

The first calling convention will only modify the names of
the index and/or the names of the Index object that is the columns.
In this case, the parameter `copy` is ignored.

The second calling convention will modify the names of the
corresponding index if mapper is a list or a scalar.
However, if mapper is dict-like or a function, it will use the
deprecated behavior of modifying the axis *labels*.

We *highly* recommend using keyword arguments to clarify your
intent.

### Examples

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     {"num_legs": [4, 4, 2], "num_arms": [0, 0, 2]}, ["dog", "cat", "monkey"]
... )
>>> df
        num_legs  num_arms
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("animal")
>>> df
        num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2
>>> df = df.rename_axis("limbs", axis="columns")
>>> df
limbs   num_legs  num_arms
animal
dog            4         0
cat            4         0
monkey         2         2
```

**MultiIndex**

```pycon
>>> df.index = pd.MultiIndex.from_product(
...     [["mammal"], ["dog", "cat", "monkey"]], names=["type", "name"]
... )
>>> df
limbs          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2
```

```pycon
>>> df.rename_axis(index={"type": "class"})
limbs          num_legs  num_arms
class  name
mammal dog            4         0
       cat            4         0
       monkey         2         2
```

```pycon
>>> df.rename_axis(columns=str.upper)
LIMBS          num_legs  num_arms
type   name
mammal dog            4         0
       cat            4         0
       monkey         2         2
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.repartition.html.md

# dask.dataframe.DataFrame.repartition

#### DataFrame.repartition(divisions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) | [None](https://docs.python.org/3/library/constants.html#None) = None, npartitions: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, partition_size: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, freq=None, force: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Repartition a collection

Exactly one of divisions, npartitions or partition_size should be
specified. A `ValueError` will be raised when that is not the case.

* **Parameters:**
  **divisions**
  : The “dividing lines” used to split the dataframe into partitions.
    For `divisions=[0, 10, 50, 100]`, there would be three output partitions,
    where the new index contained [0, 10), [10, 50), and [50, 100), respectively.
    See [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions).

  **npartitions**
  : Approximate number of partitions of output. The number of
    partitions used may be slightly lower than npartitions depending
    on data distribution, but will never be higher.
    The Callable gets the number of partitions of the input as an argument
    and should return an int.

  **partition_size**
  : Max number of bytes of memory for each partition. Use numbers or strings
    like 5MB. If specified npartitions and divisions will be ignored. Note that
    the size reflects the number of bytes used as computed by
    pandas.DataFrame.memory_usage, which will not necessarily match the size
    when storing to disk.
    <br/>
    #### WARNING
    This keyword argument triggers computation to determine
    the memory size of each partition, which may be expensive.

  **force**
  : Allows the expansion of the existing divisions.
    If False then the new divisions’ lower and upper bounds must be
    the same as the old divisions’.

  **freq**
  : A period on which to partition timeseries data like `'7D'` or
    `'12h'` or `pd.Timedelta(hours=12)`.  Assumes a datetime index.

#### SEE ALSO
[`DataFrame.memory_usage_per_partition`](dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition)

[`pandas.DataFrame.memory_usage`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.memory_usage.html#pandas.DataFrame.memory_usage)

### Notes

Exactly one of divisions, npartitions, partition_size, or freq
should be specified. A `ValueError` will be raised when that is
not the case.

Also note that `len(divisions)` is equal to `npartitions + 1`. This is because `divisions`
represents the upper and lower bounds of each partition. The first item is the
lower bound of the first partition, the second item is the lower bound of the
second partition and the upper bound of the first partition, and so on.
The second-to-last item is the lower bound of the last partition, and the last
(extra) item is the upper bound of the last partition.

### Examples

```pycon
>>> df = df.repartition(npartitions=10)
>>> df = df.repartition(divisions=[0, 5, 10, 20])
>>> df = df.repartition(freq='7d')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.replace.html.md

# dask.dataframe.DataFrame.replace

#### DataFrame.replace(to_replace=None, value=<no_default>, regex=False)

Replace values given in to_replace with value.

This docstring was copied from pandas.DataFrame.replace.

Some inconsistencies with the Dask version may exist.

Values of the Series/DataFrame are replaced with other values dynamically.
This differs from updating with `.loc` or `.iloc`, which require
you to specify a location to update with some value.

* **Parameters:**
  **to_replace**
  : How to find the values that will be replaced.
    * numeric, str or regex:
      > - numeric: numeric values equal to to_replace will be
      >   replaced with value
      > - str: string exactly matching to_replace will be replaced
      >   with value
      > - regex: regexes matching to_replace will be replaced with
      >   value
    * list of str, regex, or numeric:
      > - First, if to_replace and value are both lists, they
      >   **must** be the same length.
      > - Second, if `regex=True` then all of the strings in **both**
      >   lists will be interpreted as regexes otherwise they will match
      >   directly. This doesn’t matter much for value since there
      >   are only a few possible substitution regexes you can use.
      > - str, regex and numeric rules apply as above.
    * dict:
      > - Dicts can be used to specify different replacement values
      >   for different existing values. For example,
      >   `{'a': 'b', 'y': 'z'}` replaces the value ‘a’ with ‘b’ and
      >   ‘y’ with ‘z’. To use a dict in this way, the optional value
      >   parameter should not be given.
      > - For a DataFrame a dict can specify that different values
      >   should be replaced in different columns. For example,
      >   `{'a': 1, 'b': 'z'}` looks for the value 1 in column ‘a’
      >   and the value ‘z’ in column ‘b’ and replaces these values
      >   with whatever is specified in value. The value parameter
      >   should not be `None` in this case. You can treat this as a
      >   special case of passing two lists except that you are
      >   specifying the column to search in.
      > - For a DataFrame nested dictionaries, e.g.,
      >   `{'a': {'b': np.nan}}`, are read as follows: look in column
      >   ‘a’ for the value ‘b’ and replace it with NaN. The optional value
      >   parameter should not be specified to use a nested dict in this
      >   way. You can nest regular expressions as well. Note that
      >   column names (the top-level dictionary keys in a nested
      >   dictionary) **cannot** be regular expressions.
    * None:
      > - This means that the regex argument must be a string,
      >   compiled regular expression, or list, dict, ndarray or
      >   Series of such elements. If value is also `None` then
      >   this **must** be a nested dictionary or Series.
    <br/>
    See the examples section for examples of each of these.

  **value**
  : Value to replace any values matching to_replace with.
    For a DataFrame a dict of values can be used to specify which
    value to use for each column (columns not in the dict will not be
    filled). Regular expressions, strings and lists or dicts of such
    objects are also allowed.

  **inplace**
  : If True, performs operation inplace.

  **regex**
  : Whether to interpret to_replace and/or value as regular
    expressions. Alternatively, this could be a regular expression or a
    list, dict, or array of regular expressions in which case
    to_replace must be `None`.
* **Returns:**
  Series/DataFrame
  : Object after replacement.
* **Raises:**
  AssertionError
  : * If regex is not a `bool` and to_replace is not
      `None`.

  TypeError
  : * If to_replace is not a scalar, array-like, `dict`, or `None`
    * If to_replace is a `dict` and value is not a `list`,
      `dict`, `ndarray`, or `Series`
    * If to_replace is `None` and regex is not compilable
      into a regular expression or is a list, dict, ndarray, or
      Series.
    * When replacing multiple `bool` or `datetime64` objects and
      the arguments to to_replace does not match the type of the
      value being replaced

  ValueError
  : * If a `list` or an `ndarray` is passed to to_replace and
      value but they are not the same length.

#### SEE ALSO
[`Series.fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)
: Fill NA values.

[`DataFrame.fillna`](dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)
: Fill NA values.

[`Series.where`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Replace values based on boolean condition.

[`DataFrame.where`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Replace values based on boolean condition.

`DataFrame.map`
: Apply a function to a Dataframe elementwise.

[`Series.map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)
: Map values of Series according to an input mapping or function.

[`Series.str.replace`](dask.dataframe.Series.str.replace.md#dask.dataframe.Series.str.replace)
: Simple string replacement.

### Notes

* Regex substitution is performed under the hood with `re.sub`. The
  rules for substitution for `re.sub` are the same.
* Regular expressions will only substitute on strings, meaning you
  cannot provide, for example, a regular expression matching floating
  point numbers and expect the columns in your frame that have a
  numeric dtype to be matched. However, if those floating point
  numbers *are* strings, then you can do this.
* This method has *a lot* of options. You are encouraged to experiment
  and play with this method to gain intuition about how it works.
* When dict is used as the to_replace value, it is like
  key(s) in the dict are the to_replace part and
  value(s) in the dict are the value parameter.

### Examples

**Scalar \`to_replace\` and \`value\`**

```pycon
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> s.replace(1, 5)
0    5
1    2
2    3
3    4
4    5
dtype: int64
```

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": [5, 6, 7, 8, 9],
...         "C": ["a", "b", "c", "d", "e"],
...     }
... )
>>> df.replace(0, 5)
    A  B  C
0  5  5  a
1  1  6  b
2  2  7  c
3  3  8  d
4  4  9  e
```

**List-like \`to_replace\`**

```pycon
>>> df.replace([0, 1, 2, 3], 4)
    A  B  C
0  4  5  a
1  4  6  b
2  4  7  c
3  4  8  d
4  4  9  e
```

```pycon
>>> df.replace([0, 1, 2, 3], [4, 3, 2, 1])
    A  B  C
0  4  5  a
1  3  6  b
2  2  7  c
3  1  8  d
4  4  9  e
```

**dict-like \`to_replace\`**

```pycon
>>> df.replace({0: 10, 1: 100})
        A  B  C
0   10  5  a
1  100  6  b
2    2  7  c
3    3  8  d
4    4  9  e
```

```pycon
>>> df.replace({"A": 0, "B": 5}, 100)
        A    B  C
0  100  100  a
1    1    6  b
2    2    7  c
3    3    8  d
4    4    9  e
```

```pycon
>>> df.replace({"A": {0: 100, 4: 400}})
        A  B  C
0  100  5  a
1    1  6  b
2    2  7  c
3    3  8  d
4  400  9  e
```

**Regular expression \`to_replace\`**

```pycon
>>> df = pd.DataFrame({"A": ["bat", "foo", "bait"], "B": ["abc", "bar", "xyz"]})
>>> df.replace(to_replace=r"^ba.$", value="new", regex=True)
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace({"A": r"^ba.$"}, {"A": "new"}, regex=True)
        A    B
0   new  abc
1   foo  bar
2  bait  xyz
```

```pycon
>>> df.replace(regex=r"^ba.$", value="new")
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace(regex={r"^ba.$": "new", "foo": "xyz"})
        A    B
0   new  abc
1   xyz  new
2  bait  xyz
```

```pycon
>>> df.replace(regex=[r"^ba.$", "foo"], value="new")
        A    B
0   new  abc
1   new  new
2  bait  xyz
```

Compare the behavior of `s.replace({'a': None})` and
`s.replace('a', None)` to understand the peculiarities
of the to_replace parameter:

```pycon
>>> s = pd.Series([10, "a", "a", "b", "a"])
```

When one uses a dict as the to_replace value, it is like the
value(s) in the dict are equal to the value parameter.
`s.replace({'a': None})` is equivalent to
`s.replace(to_replace={'a': None}, value=None)`:

```pycon
>>> s.replace({"a": None})
0      10
1    None
2    None
3       b
4    None
dtype: object
```

If `None` is explicitly passed for `value`, it will be respected:

```pycon
>>> s.replace("a", None)
0      10
1    None
2    None
3       b
4    None
dtype: object
```

When `regex=True`, `value` is not `None` and to_replace is a string,
the replacement will be applied in all columns of the DataFrame.

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": ["a", "b", "c", "d", "e"],
...         "C": ["f", "g", "h", "i", "j"],
...     }
... )
```

```pycon
>>> df.replace(to_replace="^[a-g]", value="e", regex=True)
    A  B  C
0  0  e  e
1  1  e  e
2  2  e  h
3  3  e  i
4  4  e  j
```

If `value` is not `None` and to_replace is a dictionary, the dictionary
keys will be the DataFrame columns that the replacement will be applied.

```pycon
>>> df.replace(to_replace={"B": "^[a-c]", "C": "^[h-j]"}, value="e", regex=True)
    A  B  C
0  0  e  f
1  1  e  g
2  2  e  e
3  3  d  e
4  4  e  e
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.resample.html.md

# dask.dataframe.DataFrame.resample

#### DataFrame.resample(rule, closed=None, label=None)

Resample time-series data.

This docstring was copied from pandas.DataFrame.resample.

Some inconsistencies with the Dask version may exist.

Convenience method for frequency conversion and resampling of time series.
The object must have a datetime-like index (DatetimeIndex, PeriodIndex,
or TimedeltaIndex), or the caller must pass the label of a datetime-like
series/index to the `on`/`level` keyword parameter.

* **Parameters:**
  **rule**
  : The offset string or object representing target conversion.

  **closed**
  : Which side of bin interval is closed. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **label**
  : Which bin edge label to label bucket with. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **convention**
  : For PeriodIndex only, controls whether to use the start or
    end of rule.

  **on**
  : For a DataFrame, column to use instead of index for resampling.
    Column must be datetime-like.

  **level**
  : For a MultiIndex, level (name or number) to use for
    resampling. level must be datetime-like.

  **origin**
  : The timestamp on which to adjust the grouping. The timezone of origin
    must match the timezone of the index.
    If string, must be Timestamp convertible or one of the following:
    - ‘epoch’: origin is 1970-01-01
    - ‘start’: origin is the first value of the timeseries
    - ‘start_day’: origin is the first day at midnight of the timeseries
    - ‘end’: origin is the last value of the timeseries
    - ‘end_day’: origin is the ceiling midnight of the last day
    <br/>
    #### NOTE
    Only takes effect for Tick-frequencies (i.e. fixed frequencies like
    days, hours, and minutes, rather than months or quarters).

  **offset**
  : An offset timedelta added to the origin.

  **group_keys**
  : Whether to include the group keys in the result index when using
    `.apply()` on the resampled object.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `False`.
* **Returns:**
  pandas.api.typing.Resampler
  : `Resampler` object.

#### SEE ALSO
[`Series.resample`](dask.dataframe.Series.resample.md#dask.dataframe.Series.resample)
: Resample a Series.

[`DataFrame.resample`](#dask.dataframe.DataFrame.resample)
: Resample a DataFrame.

[`groupby`](dask.dataframe.DataFrame.groupby.md#dask.dataframe.DataFrame.groupby)
: Group Series/DataFrame by mapping, function, label, or list of labels.

`asfreq`
: Reindex a Series/DataFrame with the given frequency without grouping.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#resampling)
for more.

To learn more about the offset strings, please see [this link](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects).

### Examples

Start by creating a series with 9 one minute timestamps.

```pycon
>>> index = pd.date_range("1/1/2000", periods=9, freq="min")
>>> series = pd.Series(range(9), index=index)
>>> series
2000-01-01 00:00:00    0
2000-01-01 00:01:00    1
2000-01-01 00:02:00    2
2000-01-01 00:03:00    3
2000-01-01 00:04:00    4
2000-01-01 00:05:00    5
2000-01-01 00:06:00    6
2000-01-01 00:07:00    7
2000-01-01 00:08:00    8
Freq: min, dtype: int64
```

Downsample the series into 3 minute bins and sum the values
of the timestamps falling into a bin.

```pycon
>>> series.resample("3min").sum()
2000-01-01 00:00:00     3
2000-01-01 00:03:00    12
2000-01-01 00:06:00    21
Freq: 3min, dtype: int64
```

Downsample the series into 3 minute bins as above, but label each
bin using the right edge instead of the left. Please note that the
value in the bucket used as the label is not included in the bucket,
which it labels. For example, in the original series the
bucket `2000-01-01 00:03:00` contains the value 3, but the summed
value in the resampled bucket with the label `2000-01-01 00:03:00`
does not include 3 (if it did, the summed value would be 6, not 3).

```pycon
>>> series.resample("3min", label="right").sum()
2000-01-01 00:03:00     3
2000-01-01 00:06:00    12
2000-01-01 00:09:00    21
Freq: 3min, dtype: int64
```

To include this value close the right side of the bin interval,
as shown below.

```pycon
>>> series.resample("3min", label="right", closed="right").sum()
2000-01-01 00:00:00     0
2000-01-01 00:03:00     6
2000-01-01 00:06:00    15
2000-01-01 00:09:00    15
Freq: 3min, dtype: int64
```

Upsample the series into 30 second bins.

```pycon
>>> series.resample("30s").asfreq()[0:5]  # Select first 5 rows
2000-01-01 00:00:00   0.0
2000-01-01 00:00:30   NaN
2000-01-01 00:01:00   1.0
2000-01-01 00:01:30   NaN
2000-01-01 00:02:00   2.0
Freq: 30s, dtype: float64
```

Upsample the series into 30 second bins and fill the `NaN`
values using the `ffill` method.

```pycon
>>> series.resample("30s").ffill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    0
2000-01-01 00:01:00    1
2000-01-01 00:01:30    1
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Upsample the series into 30 second bins and fill the
`NaN` values using the `bfill` method.

```pycon
>>> series.resample("30s").bfill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    1
2000-01-01 00:01:00    1
2000-01-01 00:01:30    2
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Pass a custom function via `apply`

```pycon
>>> def custom_resampler(arraylike):
...     return np.sum(arraylike) + 5
>>> series.resample("3min").apply(custom_resampler)
2000-01-01 00:00:00     8
2000-01-01 00:03:00    17
2000-01-01 00:06:00    26
Freq: 3min, dtype: int64
```

For a Series with a PeriodIndex, the keyword convention can be
used to control whether to use the start or end of rule.

Resample a year by quarter using ‘start’ convention. Values are
assigned to the first quarter of the period.

```pycon
>>> s = pd.Series(
...     [1, 2], index=pd.period_range("2012-01-01", freq="Y", periods=2)
... )
>>> s
2012    1
2013    2
Freq: Y-DEC, dtype: int64
>>> s.resample("Q", convention="start").asfreq()
2012Q1    1.0
2012Q2    NaN
2012Q3    NaN
2012Q4    NaN
2013Q1    2.0
2013Q2    NaN
2013Q3    NaN
2013Q4    NaN
Freq: Q-DEC, dtype: float64
```

Resample quarters by month using ‘end’ convention. Values are
assigned to the last month of the period.

```pycon
>>> q = pd.Series(
...     [1, 2, 3, 4], index=pd.period_range("2018-01-01", freq="Q", periods=4)
... )
>>> q
2018Q1    1
2018Q2    2
2018Q3    3
2018Q4    4
Freq: Q-DEC, dtype: int64
>>> q.resample("M", convention="end").asfreq()
2018-03    1.0
2018-04    NaN
2018-05    NaN
2018-06    2.0
2018-07    NaN
2018-08    NaN
2018-09    3.0
2018-10    NaN
2018-11    NaN
2018-12    4.0
Freq: M, dtype: float64
```

For DataFrame objects, the keyword on can be used to specify the
column instead of the index for resampling.

```pycon
>>> df = pd.DataFrame([10, 11, 9, 13, 14, 18, 17, 19], columns=["price"])
>>> df["volume"] = [50, 60, 40, 100, 50, 100, 40, 50]
>>> df["week_starting"] = pd.date_range("01/01/2018", periods=8, freq="W")
>>> df
   price  volume week_starting
0     10      50    2018-01-07
1     11      60    2018-01-14
2      9      40    2018-01-21
3     13     100    2018-01-28
4     14      50    2018-02-04
5     18     100    2018-02-11
6     17      40    2018-02-18
7     19      50    2018-02-25
>>> df.resample("ME", on="week_starting").mean()
               price  volume
week_starting
2018-01-31     10.75    62.5
2018-02-28     17.00    60.0
```

For a DataFrame with MultiIndex, the keyword level can be used to
specify on which level the resampling needs to take place.

```pycon
>>> days = pd.date_range("1/1/2000", periods=4, freq="D")
>>> df2 = pd.DataFrame(
...     [
...         [10, 50],
...         [11, 60],
...         [9, 40],
...         [13, 100],
...         [14, 50],
...         [18, 100],
...         [17, 40],
...         [19, 50],
...     ],
...     columns=["price", "volume"],
...     index=pd.MultiIndex.from_product([days, ["morning", "afternoon"]]),
... )
>>> df2
                      price  volume
2000-01-01 morning       10      50
           afternoon     11      60
2000-01-02 morning        9      40
           afternoon     13     100
2000-01-03 morning       14      50
           afternoon     18     100
2000-01-04 morning       17      40
           afternoon     19      50
>>> df2.resample("D", level=0).sum()
            price  volume
2000-01-01     21     110
2000-01-02     22     140
2000-01-03     32     150
2000-01-04     36      90
```

If you want to adjust the start of the bins based on a fixed timestamp:

```pycon
>>> start, end = "2000-10-01 23:30:00", "2000-10-02 00:30:00"
>>> rng = pd.date_range(start, end, freq="7min")
>>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng)
>>> ts
2000-10-01 23:30:00     0
2000-10-01 23:37:00     3
2000-10-01 23:44:00     6
2000-10-01 23:51:00     9
2000-10-01 23:58:00    12
2000-10-02 00:05:00    15
2000-10-02 00:12:00    18
2000-10-02 00:19:00    21
2000-10-02 00:26:00    24
Freq: 7min, dtype: int64
```

```pycon
>>> ts.resample("17min").sum()
2000-10-01 23:14:00     0
2000-10-01 23:31:00     9
2000-10-01 23:48:00    21
2000-10-02 00:05:00    54
2000-10-02 00:22:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="epoch").sum()
2000-10-01 23:18:00     0
2000-10-01 23:35:00    18
2000-10-01 23:52:00    27
2000-10-02 00:09:00    39
2000-10-02 00:26:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="2000-01-01").sum()
2000-10-01 23:24:00     3
2000-10-01 23:41:00    15
2000-10-01 23:58:00    45
2000-10-02 00:15:00    45
Freq: 17min, dtype: int64
```

If you want to adjust the start of the bins with an offset Timedelta, the two
following lines are equivalent:

```pycon
>>> ts.resample("17min", origin="start").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", offset="23h30min").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

If you want to take the largest Timestamp as the end of the bins:

```pycon
>>> ts.resample("17min", origin="end").sum()
2000-10-01 23:35:00     0
2000-10-01 23:52:00    18
2000-10-02 00:09:00    27
2000-10-02 00:26:00    63
Freq: 17min, dtype: int64
```

In contrast with the start_day, you can use end_day to take the ceiling
midnight of the largest Timestamp as the end of the bins and drop the bins
not containing data:

```pycon
>>> ts.resample("17min", origin="end_day").sum()
2000-10-01 23:38:00     3
2000-10-01 23:55:00    15
2000-10-02 00:12:00    45
2000-10-02 00:29:00    45
Freq: 17min, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.reset_index.html.md

# dask.dataframe.DataFrame.reset_index

#### DataFrame.reset_index(drop: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Reset the index to the default index.

Note that unlike in `pandas`, the reset index for a Dask DataFrame will
not be monotonically increasing from 0. Instead, it will restart at 0
for each partition (e.g. `index1 = [0, ..., 10], index2 = [0, ...]`).
This is due to the inability to statically know the full length of the
index.

For DataFrame with multi-level index, returns a new DataFrame with
labeling information in the columns under the index names, defaulting
to ‘level_0’, ‘level_1’, etc. if any are None. For a standard index,
the index name will be used (if set), otherwise a default ‘index’ or
‘level_0’ (if ‘index’ is already taken) will be used.

* **Parameters:**
  **drop**
  : Do not try to insert index into dataframe columns.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rfloordiv.html.md

# dask.dataframe.DataFrame.rfloordiv

#### DataFrame.rfloordiv(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rmod.html.md

# dask.dataframe.DataFrame.rmod

#### DataFrame.rmod(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rmul.html.md

# dask.dataframe.DataFrame.rmul

#### DataFrame.rmul(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rolling.html.md

# dask.dataframe.DataFrame.rolling

#### DataFrame.rolling(window, \*\*kwargs)

Provides rolling transformations.

* **Parameters:**
  **window**
  : Size of the moving window. This is the number of observations used
    for calculating the statistic. When not using a `DatetimeIndex`,
    the window size must not be so large as to span more than one
    adjacent partition. If using an offset or offset alias like ‘5D’,
    the data must have a `DatetimeIndex`

  **min_periods**
  : Minimum number of observations in window required to have a value
    (otherwise result is NA).

  **center**
  : Set the labels at the center of the window.

  **win_type**
  : Provide a window type. The recognized window types are identical
    to pandas.

  **axis**
  : This parameter is deprecated with `pandas>=2.1`.
* **Returns:**
  a Rolling object on which to call a method to compute a statistic

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.round.html.md

# dask.dataframe.DataFrame.round

#### DataFrame.round(decimals=0)

Round numeric columns in a DataFrame to a variable number of decimal places.

This docstring was copied from pandas.DataFrame.round.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **decimals**
  : Number of decimal places to round each column to. If an int is
    given, round each column to the same number of places.
    Otherwise dict and Series round to variable numbers of places.
    Column names should be in the keys if decimals is a
    dict-like, or in the index if decimals is a Series. Any
    columns not included in decimals will be left as is. Elements
    of decimals which are not columns of the input will be
    ignored.

  **\*args**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.
* **Returns:**
  DataFrame
  : A DataFrame with the affected columns rounded to the specified
    number of decimal places.

#### SEE ALSO
[`numpy.around`](https://numpy.org/doc/stable/reference/generated/numpy.around.html#numpy.around)
: Round a numpy array to the given number of decimals.

[`Series.round`](dask.dataframe.Series.round.md#dask.dataframe.Series.round)
: Round a Series to the given number of decimals.

### Notes

For values exactly halfway between rounded decimal values, pandas rounds
to the nearest even value (e.g. -0.5 and 0.5 round to 0.0, 1.5 and 2.5
round to 2.0, etc.).

### Examples

```pycon
>>> df = pd.DataFrame(
...     [(0.21, 0.32), (0.01, 0.67), (0.66, 0.03), (0.21, 0.18)],
...     columns=["dogs", "cats"],
... )
>>> df
    dogs  cats
0  0.21  0.32
1  0.01  0.67
2  0.66  0.03
3  0.21  0.18
```

By providing an integer each column is rounded to the same number
of decimal places

```pycon
>>> df.round(1)
    dogs  cats
0   0.2   0.3
1   0.0   0.7
2   0.7   0.0
3   0.2   0.2
```

With a dict, the number of places for specific columns can be
specified with the column names as key and the number of decimal
places as value

```pycon
>>> df.round({"dogs": 1, "cats": 0})
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

Using a Series, the number of places for specific columns can be
specified with the column names as index and the number of
decimal places as value

```pycon
>>> decimals = pd.Series([0, 1], index=["cats", "dogs"])
>>> df.round(decimals)
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rpow.html.md

# dask.dataframe.DataFrame.rpow

#### DataFrame.rpow(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rsub.html.md

# dask.dataframe.DataFrame.rsub

#### DataFrame.rsub(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.rtruediv.html.md

# dask.dataframe.DataFrame.rtruediv

#### DataFrame.rtruediv(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.sample.html.md

# dask.dataframe.DataFrame.sample

#### DataFrame.sample(n=None, frac=None, replace=False, random_state=None)

Random sample of items

* **Parameters:**
  **n**
  : Number of items to return is not supported by dask. Use frac
    instead.

  **frac**
  : Approximate fraction of items to return. This sampling fraction is
    applied to all partitions equally. Note that this is an
    **approximate fraction**. You should not expect exactly `len(df) * frac`
    items to be returned, as the exact number of elements selected will
    depend on how your data is partitioned (but should be pretty close
    in practice).

  **replace**
  : Sample with or without replacement. Default = False.

  **random_state**
  : If an int, we create a new RandomState with this as the seed;
    Otherwise we draw from the passed RandomState.

#### SEE ALSO
[`DataFrame.random_split`](dask.dataframe.DataFrame.random_split.md#dask.dataframe.DataFrame.random_split)

[`pandas.DataFrame.sample`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html#pandas.DataFrame.sample)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.select_dtypes.html.md

# dask.dataframe.DataFrame.select_dtypes

#### DataFrame.select_dtypes(include=None, exclude=None)

Return a subset of the DataFrame’s columns based on the column dtypes.

This docstring was copied from pandas.DataFrame.select_dtypes.

Some inconsistencies with the Dask version may exist.

This method allows for filtering columns based on their data types.
It is useful when working with heterogeneous DataFrames where operations
need to be performed on a specific subset of data types.

* **Parameters:**
  **include, exclude**
  : A selection of dtypes or strings to be included/excluded. At least
    one of these parameters must be supplied.
* **Returns:**
  DataFrame
  : The subset of the frame including the dtypes in `include` and
    excluding the dtypes in `exclude`.
* **Raises:**
  ValueError
  : * If both of `include` and `exclude` are empty
    * If `include` and `exclude` have overlapping elements

  TypeError
  : * If any kind of string dtype is passed in.

#### SEE ALSO
[`DataFrame.dtypes`](dask.dataframe.DataFrame.dtypes.md#dask.dataframe.DataFrame.dtypes)
: Return Series with the data type of each column.

### Notes

* To select all *numeric* types, use `np.number` or `'number'`
* To select strings you must use the `object` dtype, but note that
  this will return *all* object dtype columns. With
  `pd.options.future.infer_string` enabled, using `"str"` will
  work to select all string columns.
* See the [numpy dtype hierarchy](https://numpy.org/doc/stable/reference/arrays.scalars.html)
* To select datetimes, use `np.datetime64`, `'datetime'` or
  `'datetime64'`
* To select timedeltas, use `np.timedelta64`, `'timedelta'` or
  `'timedelta64'`
* To select Pandas categorical dtypes, use `'category'`
* To select Pandas datetimetz dtypes, use `'datetimetz'`
  or `'datetime64[ns, tz]'`

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"a": [1, 2] * 3, "b": [True, False] * 3, "c": [1.0, 2.0] * 3}
... )
>>> df
        a      b  c
0       1   True  1.0
1       2  False  2.0
2       1   True  1.0
3       2  False  2.0
4       1   True  1.0
5       2  False  2.0
```

```pycon
>>> df.select_dtypes(include="bool")
   b
0  True
1  False
2  True
3  False
4  True
5  False
```

```pycon
>>> df.select_dtypes(include=["float64"])
   c
0  1.0
1  2.0
2  1.0
3  2.0
4  1.0
5  2.0
```

```pycon
>>> df.select_dtypes(exclude=["int64"])
       b    c
0   True  1.0
1  False  2.0
2   True  1.0
3  False  2.0
4   True  1.0
5  False  2.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.sem.html.md

# dask.dataframe.DataFrame.sem

#### DataFrame.sem(axis=None, skipna=True, ddof=1, split_every=False, numeric_only=False)

Return unbiased standard error of the mean over requested axis.

This docstring was copied from pandas.DataFrame.sem.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.sem with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keywords passed.
* **Returns:**
  Series or DataFrame (if level specified)
  : Unbiased standard error of the mean over requested axis.

#### SEE ALSO
[`DataFrame.var`](dask.dataframe.DataFrame.var.md#dask.dataframe.DataFrame.var)
: Return unbiased variance over requested axis.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Returns sample standard deviation over requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> round(s.sem(), 6)
0.57735
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.sem()
a   0.5
b   0.5
dtype: float64
```

Using axis=1

```pycon
>>> df.sem(axis=1)
tiger   0.5
zebra   0.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.sem(numeric_only=True)
a   0.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.set_index.html.md

# dask.dataframe.DataFrame.set_index

#### DataFrame.set_index(other, drop=True, sorted=False, npartitions: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, divisions=None, sort: [bool](https://docs.python.org/3/library/functions.html#bool) = True, shuffle_method=None, upsample: [float](https://docs.python.org/3/library/functions.html#float) = 1.0, partition_size: [float](https://docs.python.org/3/library/functions.html#float) = 128000000.0, append: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*options)

Set the DataFrame index (row labels) using an existing column.

If `sort=False`, this function operates exactly like `pandas.set_index`
and sets the index on the DataFrame. If `sort=True` (default),
this function also sorts the DataFrame by the new index. This can have a
significant impact on performance, because joins, groupbys, lookups, etc.
are all much faster on that column. However, this performance increase
comes with a cost, sorting a parallel dataset requires expensive shuffles.
Often we `set_index` once directly after data ingest and filtering and
then perform many cheap computations off of the sorted dataset.

With `sort=True`, this function is much more expensive. Under normal
operation this function does an initial pass over the index column to
compute approximate quantiles to serve as future divisions. It then passes
over the data a second time, splitting up each input partition into several
pieces and sharing those pieces to all of the output partitions now in
sorted order.

In some cases we can alleviate those costs, for example if your dataset is
sorted already then we can avoid making many small pieces or if you know
good values to split the new index column then we can avoid the initial
pass over the data. For example if your new index is a datetime index and
your data is already sorted by day then this entire operation can be done
for free. You can control these options with the following parameters.

* **Parameters:**
  **other: string or Dask Series**
  : Column to use as index.

  **drop: boolean, default True**
  : Delete column to be used as the new index.

  **sorted: bool, optional**
  : If the index column is already sorted in increasing order.
    Defaults to False

  **npartitions: int, None, or ‘auto’**
  : The ideal number of output partitions. If None, use the same as
    the input. If ‘auto’ then decide by memory use.
    Only used when `divisions` is not given. If `divisions` is given,
    the number of output partitions will be `len(divisions) - 1`.

  **divisions: list, optional**
  : The “dividing lines” used to split the new index into partitions.
    For `divisions=[0, 10, 50, 100]`, there would be three output partitions,
    where the new index contained [0, 10), [10, 50), and [50, 100), respectively.
    See [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions).
    If not given (default), good divisions are calculated by immediately computing
    the data and looking at the distribution of its values. For large datasets,
    this can be expensive.
    Note that if `sorted=True`, specified divisions are assumed to match
    the existing partitions in the data; if this is untrue you should
    leave divisions empty and call `repartition` after `set_index`.

  **inplace: bool, optional**
  : Modifying the DataFrame in place is not supported by Dask.
    Defaults to False.

  **sort: bool, optional**
  : If `True`, sort the DataFrame by the new index. Otherwise
    set the index on the individual existing partitions.
    Defaults to `True`.

  **shuffle_method: {‘disk’, ‘tasks’, ‘p2p’}, optional**
  : Either `'disk'` for single-node operation or `'tasks'` and
    `'p2p'` for distributed operation.  Will be inferred by your
    current scheduler.

  **compute: bool, default False**
  : Whether or not to trigger an immediate computation. Defaults to False.
    Note, that even if you set `compute=False`, an immediate computation
    will still be triggered if `divisions` is `None`.

  **partition_size: int, optional**
  : Desired size of each partitions in bytes.
    Only used when `npartitions='auto'`

### Examples

```pycon
>>> import dask
>>> ddf = dask.datasets.timeseries(start="2021-01-01", end="2021-01-07", freq="1h").reset_index()
>>> ddf2 = ddf.set_index("x")
>>> ddf2 = ddf.set_index(ddf.x)
>>> ddf2 = ddf.set_index(ddf.timestamp, sorted=True)
```

A common case is when we have a datetime column that we know to be
sorted and is cleanly divided by day.  We can set this index for free
by specifying both that the column is pre-sorted and the particular
divisions along which is is separated

```pycon
>>> import pandas as pd
>>> divisions = pd.date_range(start="2021-01-01", end="2021-01-07", freq='1D')
>>> divisions
DatetimeIndex(['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04',
               '2021-01-05', '2021-01-06', '2021-01-07'],
              dtype='datetime64[us]', freq='D')
```

Note that `len(divisions)` is equal to `npartitions + 1`. This is because `divisions`
represents the upper and lower bounds of each partition. The first item is the
lower bound of the first partition, the second item is the lower bound of the
second partition and the upper bound of the first partition, and so on.
The second-to-last item is the lower bound of the last partition, and the last
(extra) item is the upper bound of the last partition.

```pycon
>>> ddf2 = ddf.set_index("timestamp", sorted=True, divisions=divisions.tolist())
```

If you’ll be running set_index on the same (or similar) datasets repeatedly,
you could save time by letting Dask calculate good divisions once, then copy-pasting
them to reuse. This is especially helpful running in a Jupyter notebook:

```pycon
>>> ddf2 = ddf.set_index("name")  # slow, calculates data distribution
>>> ddf2.divisions
["Alice", "Laura", "Ursula", "Zelda"]
>>> # ^ Now copy-paste this and edit the line above to:
>>> # ddf2 = ddf.set_index("name", divisions=["Alice", "Laura", "Ursula", "Zelda"])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.shape.html.md

# dask.dataframe.DataFrame.shape

#### *property* DataFrame.shape

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.shuffle.html.md

# dask.dataframe.DataFrame.shuffle

#### DataFrame.shuffle(on: str | list | <no_default> = <no_default>, ignore_index: bool = False, npartitions: int | None = None, shuffle_method: str | None = None, on_index: bool = False, force: bool = False, \*\*options)

Rearrange DataFrame into new partitions

Uses hashing of on to map rows to output partitions. After this
operation, rows with the same value of on will be in the same
partition.

* **Parameters:**
  **on**
  : Column names to shuffle by.

  **ignore_index**
  : Whether to ignore the index. Default is `False`.

  **npartitions**
  : Number of output partitions. The partition count will
    be preserved by default.

  **shuffle_method**
  : Desired shuffle method. Default chosen at optimization time.

  **on_index**
  : Whether to shuffle on the index. Mutually exclusive with ‘on’.
    Set this to `True` if ‘on’ is not provided.

  **force**
  : This forces the optimizer to keep the shuffle even if the final
    expression could be further simplified.

  **\*\*options**
  : Algorithm-specific options.

### Notes

This does not preserve a meaningful index/partitioning scheme. This
is not deterministic if done in parallel.

### Examples

```pycon
>>> df = df.shuffle(df.columns[0])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.size.html.md

# dask.dataframe.DataFrame.size

#### *property* DataFrame.size

Size of the Series or DataFrame as a Delayed object.

### Examples

```pycon
>>> series.size
<dask_expr.expr.Scalar: expr=df.size(), dtype=int64>
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.sort_values.html.md

# dask.dataframe.DataFrame.sort_values

#### DataFrame.sort_values(by: [str](https://docs.python.org/3/library/stdtypes.html#str) | [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)], npartitions: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, ascending: [bool](https://docs.python.org/3/library/functions.html#bool) | [list](https://docs.python.org/3/library/stdtypes.html#list)[[bool](https://docs.python.org/3/library/functions.html#bool)] = True, na_position: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['first', 'last'] = 'last', partition_size: [float](https://docs.python.org/3/library/functions.html#float) = 128000000.0, sort_function: [Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame)], [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame)] | [None](https://docs.python.org/3/library/constants.html#None) = None, sort_function_kwargs: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)] | [None](https://docs.python.org/3/library/constants.html#None) = None, upsample: [float](https://docs.python.org/3/library/functions.html#float) = 1.0, ignore_index: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = False, shuffle_method: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*options)

Sort the dataset by a single column.

Sorting a parallel dataset requires expensive shuffles and is generally
not recommended. See `set_index` for implementation details.

* **Parameters:**
  **by: str or list[str]**
  : Column(s) to sort by.

  **npartitions: int, None, or ‘auto’**
  : The ideal number of output partitions. If None, use the same as
    the input. If ‘auto’ then decide by memory use.

  **ascending: bool, optional**
  : Sort ascending vs. descending.
    Defaults to True.

  **na_position: {‘last’, ‘first’}, optional**
  : Puts NaNs at the beginning if ‘first’, puts NaN at the end if ‘last’.
    Defaults to ‘last’.

  **sort_function: function, optional**
  : Sorting function to use when sorting underlying partitions.
    If None, defaults to `M.sort_values` (the partition library’s
    implementation of `sort_values`).

  **sort_function_kwargs: dict, optional**
  : Additional keyword arguments to pass to the partition sorting function.
    By default, `by`, `ascending`, and `na_position` are provided.

### Examples

```pycon
>>> df2 = df.sort_values('x')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.squeeze.html.md

# dask.dataframe.DataFrame.squeeze

#### DataFrame.squeeze(axis=None)

Squeeze 1 dimensional axis objects into scalars.

This docstring was copied from pandas.DataFrame.squeeze.

Some inconsistencies with the Dask version may exist.

Series or DataFrames with a single element are squeezed to a scalar.
DataFrames with a single column or a single row are squeezed to a
Series. Otherwise the object is unchanged.

This method is most useful when you don’t know if your
object is a Series or DataFrame, but you do know it has just a single
column. In that case you can safely call squeeze to ensure you have a
Series.

* **Parameters:**
  **axis**
  : A specific axis to squeeze. By default, all length-1 axes are
    squeezed. For Series this parameter is unused and defaults to None.
* **Returns:**
  DataFrame, Series, or scalar
  : The projection after squeezing axis or all the axes.

#### SEE ALSO
`Series.iloc`
: Integer-location based indexing for selecting scalars.

[`DataFrame.iloc`](dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)
: Integer-location based indexing for selecting Series.

[`Series.to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)
: Inverse of DataFrame.squeeze for a single-column DataFrame.

### Examples

```pycon
>>> primes = pd.Series([2, 3, 5, 7])
```

Slicing might produce a Series with a single value:

```pycon
>>> even_primes = primes[primes % 2 == 0]
>>> even_primes
0    2
dtype: int64
```

```pycon
>>> even_primes.squeeze()
np.int64(2)
```

Squeezing objects with more than one value in every axis does nothing:

```pycon
>>> odd_primes = primes[primes % 2 == 1]
>>> odd_primes
1    3
2    5
3    7
dtype: int64
```

```pycon
>>> odd_primes.squeeze()
1    3
2    5
3    7
dtype: int64
```

Squeezing is even more effective when used with DataFrames.

```pycon
>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=["a", "b"])
>>> df
   a  b
0  1  2
1  3  4
```

Slicing a single column will produce a DataFrame with the columns
having only one value:

```pycon
>>> df_a = df[["a"]]
>>> df_a
   a
0  1
1  3
```

So the columns can be squeezed down, resulting in a Series:

```pycon
>>> df_a.squeeze("columns")
0    1
1    3
Name: a, dtype: int64
```

Slicing a single row from a single column will produce a single
scalar DataFrame:

```pycon
>>> df_0a = df.loc[df.index < 1, ["a"]]
>>> df_0a
   a
0  1
```

Squeezing the rows produces a single scalar Series:

```pycon
>>> df_0a.squeeze("rows")
a    1
Name: 0, dtype: int64
```

Squeezing all axes will project directly into a scalar:

```pycon
>>> df_0a.squeeze()
np.int64(1)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.std.html.md

# dask.dataframe.DataFrame.std

#### DataFrame.std(axis=0, skipna=True, ddof=1, numeric_only=False, split_every=False, \*\*kwargs)

Return sample standard deviation over requested axis.

This docstring was copied from pandas.DataFrame.std.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument.

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.std with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Standard deviation over requested axis.

#### SEE ALSO
[`Series.std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)
: Return standard deviation over Series values.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Return the mean of the values over the requested axis.

[`DataFrame.median`](dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)
: Return the median of the values over the requested axis.

[`DataFrame.mode`](dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)
: Get the mode(s) of each element along the requested axis.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum of the values over the requested axis.

### Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the
default ddof=1)

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "person_id": [0, 1, 2, 3],
...         "age": [21, 25, 62, 43],
...         "height": [1.61, 1.87, 1.49, 2.01],
...     }
... ).set_index("person_id")
>>> df
           age  height
person_id
0           21    1.61
1           25    1.87
2           62    1.49
3           43    2.01
```

The standard deviation of the columns can be found as follows:

```pycon
>>> df.std()
age       18.786076
height     0.237417
dtype: float64
```

Alternatively, ddof=0 can be set to normalize by N instead of N-1:

```pycon
>>> df.std(ddof=0)
age       16.269219
height     0.205609
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.sub.html.md

# dask.dataframe.DataFrame.sub

#### DataFrame.sub(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.sum.html.md

# dask.dataframe.DataFrame.sum

#### DataFrame.sum(axis=0, skipna=True, numeric_only=False, min_count=0, split_every=False, \*\*kwargs)

Return the sum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.sum.

Some inconsistencies with the Dask version may exist.

This is equivalent to the method `numpy.sum`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.sum with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **min_count**
  : The required number of valid values to perform the operation. If fewer than
    `min_count` non-NA values are present the result will be NA.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Sum over requested axis.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum over Series values.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Return the mean of the values over the requested axis.

[`DataFrame.median`](dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)
: Return the median of the values over the requested axis.

[`DataFrame.mode`](dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)
: Get the mode(s) of each element along the requested axis.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Return the standard deviation of the values over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.sum()
14
```

By default, the sum of an empty or all-NA Series is `0`.

```pycon
>>> pd.Series([], dtype="float64").sum()  # min_count=0 is the default
0.0
```

This can be controlled with the `min_count` parameter. For example, if
you’d like the sum of an empty series to be NaN, pass `min_count=1`.

```pycon
>>> pd.Series([], dtype="float64").sum(min_count=1)
nan
```

Thanks to the `skipna` parameter, `min_count` handles all-NA and
empty series identically.

```pycon
>>> pd.Series([np.nan]).sum()
0.0
```

```pycon
>>> pd.Series([np.nan]).sum(min_count=1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.tail.html.md

# dask.dataframe.DataFrame.tail

#### DataFrame.tail(n: [int](https://docs.python.org/3/library/functions.html#int) = 5, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Last n rows of the dataset

Caveat, the only checks the last n rows of the last partition.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_backend.html.md

# dask.dataframe.DataFrame.to_backend

#### DataFrame.to_backend(backend: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Move to a new DataFrame backend

* **Parameters:**
  **backend**
  : The name of the new backend to move to. The default
    is the current “dataframe.backend” configuration.
* **Returns:**
  DataFrame, Series or Index

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_bag.html.md

# dask.dataframe.DataFrame.to_bag

#### DataFrame.to_bag(index=False, format='tuple')

Create a Dask Bag from a Series

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_csv.html.md

# dask.dataframe.DataFrame.to_csv

#### DataFrame.to_csv(filename, \*\*kwargs)

See dd.to_csv docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_dask_array.html.md

# dask.dataframe.DataFrame.to_dask_array

#### DataFrame.to_dask_array(lengths=None, meta=None, optimize: [bool](https://docs.python.org/3/library/functions.html#bool) = True, \*\*optimize_kwargs) → [Array](dask.array.Array.md#dask.array.Array)

Convert a dask DataFrame to a dask array.

* **Parameters:**
  **lengths**
  : How to determine the chunks sizes for the output array.
    By default, the output array will have unknown chunk lengths
    along the first axis, which can cause some later operations
    to fail.
    * True : immediately compute the length of each partition
    * Sequence : a sequence of integers to use for the chunk sizes
      on the first axis. These values are *not* validated for
      correctness, beyond ensuring that the number of items
      matches the number of partitions.

  **meta**
  : An optional meta parameter can be passed for dask to override the
    default metadata on the underlying dask array.

  **optimize**
  : Whether to optimize the expression before converting to an Array.
* **Returns:**
  A Dask Array

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_delayed.html.md

# dask.dataframe.DataFrame.to_delayed

#### DataFrame.to_delayed(optimize_graph=True)

Convert into a list of `dask.delayed` objects, one per partition.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

#### SEE ALSO
`dask_expr.from_delayed`

### Examples

```pycon
>>> partitions = df.to_delayed()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_hdf.html.md

# dask.dataframe.DataFrame.to_hdf

#### DataFrame.to_hdf(path_or_buf, key, mode='a', append=False, \*\*kwargs)

See dd.to_hdf docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_html.html.md

# dask.dataframe.DataFrame.to_html

#### DataFrame.to_html(max_rows=5)

Render a DataFrame as an HTML table.

> This docstring was copied from pandas.DataFrame.to_html.

> Some inconsistencies with the Dask version may exist.
* **Parameters:**
  **buf**
  : > Buffer to write to. If None, the output is returned as a string.
    <br/>
    columns
    : The subset of columns to write. Writes all columns by default.
    <br/>
    col_space
    : The minimum width of each column in CSS length units.  An int is assumed to be px units.
    <br/>
    header
    : Whether to print column labels, default True.
    <br/>
    index
    : Whether to print index (row) labels.
    <br/>
    na_rep
    : String representation of `NaN` to use.
    <br/>
    formatters
    : Formatter functions to apply to columns’ elements by position or
      name.
      The result of each function must be a unicode string.
      List/tuple must be of length equal to the number of columns.
    <br/>
    float_format
    : Formatter function to apply to columns’ elements if they are
      floats. This function must return a unicode string and will be
      applied only to the non-`NaN` elements, with `NaN` being
      handled by `na_rep`.
    <br/>
    sparsify
    : Set to False for a DataFrame with a hierarchical index to print
      every multiindex key at each row.
    <br/>
    index_names
    : Prints the names of the indexes.
    <br/>
    justify
    : How to justify the column labels. If None uses the option from
      the print configuration (controlled by set_option), ‘right’ out
      of the box. Valid values are
      * left
      * right
      * center
      * justify
      * justify-all
      * start
      * end
      * inherit
      * match-parent
      * initial
      * unset.
    <br/>
    max_rows
    : Maximum number of rows to display in the console.
    <br/>
    max_cols
    : Maximum number of columns to display in the console.
    <br/>
    show_dimensions
    : Display DataFrame dimensions (number of rows by number of columns).
    <br/>
    decimal
    : Character recognized as decimal separator, e.g. ‘,’ in Europe.

  **bold_rows**
  : Make the row labels bold in the output.

  **classes**
  : CSS class(es) to apply to the resulting html table.

  **escape**
  : Convert the characters <, >, and & to HTML-safe sequences.

  **notebook**
  : Whether the generated HTML is for IPython Notebook.

  **border**
  : When an integer value is provided, it sets the border attribute in
    the opening tag, specifying the thickness of the border.
    If `False` or `0` is passed, the border attribute will not
    be present in the `<table>` tag.
    The default value for this parameter is governed by
    `pd.options.display.html.border`.

  **table_id**
  : A css id is included in the opening <table> tag if specified.

  **render_links**
  : Convert URLs to HTML links.

  **encoding**
  : Set character encoding.
* **Returns:**
  str or None
  : If buf is None, returns the result as a string. Otherwise returns
    None.

#### SEE ALSO
[`to_string`](dask.dataframe.DataFrame.to_string.md#dask.dataframe.DataFrame.to_string)
: Convert DataFrame to a string.

### Examples

```pycon
>>> df = pd.DataFrame(data={"col1": [1, 2], "col2": [4, 3]})
>>> html_string = df.to_html()
>>> print(html_string)
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>col1</th>
      <th>col2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>4</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
```

HTML output

|    |   col1 |   col2 |
|----|--------|--------|
|  0 |      1 |      4 |
|  1 |      2 |      3 |
```pycon
>>> df = pd.DataFrame(data={"col1": [1, 2], "col2": [4, 3]})
>>> html_string = df.to_html(index=False)
>>> print(html_string)
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>col1</th>
      <th>col2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1</td>
      <td>4</td>
    </tr>
    <tr>
      <td>2</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
```

HTML output

|   col1 |   col2 |
|--------|--------|
|      1 |      4 |
|      2 |      3 |
<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_json.html.md

# dask.dataframe.DataFrame.to_json

#### DataFrame.to_json(filename, \*args, \*\*kwargs)

See dd.to_json docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_orc.html.md

# dask.dataframe.DataFrame.to_orc

#### DataFrame.to_orc(path, \*args, \*\*kwargs)

See dd.to_orc docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_parquet.html.md

# dask.dataframe.DataFrame.to_parquet

#### DataFrame.to_parquet(path, \*\*kwargs)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_records.html.md

# dask.dataframe.DataFrame.to_records

#### DataFrame.to_records(index=False, lengths=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_sql.html.md

# dask.dataframe.DataFrame.to_sql

#### DataFrame.to_sql(name: [str](https://docs.python.org/3/library/stdtypes.html#str), uri: [str](https://docs.python.org/3/library/stdtypes.html#str), schema=None, if_exists: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'fail', index: [bool](https://docs.python.org/3/library/functions.html#bool) = True, index_label=None, chunksize=None, dtype=None, method=None, compute=True, parallel=False, engine_kwargs=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_string.html.md

# dask.dataframe.DataFrame.to_string

#### DataFrame.to_string(max_rows=5)

Render a DataFrame to a console-friendly tabular output.

> This docstring was copied from pandas.DataFrame.to_string.

> Some inconsistencies with the Dask version may exist.
* **Parameters:**
  **buf**
  : > Buffer to write to. If None, the output is returned as a string.
    <br/>
    columns
    : The subset of columns to write. Writes all columns by default.
    <br/>
    col_space
    : The minimum width of each column. If a list of ints is given every integers corresponds with one column. If a dict is given, the key references the column, while the value defines the space to use.
    <br/>
    header
    : Write out the column names. If a list of columns is given, it is assumed to be aliases for the column names.
    <br/>
    index
    : Whether to print index (row) labels.
    <br/>
    na_rep
    : String representation of `NaN` to use.
    <br/>
    formatters
    : Formatter functions to apply to columns’ elements by position or
      name.
      The result of each function must be a unicode string.
      List/tuple must be of length equal to the number of columns.
    <br/>
    float_format
    : Formatter function to apply to columns’ elements if they are
      floats. This function must return a unicode string and will be
      applied only to the non-`NaN` elements, with `NaN` being
      handled by `na_rep`.
    <br/>
    sparsify
    : Set to False for a DataFrame with a hierarchical index to print
      every multiindex key at each row.
    <br/>
    index_names
    : Prints the names of the indexes.
    <br/>
    justify
    : How to justify the column labels. If None uses the option from
      the print configuration (controlled by set_option), ‘right’ out
      of the box. Valid values are
      * left
      * right
      * center
      * justify
      * justify-all
      * start
      * end
      * inherit
      * match-parent
      * initial
      * unset.
    <br/>
    max_rows
    : Maximum number of rows to display in the console.
    <br/>
    max_cols
    : Maximum number of columns to display in the console.
    <br/>
    show_dimensions
    : Display DataFrame dimensions (number of rows by number of columns).
    <br/>
    decimal
    : Character recognized as decimal separator, e.g. ‘,’ in Europe.

  **line_width**
  : Width to wrap a line in characters.

  **min_rows**
  : The number of rows to display in the console in a truncated repr
    (when number of rows is above max_rows).

  **max_colwidth**
  : Max width to truncate each column in characters. By default, no limit.

  **encoding**
  : Set character encoding.
* **Returns:**
  str or None
  : If buf is None, returns the result as a string. Otherwise returns
    None.

#### SEE ALSO
[`to_html`](dask.dataframe.DataFrame.to_html.md#dask.dataframe.DataFrame.to_html)
: Convert DataFrame to HTML.

### Examples

```pycon
>>> d = {"col1": [1, 2, 3], "col2": [4, 5, 6]}
>>> df = pd.DataFrame(d)
>>> print(df.to_string())
   col1  col2
0     1     4
1     2     5
2     3     6
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.to_timestamp.html.md

# dask.dataframe.DataFrame.to_timestamp

#### DataFrame.to_timestamp(freq=None, how='start')

Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.

This docstring was copied from pandas.DataFrame.to_timestamp.

Some inconsistencies with the Dask version may exist.

This can be changed to the *end* of the period, by specifying how=”e”.

* **Parameters:**
  **freq**
  : Desired frequency.

  **how**
  : Convention for converting period to timestamp; start of period
    vs. end.

  **axis**
  : The axis to convert (the index by default).

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.
* **Returns:**
  DataFrame with DatetimeIndex
  : DataFrame with the PeriodIndex cast to DatetimeIndex.

#### SEE ALSO
`DataFrame.to_period`
: Inverse method to cast DatetimeIndex to PeriodIndex.

[`Series.to_timestamp`](dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)
: Equivalent method for Series.

### Examples

```pycon
>>> idx = pd.PeriodIndex(["2023", "2024"], freq="Y")
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df1 = pd.DataFrame(data=d, index=idx)
>>> df1
      col1   col2
2023     1      3
2024     2      4
```

The resulting timestamps will be at the beginning of the year in this case

```pycon
>>> df1 = df1.to_timestamp()
>>> df1
            col1   col2
2023-01-01     1      3
2024-01-01     2      4
>>> df1.index
DatetimeIndex(['2023-01-01', '2024-01-01'], dtype='datetime64[us]', freq=None)
```

Using freq which is the offset that the Timestamps will have

```pycon
>>> df2 = pd.DataFrame(data=d, index=idx)
>>> df2 = df2.to_timestamp(freq="M")
>>> df2
            col1   col2
2023-01-31     1      3
2024-01-31     2      4
>>> df2.index
DatetimeIndex(['2023-01-31', '2024-01-31'], dtype='datetime64[us]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.truediv.html.md

# dask.dataframe.DataFrame.truediv

#### DataFrame.truediv(other, axis='columns', level=None, fill_value=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.values.html.md

# dask.dataframe.DataFrame.values

#### *property* DataFrame.values

Return a dask.array of the values of this dataframe

Warning: This creates a dask.array without precise shape information.
Operations that depend on shape information, like slicing or reshaping,
will not work.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.var.html.md

# dask.dataframe.DataFrame.var

#### DataFrame.var(axis=0, skipna=True, ddof=1, numeric_only=False, split_every=False, \*\*kwargs)

Return unbiased variance over requested axis.

This docstring was copied from pandas.DataFrame.var.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument.

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.var with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keywords passed.
* **Returns:**
  Series or scalaer
  : Unbiased variance over requested axis.

#### SEE ALSO
[`numpy.var`](https://numpy.org/doc/stable/reference/generated/numpy.var.html#numpy.var)
: Equivalent function in NumPy.

[`Series.var`](dask.dataframe.Series.var.md#dask.dataframe.Series.var)
: Return unbiased variance over Series values.

[`Series.std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)
: Return standard deviation over Series values.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Return standard deviation of the values over the requested axis.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "person_id": [0, 1, 2, 3],
...         "age": [21, 25, 62, 43],
...         "height": [1.61, 1.87, 1.49, 2.01],
...     }
... ).set_index("person_id")
>>> df
           age  height
person_id
0           21    1.61
1           25    1.87
2           62    1.49
3           43    2.01
```

```pycon
>>> df.var()
age       352.916667
height      0.056367
dtype: float64
```

Alternatively, `ddof=0` can be set to normalize by N instead of N-1:

```pycon
>>> df.var(ddof=0)
age       264.687500
height      0.042275
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.visualize.html.md

# dask.dataframe.DataFrame.visualize

#### DataFrame.visualize(tasks: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs)

Visualize the expression or task graph

* **Parameters:**
  **tasks:**
  : Whether to visualize the task graph. By default
    the expression graph will be visualized instead.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.DataFrame.where.html.md

# dask.dataframe.DataFrame.where

#### DataFrame.where(cond, other=nan)

Replace values where the condition is False.

This docstring was copied from pandas.DataFrame.where.

Some inconsistencies with the Dask version may exist.

This method allows conditional replacement of values. Where the
condition evaluates to True, the original values are retained; where
it evaluates to False, values are replaced with corresponding entries
from `other`.

* **Parameters:**
  **cond**
  : Where cond is True, keep the original value. Where
    False, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is False are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.mask()`](dask.dataframe.DataFrame.mask.md#dask.dataframe.DataFrame.mask)
: Return an object of same shape as caller.

[`Series.mask()`](dask.dataframe.Series.mask.md#dask.dataframe.Series.mask)
: Return an object of same shape as caller.

### Notes

The where method is an application of the if-then idiom. For each
element in the caller, if `cond` is `True` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with False.

The signature for [`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where) or
[`DataFrame.where()`](#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `where` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.add.html.md

# dask.dataframe.Index.add

#### Index.add(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.align.html.md

# dask.dataframe.Index.align

#### Index.align(other, join='outer', axis=None, fill_value=None)

Align two objects on their axes with the specified join method.

This docstring was copied from pandas.DataFrame.align.

Some inconsistencies with the Dask version may exist.

Join method is specified for each axis Index.

* **Parameters:**
  **other**
  : The object to align with.

  **join**
  : Type of alignment to be performed.
    * left: use only keys from left frame, preserve key order.
    * right: use only keys from right frame, preserve key order.
    * outer: use union of keys from both frames, sort keys lexicographically.
    * inner: use intersection of keys from both frames,
      preserve the order of the left keys.

  **axis**
  : Align on index (0), columns (1), or both (None).

  **level**
  : Broadcast across a level, matching Index values on the
    passed MultiIndex level.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **fill_value**
  : Value to use for missing values. Defaults to NaN, but can be any
    “compatible” value.
* **Returns:**
  tuple of (Series/DataFrame, type of other)
  : Aligned objects.

#### SEE ALSO
[`Series.align`](dask.dataframe.Series.align.md#dask.dataframe.Series.align)
: Align two objects on their axes with specified join method.

[`DataFrame.align`](dask.dataframe.DataFrame.align.md#dask.dataframe.DataFrame.align)
: Align two objects on their axes with specified join method.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [[1, 2, 3, 4], [6, 7, 8, 9]], columns=["D", "B", "E", "A"], index=[1, 2]
... )
>>> other = pd.DataFrame(
...     [[10, 20, 30, 40], [60, 70, 80, 90], [600, 700, 800, 900]],
...     columns=["A", "B", "C", "D"],
...     index=[2, 3, 4],
... )
>>> df
   D  B  E  A
1  1  2  3  4
2  6  7  8  9
>>> other
    A    B    C    D
2   10   20   30   40
3   60   70   80   90
4  600  700  800  900
```

Align on columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=1)
>>> left
   A  B   C  D  E
1  4  2 NaN  1  3
2  9  7 NaN  6  8
>>> right
    A    B    C    D   E
2   10   20   30   40 NaN
3   60   70   80   90 NaN
4  600  700  800  900 NaN
```

We can also align on the index:

```pycon
>>> left, right = df.align(other, join="outer", axis=0)
>>> left
    D    B    E    A
1  1.0  2.0  3.0  4.0
2  6.0  7.0  8.0  9.0
3  NaN  NaN  NaN  NaN
4  NaN  NaN  NaN  NaN
>>> right
    A      B      C      D
1    NaN    NaN    NaN    NaN
2   10.0   20.0   30.0   40.0
3   60.0   70.0   80.0   90.0
4  600.0  700.0  800.0  900.0
```

Finally, the default axis=None will align on both index and columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=None)
>>> left
     A    B   C    D    E
1  4.0  2.0 NaN  1.0  3.0
2  9.0  7.0 NaN  6.0  8.0
3  NaN  NaN NaN  NaN  NaN
4  NaN  NaN NaN  NaN  NaN
>>> right
       A      B      C      D   E
1    NaN    NaN    NaN    NaN NaN
2   10.0   20.0   30.0   40.0 NaN
3   60.0   70.0   80.0   90.0 NaN
4  600.0  700.0  800.0  900.0 NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.all.html.md

# dask.dataframe.Index.all

#### Index.all(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether all elements are True, potentially over an axis.

This docstring was copied from pandas.DataFrame.all.

Some inconsistencies with the Dask version may exist.

Returns True unless there at least one element within a series or
along a Dataframe axis that is False or equivalent (e.g. zero or
empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be True, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`Series.all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)
: Return True if all elements are True.

[`DataFrame.any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)
: Return True if one (or more) elements are True.

### Examples

**Series**

```pycon
>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([], dtype="float64").all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True
```

**DataFrames**

Create a DataFrame from a dictionary.

```pycon
>>> df = pd.DataFrame({"col1": [True, True], "col2": [True, False]})
>>> df
   col1   col2
0  True   True
1  True  False
```

Default behaviour checks if values in each column all return True.

```pycon
>>> df.all()
col1     True
col2    False
dtype: bool
```

Specify `axis='columns'` to check if values in each row all return True.

```pycon
>>> df.all(axis="columns")
0     True
1    False
dtype: bool
```

Or `axis=None` for whether every value is True.

```pycon
>>> df.all(axis=None)
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.any.html.md

# dask.dataframe.Index.any

#### Index.any(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether any element is True, potentially over an axis.

This docstring was copied from pandas.DataFrame.any.

Some inconsistencies with the Dask version may exist.

Returns False unless there is at least one element within a series or
along a Dataframe axis that is True or equivalent (e.g. non-zero or
non-empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be False, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`numpy.any`](https://numpy.org/doc/stable/reference/generated/numpy.any.html#numpy.any)
: Numpy version of this method.

[`Series.any`](dask.dataframe.Series.any.md#dask.dataframe.Series.any)
: Return whether any element is True.

[`Series.all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)
: Return whether all elements are True.

[`DataFrame.any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)
: Return whether any element is True over requested axis.

[`DataFrame.all`](dask.dataframe.DataFrame.all.md#dask.dataframe.DataFrame.all)
: Return whether all elements are True over requested axis.

### Examples

**Series**

For Series input, the output is a scalar indicating whether any element
is True.

```pycon
>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([], dtype="float64").any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True
```

**DataFrame**

Whether each column contains at least one True element (the default).

```pycon
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
   A  B  C
0  1  0  0
1  2  2  0
```

```pycon
>>> df.any()
A     True
B     True
C    False
dtype: bool
```

Aggregating over the columns.

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
       A  B
0   True  1
1  False  2
```

```pycon
>>> df.any(axis="columns")
0    True
1    True
dtype: bool
```

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
       A  B
0   True  1
1  False  0
```

```pycon
>>> df.any(axis="columns")
0    True
1    False
dtype: bool
```

Aggregating over the entire DataFrame with `axis=None`.

```pycon
>>> df.any(axis=None)
True
```

any for an empty DataFrame is an empty Series.

```pycon
>>> pd.DataFrame([]).any()
Series([], dtype: bool)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.apply.html.md

# dask.dataframe.Index.apply

#### Index.apply(function, \*args, meta=<no_default>, axis=0, \*\*kwargs)

Parallel version of pandas.Series.apply

* **Parameters:**
  **func**
  : Function to apply

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

  **args**
  : Positional arguments to pass to function in addition to the value.

  **Additional keyword arguments will be passed as keywords to the function.**
* **Returns:**
  **applied**

#### SEE ALSO
[`Series.map_partitions`](dask.dataframe.Series.map_partitions.md#dask.dataframe.Series.map_partitions)

### Examples

```pycon
>>> import dask.dataframe as dd
>>> s = pd.Series(range(5), name='x')
>>> ds = dd.from_pandas(s, npartitions=2)
```

Apply a function elementwise across the Series, passing in extra
arguments in `args` and `kwargs`:

```pycon
>>> def myadd(x, a, b=1):
...     return x + a + b
>>> res = ds.apply(myadd, args=(2,), b=1.5)
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with name `'x'`, and dtype
`float64`:

```pycon
>>> res = ds.apply(myadd, args=(2,), b=1.5, meta=('x', 'f8'))
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ds.apply(lambda x: x + 1, meta=ds)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.astype.html.md

# dask.dataframe.Index.astype

#### Index.astype(dtypes)

Cast a pandas object to a specified dtype `dtype`.

This docstring was copied from pandas.DataFrame.astype.

Some inconsistencies with the Dask version may exist.

This method allows the conversion of the data types of pandas objects,
including DataFrames and Series, to the specified dtype. It supports casting
entire objects to a single data type or applying different data types to
individual columns using a mapping.

* **Parameters:**
  **dtype**
  : Use a str, numpy.dtype, pandas.ExtensionDtype or Python type to
    cast entire pandas object to the same type. Alternatively, use a
    mapping, e.g. {col: dtype, …}, where col is a column label and dtype is
    a numpy.dtype or Python type to cast one or more of the DataFrame’s
    columns to column-specific types.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **errors**
  : Control raising of exceptions on invalid data for provided dtype.
    - `raise` : allow exceptions to be raised
    - `ignore` : suppress exceptions. On error return original object.
* **Returns:**
  same type as caller
  : The pandas object casted to the specified `dtype`.

#### SEE ALSO
[`to_datetime`](dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)
: Convert argument to datetime.

[`to_timedelta`](dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)
: Convert argument to timedelta.

[`to_numeric`](dask.dataframe.to_numeric.md#dask.dataframe.to_numeric)
: Convert argument to a numeric type.

[`numpy.ndarray.astype`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html#numpy.ndarray.astype)
: Cast a numpy array to a specified type.

### Notes

#### Versionchanged
Changed in version 2.0.0: Using `astype` to convert from timezone-naive dtype to
timezone-aware dtype will raise an exception.
Use `Series.dt.tz_localize()` instead.

### Examples

Create a DataFrame:

```pycon
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object
```

Cast all columns to int32:

```pycon
>>> df.astype("int32").dtypes
col1    int32
col2    int32
dtype: object
```

Cast col1 to int32 using a dictionary:

```pycon
>>> df.astype({"col1": "int32"}).dtypes
col1    int32
col2    int64
dtype: object
```

Create a series:

```pycon
>>> ser = pd.Series([1, 2], dtype="int32")
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype("int64")
0    1
1    2
dtype: int64
```

Convert to categorical type:

```pycon
>>> ser.astype("category")
0    1
1    2
dtype: category
Categories (2, int32): [1, 2]
```

Convert to ordered categorical type with custom ordering:

```pycon
>>> from pandas.api.types import CategoricalDtype
>>> cat_dtype = CategoricalDtype(categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]
```

Create a series of dates:

```pycon
>>> ser_date = pd.Series(pd.date_range("20200101", periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[us]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.autocorr.html.md

# dask.dataframe.Index.autocorr

#### Index.autocorr(lag=1, split_every=False)

Compute the lag-N autocorrelation.

This docstring was copied from pandas.Series.autocorr.

Some inconsistencies with the Dask version may exist.

This method computes the Pearson correlation between
the Series and its shifted self.

* **Parameters:**
  **lag**
  : Number of lags to apply before performing autocorrelation.
* **Returns:**
  float
  : The Pearson correlation between self and self.shift(lag).

#### SEE ALSO
[`Series.corr`](dask.dataframe.Series.corr.md#dask.dataframe.Series.corr)
: Compute the correlation between two Series.

[`Series.shift`](dask.dataframe.Series.shift.md#dask.dataframe.Series.shift)
: Shift index by desired number of periods.

[`DataFrame.corr`](dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)
: Compute pairwise correlation of columns.

`DataFrame.corrwith`
: Compute pairwise correlation between rows or columns of two DataFrame objects.

### Notes

If the Pearson correlation is not well defined return ‘NaN’.

### Examples

```pycon
>>> s = pd.Series([0.25, 0.5, 0.2, -0.05])
>>> s.autocorr()
0.10355...
>>> s.autocorr(lag=2)
-0.99999...
```

If the Pearson correlation is not well defined, then ‘NaN’ is returned.

```pycon
>>> s = pd.Series([1, 0, 0, 0])
>>> s.autocorr()
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.between.html.md

# dask.dataframe.Index.between

#### Index.between(left, right, inclusive='both')

Return boolean Series equivalent to left <= series <= right.

This docstring was copied from pandas.Series.between.

Some inconsistencies with the Dask version may exist.

This function returns a boolean vector containing True wherever the
corresponding Series element is between the boundary values left and
right. NA values are treated as False.

* **Parameters:**
  **left**
  : Left boundary.

  **right**
  : Right boundary.

  **inclusive**
  : Include boundaries. Whether to set each bound as closed or open.
* **Returns:**
  Series
  : Series representing whether each element is between left and
    right (inclusive).

#### SEE ALSO
[`Series.gt`](dask.dataframe.Series.gt.md#dask.dataframe.Series.gt)
: Greater than of series and other.

[`Series.lt`](dask.dataframe.Series.lt.md#dask.dataframe.Series.lt)
: Less than of series and other.

### Notes

This function is equivalent to `(left <= ser) & (ser <= right)`

### Examples

```pycon
>>> s = pd.Series([2, 0, 4, 8, np.nan])
```

Boundary values are included by default:

```pycon
>>> s.between(1, 4)
0     True
1    False
2     True
3    False
4    False
dtype: bool
```

With inclusive set to `"neither"` boundary values are excluded:

```pycon
>>> s.between(1, 4, inclusive="neither")
0     True
1    False
2    False
3    False
4    False
dtype: bool
```

left and right can be any scalar value:

```pycon
>>> s = pd.Series(["Alice", "Bob", "Carol", "Eve"])
>>> s.between("Anna", "Daniel")
0    False
1     True
2     True
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.bfill.html.md

# dask.dataframe.Index.bfill

#### Index.bfill(axis=0, limit=None)

Fill NA/NaN values by using the next valid observation to fill the gap.

This docstring was copied from pandas.DataFrame.bfill.

Some inconsistencies with the Dask version may exist.

This method fills missing values in a backward direction along the
specified axis, propagating non-null values from later positions to
earlier positions containing NaN.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.ffill`](dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)
: Fill NA/NaN values by propagating the last valid observation to next valid.

### Examples

For Series:

```pycon
>>> s = pd.Series([1, None, None, 2])
>>> s.bfill()
0    1.0
1    2.0
2    2.0
3    2.0
dtype: float64
>>> s.bfill(limit=1)
0    1.0
1    NaN
2    2.0
3    2.0
dtype: float64
```

With DataFrame:

```pycon
>>> df = pd.DataFrame({"A": [1, None, None, 4], "B": [None, 5, None, 7]})
>>> df
      A     B
0   1.0   NaN
1   NaN   5.0
2   NaN   NaN
3   4.0   7.0
>>> df.bfill()
      A     B
0   1.0   5.0
1   4.0   5.0
2   4.0   7.0
3   4.0   7.0
>>> df.bfill(limit=1)
      A     B
0   1.0   5.0
1   NaN   5.0
2   4.0   7.0
3   4.0   7.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.clear_divisions.html.md

# dask.dataframe.Index.clear_divisions

#### Index.clear_divisions()

Forget division information.

This is useful if the divisions are no longer meaningful.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.clip.html.md

# dask.dataframe.Index.clip

#### Index.clip(lower=None, upper=None, axis=None, \*\*kwargs)

Trim values at input threshold(s).

This docstring was copied from pandas.Series.clip.

Some inconsistencies with the Dask version may exist.

Assigns values outside boundary to boundary values. Thresholds
can be singular values or array like, and in the latter case
the clipping is performed element-wise in the specified axis.

* **Parameters:**
  **lower**
  : Minimum threshold value. All values below this
    threshold will be set to it. A missing
    threshold (e.g NA) will not clip the value.

  **upper**
  : Maximum threshold value. All values above this
    threshold will be set to it. A missing
    threshold (e.g NA) will not clip the value.

  **axis**
  : Align object with lower and upper along the given axis.
    For Series this parameter is unused and defaults to None.

  **inplace**
  : Whether to perform the operation in place on the data.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted
    for compatibility with numpy.
* **Returns:**
  Series or DataFrame
  : Same type as calling object with the values outside the
    clip boundaries replaced.

#### SEE ALSO
[`Series.clip`](dask.dataframe.Series.clip.md#dask.dataframe.Series.clip)
: Trim values at input threshold in series.

`DataFrame.clip`
: Trim values at input threshold in DataFrame.

[`numpy.clip`](https://numpy.org/doc/stable/reference/generated/numpy.clip.html#numpy.clip)
: Clip (limit) the values in an array.

### Examples

```pycon
>>> data = {"col_0": [9, -3, 0, -1, 5], "col_1": [-2, -7, 6, 8, -5]}
>>> df = pd.DataFrame(data)
>>> df
   col_0  col_1
0      9     -2
1     -3     -7
2      0      6
3     -1      8
4      5     -5
```

Clips per column using lower and upper thresholds:

```pycon
>>> df.clip(-4, 6)
   col_0  col_1
0      6     -2
1     -3     -4
2      0      6
3     -1      6
4      5     -4
```

Clips using specific lower and upper thresholds per column:

```pycon
>>> df.clip([-2, -1], [4, 5])
    col_0  col_1
0      4     -1
1     -2     -1
2      0      5
3     -1      5
4      4     -1
```

Clips using specific lower and upper thresholds per column element:

```pycon
>>> t = pd.Series([2, -4, -1, 6, 3])
>>> t
0    2
1   -4
2   -1
3    6
4    3
dtype: int64
```

```pycon
>>> df.clip(t, t + 4, axis=0)
   col_0  col_1
0      6      2
1     -3     -4
2      0      3
3      6      8
4      5      3
```

Clips using specific lower threshold per column element, with missing values:

```pycon
>>> t = pd.Series([2, -4, np.nan, 6, 3])
>>> t
0    2.0
1   -4.0
2    NaN
3    6.0
4    3.0
dtype: float64
```

```pycon
>>> df.clip(t, axis=0)
col_0  col_1
0      9.0    2.0
1     -3.0   -4.0
2      0.0    6.0
3      6.0    8.0
4      5.0    3.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.compute.html.md

# dask.dataframe.Index.compute

#### Index.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.copy.html.md

# dask.dataframe.Index.copy

#### Index.copy(deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Make a copy of the dataframe

This is strictly a shallow copy of the underlying computational graph.
It does not affect the underlying data

* **Parameters:**
  **deep**
  : The deep value must be False and it is declared as a parameter just for
    compatibility with third-party libraries like cuDF and pandas

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.corr.html.md

# dask.dataframe.Index.corr

#### Index.corr(other, method='pearson', min_periods=None, split_every=False)

Compute correlation with other Series, excluding missing values.

This docstring was copied from pandas.Series.corr.

Some inconsistencies with the Dask version may exist.

The two Series objects are not required to be the same length and will be
aligned internally before the correlation function is applied.

* **Parameters:**
  **other**
  : Series with which to compute the correlation.

  **method**
  : Method used to compute correlation:
    - pearson : Standard correlation coefficient
    - kendall : Kendall Tau correlation coefficient
    - spearman : Spearman rank correlation
    - callable: Callable with input two 1d ndarrays and returning a float.
    <br/>
    #### WARNING
    Note that the returned matrix from corr will have 1 along the
    diagonals and will be symmetric regardless of the callable’s
    behavior.

  **min_periods**
  : Minimum number of observations needed to have a valid result.
* **Returns:**
  float
  : Correlation with other.

#### SEE ALSO
[`DataFrame.corr`](dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)
: Compute pairwise correlation between columns.

`DataFrame.corrwith`
: Compute pairwise correlation with another DataFrame or Series.

### Notes

Pearson, Kendall and Spearman correlation are currently computed using pairwise complete observations.

* [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)
* [Kendall rank correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)
* [Spearman’s rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient)

Automatic data alignment: as with all pandas operations, automatic data alignment is performed for this method.
`corr()` automatically considers values with matching indices.

### Examples

```pycon
>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> s1 = pd.Series([0.2, 0.0, 0.6, 0.2])
>>> s2 = pd.Series([0.3, 0.6, 0.0, 0.1])
>>> s1.corr(s2, method=histogram_intersection)
0.3
```

Pandas auto-aligns the values with matching indices

```pycon
>>> s1 = pd.Series([1, 2, 3], index=[0, 1, 2])
>>> s2 = pd.Series([1, 2, 3], index=[2, 1, 0])
>>> s1.corr(s2)
-1.0
```

If the input is a constant array, the correlation is not defined in this case,
and `np.nan` is returned.

```pycon
>>> s1 = pd.Series([0.45, 0.45])
>>> s1.corr(s1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.count.html.md

# dask.dataframe.Index.count

#### Index.count(split_every=False)

Count non-NA cells for each column or row.

This docstring was copied from pandas.DataFrame.count.

Some inconsistencies with the Dask version may exist.

The values None, NaN, NaT, `pandas.NA` are considered NA.

* **Parameters:**
  **axis**
  : If 0 or ‘index’ counts are generated for each column.
    If 1 or ‘columns’ counts are generated for each row.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : For each column/row the number of non-NA/null entries.

#### SEE ALSO
[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Number of non-NA elements in a Series.

`DataFrame.value_counts`
: Count unique combinations of columns.

[`DataFrame.shape`](dask.dataframe.DataFrame.shape.md#dask.dataframe.DataFrame.shape)
: Number of DataFrame rows and columns (including NA elements).

[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Boolean same-sized DataFrame showing places of NA elements.

### Examples

Constructing DataFrame from a dictionary:

```pycon
>>> df = pd.DataFrame(
...     {
...         "Person": ["John", "Myla", "Lewis", "John", "Myla"],
...         "Age": [24.0, np.nan, 21.0, 33, 26],
...         "Single": [False, True, True, True, False],
...     }
... )
>>> df
   Person   Age  Single
0    John  24.0   False
1    Myla   NaN    True
2   Lewis  21.0    True
3    John  33.0    True
4    Myla  26.0   False
```

Notice the uncounted NA values:

```pycon
>>> df.count()
Person    5
Age       4
Single    5
dtype: int64
```

Counts for each **row**:

```pycon
>>> df.count(axis="columns")
0    3
1    2
2    3
3    3
4    3
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.cov.html.md

# dask.dataframe.Index.cov

#### Index.cov(other, min_periods=None, split_every=False)

Compute covariance with Series, excluding missing values.

This docstring was copied from pandas.Series.cov.

Some inconsistencies with the Dask version may exist.

The two Series objects are not required to be the same length and
will be aligned internally before the covariance is calculated.

* **Parameters:**
  **other**
  : Series with which to compute the covariance.

  **min_periods**
  : Minimum number of observations needed to have a valid result.

  **ddof**
  : Delta degrees of freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.
* **Returns:**
  float
  : Covariance between Series and other normalized by N-1
    (unbiased estimator).

#### SEE ALSO
[`DataFrame.cov`](dask.dataframe.DataFrame.cov.md#dask.dataframe.DataFrame.cov)
: Compute pairwise covariance of columns.

### Examples

```pycon
>>> s1 = pd.Series([0.90010907, 0.13484424, 0.62036035])
>>> s2 = pd.Series([0.12528585, 0.26962463, 0.51111198])
>>> s1.cov(s2)
-0.01685762652715874
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.cummax.html.md

# dask.dataframe.Index.cummax

#### Index.cummax(axis=0, skipna=True)

Return cumulative maximum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummax.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
maximum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative maximum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.max`
: Similar functionality but ignores `NaN` values.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummax()
0    2.0
1    NaN
2    5.0
3    5.0
4    5.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummax(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the maximum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummax()
     A    B
0  2.0  1.0
1  3.0  NaN
2  3.0  1.0
```

To iterate over columns and find the maximum in each row,
use `axis=1`

```pycon
>>> df.cummax(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.cummin.html.md

# dask.dataframe.Index.cummin

#### Index.cummin(axis=0, skipna=True)

Return cumulative minimum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummin.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
minimum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative minimum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.min`
: Similar functionality but ignores `NaN` values.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummin()
0    2.0
1    NaN
2    2.0
3   -1.0
4   -1.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummin(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the minimum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummin()
     A    B
0  2.0  1.0
1  2.0  NaN
2  1.0  0.0
```

To iterate over columns and find the minimum in each row,
use `axis=1`

```pycon
>>> df.cummin(axis=1)
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.cumprod.html.md

# dask.dataframe.Index.cumprod

#### Index.cumprod(axis=0, skipna=True, \*\*kwargs)

Return cumulative product over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumprod.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
product.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative product of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.prod`
: Similar functionality but ignores `NaN` values.

[`DataFrame.prod`](dask.dataframe.DataFrame.prod.md#dask.dataframe.DataFrame.prod)
: Return the product over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumprod()
0     2.0
1     NaN
2    10.0
3   -10.0
4    -0.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumprod(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the product
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumprod()
     A    B
0  2.0  1.0
1  6.0  NaN
2  6.0  0.0
```

To iterate over columns and find the product in each row,
use `axis=1`

```pycon
>>> df.cumprod(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.cumsum.html.md

# dask.dataframe.Index.cumsum

#### Index.cumsum(axis=0, skipna=True, \*\*kwargs)

Return cumulative sum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumsum.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
sum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative sum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.sum`
: Similar functionality but ignores `NaN` values.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumsum()
0    2.0
1    NaN
2    7.0
3    6.0
4    6.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumsum(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the sum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumsum()
     A    B
0  2.0  1.0
1  5.0  NaN
2  6.0  1.0
```

To iterate over columns and find the sum in each row,
use `axis=1`

```pycon
>>> df.cumsum(axis=1)
     A    B
0  2.0  3.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.describe.html.md

# dask.dataframe.Index.describe

#### Index.describe(split_every=False, percentiles=None, percentiles_method='default', include=None, exclude=None)

Generate descriptive statistics.

This docstring was copied from pandas.Series.describe.

Some inconsistencies with the Dask version may exist.

Generate descriptive statistics.

Dask computes percentiles (used for the `25%`, `50%`, and
`75%` statistics) using an **approximate algorithm** by default.
Results may therefore differ slightly from pandas.  Use
`percentiles_method="dask"` for the built-in Dask algorithm or
`percentiles_method="tdigest"` for the t-digest algorithm.
See [`dask.dataframe.Series.quantile()`](dask.dataframe.Series.quantile.md#dask.dataframe.Series.quantile) for details.

* **Parameters:**
  **split_every**
  : Number of partitions to aggregate at once. Defaults to `False`
    which uses a single-pass reduction over all partitions.

  **percentiles**
  : The percentiles to include in the output. All should fall
    between 0 and 1. By default, `[0.25, 0.5, 0.75]` is used.

  **percentiles_method**
  : Method for computing percentiles. `"default"` uses the internal
    Dask algorithm. `"tdigest"` uses the t-digest algorithm for
    floats and ints and falls back to `"dask"` otherwise.

  **Descriptive statistics include those that summarize the central**

  **tendency, dispersion and shape of a**

  **dataset’s distribution, excluding \`\`NaN\`\` values.**

  **Analyzes both numeric and object series, as well**

  **as \`\`DataFrame\`\` column sets of mixed data types. The output**

  **will vary depending on what is provided. Refer to the notes**

  **below for more detail.**
* **Returns:**
  Series or DataFrame
  : Summary statistics of the Series or Dataframe provided.

#### SEE ALSO
[`DataFrame.count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)
: Count number of non-NA/null observations.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Maximum of the values in the object.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Minimum of the values in the object.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Mean of the values.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Standard deviation of the observations.

[`DataFrame.select_dtypes`](dask.dataframe.DataFrame.select_dtypes.md#dask.dataframe.DataFrame.select_dtypes)
: Subset of a DataFrame including/excluding columns based on their dtype.

### Notes

For numeric data, the result’s index will include `count`,
`mean`, `std`, `min`, `max` as well as lower, `50` and
upper percentiles. By default the lower percentile is `25` and the
upper percentile is `75`. The `50` percentile is the
same as the median.

For object data (e.g. strings), the result’s index
will include `count`, `unique`, `top`, and `freq`. The `top`
is the most common value. The `freq` is the most common value’s
frequency.

If multiple object values have the highest count, then the
`count` and `top` results will be arbitrarily chosen from
among those with the highest count.

For mixed data types provided via a `DataFrame`, the default is to
return only an analysis of numeric columns. If the DataFrame consists
only of object and categorical data without any numeric columns, the
default is to return an analysis of both the object and categorical
columns. If `include='all'` is provided as an option, the result
will include a union of attributes of each type.

The include and exclude parameters can be used to limit
which columns in a `DataFrame` are analyzed for the output.
The parameters are ignored when analyzing a `Series`.

### Examples

Describing a numeric `Series`.

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
dtype: float64
```

Describing a categorical `Series`.

```pycon
>>> s = pd.Series(["a", "a", "b", "c"])
>>> s.describe()
count     4
unique    3
top       a
freq      2
dtype: object
```

Describing a timestamp `Series`.

```pycon
>>> s = pd.Series(
...     [
...         np.datetime64("2000-01-01"),
...         np.datetime64("2010-01-01"),
...         np.datetime64("2010-01-01"),
...     ]
... )
>>> s.describe()
count                      3
mean     2006-09-01 08:00:00
min      2000-01-01 00:00:00
25%      2004-12-31 12:00:00
50%      2010-01-01 00:00:00
75%      2010-01-01 00:00:00
max      2010-01-01 00:00:00
dtype: object
```

Describing a `DataFrame`. By default only numeric fields
are returned.

```pycon
>>> df = pd.DataFrame(
...     {
...         "categorical": pd.Categorical(["d", "e", "f"]),
...         "numeric": [1, 2, 3],
...         "object": ["a", "b", "c"],
...     }
... )
>>> df.describe()
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Describing all columns of a `DataFrame` regardless of data type.

```pycon
>>> df.describe(include="all")
       categorical  numeric object
count            3      3.0      3
unique           3      NaN      3
top              f      NaN      a
freq             1      NaN      1
mean           NaN      2.0    NaN
std            NaN      1.0    NaN
min            NaN      1.0    NaN
25%            NaN      1.5    NaN
50%            NaN      2.0    NaN
75%            NaN      2.5    NaN
max            NaN      3.0    NaN
```

Describing a column from a `DataFrame` by accessing it as
an attribute.

```pycon
>>> df.numeric.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
Name: numeric, dtype: float64
```

Including only numeric columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[np.number])
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Including only string columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[object])
       object
count       3
unique      3
top         a
freq        1
```

Including only categorical columns from a `DataFrame` description.

```pycon
>>> df.describe(include=["category"])
       categorical
count            3
unique           3
top              d
freq             1
```

Excluding numeric columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[np.number])
       categorical object
count            3      3
unique           3      3
top              f      a
freq             1      1
```

Excluding object columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[object])
       categorical  numeric
count            3      3.0
unique           3      NaN
top              f      NaN
freq             1      NaN
mean           NaN      2.0
std            NaN      1.0
min            NaN      1.0
25%            NaN      1.5
50%            NaN      2.0
75%            NaN      2.5
max            NaN      3.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.diff.html.md

# dask.dataframe.Index.diff

#### Index.diff(periods=1, axis=0)

First discrete difference of element.

This docstring was copied from pandas.DataFrame.diff.

Some inconsistencies with the Dask version may exist.

#### NOTE
Pandas currently uses an `object`-dtype column to represent
boolean data with missing values. This can cause issues for
boolean-specific operations, like `|`. To enable boolean-
specific operations, at the cost of metadata that doesn’t match
pandas, use `.astype(bool)` after the `shift`.

Calculates the difference of a DataFrame element compared with another
element in the DataFrame (default is element in previous row).

* **Parameters:**
  **periods**
  : Periods to shift for calculating difference, accepts negative
    values.

  **axis**
  : Take difference over rows (0) or columns (1).
* **Returns:**
  DataFrame
  : First differences of the Series.

#### SEE ALSO
`DataFrame.pct_change`
: Percent change over given number of periods.

`DataFrame.shift`
: Shift index by desired number of periods with an optional time freq.

[`Series.diff`](dask.dataframe.Series.diff.md#dask.dataframe.Series.diff)
: First discrete difference of object.

### Notes

For boolean dtypes, this uses `operator.xor()` rather than
`operator.sub()`.
The result is calculated according to current dtype in DataFrame,
however dtype of the result is always float64.

### Examples

Difference with previous row

```pycon
>>> df = pd.DataFrame(
...     {
...         "a": [1, 2, 3, 4, 5, 6],
...         "b": [1, 1, 2, 3, 5, 8],
...         "c": [1, 4, 9, 16, 25, 36],
...     }
... )
>>> df
   a  b   c
0  1  1   1
1  2  1   4
2  3  2   9
3  4  3  16
4  5  5  25
5  6  8  36
>>> df.diff()
     a    b     c
0  NaN  NaN   NaN
1  1.0  0.0   3.0
2  1.0  1.0   5.0
3  1.0  1.0   7.0
4  1.0  2.0   9.0
5  1.0  3.0  11.0
```

Difference with previous column

```pycon
>>> df.diff(axis=1)
    a  b   c
0 NaN  0   0
1 NaN -1   3
2 NaN -1   7
3 NaN -1  13
4 NaN  0  20
5 NaN  2  28
```

Difference with 3rd previous row

```pycon
>>> df.diff(periods=3)
     a    b     c
0  NaN  NaN   NaN
1  NaN  NaN   NaN
2  NaN  NaN   NaN
3  3.0  2.0  15.0
4  3.0  4.0  21.0
5  3.0  6.0  27.0
```

Difference with following row

```pycon
>>> df.diff(periods=-1)
     a    b     c
0 -1.0  0.0  -3.0
1 -1.0 -1.0  -5.0
2 -1.0 -1.0  -7.0
3 -1.0 -2.0  -9.0
4 -1.0 -3.0 -11.0
5  NaN  NaN   NaN
```

Overflow in input dtype

```pycon
>>> df = pd.DataFrame({"a": [1, 0]}, dtype=np.uint8)
>>> df.diff()
       a
0    NaN
1  255.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.div.html.md

# dask.dataframe.Index.div

#### Index.div(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.drop_duplicates.html.md

# dask.dataframe.Index.drop_duplicates

#### Index.drop_duplicates(ignore_index=False, split_every=None, split_out=True, shuffle_method=None, keep='first')

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.dropna.html.md

# dask.dataframe.Index.dropna

#### Index.dropna()

Return a new Series with missing values removed.

This docstring was copied from pandas.Series.dropna.

Some inconsistencies with the Dask version may exist.

See the [User Guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data) for more on which values are
considered missing, and how to work with missing data.

* **Parameters:**
  **axis**
  : Unused. Parameter needed for compatibility with DataFrame.

  **inplace**
  : If True, do operation inplace and return None.

  **how**
  : Not in use. Kept for compatibility.

  **ignore_index**
  : If `True`, the resulting axis will be labeled 0, 1, …, n - 1.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  Series or None
  : Series with NA entries dropped from it or None if `inplace=True`.

#### SEE ALSO
[`Series.isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)
: Indicate missing values.

`Series.notna`
: Indicate existing (non-missing) values.

[`Series.fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)
: Replace missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Drop rows or columns which contain NA values.

[`Index.dropna`](#dask.dataframe.Index.dropna)
: Drop missing indices.

### Examples

```pycon
>>> ser = pd.Series([1.0, 2.0, np.nan])
>>> ser
0    1.0
1    2.0
2    NaN
dtype: float64
```

Drop NA values from a Series.

```pycon
>>> ser.dropna()
0    1.0
1    2.0
dtype: float64
```

Empty strings are not considered NA values. `None` is considered an
NA value.

```pycon
>>> ser = pd.Series([np.nan, 2, pd.NaT, "", None, "I stay"])
>>> ser
0       NaN
1         2
2       NaT
3
4      None
5    I stay
dtype: object
>>> ser.dropna()
1         2
3
5    I stay
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.dtype.html.md

# dask.dataframe.Index.dtype

#### *property* Index.dtype

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.eq.html.md

# dask.dataframe.Index.eq

#### Index.eq(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.explode.html.md

# dask.dataframe.Index.explode

#### Index.explode()

Transform each element of a list-like to a row.

This docstring was copied from pandas.Series.explode.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ignore_index**
  : If True, the resulting index will be labeled 0, 1, …, n - 1.
* **Returns:**
  Series
  : Exploded lists to rows; index will be duplicated for these rows.

#### SEE ALSO
[`Series.str.split`](dask.dataframe.Series.str.split.md#dask.dataframe.Series.str.split)
: Split string values on specified separator.

`Series.unstack`
: Unstack, a.k.a. pivot, Series with MultiIndex to produce DataFrame.

[`DataFrame.melt`](dask.dataframe.DataFrame.melt.md#dask.dataframe.DataFrame.melt)
: Unpivot a DataFrame from wide format to long format.

[`DataFrame.explode`](dask.dataframe.DataFrame.explode.md#dask.dataframe.DataFrame.explode)
: Explode a DataFrame from list-like columns to long format.

### Notes

This routine will explode list-likes including lists, tuples, sets,
Series, and np.ndarray. The result dtype of the subset rows will
be object. Scalars will be returned unchanged, and empty list-likes will
result in an np.nan for that row. In addition, the ordering of elements in
the output will be non-deterministic when exploding sets.

Reference [the user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html#reshaping-explode) for more examples.

### Examples

```pycon
>>> s = pd.Series([[1, 2, 3], "foo", [], [3, 4]])
>>> s
0    [1, 2, 3]
1          foo
2           []
3       [3, 4]
dtype: object
```

```pycon
>>> s.explode()
0      1
0      2
0      3
1    foo
2    NaN
3      3
3      4
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.ffill.html.md

# dask.dataframe.Index.ffill

#### Index.ffill(axis=0, limit=None)

Fill NA/NaN values by propagating the last valid observation to next valid.

This docstring was copied from pandas.DataFrame.ffill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.bfill`](dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)
: Fill NA/NaN values by using the next valid observation to fill the gap.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

```pycon
>>> df.ffill()
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  3.0  4.0 NaN  1.0
3  3.0  3.0 NaN  4.0
```

```pycon
>>> ser = pd.Series([1, np.nan, 2, 3])
>>> ser.ffill()
0   1.0
1   1.0
2   2.0
3   3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.fillna.html.md

# dask.dataframe.Index.fillna

#### Index.fillna(value=None, axis=None)

Fill NA/NaN values with value.

This docstring was copied from pandas.DataFrame.fillna.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **value**
  : Value to use to fill holes (e.g. 0), alternately a
    dict/Series/DataFrame of values specifying which value to use for
    each index (for a Series) or column (for a DataFrame).  Values not
    in the dict/Series/DataFrame will not be filled. This value cannot
    be a list.

  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : This is the maximum number of entries along the entire axis
    where NaNs will be filled. Must be greater than 0 if not None.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`ffill`](dask.dataframe.Index.ffill.md#dask.dataframe.Index.ffill)
: Fill values by propagating the last valid observation to next valid.

[`bfill`](dask.dataframe.Index.bfill.md#dask.dataframe.Index.bfill)
: Fill values by using the next valid observation to fill the gap.

`interpolate`
: Fill NaN values using interpolation.

`reindex`
: Conform object to new index.

`asfreq`
: Convert TimeSeries to specified frequency.

### Notes

For non-object dtype, `value=None` will use the NA value of the dtype.
See more details in the [Filling missing data](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data-fillna)
section.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

Replace all NaN elements with 0s.

```pycon
>>> df.fillna(0)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  0.0
3  0.0  3.0  0.0  4.0
```

Replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1,
2, and 3 respectively.

```pycon
>>> values = {"A": 0, "B": 1, "C": 2, "D": 3}
>>> df.fillna(value=values)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  2.0  1.0
2  0.0  1.0  2.0  3.0
3  0.0  3.0  2.0  4.0
```

Only replace the first NaN element.

```pycon
>>> df.fillna(value=values, limit=1)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  NaN  1.0
2  NaN  1.0  NaN  3.0
3  NaN  3.0  NaN  4.0
```

When filling using a DataFrame, replacement happens along
the same column names and same indices

```pycon
>>> df2 = pd.DataFrame(np.zeros((4, 4)), columns=list("ABCE"))
>>> df.fillna(df2)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  NaN
3  0.0  3.0  0.0  4.0
```

Note that column D is not affected since it is not present in df2.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.floordiv.html.md

# dask.dataframe.Index.floordiv

#### Index.floordiv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.ge.html.md

# dask.dataframe.Index.ge

#### Index.ge(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.get_partition.html.md

# dask.dataframe.Index.get_partition

#### Index.get_partition(n)

Get a dask DataFrame/Series representing the nth partition.

* **Parameters:**
  **n**
  : The 0-indexed partition number to select.
* **Returns:**
  Dask DataFrame or Series
  : The same type as the original object.

#### SEE ALSO
[`DataFrame.partitions`](dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.groupby.html.md

# dask.dataframe.Index.groupby

#### Index.groupby(by, \*\*kwargs)

Group Series using a mapper or by a Series of columns.

This docstring was copied from pandas.Series.groupby.

Some inconsistencies with the Dask version may exist.

A groupby operation involves some combination of splitting the
object, applying a function, and combining the results. This can be
used to group large amounts of data and compute operations on these
groups.

* **Parameters:**
  **by**
  : Used to determine the groups for the groupby.
    If `by` is a function, it’s called on each value of the object’s
    index. If a dict or Series is passed, the Series or dict VALUES
    will be used to determine the groups (the Series’ values are first
    aligned; see `.align()` method). If a list or ndarray of length
    equal to the selected axis is passed (see the [groupby user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#splitting-an-object-into-groups)),
    the values are used as-is to determine the groups. A label or list
    of labels may be passed to group by the columns in `self`.
    Notice that a tuple is interpreted as a (single) key.

  **level**
  : If the axis is a MultiIndex (hierarchical), group by a particular
    level or levels. Do not specify both `by` and `level`.

  **as_index**
  : Return object with group labels as the
    index. Only relevant for DataFrame input. as_index=False is
    effectively “SQL-style” grouped output. This argument has no effect
    on filtrations (see the [filtrations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).

  **sort**
  : Sort group keys. Get better performance by turning this off.
    Note this does not influence the order of observations within each
    group. Groupby preserves the order of rows within each group. If False,
    the groups will appear in the same order as they did in the original DataFrame.
    This argument has no effect on filtrations (see the [filtrations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: Specifying `sort=False` with an ordered categorical grouper will no
    longer sort the values.

  **group_keys**
  : When calling apply and the `by` argument produces a like-indexed
    (i.e. [a transform](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#groupby-transform)) result, add group keys to
    index to identify pieces. By default group keys are not included
    when the result’s index (and column) labels match the inputs, and
    are included otherwise.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `True`.

  **observed**
  : This only applies if any of the groupers are Categoricals.
    If True: only show observed values for categorical groupers.
    If False: show all values for categorical groupers.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0: The default value is now `True`.

  **dropna**
  : If True, and if group keys contain NA values, NA values together
    with row/column will be dropped.
    If False, NA values will also be treated as the key in groups.
* **Returns:**
  pandas.api.typing.SeriesGroupBy
  : Returns a groupby object that contains information about the groups.

#### SEE ALSO
[`resample`](dask.dataframe.Index.resample.md#dask.dataframe.Index.resample)
: Convenience method for frequency conversion and resampling of time series.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/groupby.html) for more
detailed usage and examples, including splitting an object into groups,
iterating through groups, selecting a group, aggregation, and more.

The implementation of groupby is hash-based, meaning in particular that
objects that compare as equal will be considered to be in the same group.
An exception to this is that pandas has special handling of NA values:
any NA values will be collapsed to a single group, regardless of how
they compare. See the user guide linked above for more details.

### Examples

```pycon
>>> ser = pd.Series([390., 350., 30., 20.],
...                 index=['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...                 name="Max Speed")
>>> ser
Falcon    390.0
Falcon    350.0
Parrot     30.0
Parrot     20.0
Name: Max Speed, dtype: float64
```

We can pass a list of values to group the Series data by custom labels:

```pycon
>>> ser.groupby(["a", "b", "a", "b"]).mean()
a    210.0
b    185.0
Name: Max Speed, dtype: float64
```

Grouping by numeric labels yields similar results:

```pycon
>>> ser.groupby([0, 1, 0, 1]).mean()
0    210.0
1    185.0
Name: Max Speed, dtype: float64
```

We can group by a level of the index:

```pycon
>>> ser.groupby(level=0).mean()
Falcon    370.0
Parrot     25.0
Name: Max Speed, dtype: float64
```

We can group by a condition applied to the Series values:

```pycon
>>> ser.groupby(ser > 100).mean()
Max Speed
False     25.0
True     370.0
Name: Max Speed, dtype: float64
```

**Grouping by Indexes**

We can groupby different levels of a hierarchical index
using the level parameter:

```pycon
>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...           ['Captive', 'Wild', 'Captive', 'Wild']]
>>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
>>> ser = pd.Series([390., 350., 30., 20.], index=index, name="Max Speed")
>>> ser
Animal  Type
Falcon  Captive    390.0
        Wild       350.0
Parrot  Captive     30.0
        Wild        20.0
Name: Max Speed, dtype: float64
```

```pycon
>>> ser.groupby(level=0).mean()
Animal
Falcon    370.0
Parrot     25.0
Name: Max Speed, dtype: float64
```

We can also group by the ‘Type’ level of the hierarchical index
to get the mean speed for each type:

```pycon
>>> ser.groupby(level="Type").mean()
Type
Captive    210.0
Wild       185.0
Name: Max Speed, dtype: float64
```

We can also choose to include NA in group keys or not by defining
dropna parameter, the default setting is True.

```pycon
>>> ser = pd.Series([1, 2, 3, 3], index=["a", 'a', 'b', np.nan])
>>> ser.groupby(level=0).sum()
a    3
b    3
dtype: int64
```

To include NA values in the group keys, set dropna=False:

```pycon
>>> ser.groupby(level=0, dropna=False).sum()
a    3
b    3
NaN  3
dtype: int64
```

We can also group by a custom list with NaN values to handle
missing group labels:

```pycon
>>> arrays = ['Falcon', 'Falcon', 'Parrot', 'Parrot']
>>> ser = pd.Series([390., 350., 30., 20.], index=arrays, name="Max Speed")
>>> ser.groupby(["a", "b", "a", np.nan]).mean()
a    210.0
b    350.0
Name: Max Speed, dtype: float64
```

```pycon
>>> ser.groupby(["a", "b", "a", np.nan], dropna=False).mean()
a    210.0
b    350.0
NaN   20.0
Name: Max Speed, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.gt.html.md

# dask.dataframe.Index.gt

#### Index.gt(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.head.html.md

# dask.dataframe.Index.head

#### Index.head(n: [int](https://docs.python.org/3/library/functions.html#int) = 5, npartitions=1, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

First n rows of the dataset

* **Parameters:**
  **n**
  : The number of rows to return. Default is 5.

  **npartitions**
  : Elements are only taken from the first `npartitions`, with a
    default of 1. If there are fewer than `n` rows in the first
    `npartitions` a warning will be raised and any found rows
    returned. Pass -1 to use all partitions.

  **compute**
  : Whether to compute the result, default is True.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.html.md

# dask.dataframe.Index

### *class* dask.dataframe.Index(expr)

Index-like Expr Collection.

The constructor takes the expression that represents the query as input. The class
is not meant to be instantiated directly. Instead, use one of the IO connectors from
Dask.

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(expr)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.dataframe.Index.__init__)(expr)                                                                                                |                                                                              |
|---------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| `abs`()                                                                                                                                           | Return a Series/DataFrame with absolute numeric value of each element.       |
| [`add`](dask.dataframe.Index.add.md#dask.dataframe.Index.add)(other[, level, fill_value, axis])                                                   |                                                                              |
| `add_prefix`(prefix)                                                                                                                              | Prefix labels with string prefix.                                            |
| `add_suffix`(suffix)                                                                                                                              | Suffix labels with string suffix.                                            |
| [`align`](dask.dataframe.Index.align.md#dask.dataframe.Index.align)(other[, join, axis, fill_value])                                              | Align two objects on their axes with the specified join method.              |
| [`all`](dask.dataframe.Index.all.md#dask.dataframe.Index.all)([axis, skipna, split_every])                                                        | Return whether all elements are True, potentially over an axis.              |
| `analyze`([filename, format])                                                                                                                     | Outputs statistics about every node in the expression.                       |
| [`any`](dask.dataframe.Index.any.md#dask.dataframe.Index.any)([axis, skipna, split_every])                                                        | Return whether any element is True, potentially over an axis.                |
| [`apply`](dask.dataframe.Index.apply.md#dask.dataframe.Index.apply)(function, \*args[, meta, axis])                                               | Parallel version of pandas.Series.apply                                      |
| [`astype`](dask.dataframe.Index.astype.md#dask.dataframe.Index.astype)(dtypes)                                                                    | Cast a pandas object to a specified dtype `dtype`.                           |
| [`autocorr`](dask.dataframe.Index.autocorr.md#dask.dataframe.Index.autocorr)([lag, split_every])                                                  | Compute the lag-N autocorrelation.                                           |
| [`between`](dask.dataframe.Index.between.md#dask.dataframe.Index.between)(left, right[, inclusive])                                               | Return boolean Series equivalent to left <= series <= right.                 |
| [`bfill`](dask.dataframe.Index.bfill.md#dask.dataframe.Index.bfill)([axis, limit])                                                                | Fill NA/NaN values by using the next valid observation to fill the gap.      |
| `case_when`(caselist)                                                                                                                             | Replace values where the conditions are True.                                |
| [`clear_divisions`](dask.dataframe.Index.clear_divisions.md#dask.dataframe.Index.clear_divisions)()                                               | Forget division information.                                                 |
| [`clip`](dask.dataframe.Index.clip.md#dask.dataframe.Index.clip)([lower, upper, axis])                                                            | Trim values at input threshold(s).                                           |
| `combine`(other, func[, fill_value])                                                                                                              | Combine the Series with a Series or scalar according to func.                |
| `combine_first`(other)                                                                                                                            | Update null elements with value in the same location in other.               |
| [`compute`](dask.dataframe.Index.compute.md#dask.dataframe.Index.compute)(\*\*kwargs)                                                             | Compute this dask collection                                                 |
| `compute_current_divisions`([col, set_divisions])                                                                                                 | Compute the current divisions of the DataFrame.                              |
| [`copy`](dask.dataframe.Index.copy.md#dask.dataframe.Index.copy)([deep])                                                                          | Make a copy of the dataframe                                                 |
| [`corr`](dask.dataframe.Index.corr.md#dask.dataframe.Index.corr)(other[, method, min_periods, split_every])                                       | Compute correlation with other Series, excluding missing values.             |
| [`count`](dask.dataframe.Index.count.md#dask.dataframe.Index.count)([split_every])                                                                | Count non-NA cells for each column or row.                                   |
| [`cov`](dask.dataframe.Index.cov.md#dask.dataframe.Index.cov)(other[, min_periods, split_every])                                                  | Compute covariance with Series, excluding missing values.                    |
| [`cummax`](dask.dataframe.Index.cummax.md#dask.dataframe.Index.cummax)([axis, skipna])                                                            | Return cumulative maximum over a DataFrame or Series axis.                   |
| [`cummin`](dask.dataframe.Index.cummin.md#dask.dataframe.Index.cummin)([axis, skipna])                                                            | Return cumulative minimum over a DataFrame or Series axis.                   |
| [`cumprod`](dask.dataframe.Index.cumprod.md#dask.dataframe.Index.cumprod)([axis, skipna])                                                         | Return cumulative product over a DataFrame or Series axis.                   |
| [`cumsum`](dask.dataframe.Index.cumsum.md#dask.dataframe.Index.cumsum)([axis, skipna])                                                            | Return cumulative sum over a DataFrame or Series axis.                       |
| [`describe`](dask.dataframe.Index.describe.md#dask.dataframe.Index.describe)([split_every, percentiles, ...])                                     | Generate descriptive statistics.                                             |
| [`diff`](dask.dataframe.Index.diff.md#dask.dataframe.Index.diff)([periods, axis])                                                                 | First discrete difference of element.                                        |
| [`div`](dask.dataframe.Index.div.md#dask.dataframe.Index.div)(other[, level, fill_value, axis])                                                   |                                                                              |
| `divide`(other[, level, fill_value, axis])                                                                                                        |                                                                              |
| `dot`(other[, meta])                                                                                                                              | Compute the dot product between the Series and the columns of other.         |
| [`drop_duplicates`](dask.dataframe.Index.drop_duplicates.md#dask.dataframe.Index.drop_duplicates)([ignore_index, split_every, ...])               |                                                                              |
| [`dropna`](dask.dataframe.Index.dropna.md#dask.dataframe.Index.dropna)()                                                                          | Return a new Series with missing values removed.                             |
| `enforce_runtime_divisions`()                                                                                                                     | Enforce the current divisions at runtime.                                    |
| [`eq`](dask.dataframe.Index.eq.md#dask.dataframe.Index.eq)(other[, level, fill_value, axis])                                                      |                                                                              |
| `explain`([stage, format])                                                                                                                        | Create a graph representation of the Expression.                             |
| [`explode`](dask.dataframe.Index.explode.md#dask.dataframe.Index.explode)()                                                                       | Transform each element of a list-like to a row.                              |
| [`ffill`](dask.dataframe.Index.ffill.md#dask.dataframe.Index.ffill)([axis, limit])                                                                | Fill NA/NaN values by propagating the last valid observation to next valid.  |
| [`fillna`](dask.dataframe.Index.fillna.md#dask.dataframe.Index.fillna)([value, axis])                                                             | Fill NA/NaN values with value.                                               |
| [`floordiv`](dask.dataframe.Index.floordiv.md#dask.dataframe.Index.floordiv)(other[, level, fill_value, axis])                                    |                                                                              |
| `from_dict`(data, \*[, npartitions, orient, ...])                                                                                                 | Construct a Dask DataFrame from a Python Dictionary                          |
| [`ge`](dask.dataframe.Index.ge.md#dask.dataframe.Index.ge)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`get_partition`](dask.dataframe.Index.get_partition.md#dask.dataframe.Index.get_partition)(n)                                                    | Get a dask DataFrame/Series representing the nth partition.                  |
| [`groupby`](dask.dataframe.Index.groupby.md#dask.dataframe.Index.groupby)(by, \*\*kwargs)                                                         | Group Series using a mapper or by a Series of columns.                       |
| [`gt`](dask.dataframe.Index.gt.md#dask.dataframe.Index.gt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`head`](dask.dataframe.Index.head.md#dask.dataframe.Index.head)([n, npartitions, compute])                                                       | First n rows of the dataset                                                  |
| `idxmax`(\*args, \*\*kwargs)                                                                                                                      | Return index of first occurrence of maximum over requested axis.             |
| `idxmin`(\*args, \*\*kwargs)                                                                                                                      | Return index of first occurrence of minimum over requested axis.             |
| [`isin`](dask.dataframe.Index.isin.md#dask.dataframe.Index.isin)(values)                                                                          | Whether each element in the DataFrame is contained in values.                |
| [`isna`](dask.dataframe.Index.isna.md#dask.dataframe.Index.isna)()                                                                                | Detect missing values.                                                       |
| [`isnull`](dask.dataframe.Index.isnull.md#dask.dataframe.Index.isnull)()                                                                          | DataFrame.isnull is an alias for DataFrame.isna.                             |
| `kurt`([axis, fisher, bias, nan_policy, ...])                                                                                                     | Return unbiased kurtosis over requested axis.                                |
| `kurtosis`([axis, fisher, bias, nan_policy, ...])                                                                                                 | Return unbiased kurtosis over requested axis.                                |
| [`le`](dask.dataframe.Index.le.md#dask.dataframe.Index.le)(other[, level, fill_value, axis])                                                      |                                                                              |
| `lower_once`()                                                                                                                                    |                                                                              |
| [`lt`](dask.dataframe.Index.lt.md#dask.dataframe.Index.lt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`map`](dask.dataframe.Index.map.md#dask.dataframe.Index.map)(arg[, na_action, meta, is_monotonic])                                               | Map values using an input mapping or function.                               |
| [`map_overlap`](dask.dataframe.Index.map_overlap.md#dask.dataframe.Index.map_overlap)(func, before, after, \*args[, ...])                         | Apply a function to each partition, sharing rows with adjacent partitions.   |
| [`map_partitions`](dask.dataframe.Index.map_partitions.md#dask.dataframe.Index.map_partitions)(func, \*args[, meta, ...])                         | Apply a Python function to each partition                                    |
| [`mask`](dask.dataframe.Index.mask.md#dask.dataframe.Index.mask)(cond[, other])                                                                   | Replace values where the condition is True.                                  |
| [`max`](dask.dataframe.Index.max.md#dask.dataframe.Index.max)([axis, skipna, numeric_only, split_every])                                          | Return the maximum of the values over the requested axis.                    |
| `mean`(\*args, \*\*kwargs)                                                                                                                        | Return the mean of the values over the requested axis.                       |
| [`median`](dask.dataframe.Index.median.md#dask.dataframe.Index.median)()                                                                          | Return the median of the values over the requested axis.                     |
| [`median_approximate`](dask.dataframe.Index.median_approximate.md#dask.dataframe.Index.median_approximate)([method])                              | Return the approximate median of the values over the requested axis.         |
| [`memory_usage`](dask.dataframe.Index.memory_usage.md#dask.dataframe.Index.memory_usage)([deep])                                                  | Memory usage of the values.                                                  |
| [`memory_usage_per_partition`](dask.dataframe.Index.memory_usage_per_partition.md#dask.dataframe.Index.memory_usage_per_partition)([index, deep]) | Return the memory usage of each partition                                    |
| [`min`](dask.dataframe.Index.min.md#dask.dataframe.Index.min)([axis, skipna, numeric_only, split_every])                                          | Return the minimum of the values over the requested axis.                    |
| [`mod`](dask.dataframe.Index.mod.md#dask.dataframe.Index.mod)(other[, level, fill_value, axis])                                                   |                                                                              |
| `mode`([dropna, split_every])                                                                                                                     | Return the mode(s) of the Series.                                            |
| [`mul`](dask.dataframe.Index.mul.md#dask.dataframe.Index.mul)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`ne`](dask.dataframe.Index.ne.md#dask.dataframe.Index.ne)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`nlargest`](dask.dataframe.Index.nlargest.md#dask.dataframe.Index.nlargest)([n, split_every])                                                    | Return the largest n elements.                                               |
| [`notnull`](dask.dataframe.Index.notnull.md#dask.dataframe.Index.notnull)()                                                                       | DataFrame.notnull is an alias for DataFrame.notna.                           |
| [`nsmallest`](dask.dataframe.Index.nsmallest.md#dask.dataframe.Index.nsmallest)([n, split_every])                                                 | Return the smallest n elements.                                              |
| [`nunique`](dask.dataframe.Index.nunique.md#dask.dataframe.Index.nunique)([dropna, split_every, split_out])                                       | Return number of unique elements in the object.                              |
| [`nunique_approx`](dask.dataframe.Index.nunique_approx.md#dask.dataframe.Index.nunique_approx)([split_every])                                     | Approximate number of unique rows.                                           |
| `optimize`([fuse])                                                                                                                                | Optimizes the DataFrame.                                                     |
| [`persist`](dask.dataframe.Index.persist.md#dask.dataframe.Index.persist)([fuse])                                                                 | Persist this dask collection into memory                                     |
| [`pipe`](dask.dataframe.Index.pipe.md#dask.dataframe.Index.pipe)(func, \*args, \*\*kwargs)                                                        | Apply chainable functions that expect Series or DataFrames.                  |
| [`pow`](dask.dataframe.Index.pow.md#dask.dataframe.Index.pow)(other[, level, fill_value, axis])                                                   |                                                                              |
| `pprint`()                                                                                                                                        | Outputs a string representation of the DataFrame.                            |
| `prod`(\*args, \*\*kwargs)                                                                                                                        | Return the product of the values over the requested axis.                    |
| `product`([axis, skipna, numeric_only, ...])                                                                                                      | Return the product of the values over the requested axis.                    |
| [`quantile`](dask.dataframe.Index.quantile.md#dask.dataframe.Index.quantile)([q, method])                                                         | Approximate quantiles of Series                                              |
| [`radd`](dask.dataframe.Index.radd.md#dask.dataframe.Index.radd)(other[, level, fill_value, axis])                                                |                                                                              |
| [`random_split`](dask.dataframe.Index.random_split.md#dask.dataframe.Index.random_split)(frac[, random_state, shuffle])                           | Pseudorandomly split dataframe into different pieces row-wise                |
| [`rdiv`](dask.dataframe.Index.rdiv.md#dask.dataframe.Index.rdiv)(other[, level, fill_value, axis])                                                |                                                                              |
| `reduction`(chunk[, aggregate, combine, meta, ...])                                                                                               | Generic row-wise reductions.                                                 |
| [`rename`](dask.dataframe.Index.rename.md#dask.dataframe.Index.rename)(index[, sorted_index])                                                     | Alter Series index labels or name                                            |
| `rename_axis`([mapper, index, columns, axis])                                                                                                     | Set the name of the axis for the index or columns.                           |
| [`repartition`](dask.dataframe.Index.repartition.md#dask.dataframe.Index.repartition)([divisions, npartitions, ...])                              | Repartition a collection                                                     |
| [`replace`](dask.dataframe.Index.replace.md#dask.dataframe.Index.replace)([to_replace, value, regex])                                             | Replace values given in to_replace with value.                               |
| [`resample`](dask.dataframe.Index.resample.md#dask.dataframe.Index.resample)(rule[, closed, label])                                               | Resample time-series data.                                                   |
| [`reset_index`](dask.dataframe.Index.reset_index.md#dask.dataframe.Index.reset_index)([drop])                                                     | Reset the index to the default index.                                        |
| `rfloordiv`(other[, level, fill_value, axis])                                                                                                     |                                                                              |
| `rmod`(other[, level, fill_value, axis])                                                                                                          |                                                                              |
| `rmul`(other[, level, fill_value, axis])                                                                                                          |                                                                              |
| [`rolling`](dask.dataframe.Index.rolling.md#dask.dataframe.Index.rolling)(window, \*\*kwargs)                                                     | Provides rolling transformations.                                            |
| [`round`](dask.dataframe.Index.round.md#dask.dataframe.Index.round)([decimals])                                                                   | Round numeric columns in a DataFrame to a variable number of decimal places. |
| `rpow`(other[, level, fill_value, axis])                                                                                                          |                                                                              |
| `rsub`(other[, level, fill_value, axis])                                                                                                          |                                                                              |
| `rtruediv`(other[, level, fill_value, axis])                                                                                                      |                                                                              |
| [`sample`](dask.dataframe.Index.sample.md#dask.dataframe.Index.sample)([n, frac, replace, random_state])                                          | Random sample of items                                                       |
| [`sem`](dask.dataframe.Index.sem.md#dask.dataframe.Index.sem)([axis, skipna, ddof, split_every, ...])                                             | Return unbiased standard error of the mean over requested axis.              |
| [`shift`](dask.dataframe.Index.shift.md#dask.dataframe.Index.shift)([periods, freq])                                                              | Shift index by desired number of periods with an optional time freq.         |
| `shuffle`([on, ignore_index, npartitions, ...])                                                                                                   | Rearrange DataFrame into new partitions                                      |
| `simplify`()                                                                                                                                      |                                                                              |
| `skew`([axis, bias, nan_policy, numeric_only])                                                                                                    | Return unbiased skew over requested axis.                                    |
| `squeeze`()                                                                                                                                       | Squeeze 1 dimensional axis objects into scalars.                             |
| `std`(\*args, \*\*kwargs)                                                                                                                         | Return sample standard deviation over requested axis.                        |
| [`sub`](dask.dataframe.Index.sub.md#dask.dataframe.Index.sub)(other[, level, fill_value, axis])                                                   |                                                                              |
| `sum`(\*args, \*\*kwargs)                                                                                                                         | Return the sum of the values over the requested axis.                        |
| `tail`([n, compute])                                                                                                                              | Last n rows of the dataset                                                   |
| [`to_backend`](dask.dataframe.Index.to_backend.md#dask.dataframe.Index.to_backend)([backend])                                                     | Move to a new DataFrame backend                                              |
| [`to_bag`](dask.dataframe.Index.to_bag.md#dask.dataframe.Index.to_bag)([index, format])                                                           | Create a Dask Bag from a Series                                              |
| [`to_csv`](dask.dataframe.Index.to_csv.md#dask.dataframe.Index.to_csv)(filename, \*\*kwargs)                                                      | See dd.to_csv docstring for more information                                 |
| [`to_dask_array`](dask.dataframe.Index.to_dask_array.md#dask.dataframe.Index.to_dask_array)([lengths, meta, optimize])                            | Convert a dask DataFrame to a dask array.                                    |
| [`to_delayed`](dask.dataframe.Index.to_delayed.md#dask.dataframe.Index.to_delayed)([optimize_graph])                                              | Convert into a list of `dask.delayed` objects, one per partition.            |
| [`to_frame`](dask.dataframe.Index.to_frame.md#dask.dataframe.Index.to_frame)([index, name])                                                       | Create a DataFrame with a column containing the Index.                       |
| [`to_hdf`](dask.dataframe.Index.to_hdf.md#dask.dataframe.Index.to_hdf)(path_or_buf, key[, mode, append])                                          | See dd.to_hdf docstring for more information                                 |
| `to_json`(filename, \*args, \*\*kwargs)                                                                                                           | See dd.to_json docstring for more information                                |
| `to_orc`(path, \*args, \*\*kwargs)                                                                                                                | See dd.to_orc docstring for more information                                 |
| `to_records`([index, lengths])                                                                                                                    |                                                                              |
| [`to_series`](dask.dataframe.Index.to_series.md#dask.dataframe.Index.to_series)([index, name])                                                    | Create a Series with both index and values equal to the index keys.          |
| `to_sql`(name, uri[, schema, if_exists, ...])                                                                                                     |                                                                              |
| [`to_string`](dask.dataframe.Index.to_string.md#dask.dataframe.Index.to_string)([max_rows])                                                       | Render a string representation of the Series.                                |
| [`to_timestamp`](dask.dataframe.Index.to_timestamp.md#dask.dataframe.Index.to_timestamp)([freq, how])                                             | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.   |
| [`truediv`](dask.dataframe.Index.truediv.md#dask.dataframe.Index.truediv)(other[, level, fill_value, axis])                                       |                                                                              |
| [`unique`](dask.dataframe.Index.unique.md#dask.dataframe.Index.unique)([split_every, split_out, shuffle_method])                                  | Return Series of unique values in the object.                                |
| [`value_counts`](dask.dataframe.Index.value_counts.md#dask.dataframe.Index.value_counts)([sort, ascending, dropna, ...])                          | Return a Series containing counts of unique values.                          |
| `var`(\*args, \*\*kwargs)                                                                                                                         | Return unbiased variance over requested axis.                                |
| [`visualize`](dask.dataframe.Index.visualize.md#dask.dataframe.Index.visualize)([tasks])                                                          | Visualize the expression or task graph                                       |
| [`where`](dask.dataframe.Index.where.md#dask.dataframe.Index.where)(cond[, other])                                                                | Replace values where the condition is False.                                 |

### Attributes

| `axes`                                                                                                                    |                                                                                                                  |
|---------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| `columns`                                                                                                                 |                                                                                                                  |
| `dask`                                                                                                                    |                                                                                                                  |
| `divisions`                                                                                                               | Tuple of `npartitions + 1` values, in ascending order, marking the lower/upper bounds of each partition's index. |
| [`dtype`](dask.dataframe.Index.dtype.md#dask.dataframe.Index.dtype)                                                       |                                                                                                                  |
| `dtypes`                                                                                                                  | Return data types                                                                                                |
| `expr`                                                                                                                    |                                                                                                                  |
| `index`                                                                                                                   | Return dask Index instance                                                                                       |
| [`is_monotonic_decreasing`](dask.dataframe.Index.is_monotonic_decreasing.md#dask.dataframe.Index.is_monotonic_decreasing) | Return True if values in the object are monotonically decreasing.                                                |
| [`is_monotonic_increasing`](dask.dataframe.Index.is_monotonic_increasing.md#dask.dataframe.Index.is_monotonic_increasing) | Return True if values in the object are monotonically increasing.                                                |
| [`known_divisions`](dask.dataframe.Index.known_divisions.md#dask.dataframe.Index.known_divisions)                         | Whether the divisions are known.                                                                                 |
| [`loc`](dask.dataframe.Index.loc.md#dask.dataframe.Index.loc)                                                             | Purely label-location based indexer for selection by label.                                                      |
| `name`                                                                                                                    |                                                                                                                  |
| [`nbytes`](dask.dataframe.Index.nbytes.md#dask.dataframe.Index.nbytes)                                                    | Number of bytes                                                                                                  |
| [`ndim`](dask.dataframe.Index.ndim.md#dask.dataframe.Index.ndim)                                                          | Return dimensionality                                                                                            |
| `npartitions`                                                                                                             | Return number of partitions                                                                                      |
| `partitions`                                                                                                              | Slice dataframe by partitions                                                                                    |
| [`shape`](dask.dataframe.Index.shape.md#dask.dataframe.Index.shape)                                                       | Return a tuple representing the dimensionality of the DataFrame.                                                 |
| [`size`](dask.dataframe.Index.size.md#dask.dataframe.Index.size)                                                          | Size of the Series or DataFrame as a Delayed object.                                                             |
| [`values`](dask.dataframe.Index.values.md#dask.dataframe.Index.values)                                                    | Return a dask.array of the values of this dataframe                                                              |


# dask.dataframe.Index.is_monotonic_decreasing.html.md

# dask.dataframe.Index.is_monotonic_decreasing

#### *property* Index.is_monotonic_decreasing

Return True if values in the object are monotonically decreasing.

This docstring was copied from pandas.Series.is_monotonic_decreasing.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  bool

#### SEE ALSO
`Series.is_monotonic_increasing`
: Return boolean if values in the object are monotonically increasing.

### Examples

```pycon
>>> s = pd.Series([3, 2, 2, 1])
>>> s.is_monotonic_decreasing
True
```

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.is_monotonic_decreasing
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.is_monotonic_increasing.html.md

# dask.dataframe.Index.is_monotonic_increasing

#### *property* Index.is_monotonic_increasing

Return True if values in the object are monotonically increasing.

This docstring was copied from pandas.Series.is_monotonic_increasing.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  bool

#### SEE ALSO
`Series.is_monotonic_decreasing`
: Return boolean if values in the object are monotonically decreasing.

### Examples

```pycon
>>> s = pd.Series([1, 2, 2])
>>> s.is_monotonic_increasing
True
```

```pycon
>>> s = pd.Series([3, 2, 1])
>>> s.is_monotonic_increasing
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.isin.html.md

# dask.dataframe.Index.isin

#### Index.isin(values)

Whether each element in the DataFrame is contained in values.

This docstring was copied from pandas.DataFrame.isin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **values**
  : The result will only be true at a location if all the
    labels match. If values is a Series, that’s the index. If
    values is a dict, the keys must be the column names,
    which must match. If values is a DataFrame,
    then both the index and column labels must match.
* **Returns:**
  DataFrame
  : DataFrame of booleans showing whether each element in the DataFrame
    is contained in values.

#### SEE ALSO
[`DataFrame.eq`](dask.dataframe.DataFrame.eq.md#dask.dataframe.DataFrame.eq)
: Equality test for DataFrame.

[`Series.isin`](dask.dataframe.Series.isin.md#dask.dataframe.Series.isin)
: Equivalent method on Series.

[`Series.str.contains`](dask.dataframe.Series.str.contains.md#dask.dataframe.Series.str.contains)
: Test if pattern or regex is contained within a string of a Series or Index.

### Notes

`__iter__` is used (and not `__contains__`) to iterate over values
when checking if it contains the elements in DataFrame.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"num_legs": [2, 4], "num_wings": [2, 0]}, index=["falcon", "dog"]
... )
>>> df
        num_legs  num_wings
falcon         2          2
dog            4          0
```

When `values` is a list check whether every value in the DataFrame
is present in the list (which animals have 0 or 2 legs or wings)

```pycon
>>> df.isin([0, 2])
        num_legs  num_wings
falcon      True       True
dog        False       True
```

To check if `values` is *not* in the DataFrame, use the `~` operator:

```pycon
>>> ~df.isin([0, 2])
        num_legs  num_wings
falcon     False      False
dog         True      False
```

When `values` is a dict, we can pass values to check for each
column separately:

```pycon
>>> df.isin({"num_wings": [0, 3]})
        num_legs  num_wings
falcon     False      False
dog        False       True
```

When `values` is a Series or DataFrame the index and column must
match. Note that ‘falcon’ does not match based on the number of legs
in other.

```pycon
>>> other = pd.DataFrame(
...     {"num_legs": [8, 3], "num_wings": [0, 2]}, index=["spider", "falcon"]
... )
>>> df.isin(other)
        num_legs  num_wings
falcon     False       True
dog        False      False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.isna.html.md

# dask.dataframe.Index.isna

#### Index.isna()

Detect missing values.

This docstring was copied from pandas.DataFrame.isna.

Some inconsistencies with the Dask version may exist.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](#dask.dataframe.Index.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.isnull.html.md

# dask.dataframe.Index.isnull

#### Index.isnull()

DataFrame.isnull is an alias for DataFrame.isna.

This docstring was copied from pandas.DataFrame.isnull.

Some inconsistencies with the Dask version may exist.

Detect missing values.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](dask.dataframe.Index.isna.md#dask.dataframe.Index.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.known_divisions.html.md

# dask.dataframe.Index.known_divisions

#### *property* Index.known_divisions

Whether the divisions are known.

This check can be expensive if the division calculation is expensive.
DataFrame.set_index is a good example where the calculation needs an
inspection of the data.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.le.html.md

# dask.dataframe.Index.le

#### Index.le(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.loc.html.md

# dask.dataframe.Index.loc

#### *property* Index.loc

Purely label-location based indexer for selection by label.

```pycon
>>> df.loc["b"]
>>> df.loc["b":"d"]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.lt.html.md

# dask.dataframe.Index.lt

#### Index.lt(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.map.html.md

# dask.dataframe.Index.map

#### Index.map(arg, na_action=None, meta=None, is_monotonic=False)

Map values using an input mapping or function.

This docstring was copied from pandas.Index.map.

Some inconsistencies with the Dask version may exist.

Note that this method clears any known divisions.

If your mapping function is monotonically increasing then use is_monotonic
to apply the mapping function to the old divisions and assign the new
divisions to the output.

* **Parameters:**
  **mapper**
  : Mapping correspondence.

  **na_action**
  : If ‘ignore’, propagate NA values, without passing them to the
    mapping correspondence.
* **Returns:**
  Union[Index, MultiIndex]
  : The output of the mapping function applied to the index.
    If the function returns a tuple with more than one element
    a MultiIndex will be returned.

#### SEE ALSO
[`Index.where`](dask.dataframe.Index.where.md#dask.dataframe.Index.where)
: Replace values where the condition is False.

### Examples

```pycon
>>> idx = pd.Index([1, 2, 3])
>>> idx.map({1: "a", 2: "b", 3: "c"})
Index(['a', 'b', 'c'], dtype='str')
```

Using map with a function:

```pycon
>>> idx = pd.Index([1, 2, 3])
>>> idx.map("I am a {}".format)
Index(['I am a 1', 'I am a 2', 'I am a 3'], dtype='str')
```

```pycon
>>> idx = pd.Index(["a", "b", "c"])
>>> idx.map(lambda x: x.upper())
Index(['A', 'B', 'C'], dtype='str')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.map_overlap.html.md

# dask.dataframe.Index.map_overlap

#### Index.map_overlap(func, before, after, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, \*\*kwargs)

Apply a function to each partition, sharing rows with adjacent partitions.

This can be useful for implementing windowing functions such as
`df.rolling(...).mean()` or `df.diff()`.

* **Parameters:**
  **func**
  : Function applied to each partition.

  **before**
  : The rows to prepend to partition `i` from the end of
    partition `i - 1`.

  **after**
  : The rows to append to partition `i` from the beginning
    of partition `i + 1`.

  **args, kwargs**
  : Positional and keyword arguments to pass to the function.
    Positional arguments are computed on a per-partition basis, while
    keyword arguments are shared across all partitions. The partition
    itself will be the first positional argument, with all other
    arguments passed *after*. Arguments can be `Scalar`, `Delayed`,
    or regular Python objects. DataFrame-like args (both dask and
    pandas) will be repartitioned to align (if necessary) before
    applying the function; see `align_dataframes` to control this
    behavior.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition,
    and will raise an error if this doesn’t work,
    but it won’t raise if dtypes don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **align_dataframes**
  : Whether to repartition DataFrame- or Series-like args
    (both dask and pandas) so their divisions align before applying
    the function. This requires all inputs to have known divisions.
    Single-partition inputs will be split into multiple partitions.
    <br/>
    If False, all inputs must have either the same number of partitions
    or a single partition. Single-partition inputs will be broadcast to
    every partition of multi-partition inputs.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

### Notes

Given positive integers `before` and `after`, and a function
`func`, `map_overlap` does the following:

1. Prepend `before` rows to each partition `i` from the end of
   partition `i - 1`. The first partition has no rows prepended.
2. Append `after` rows to each partition `i` from the beginning of
   partition `i + 1`. The last partition has no rows appended.
3. Apply `func` to each partition, passing in any extra `args` and
   `kwargs` if provided.
4. Trim `before` rows from the beginning of all but the first
   partition.
5. Trim `after` rows from the end of all but the last partition.

### Examples

Given a DataFrame, Series, or Index, such as:

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 4, 7, 11],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

A rolling sum with a trailing moving window of size 2 can be computed by
overlapping 2 rows before each partition, and then mapping calls to
`df.rolling(2).sum()`:

```pycon
>>> ddf.compute()
    x    y
0   1  1.0
1   2  2.0
2   4  3.0
3   7  4.0
4  11  5.0
>>> ddf.map_overlap(lambda df: df.rolling(2).sum(), 2, 0).compute()
      x    y
0   NaN  NaN
1   3.0  3.0
2   6.0  5.0
3  11.0  7.0
4  18.0  9.0
```

The pandas `diff` method computes a discrete difference shifted by a
number of periods (can be positive or negative). This can be
implemented by mapping calls to `df.diff` to each partition after
prepending/appending that many rows, depending on sign:

```pycon
>>> def diff(df, periods=1):
...     before, after = (periods, 0) if periods > 0 else (0, -periods)
...     return df.map_overlap(lambda df, periods=1: df.diff(periods),
...                           periods, 0, periods=periods)
>>> diff(ddf, 1).compute()
     x    y
0  NaN  NaN
1  1.0  1.0
2  2.0  1.0
3  3.0  1.0
4  4.0  1.0
```

If you have a `DatetimeIndex`, you can use a `pd.Timedelta` for time-
based windows or any `pd.Timedelta` convertible string:

```pycon
>>> ts = pd.Series(range(10), index=pd.date_range('2017', periods=10))
>>> dts = dd.from_pandas(ts, npartitions=2)
>>> dts.map_overlap(lambda df: df.rolling('2D').sum(),
...                 pd.Timedelta('2D'), 0).compute()
2017-01-01     0.0
2017-01-02     1.0
2017-01-03     3.0
2017-01-04     5.0
2017-01-05     7.0
2017-01-06     9.0
2017-01-07    11.0
2017-01-08    13.0
2017-01-09    15.0
2017-01-10    17.0
Freq: D, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.map_partitions.html.md

# dask.dataframe.Index.map_partitions

#### Index.map_partitions(func, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, parent_meta=None, required_columns=None, \*\*kwargs)

Apply a Python function to each partition

* **Parameters:**
  **func**
  : Function applied to each partition.

  **args, kwargs**
  : Arguments and keywords to pass to the function. Arguments and
    keywords may contain `FrameBase` or regular python objects.
    DataFrame-like args (both dask and pandas) must have the same
    number of partitions as `self` or comprise a single partition.
    Key-word arguments, Single-partition arguments, and general
    python-object arguments will be broadcasted to all partitions.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition, and will
    raise an error if this doesn’t work, but it won’t raise if dtypes
    don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **clear_divisions**
  : Whether divisions should be cleared. If True, transform_divisions
    will be ignored.

  **required_columns**
  : List of columns that `func` requires for execution. These columns
    must belong to the first DataFrame argument (in `args`). If None
    is specified (the default), the query optimizer will assume that
    all input columns are required.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

### Examples

Given a DataFrame, Series, or Index, such as:

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 3, 4, 5],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

One can use `map_partitions` to apply a function on each partition.
Extra arguments and keywords can optionally be provided, and will be
passed to the function after the partition.

Here we apply a function with arguments and keywords to a DataFrame,
resulting in a Series:

```pycon
>>> def myadd(df, a, b=1):
...     return df.x + df.y + a + b
>>> res = ddf.map_partitions(myadd, 1, b=2)
>>> res.dtype
dtype('float64')
```

Here we apply a function to a Series resulting in a Series:

```pycon
>>> res = ddf.x.map_partitions(lambda x: len(x)) # ddf.x is a Dask Series Structure
>>> res.dtype
dtype('int64')
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with no name, and dtype
`float64`:

```pycon
>>> res = ddf.map_partitions(myadd, 1, b=2, meta=(None, 'f8'))
```

Here we map a function that takes in a DataFrame, and returns a
DataFrame with a new column:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y))
>>> res.dtypes
x      int64
y    float64
z    float64
dtype: object
```

As before, the output metadata can also be specified manually. This
time we pass in a `dict`, as the output is a DataFrame:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y),
...                          meta={'x': 'i8', 'y': 'f8', 'z': 'f8'})
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ddf.map_partitions(lambda df: df.head(), meta=ddf)
```

Also note that the index and divisions are assumed to remain unchanged.
If the function you’re mapping changes the index/divisions, you’ll need
to pass `clear_divisions=True`.

```pycon
>>> ddf.map_partitions(func, clear_divisions=True)
```

Your map function gets information about where it is in the dataframe by
accepting a special `partition_info` keyword argument.

```pycon
>>> def func(partition, partition_info=None):
...     pass
```

This will receive the following information:

```pycon
>>> partition_info
{'number': 1, 'division': 3}
```

For each argument and keyword arguments that are dask dataframes you will
receive the number (n) which represents the nth partition of the dataframe
and the division (the first index value in the partition). If divisions
are not known (for instance if the index is not sorted) then you will get
None as the division.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.mask.html.md

# dask.dataframe.Index.mask

#### Index.mask(cond, other=nan)

Replace values where the condition is True.

This docstring was copied from pandas.DataFrame.mask.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **cond**
  : Where cond is False, keep the original value. Where
    True, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is True are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Return an object of same shape as caller.

[`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Return an object of same shape as caller.

### Notes

The mask method is an application of the if-then idiom. For each
element in the caller, if `cond` is `False` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with True.

The signature for [`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where) or
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `mask` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.max.html.md

# dask.dataframe.Index.max

#### Index.max(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the maximum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.max.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the maximum, use `idxmax`.
This is the equivalent of the `numpy.ndarray` method `argmax`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.max()
8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.median.html.md

# dask.dataframe.Index.median

#### Index.median()

Return the median of the values over the requested axis.

This docstring was copied from pandas.Series.median.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  scalar or Series (if level specified)
  : Median of the values for the requested axis.

#### SEE ALSO
[`numpy.median`](https://numpy.org/doc/stable/reference/generated/numpy.median.html#numpy.median)
: Equivalent numpy function for computing median.

[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Sum of the values.

[`Series.median`](dask.dataframe.Series.median.md#dask.dataframe.Series.median)
: Median of the values.

[`Series.std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)
: Standard deviation of the values.

[`Series.var`](dask.dataframe.Series.var.md#dask.dataframe.Series.var)
: Variance of the values.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Minimum value.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Maximum value.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.median()
2.0
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.median()
a   1.5
b   2.5
dtype: float64
```

Using axis=1

```pycon
>>> df.median(axis=1)
tiger   1.5
zebra   2.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.median(numeric_only=True)
a   1.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.median_approximate.html.md

# dask.dataframe.Index.median_approximate

#### Index.median_approximate(method='default')

Return the approximate median of the values over the requested axis.

* **Parameters:**
  **method**
  : What method to use. By default will use Dask’s internal custom
    algorithm (`"dask"`).  If set to `"tdigest"` will use tdigest
    for floats and ints and fallback to the `"dask"` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.memory_usage.html.md

# dask.dataframe.Index.memory_usage

#### Index.memory_usage(deep=False)

Memory usage of the values.

This docstring was copied from pandas.Index.memory_usage.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **deep**
  : Introspect the data deeply, interrogate
    object dtypes for system-level memory consumption.
* **Returns:**
  bytes used
  : Returns memory usage of the values in the Index in bytes.

#### SEE ALSO
[`numpy.ndarray.nbytes`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.nbytes.html#numpy.ndarray.nbytes)
: Total bytes consumed by the elements of the array.

### Notes

Memory usage does not include memory consumed by elements that
are not components of the array if deep=False or if used on PyPy

### Examples

```pycon
>>> idx = pd.Index([1, 2, 3])
>>> idx.memory_usage()
24
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.memory_usage_per_partition.html.md

# dask.dataframe.Index.memory_usage_per_partition

#### Index.memory_usage_per_partition(index: [bool](https://docs.python.org/3/library/functions.html#bool) = True, deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Return the memory usage of each partition

* **Parameters:**
  **index**
  : Specifies whether to include the memory usage of the index in
    returned Series.

  **deep**
  : If True, introspect the data deeply by interrogating
    `object` dtypes for system-level memory consumption, and include
    it in the returned values.
* **Returns:**
  Series
  : A Series whose index is the partition number and whose values
    are the memory usage of each partition in bytes.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.min.html.md

# dask.dataframe.Index.min

#### Index.min(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the minimum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.min.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the minimum, use `idxmin`.
This is the equivalent of the `numpy.ndarray` method `argmin`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.min()
0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.mod.html.md

# dask.dataframe.Index.mod

#### Index.mod(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.mul.html.md

# dask.dataframe.Index.mul

#### Index.mul(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.nbytes.html.md

# dask.dataframe.Index.nbytes

#### *property* Index.nbytes

Number of bytes

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.ndim.html.md

# dask.dataframe.Index.ndim

#### *property* Index.ndim

Return dimensionality

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.ne.html.md

# dask.dataframe.Index.ne

#### Index.ne(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.nlargest.html.md

# dask.dataframe.Index.nlargest

#### Index.nlargest(n=5, split_every=None)

Return the largest n elements.

This docstring was copied from pandas.Series.nlargest.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : Return this many descending sorted values.

  **keep**
  : When there are duplicate values that cannot all fit in a
    Series of n elements:
    - `first` : return the first n occurrences in order
      of appearance.
    - `last` : return the last n occurrences in reverse
      order of appearance.
    - `all` : keep all occurrences. This can result in a Series of
      size larger than n.
* **Returns:**
  Series
  : The n largest values in the Series, sorted in decreasing order.

#### SEE ALSO
[`Series.nsmallest`](dask.dataframe.Series.nsmallest.md#dask.dataframe.Series.nsmallest)
: Get the n smallest elements.

`Series.sort_values`
: Sort Series by values.

[`Series.head`](dask.dataframe.Series.head.md#dask.dataframe.Series.head)
: Return the first n rows.

### Notes

Faster than `.sort_values(ascending=False).head(n)` for small n
relative to the size of the `Series` object.

### Examples

```pycon
>>> countries_population = {
...     "Italy": 59000000,
...     "France": 65000000,
...     "Malta": 434000,
...     "Maldives": 434000,
...     "Brunei": 434000,
...     "Iceland": 337000,
...     "Nauru": 11300,
...     "Tuvalu": 11300,
...     "Anguilla": 11300,
...     "Montserrat": 5200,
... }
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Malta         434000
Maldives      434000
Brunei        434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64
```

The n largest elements where `n=5` by default.

```pycon
>>> s.nlargest()
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64
```

The n largest elements where `n=3`. Default keep value is ‘first’
so Malta will be kept.

```pycon
>>> s.nlargest(3)
France    65000000
Italy     59000000
Malta       434000
dtype: int64
```

The n largest elements where `n=3` and keeping the last duplicates.
Brunei will be kept since it is the last with value 434000 based on
the index order.

```pycon
>>> s.nlargest(3, keep="last")
France      65000000
Italy       59000000
Brunei        434000
dtype: int64
```

The n largest elements where `n=3` with all duplicates kept. Note
that the returned Series has five elements due to the three duplicates.

```pycon
>>> s.nlargest(3, keep="all")
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.notnull.html.md

# dask.dataframe.Index.notnull

#### Index.notnull()

DataFrame.notnull is an alias for DataFrame.notna.

This docstring was copied from pandas.DataFrame.notnull.

Some inconsistencies with the Dask version may exist.

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA.
Non-missing values get mapped to True. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.
NA values, such as None or `numpy.NaN`, get mapped to False
values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is not an NA value.

#### SEE ALSO
[`Series.notnull`](dask.dataframe.Series.notnull.md#dask.dataframe.Series.notnull)
: Alias of notna.

`DataFrame.notnull`
: Alias of notna.

[`Series.isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)
: Boolean inverse of notna.

[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Boolean inverse of notna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

`notna`
: Top-level notna.

### Examples

Show which entries in a DataFrame are not NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.notnull()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True
```

Show which entries in a Series are not NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.notnull()
0     True
1     True
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.nsmallest.html.md

# dask.dataframe.Index.nsmallest

#### Index.nsmallest(n=5, split_every=None)

Return the smallest n elements.

This docstring was copied from pandas.Series.nsmallest.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : Return this many ascending sorted values.

  **keep**
  : When there are duplicate values that cannot all fit in a
    Series of n elements:
    - `first` : return the first n occurrences in order
      of appearance.
    - `last` : return the last n occurrences in reverse
      order of appearance.
    - `all` : keep all occurrences. This can result in a Series of
      size larger than n.
* **Returns:**
  Series
  : The n smallest values in the Series, sorted in increasing order.

#### SEE ALSO
[`Series.nlargest`](dask.dataframe.Series.nlargest.md#dask.dataframe.Series.nlargest)
: Get the n largest elements.

`Series.sort_values`
: Sort Series by values.

[`Series.head`](dask.dataframe.Series.head.md#dask.dataframe.Series.head)
: Return the first n rows.

### Notes

Faster than `.sort_values().head(n)` for small n relative to
the size of the `Series` object.

### Examples

```pycon
>>> countries_population = {
...     "Italy": 59000000,
...     "France": 65000000,
...     "Brunei": 434000,
...     "Malta": 434000,
...     "Maldives": 434000,
...     "Iceland": 337000,
...     "Nauru": 11300,
...     "Tuvalu": 11300,
...     "Anguilla": 11300,
...     "Montserrat": 5200,
... }
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Brunei        434000
Malta         434000
Maldives      434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64
```

The n smallest elements where `n=5` by default.

```pycon
>>> s.nsmallest()
Montserrat    5200
Nauru        11300
Tuvalu       11300
Anguilla     11300
Iceland     337000
dtype: int64
```

The n smallest elements where `n=3`. Default keep value is
‘first’ so Nauru and Tuvalu will be kept.

```pycon
>>> s.nsmallest(3)
Montserrat   5200
Nauru       11300
Tuvalu      11300
dtype: int64
```

The n smallest elements where `n=3` and keeping the last
duplicates. Anguilla and Tuvalu will be kept since they are the last
with value 11300 based on the index order.

```pycon
>>> s.nsmallest(3, keep="last")
Montserrat   5200
Anguilla    11300
Tuvalu      11300
dtype: int64
```

The n smallest elements where `n=3` with all duplicates kept. Note
that the returned Series has four elements due to the three duplicates.

```pycon
>>> s.nsmallest(3, keep="all")
Montserrat   5200
Nauru       11300
Tuvalu      11300
Anguilla    11300
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.nunique.html.md

# dask.dataframe.Index.nunique

#### Index.nunique(dropna=True, split_every=False, split_out=True)

Return number of unique elements in the object.

This docstring was copied from pandas.Series.nunique.

Some inconsistencies with the Dask version may exist.

Excludes NA values by default.

* **Parameters:**
  **dropna**
  : Don’t include NaN in the count.
* **Returns:**
  int
  : An integer indicating the number of unique elements in the object.

#### SEE ALSO
`DataFrame.nunique`
: Method nunique for DataFrame.

[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Count non-NA/null observations in the Series.

### Examples

```pycon
>>> s = pd.Series([1, 3, 5, 7, 7])
>>> s
0    1
1    3
2    5
3    7
4    7
dtype: int64
```

```pycon
>>> s.nunique()
4
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.nunique_approx.html.md

# dask.dataframe.Index.nunique_approx

#### Index.nunique_approx(split_every=None)

Approximate number of unique rows.

This method uses the HyperLogLog algorithm for cardinality
estimation to compute the approximate number of unique rows.
The approximate error is 0.406%.

* **Parameters:**
  **split_every**
  : Group partitions into groups of this size while performing a
    tree-reduction. If set to False, no tree-reduction will be used.
    Default is 8.
* **Returns:**
  a float representing the approximate number of elements

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.persist.html.md

# dask.dataframe.Index.persist

#### Index.persist(fuse=True, \*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.pipe.html.md

# dask.dataframe.Index.pipe

#### Index.pipe(func, \*args, \*\*kwargs)

Apply chainable functions that expect Series or DataFrames.

This docstring was copied from pandas.DataFrame.pipe.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **func**
  : Function to apply to the Series/DataFrame.
    `args`, and `kwargs` are passed into `func`.
    Alternatively a `(callable, data_keyword)` tuple where
    `data_keyword` is a string indicating the keyword of
    `callable` that expects the Series/DataFrame.

  **\*args**
  : Positional arguments passed into `func`.

  **\*\*kwargs**
  : A dictionary of keyword arguments passed into `func`.
* **Returns:**
  The return type of `func`.
  : The result of applying `func` to the Series or DataFrame.

#### SEE ALSO
[`DataFrame.apply`](dask.dataframe.DataFrame.apply.md#dask.dataframe.DataFrame.apply)
: Apply a function along input axis of DataFrame.

`DataFrame.map`
: Apply a function elementwise on a whole DataFrame.

[`Series.map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)
: Apply a mapping correspondence on a [`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series).

### Notes

Use `.pipe` when chaining together functions that expect
Series, DataFrames or GroupBy objects.

### Examples

Constructing an income DataFrame from a dictionary.

```pycon
>>> data = [[8000, 1000], [9500, np.nan], [5000, 2000]]
>>> df = pd.DataFrame(data, columns=["Salary", "Others"])
>>> df
   Salary  Others
0    8000  1000.0
1    9500     NaN
2    5000  2000.0
```

Functions that perform tax reductions on an income DataFrame.

```pycon
>>> def subtract_federal_tax(df):
...     return df * 0.9
>>> def subtract_state_tax(df, rate):
...     return df * (1 - rate)
>>> def subtract_national_insurance(df, rate, rate_increase):
...     new_rate = rate + rate_increase
...     return df * (1 - new_rate)
```

Instead of writing

```pycon
>>> subtract_national_insurance(
...     subtract_state_tax(subtract_federal_tax(df), rate=0.12),
...     rate=0.05,
...     rate_increase=0.02,
... )
```

You can write

```pycon
>>> (
...     df.pipe(subtract_federal_tax)
...     .pipe(subtract_state_tax, rate=0.12)
...     .pipe(subtract_national_insurance, rate=0.05, rate_increase=0.02)
... )
    Salary   Others
0  5892.48   736.56
1  6997.32      NaN
2  3682.80  1473.12
```

If you have a function that takes the data as (say) the second
argument, pass a tuple indicating which keyword expects the
data. For example, suppose `national_insurance` takes its data as `df`
in the second argument:

```pycon
>>> def subtract_national_insurance(rate, df, rate_increase):
...     new_rate = rate + rate_increase
...     return df * (1 - new_rate)
>>> (
...     df.pipe(subtract_federal_tax)
...     .pipe(subtract_state_tax, rate=0.12)
...     .pipe(
...         (subtract_national_insurance, "df"), rate=0.05, rate_increase=0.02
...     )
... )
    Salary   Others
0  5892.48   736.56
1  6997.32      NaN
2  3682.80  1473.12
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.pow.html.md

# dask.dataframe.Index.pow

#### Index.pow(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.quantile.html.md

# dask.dataframe.Index.quantile

#### Index.quantile(q=0.5, method='default')

Approximate quantiles of Series

* **Parameters:**
  **q**
  : Iterable of numbers ranging from 0 to 1 for the desired quantiles

  **method**
  : What method to use. By default will use dask’s internal custom
    algorithm (`'dask'`).  If set to `'tdigest'` will use tdigest
    for floats and ints and fallback to the `'dask'` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.radd.html.md

# dask.dataframe.Index.radd

#### Index.radd(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.random_split.html.md

# dask.dataframe.Index.random_split

#### Index.random_split(frac, random_state=None, shuffle=False)

Pseudorandomly split dataframe into different pieces row-wise

* **Parameters:**
  **frac**
  : List of floats that should sum to one.

  **random_state**
  : If int or None create a new RandomState with this as the seed.
    Otherwise draw from the passed RandomState.

  **shuffle**
  : If set to True, the dataframe is shuffled (within partition)
    before the split.

#### SEE ALSO
`dask.DataFrame.sample`

### Examples

50/50 split

```pycon
>>> a, b = df.random_split([0.5, 0.5])
```

80/10/10 split, consistent random_state

```pycon
>>> a, b, c = df.random_split([0.8, 0.1, 0.1], random_state=123)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.rdiv.html.md

# dask.dataframe.Index.rdiv

#### Index.rdiv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.rename.html.md

# dask.dataframe.Index.rename

#### Index.rename(index, sorted_index=False)

Alter Series index labels or name

Function / dict values must be unique (1-to-1). Labels not contained in
a dict / Series will be left as-is. Extra labels listed don’t throw an
error.

Alternatively, change `Series.name` with a scalar value.

* **Parameters:**
  **index**
  : If dict-like or callable, the transformation is applied to the
    index. Scalar or hashable sequence-like will alter the
    `Series.name` attribute.

  **inplace**
  : Whether to return a new Series or modify this one inplace.

  **sorted_index**
  : If true, the output `Series` will have known divisions inferred
    from the input series and the transformation. Ignored for
    non-callable/dict-like `index` or when the input series has
    unknown divisions. Note that this may only be set to `True` if
    you know that the transformed index is monotonically increasing. Dask
    will check that transformed divisions are monotonic, but cannot
    check all the values between divisions, so incorrectly setting this
    can result in bugs.
* **Returns:**
  **renamed**

#### SEE ALSO
[`pandas.Series.rename`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.rename.html#pandas.Series.rename)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.repartition.html.md

# dask.dataframe.Index.repartition

#### Index.repartition(divisions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) | [None](https://docs.python.org/3/library/constants.html#None) = None, npartitions: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, partition_size: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, freq=None, force: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Repartition a collection

Exactly one of divisions, npartitions or partition_size should be
specified. A `ValueError` will be raised when that is not the case.

* **Parameters:**
  **divisions**
  : The “dividing lines” used to split the dataframe into partitions.
    For `divisions=[0, 10, 50, 100]`, there would be three output partitions,
    where the new index contained [0, 10), [10, 50), and [50, 100), respectively.
    See [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions).

  **npartitions**
  : Approximate number of partitions of output. The number of
    partitions used may be slightly lower than npartitions depending
    on data distribution, but will never be higher.
    The Callable gets the number of partitions of the input as an argument
    and should return an int.

  **partition_size**
  : Max number of bytes of memory for each partition. Use numbers or strings
    like 5MB. If specified npartitions and divisions will be ignored. Note that
    the size reflects the number of bytes used as computed by
    pandas.DataFrame.memory_usage, which will not necessarily match the size
    when storing to disk.
    <br/>
    #### WARNING
    This keyword argument triggers computation to determine
    the memory size of each partition, which may be expensive.

  **force**
  : Allows the expansion of the existing divisions.
    If False then the new divisions’ lower and upper bounds must be
    the same as the old divisions’.

  **freq**
  : A period on which to partition timeseries data like `'7D'` or
    `'12h'` or `pd.Timedelta(hours=12)`.  Assumes a datetime index.

#### SEE ALSO
[`DataFrame.memory_usage_per_partition`](dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition)

[`pandas.DataFrame.memory_usage`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.memory_usage.html#pandas.DataFrame.memory_usage)

### Notes

Exactly one of divisions, npartitions, partition_size, or freq
should be specified. A `ValueError` will be raised when that is
not the case.

Also note that `len(divisions)` is equal to `npartitions + 1`. This is because `divisions`
represents the upper and lower bounds of each partition. The first item is the
lower bound of the first partition, the second item is the lower bound of the
second partition and the upper bound of the first partition, and so on.
The second-to-last item is the lower bound of the last partition, and the last
(extra) item is the upper bound of the last partition.

### Examples

```pycon
>>> df = df.repartition(npartitions=10)
>>> df = df.repartition(divisions=[0, 5, 10, 20])
>>> df = df.repartition(freq='7d')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.replace.html.md

# dask.dataframe.Index.replace

#### Index.replace(to_replace=None, value=<no_default>, regex=False)

Replace values given in to_replace with value.

This docstring was copied from pandas.DataFrame.replace.

Some inconsistencies with the Dask version may exist.

Values of the Series/DataFrame are replaced with other values dynamically.
This differs from updating with `.loc` or `.iloc`, which require
you to specify a location to update with some value.

* **Parameters:**
  **to_replace**
  : How to find the values that will be replaced.
    * numeric, str or regex:
      > - numeric: numeric values equal to to_replace will be
      >   replaced with value
      > - str: string exactly matching to_replace will be replaced
      >   with value
      > - regex: regexes matching to_replace will be replaced with
      >   value
    * list of str, regex, or numeric:
      > - First, if to_replace and value are both lists, they
      >   **must** be the same length.
      > - Second, if `regex=True` then all of the strings in **both**
      >   lists will be interpreted as regexes otherwise they will match
      >   directly. This doesn’t matter much for value since there
      >   are only a few possible substitution regexes you can use.
      > - str, regex and numeric rules apply as above.
    * dict:
      > - Dicts can be used to specify different replacement values
      >   for different existing values. For example,
      >   `{'a': 'b', 'y': 'z'}` replaces the value ‘a’ with ‘b’ and
      >   ‘y’ with ‘z’. To use a dict in this way, the optional value
      >   parameter should not be given.
      > - For a DataFrame a dict can specify that different values
      >   should be replaced in different columns. For example,
      >   `{'a': 1, 'b': 'z'}` looks for the value 1 in column ‘a’
      >   and the value ‘z’ in column ‘b’ and replaces these values
      >   with whatever is specified in value. The value parameter
      >   should not be `None` in this case. You can treat this as a
      >   special case of passing two lists except that you are
      >   specifying the column to search in.
      > - For a DataFrame nested dictionaries, e.g.,
      >   `{'a': {'b': np.nan}}`, are read as follows: look in column
      >   ‘a’ for the value ‘b’ and replace it with NaN. The optional value
      >   parameter should not be specified to use a nested dict in this
      >   way. You can nest regular expressions as well. Note that
      >   column names (the top-level dictionary keys in a nested
      >   dictionary) **cannot** be regular expressions.
    * None:
      > - This means that the regex argument must be a string,
      >   compiled regular expression, or list, dict, ndarray or
      >   Series of such elements. If value is also `None` then
      >   this **must** be a nested dictionary or Series.
    <br/>
    See the examples section for examples of each of these.

  **value**
  : Value to replace any values matching to_replace with.
    For a DataFrame a dict of values can be used to specify which
    value to use for each column (columns not in the dict will not be
    filled). Regular expressions, strings and lists or dicts of such
    objects are also allowed.

  **inplace**
  : If True, performs operation inplace.

  **regex**
  : Whether to interpret to_replace and/or value as regular
    expressions. Alternatively, this could be a regular expression or a
    list, dict, or array of regular expressions in which case
    to_replace must be `None`.
* **Returns:**
  Series/DataFrame
  : Object after replacement.
* **Raises:**
  AssertionError
  : * If regex is not a `bool` and to_replace is not
      `None`.

  TypeError
  : * If to_replace is not a scalar, array-like, `dict`, or `None`
    * If to_replace is a `dict` and value is not a `list`,
      `dict`, `ndarray`, or `Series`
    * If to_replace is `None` and regex is not compilable
      into a regular expression or is a list, dict, ndarray, or
      Series.
    * When replacing multiple `bool` or `datetime64` objects and
      the arguments to to_replace does not match the type of the
      value being replaced

  ValueError
  : * If a `list` or an `ndarray` is passed to to_replace and
      value but they are not the same length.

#### SEE ALSO
[`Series.fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)
: Fill NA values.

[`DataFrame.fillna`](dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)
: Fill NA values.

[`Series.where`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Replace values based on boolean condition.

[`DataFrame.where`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Replace values based on boolean condition.

`DataFrame.map`
: Apply a function to a Dataframe elementwise.

[`Series.map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)
: Map values of Series according to an input mapping or function.

[`Series.str.replace`](dask.dataframe.Series.str.replace.md#dask.dataframe.Series.str.replace)
: Simple string replacement.

### Notes

* Regex substitution is performed under the hood with `re.sub`. The
  rules for substitution for `re.sub` are the same.
* Regular expressions will only substitute on strings, meaning you
  cannot provide, for example, a regular expression matching floating
  point numbers and expect the columns in your frame that have a
  numeric dtype to be matched. However, if those floating point
  numbers *are* strings, then you can do this.
* This method has *a lot* of options. You are encouraged to experiment
  and play with this method to gain intuition about how it works.
* When dict is used as the to_replace value, it is like
  key(s) in the dict are the to_replace part and
  value(s) in the dict are the value parameter.

### Examples

**Scalar \`to_replace\` and \`value\`**

```pycon
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> s.replace(1, 5)
0    5
1    2
2    3
3    4
4    5
dtype: int64
```

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": [5, 6, 7, 8, 9],
...         "C": ["a", "b", "c", "d", "e"],
...     }
... )
>>> df.replace(0, 5)
    A  B  C
0  5  5  a
1  1  6  b
2  2  7  c
3  3  8  d
4  4  9  e
```

**List-like \`to_replace\`**

```pycon
>>> df.replace([0, 1, 2, 3], 4)
    A  B  C
0  4  5  a
1  4  6  b
2  4  7  c
3  4  8  d
4  4  9  e
```

```pycon
>>> df.replace([0, 1, 2, 3], [4, 3, 2, 1])
    A  B  C
0  4  5  a
1  3  6  b
2  2  7  c
3  1  8  d
4  4  9  e
```

**dict-like \`to_replace\`**

```pycon
>>> df.replace({0: 10, 1: 100})
        A  B  C
0   10  5  a
1  100  6  b
2    2  7  c
3    3  8  d
4    4  9  e
```

```pycon
>>> df.replace({"A": 0, "B": 5}, 100)
        A    B  C
0  100  100  a
1    1    6  b
2    2    7  c
3    3    8  d
4    4    9  e
```

```pycon
>>> df.replace({"A": {0: 100, 4: 400}})
        A  B  C
0  100  5  a
1    1  6  b
2    2  7  c
3    3  8  d
4  400  9  e
```

**Regular expression \`to_replace\`**

```pycon
>>> df = pd.DataFrame({"A": ["bat", "foo", "bait"], "B": ["abc", "bar", "xyz"]})
>>> df.replace(to_replace=r"^ba.$", value="new", regex=True)
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace({"A": r"^ba.$"}, {"A": "new"}, regex=True)
        A    B
0   new  abc
1   foo  bar
2  bait  xyz
```

```pycon
>>> df.replace(regex=r"^ba.$", value="new")
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace(regex={r"^ba.$": "new", "foo": "xyz"})
        A    B
0   new  abc
1   xyz  new
2  bait  xyz
```

```pycon
>>> df.replace(regex=[r"^ba.$", "foo"], value="new")
        A    B
0   new  abc
1   new  new
2  bait  xyz
```

Compare the behavior of `s.replace({'a': None})` and
`s.replace('a', None)` to understand the peculiarities
of the to_replace parameter:

```pycon
>>> s = pd.Series([10, "a", "a", "b", "a"])
```

When one uses a dict as the to_replace value, it is like the
value(s) in the dict are equal to the value parameter.
`s.replace({'a': None})` is equivalent to
`s.replace(to_replace={'a': None}, value=None)`:

```pycon
>>> s.replace({"a": None})
0      10
1    None
2    None
3       b
4    None
dtype: object
```

If `None` is explicitly passed for `value`, it will be respected:

```pycon
>>> s.replace("a", None)
0      10
1    None
2    None
3       b
4    None
dtype: object
```

When `regex=True`, `value` is not `None` and to_replace is a string,
the replacement will be applied in all columns of the DataFrame.

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": ["a", "b", "c", "d", "e"],
...         "C": ["f", "g", "h", "i", "j"],
...     }
... )
```

```pycon
>>> df.replace(to_replace="^[a-g]", value="e", regex=True)
    A  B  C
0  0  e  e
1  1  e  e
2  2  e  h
3  3  e  i
4  4  e  j
```

If `value` is not `None` and to_replace is a dictionary, the dictionary
keys will be the DataFrame columns that the replacement will be applied.

```pycon
>>> df.replace(to_replace={"B": "^[a-c]", "C": "^[h-j]"}, value="e", regex=True)
    A  B  C
0  0  e  f
1  1  e  g
2  2  e  e
3  3  d  e
4  4  e  e
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.resample.html.md

# dask.dataframe.Index.resample

#### Index.resample(rule, closed=None, label=None)

Resample time-series data.

This docstring was copied from pandas.DataFrame.resample.

Some inconsistencies with the Dask version may exist.

Convenience method for frequency conversion and resampling of time series.
The object must have a datetime-like index (DatetimeIndex, PeriodIndex,
or TimedeltaIndex), or the caller must pass the label of a datetime-like
series/index to the `on`/`level` keyword parameter.

* **Parameters:**
  **rule**
  : The offset string or object representing target conversion.

  **closed**
  : Which side of bin interval is closed. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **label**
  : Which bin edge label to label bucket with. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **convention**
  : For PeriodIndex only, controls whether to use the start or
    end of rule.

  **on**
  : For a DataFrame, column to use instead of index for resampling.
    Column must be datetime-like.

  **level**
  : For a MultiIndex, level (name or number) to use for
    resampling. level must be datetime-like.

  **origin**
  : The timestamp on which to adjust the grouping. The timezone of origin
    must match the timezone of the index.
    If string, must be Timestamp convertible or one of the following:
    - ‘epoch’: origin is 1970-01-01
    - ‘start’: origin is the first value of the timeseries
    - ‘start_day’: origin is the first day at midnight of the timeseries
    - ‘end’: origin is the last value of the timeseries
    - ‘end_day’: origin is the ceiling midnight of the last day
    <br/>
    #### NOTE
    Only takes effect for Tick-frequencies (i.e. fixed frequencies like
    days, hours, and minutes, rather than months or quarters).

  **offset**
  : An offset timedelta added to the origin.

  **group_keys**
  : Whether to include the group keys in the result index when using
    `.apply()` on the resampled object.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `False`.
* **Returns:**
  pandas.api.typing.Resampler
  : `Resampler` object.

#### SEE ALSO
[`Series.resample`](dask.dataframe.Series.resample.md#dask.dataframe.Series.resample)
: Resample a Series.

[`DataFrame.resample`](dask.dataframe.DataFrame.resample.md#dask.dataframe.DataFrame.resample)
: Resample a DataFrame.

[`groupby`](dask.dataframe.Index.groupby.md#dask.dataframe.Index.groupby)
: Group Series/DataFrame by mapping, function, label, or list of labels.

`asfreq`
: Reindex a Series/DataFrame with the given frequency without grouping.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#resampling)
for more.

To learn more about the offset strings, please see [this link](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects).

### Examples

Start by creating a series with 9 one minute timestamps.

```pycon
>>> index = pd.date_range("1/1/2000", periods=9, freq="min")
>>> series = pd.Series(range(9), index=index)
>>> series
2000-01-01 00:00:00    0
2000-01-01 00:01:00    1
2000-01-01 00:02:00    2
2000-01-01 00:03:00    3
2000-01-01 00:04:00    4
2000-01-01 00:05:00    5
2000-01-01 00:06:00    6
2000-01-01 00:07:00    7
2000-01-01 00:08:00    8
Freq: min, dtype: int64
```

Downsample the series into 3 minute bins and sum the values
of the timestamps falling into a bin.

```pycon
>>> series.resample("3min").sum()
2000-01-01 00:00:00     3
2000-01-01 00:03:00    12
2000-01-01 00:06:00    21
Freq: 3min, dtype: int64
```

Downsample the series into 3 minute bins as above, but label each
bin using the right edge instead of the left. Please note that the
value in the bucket used as the label is not included in the bucket,
which it labels. For example, in the original series the
bucket `2000-01-01 00:03:00` contains the value 3, but the summed
value in the resampled bucket with the label `2000-01-01 00:03:00`
does not include 3 (if it did, the summed value would be 6, not 3).

```pycon
>>> series.resample("3min", label="right").sum()
2000-01-01 00:03:00     3
2000-01-01 00:06:00    12
2000-01-01 00:09:00    21
Freq: 3min, dtype: int64
```

To include this value close the right side of the bin interval,
as shown below.

```pycon
>>> series.resample("3min", label="right", closed="right").sum()
2000-01-01 00:00:00     0
2000-01-01 00:03:00     6
2000-01-01 00:06:00    15
2000-01-01 00:09:00    15
Freq: 3min, dtype: int64
```

Upsample the series into 30 second bins.

```pycon
>>> series.resample("30s").asfreq()[0:5]  # Select first 5 rows
2000-01-01 00:00:00   0.0
2000-01-01 00:00:30   NaN
2000-01-01 00:01:00   1.0
2000-01-01 00:01:30   NaN
2000-01-01 00:02:00   2.0
Freq: 30s, dtype: float64
```

Upsample the series into 30 second bins and fill the `NaN`
values using the `ffill` method.

```pycon
>>> series.resample("30s").ffill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    0
2000-01-01 00:01:00    1
2000-01-01 00:01:30    1
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Upsample the series into 30 second bins and fill the
`NaN` values using the `bfill` method.

```pycon
>>> series.resample("30s").bfill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    1
2000-01-01 00:01:00    1
2000-01-01 00:01:30    2
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Pass a custom function via `apply`

```pycon
>>> def custom_resampler(arraylike):
...     return np.sum(arraylike) + 5
>>> series.resample("3min").apply(custom_resampler)
2000-01-01 00:00:00     8
2000-01-01 00:03:00    17
2000-01-01 00:06:00    26
Freq: 3min, dtype: int64
```

For a Series with a PeriodIndex, the keyword convention can be
used to control whether to use the start or end of rule.

Resample a year by quarter using ‘start’ convention. Values are
assigned to the first quarter of the period.

```pycon
>>> s = pd.Series(
...     [1, 2], index=pd.period_range("2012-01-01", freq="Y", periods=2)
... )
>>> s
2012    1
2013    2
Freq: Y-DEC, dtype: int64
>>> s.resample("Q", convention="start").asfreq()
2012Q1    1.0
2012Q2    NaN
2012Q3    NaN
2012Q4    NaN
2013Q1    2.0
2013Q2    NaN
2013Q3    NaN
2013Q4    NaN
Freq: Q-DEC, dtype: float64
```

Resample quarters by month using ‘end’ convention. Values are
assigned to the last month of the period.

```pycon
>>> q = pd.Series(
...     [1, 2, 3, 4], index=pd.period_range("2018-01-01", freq="Q", periods=4)
... )
>>> q
2018Q1    1
2018Q2    2
2018Q3    3
2018Q4    4
Freq: Q-DEC, dtype: int64
>>> q.resample("M", convention="end").asfreq()
2018-03    1.0
2018-04    NaN
2018-05    NaN
2018-06    2.0
2018-07    NaN
2018-08    NaN
2018-09    3.0
2018-10    NaN
2018-11    NaN
2018-12    4.0
Freq: M, dtype: float64
```

For DataFrame objects, the keyword on can be used to specify the
column instead of the index for resampling.

```pycon
>>> df = pd.DataFrame([10, 11, 9, 13, 14, 18, 17, 19], columns=["price"])
>>> df["volume"] = [50, 60, 40, 100, 50, 100, 40, 50]
>>> df["week_starting"] = pd.date_range("01/01/2018", periods=8, freq="W")
>>> df
   price  volume week_starting
0     10      50    2018-01-07
1     11      60    2018-01-14
2      9      40    2018-01-21
3     13     100    2018-01-28
4     14      50    2018-02-04
5     18     100    2018-02-11
6     17      40    2018-02-18
7     19      50    2018-02-25
>>> df.resample("ME", on="week_starting").mean()
               price  volume
week_starting
2018-01-31     10.75    62.5
2018-02-28     17.00    60.0
```

For a DataFrame with MultiIndex, the keyword level can be used to
specify on which level the resampling needs to take place.

```pycon
>>> days = pd.date_range("1/1/2000", periods=4, freq="D")
>>> df2 = pd.DataFrame(
...     [
...         [10, 50],
...         [11, 60],
...         [9, 40],
...         [13, 100],
...         [14, 50],
...         [18, 100],
...         [17, 40],
...         [19, 50],
...     ],
...     columns=["price", "volume"],
...     index=pd.MultiIndex.from_product([days, ["morning", "afternoon"]]),
... )
>>> df2
                      price  volume
2000-01-01 morning       10      50
           afternoon     11      60
2000-01-02 morning        9      40
           afternoon     13     100
2000-01-03 morning       14      50
           afternoon     18     100
2000-01-04 morning       17      40
           afternoon     19      50
>>> df2.resample("D", level=0).sum()
            price  volume
2000-01-01     21     110
2000-01-02     22     140
2000-01-03     32     150
2000-01-04     36      90
```

If you want to adjust the start of the bins based on a fixed timestamp:

```pycon
>>> start, end = "2000-10-01 23:30:00", "2000-10-02 00:30:00"
>>> rng = pd.date_range(start, end, freq="7min")
>>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng)
>>> ts
2000-10-01 23:30:00     0
2000-10-01 23:37:00     3
2000-10-01 23:44:00     6
2000-10-01 23:51:00     9
2000-10-01 23:58:00    12
2000-10-02 00:05:00    15
2000-10-02 00:12:00    18
2000-10-02 00:19:00    21
2000-10-02 00:26:00    24
Freq: 7min, dtype: int64
```

```pycon
>>> ts.resample("17min").sum()
2000-10-01 23:14:00     0
2000-10-01 23:31:00     9
2000-10-01 23:48:00    21
2000-10-02 00:05:00    54
2000-10-02 00:22:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="epoch").sum()
2000-10-01 23:18:00     0
2000-10-01 23:35:00    18
2000-10-01 23:52:00    27
2000-10-02 00:09:00    39
2000-10-02 00:26:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="2000-01-01").sum()
2000-10-01 23:24:00     3
2000-10-01 23:41:00    15
2000-10-01 23:58:00    45
2000-10-02 00:15:00    45
Freq: 17min, dtype: int64
```

If you want to adjust the start of the bins with an offset Timedelta, the two
following lines are equivalent:

```pycon
>>> ts.resample("17min", origin="start").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", offset="23h30min").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

If you want to take the largest Timestamp as the end of the bins:

```pycon
>>> ts.resample("17min", origin="end").sum()
2000-10-01 23:35:00     0
2000-10-01 23:52:00    18
2000-10-02 00:09:00    27
2000-10-02 00:26:00    63
Freq: 17min, dtype: int64
```

In contrast with the start_day, you can use end_day to take the ceiling
midnight of the largest Timestamp as the end of the bins and drop the bins
not containing data:

```pycon
>>> ts.resample("17min", origin="end_day").sum()
2000-10-01 23:38:00     3
2000-10-01 23:55:00    15
2000-10-02 00:12:00    45
2000-10-02 00:29:00    45
Freq: 17min, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.reset_index.html.md

# dask.dataframe.Index.reset_index

#### Index.reset_index(drop: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Reset the index to the default index.

Note that unlike in `pandas`, the reset index for a Dask DataFrame will
not be monotonically increasing from 0. Instead, it will restart at 0
for each partition (e.g. `index1 = [0, ..., 10], index2 = [0, ...]`).
This is due to the inability to statically know the full length of the
index.

For DataFrame with multi-level index, returns a new DataFrame with
labeling information in the columns under the index names, defaulting
to ‘level_0’, ‘level_1’, etc. if any are None. For a standard index,
the index name will be used (if set), otherwise a default ‘index’ or
‘level_0’ (if ‘index’ is already taken) will be used.

* **Parameters:**
  **drop**
  : Do not try to insert index into dataframe columns.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.rolling.html.md

# dask.dataframe.Index.rolling

#### Index.rolling(window, \*\*kwargs)

Provides rolling transformations.

* **Parameters:**
  **window**
  : Size of the moving window. This is the number of observations used
    for calculating the statistic. When not using a `DatetimeIndex`,
    the window size must not be so large as to span more than one
    adjacent partition. If using an offset or offset alias like ‘5D’,
    the data must have a `DatetimeIndex`

  **min_periods**
  : Minimum number of observations in window required to have a value
    (otherwise result is NA).

  **center**
  : Set the labels at the center of the window.

  **win_type**
  : Provide a window type. The recognized window types are identical
    to pandas.

  **axis**
  : This parameter is deprecated with `pandas>=2.1`.
* **Returns:**
  a Rolling object on which to call a method to compute a statistic

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.round.html.md

# dask.dataframe.Index.round

#### Index.round(decimals=0)

Round numeric columns in a DataFrame to a variable number of decimal places.

This docstring was copied from pandas.DataFrame.round.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **decimals**
  : Number of decimal places to round each column to. If an int is
    given, round each column to the same number of places.
    Otherwise dict and Series round to variable numbers of places.
    Column names should be in the keys if decimals is a
    dict-like, or in the index if decimals is a Series. Any
    columns not included in decimals will be left as is. Elements
    of decimals which are not columns of the input will be
    ignored.

  **\*args**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.
* **Returns:**
  DataFrame
  : A DataFrame with the affected columns rounded to the specified
    number of decimal places.

#### SEE ALSO
[`numpy.around`](https://numpy.org/doc/stable/reference/generated/numpy.around.html#numpy.around)
: Round a numpy array to the given number of decimals.

[`Series.round`](dask.dataframe.Series.round.md#dask.dataframe.Series.round)
: Round a Series to the given number of decimals.

### Notes

For values exactly halfway between rounded decimal values, pandas rounds
to the nearest even value (e.g. -0.5 and 0.5 round to 0.0, 1.5 and 2.5
round to 2.0, etc.).

### Examples

```pycon
>>> df = pd.DataFrame(
...     [(0.21, 0.32), (0.01, 0.67), (0.66, 0.03), (0.21, 0.18)],
...     columns=["dogs", "cats"],
... )
>>> df
    dogs  cats
0  0.21  0.32
1  0.01  0.67
2  0.66  0.03
3  0.21  0.18
```

By providing an integer each column is rounded to the same number
of decimal places

```pycon
>>> df.round(1)
    dogs  cats
0   0.2   0.3
1   0.0   0.7
2   0.7   0.0
3   0.2   0.2
```

With a dict, the number of places for specific columns can be
specified with the column names as key and the number of decimal
places as value

```pycon
>>> df.round({"dogs": 1, "cats": 0})
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

Using a Series, the number of places for specific columns can be
specified with the column names as index and the number of
decimal places as value

```pycon
>>> decimals = pd.Series([0, 1], index=["cats", "dogs"])
>>> df.round(decimals)
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.sample.html.md

# dask.dataframe.Index.sample

#### Index.sample(n=None, frac=None, replace=False, random_state=None)

Random sample of items

* **Parameters:**
  **n**
  : Number of items to return is not supported by dask. Use frac
    instead.

  **frac**
  : Approximate fraction of items to return. This sampling fraction is
    applied to all partitions equally. Note that this is an
    **approximate fraction**. You should not expect exactly `len(df) * frac`
    items to be returned, as the exact number of elements selected will
    depend on how your data is partitioned (but should be pretty close
    in practice).

  **replace**
  : Sample with or without replacement. Default = False.

  **random_state**
  : If an int, we create a new RandomState with this as the seed;
    Otherwise we draw from the passed RandomState.

#### SEE ALSO
[`DataFrame.random_split`](dask.dataframe.DataFrame.random_split.md#dask.dataframe.DataFrame.random_split)

[`pandas.DataFrame.sample`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html#pandas.DataFrame.sample)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.sem.html.md

# dask.dataframe.Index.sem

#### Index.sem(axis=None, skipna=True, ddof=1, split_every=False, numeric_only=False)

Return unbiased standard error of the mean over requested axis.

This docstring was copied from pandas.DataFrame.sem.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.sem with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keywords passed.
* **Returns:**
  Series or DataFrame (if level specified)
  : Unbiased standard error of the mean over requested axis.

#### SEE ALSO
[`DataFrame.var`](dask.dataframe.DataFrame.var.md#dask.dataframe.DataFrame.var)
: Return unbiased variance over requested axis.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Returns sample standard deviation over requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> round(s.sem(), 6)
0.57735
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.sem()
a   0.5
b   0.5
dtype: float64
```

Using axis=1

```pycon
>>> df.sem(axis=1)
tiger   0.5
zebra   0.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.sem(numeric_only=True)
a   0.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.shape.html.md

# dask.dataframe.Index.shape

#### *property* Index.shape

Return a tuple representing the dimensionality of the DataFrame.

The number of rows is a Delayed result. The number of columns
is a concrete integer.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.shift.html.md

# dask.dataframe.Index.shift

#### Index.shift(periods=1, freq=None)

Shift index by desired number of periods with an optional time freq.

This docstring was copied from pandas.DataFrame.shift.

Some inconsistencies with the Dask version may exist.

When freq is not passed, shift the index without realigning the data.
If freq is passed (in this case, the index must be date or datetime,
or it will raise a NotImplementedError), the index will be
increased using the periods and the freq. freq can be inferred
when specified as “infer” as long as either freq or inferred_freq
attribute is set in the index.

* **Parameters:**
  **periods**
  : Number of periods to shift. Can be positive or negative.
    If an iterable of ints, the data will be shifted once by each int.
    This is equivalent to shifting by one value at a time and
    concatenating all resulting frames. The resulting columns will have
    the shift suffixed to their column names. For multiple periods,
    axis must not be 1.

  **freq**
  : Offset to use from the tseries module or time rule (e.g. ‘EOM’).
    If freq is specified then the index values are shifted but the
    data is not realigned. That is, use freq if you would like to
    extend the index when shifting and preserve the original data.
    If freq is specified as “infer” then it will be inferred from
    the freq or inferred_freq attributes of the index. If neither of
    those attributes exist, a ValueError is thrown.

  **axis**
  : Shift direction. For Series this parameter is unused and defaults to 0.

  **fill_value**
  : The scalar value to use for newly introduced missing values.
    the default depends on the dtype of self.
    For Boolean and numeric NumPy data types, `np.nan` is used.
    For datetime, timedelta, or period data, etc. `NaT` is used.
    For extension dtypes, `self.dtype.na_value` is used.

  **suffix**
  : If str and periods is an iterable, this is added after the column
    name and before the shift value for each shifted column name.
    For Series this parameter is unused and defaults to None.
* **Returns:**
  DataFrame
  : Copy of input object, shifted.

#### SEE ALSO
[`Index.shift`](#dask.dataframe.Index.shift)
: Shift values of Index.

`DatetimeIndex.shift`
: Shift values of DatetimeIndex.

`PeriodIndex.shift`
: Shift values of PeriodIndex.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [[10, 13, 17], [20, 23, 27], [15, 18, 22], [30, 33, 37], [45, 48, 52]],
...     columns=["Col1", "Col2", "Col3"],
...     index=pd.date_range("2020-01-01", "2020-01-05"),
... )
>>> df
            Col1  Col2  Col3
2020-01-01    10    13    17
2020-01-02    20    23    27
2020-01-03    15    18    22
2020-01-04    30    33    37
2020-01-05    45    48    52
```

```pycon
>>> df.shift(periods=3)
            Col1  Col2  Col3
2020-01-01   NaN   NaN   NaN
2020-01-02   NaN   NaN   NaN
2020-01-03   NaN   NaN   NaN
2020-01-04  10.0  13.0  17.0
2020-01-05  20.0  23.0  27.0
```

```pycon
>>> df.shift(periods=1, axis="columns")
            Col1  Col2  Col3
2020-01-01   NaN    10    13
2020-01-02   NaN    20    23
2020-01-03   NaN    15    18
2020-01-04   NaN    30    33
2020-01-05   NaN    45    48
```

```pycon
>>> df.shift(periods=3, fill_value=0)
            Col1  Col2  Col3
2020-01-01     0     0     0
2020-01-02     0     0     0
2020-01-03     0     0     0
2020-01-04    10    13    17
2020-01-05    20    23    27
```

```pycon
>>> df.shift(periods=3, freq="D")
            Col1  Col2  Col3
2020-01-04    10    13    17
2020-01-05    20    23    27
2020-01-06    15    18    22
2020-01-07    30    33    37
2020-01-08    45    48    52
```

```pycon
>>> df.shift(periods=3, freq="infer")
            Col1  Col2  Col3
2020-01-04    10    13    17
2020-01-05    20    23    27
2020-01-06    15    18    22
2020-01-07    30    33    37
2020-01-08    45    48    52
```

```pycon
>>> df["Col1"].shift(periods=[0, 1, 2])
            Col1_0  Col1_1  Col1_2
2020-01-01      10     NaN     NaN
2020-01-02      20    10.0     NaN
2020-01-03      15    20.0    10.0
2020-01-04      30    15.0    20.0
2020-01-05      45    30.0    15.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.size.html.md

# dask.dataframe.Index.size

#### *property* Index.size

Size of the Series or DataFrame as a Delayed object.

### Examples

```pycon
>>> series.size
<dask_expr.expr.Scalar: expr=df.size(), dtype=int64>
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.sub.html.md

# dask.dataframe.Index.sub

#### Index.sub(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_backend.html.md

# dask.dataframe.Index.to_backend

#### Index.to_backend(backend: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Move to a new DataFrame backend

* **Parameters:**
  **backend**
  : The name of the new backend to move to. The default
    is the current “dataframe.backend” configuration.
* **Returns:**
  DataFrame, Series or Index

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_bag.html.md

# dask.dataframe.Index.to_bag

#### Index.to_bag(index=False, format='tuple')

Create a Dask Bag from a Series

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_csv.html.md

# dask.dataframe.Index.to_csv

#### Index.to_csv(filename, \*\*kwargs)

See dd.to_csv docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_dask_array.html.md

# dask.dataframe.Index.to_dask_array

#### Index.to_dask_array(lengths=None, meta=None, optimize: [bool](https://docs.python.org/3/library/functions.html#bool) = True, \*\*optimize_kwargs) → [Array](dask.array.Array.md#dask.array.Array)

Convert a dask DataFrame to a dask array.

* **Parameters:**
  **lengths**
  : How to determine the chunks sizes for the output array.
    By default, the output array will have unknown chunk lengths
    along the first axis, which can cause some later operations
    to fail.
    * True : immediately compute the length of each partition
    * Sequence : a sequence of integers to use for the chunk sizes
      on the first axis. These values are *not* validated for
      correctness, beyond ensuring that the number of items
      matches the number of partitions.

  **meta**
  : An optional meta parameter can be passed for dask to override the
    default metadata on the underlying dask array.

  **optimize**
  : Whether to optimize the expression before converting to an Array.
* **Returns:**
  A Dask Array

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_delayed.html.md

# dask.dataframe.Index.to_delayed

#### Index.to_delayed(optimize_graph=True)

Convert into a list of `dask.delayed` objects, one per partition.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

#### SEE ALSO
`dask_expr.from_delayed`

### Examples

```pycon
>>> partitions = df.to_delayed()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_frame.html.md

# dask.dataframe.Index.to_frame

#### Index.to_frame(index=True, name=<no_default>)

Create a DataFrame with a column containing the Index.

This docstring was copied from pandas.Index.to_frame.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **index**
  : Set the index of the returned DataFrame as the original Index.

  **name**
  : The passed name should substitute for the index name (if it has
    one).
* **Returns:**
  DataFrame
  : DataFrame containing the original Index data.

#### SEE ALSO
[`Index.to_series`](dask.dataframe.Index.to_series.md#dask.dataframe.Index.to_series)
: Convert an Index to a Series.

[`Series.to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)
: Convert Series to DataFrame.

### Examples

```pycon
>>> idx = pd.Index(["Ant", "Bear", "Cow"], name="animal")
>>> idx.to_frame()
       animal
animal
Ant       Ant
Bear     Bear
Cow       Cow
```

By default, the original Index is reused. To enforce a new Index:

```pycon
>>> idx.to_frame(index=False)
    animal
0   Ant
1  Bear
2   Cow
```

To override the name of the resulting column, specify name:

```pycon
>>> idx.to_frame(index=False, name="zoo")
    zoo
0   Ant
1  Bear
2   Cow
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_hdf.html.md

# dask.dataframe.Index.to_hdf

#### Index.to_hdf(path_or_buf, key, mode='a', append=False, \*\*kwargs)

See dd.to_hdf docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_series.html.md

# dask.dataframe.Index.to_series

#### Index.to_series(index=None, name=<no_default>)

Create a Series with both index and values equal to the index keys.

This docstring was copied from pandas.Index.to_series.

Some inconsistencies with the Dask version may exist.

Useful with map for returning an indexer based on an index.

* **Parameters:**
  **index**
  : Index of resulting Series. If None, defaults to original index.

  **name**
  : Name of resulting Series. If None, defaults to name of original
    index.
* **Returns:**
  Series
  : The dtype will be based on the type of the Index values.

#### SEE ALSO
[`Index.to_frame`](dask.dataframe.Index.to_frame.md#dask.dataframe.Index.to_frame)
: Convert an Index to a DataFrame.

[`Series.to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)
: Convert Series to DataFrame.

### Examples

```pycon
>>> idx = pd.Index(["Ant", "Bear", "Cow"], name="animal")
```

By default, the original index and original name is reused.

```pycon
>>> idx.to_series()
animal
Ant      Ant
Bear    Bear
Cow      Cow
Name: animal, dtype: str
```

To enforce a new index, specify new labels to `index`:

```pycon
>>> idx.to_series(index=[0, 1, 2])
0     Ant
1    Bear
2     Cow
Name: animal, dtype: str
```

To override the name of the resulting column, specify `name`:

```pycon
>>> idx.to_series(name="zoo")
animal
Ant      Ant
Bear    Bear
Cow      Cow
Name: zoo, dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_string.html.md

# dask.dataframe.Index.to_string

#### Index.to_string(max_rows=5)

Render a string representation of the Series.

This docstring was copied from pandas.Series.to_string.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **buf**
  : Buffer to write to.

  **na_rep**
  : String representation of NaN to use, default ‘NaN’.

  **float_format**
  : Formatter function to apply to columns’ elements if they are
    floats, default None.

  **header**
  : Add the Series header (index name).

  **index**
  : Add index (row) labels, default True.

  **length**
  : Add the Series length.

  **dtype**
  : Add the Series dtype.

  **name**
  : Add the Series name if not None.

  **max_rows**
  : Maximum number of rows to show before truncating. If None, show
    all.

  **min_rows**
  : The number of rows to display in a truncated repr (when number
    of rows is above max_rows).
* **Returns:**
  str or None
  : String representation of Series if `buf=None`, otherwise None.

#### SEE ALSO
`Series.to_dict`
: Convert Series to dict object.

[`Series.to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)
: Convert Series to DataFrame object.

`Series.to_markdown`
: Print Series in Markdown-friendly format.

[`Series.to_timestamp`](dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)
: Cast to DatetimeIndex of Timestamps.

### Examples

```pycon
>>> ser = pd.Series([1, 2, 3]).to_string()
>>> ser
'0    1\n1    2\n2    3'
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.to_timestamp.html.md

# dask.dataframe.Index.to_timestamp

#### Index.to_timestamp(freq=None, how='start')

Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.

This docstring was copied from pandas.DataFrame.to_timestamp.

Some inconsistencies with the Dask version may exist.

This can be changed to the *end* of the period, by specifying how=”e”.

* **Parameters:**
  **freq**
  : Desired frequency.

  **how**
  : Convention for converting period to timestamp; start of period
    vs. end.

  **axis**
  : The axis to convert (the index by default).

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.
* **Returns:**
  DataFrame with DatetimeIndex
  : DataFrame with the PeriodIndex cast to DatetimeIndex.

#### SEE ALSO
`DataFrame.to_period`
: Inverse method to cast DatetimeIndex to PeriodIndex.

[`Series.to_timestamp`](dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)
: Equivalent method for Series.

### Examples

```pycon
>>> idx = pd.PeriodIndex(["2023", "2024"], freq="Y")
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df1 = pd.DataFrame(data=d, index=idx)
>>> df1
      col1   col2
2023     1      3
2024     2      4
```

The resulting timestamps will be at the beginning of the year in this case

```pycon
>>> df1 = df1.to_timestamp()
>>> df1
            col1   col2
2023-01-01     1      3
2024-01-01     2      4
>>> df1.index
DatetimeIndex(['2023-01-01', '2024-01-01'], dtype='datetime64[us]', freq=None)
```

Using freq which is the offset that the Timestamps will have

```pycon
>>> df2 = pd.DataFrame(data=d, index=idx)
>>> df2 = df2.to_timestamp(freq="M")
>>> df2
            col1   col2
2023-01-31     1      3
2024-01-31     2      4
>>> df2.index
DatetimeIndex(['2023-01-31', '2024-01-31'], dtype='datetime64[us]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.truediv.html.md

# dask.dataframe.Index.truediv

#### Index.truediv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.unique.html.md

# dask.dataframe.Index.unique

#### Index.unique(split_every=None, split_out=True, shuffle_method=None)

Return Series of unique values in the object. Includes NA values.

* **Returns:**
  **uniques**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.value_counts.html.md

# dask.dataframe.Index.value_counts

#### Index.value_counts(sort=None, ascending=False, dropna=True, normalize=False, split_every=None, split_out=<no_default>)

Return a Series containing counts of unique values.

This docstring was copied from pandas.Series.value_counts.

Some inconsistencies with the Dask version may exist.

The resulting object will be in descending order so that the
first element is the most frequently-occurring element.
Excludes NA values by default.

* **Parameters:**
  **normalize**
  : If True then the object returned will contain the relative
    frequencies of the unique values.

  **sort**
  : Stable sort by frequencies when True. Preserve the order of the data
    when False.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0: Prior to 3.0.0, the sort was unstable.

  **ascending**
  : Sort in ascending order.

  **bins**
  : Rather than count values, group them into half-open bins,
    a convenience for `pd.cut`, only works with numeric data.

  **dropna**
  : Don’t include counts of NaN.
* **Returns:**
  Series
  : Series containing counts of unique values.

#### SEE ALSO
[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Number of non-NA elements in a Series.

[`DataFrame.count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)
: Number of non-NA elements in a DataFrame.

`DataFrame.value_counts`
: Equivalent method on DataFrames.

### Examples

```pycon
>>> index = pd.Index([3, 1, 2, 3, 4, np.nan])
>>> index.value_counts()
3.0    2
1.0    1
2.0    1
4.0    1
Name: count, dtype: int64
```

With normalize set to True, returns the relative frequency by
dividing all values by the sum of values.

```pycon
>>> s = pd.Series([3, 1, 2, 3, 4, np.nan])
>>> s.value_counts(normalize=True)
3.0    0.4
1.0    0.2
2.0    0.2
4.0    0.2
Name: proportion, dtype: float64
```

**bins**

Bins can be useful for going from a continuous variable to a
categorical variable; instead of counting unique
apparitions of values, divide the index in the specified
number of half-open bins.

```pycon
>>> s.value_counts(bins=3)
(0.996, 2.0]    2
(2.0, 3.0]      2
(3.0, 4.0]      1
Name: count, dtype: int64
```

**dropna**

With dropna set to False we can also see NaN index values.

```pycon
>>> s.value_counts(dropna=False)
3.0    2
1.0    1
2.0    1
4.0    1
NaN    1
Name: count, dtype: int64
```

**Categorical Dtypes**

Rows with categorical type will be counted as one group
if they have same categories and order.
In the example below, even though `a`, `c`, and `d`
all have the same data types of `category`,
only `c` and `d` will be counted as one group
since `a` doesn’t have the same categories.

```pycon
>>> df = pd.DataFrame({"a": [1], "b": ["2"], "c": [3], "d": [3]})
>>> df = df.astype({"a": "category", "c": "category", "d": "category"})
>>> df
   a  b  c  d
0  1  2  3  3
```

```pycon
>>> df.dtypes
a    category
b      str
c    category
d    category
dtype: object
```

```pycon
>>> df.dtypes.value_counts()
category    2
category    1
str         1
Name: count, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.values.html.md

# dask.dataframe.Index.values

#### *property* Index.values

Return a dask.array of the values of this dataframe

Warning: This creates a dask.array without precise shape information.
Operations that depend on shape information, like slicing or reshaping,
will not work.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.visualize.html.md

# dask.dataframe.Index.visualize

#### Index.visualize(tasks: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs)

Visualize the expression or task graph

* **Parameters:**
  **tasks:**
  : Whether to visualize the task graph. By default
    the expression graph will be visualized instead.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Index.where.html.md

# dask.dataframe.Index.where

#### Index.where(cond, other=nan)

Replace values where the condition is False.

This docstring was copied from pandas.DataFrame.where.

Some inconsistencies with the Dask version may exist.

This method allows conditional replacement of values. Where the
condition evaluates to True, the original values are retained; where
it evaluates to False, values are replaced with corresponding entries
from `other`.

* **Parameters:**
  **cond**
  : Where cond is True, keep the original value. Where
    False, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is False are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.mask()`](dask.dataframe.DataFrame.mask.md#dask.dataframe.DataFrame.mask)
: Return an object of same shape as caller.

[`Series.mask()`](dask.dataframe.Series.mask.md#dask.dataframe.Series.mask)
: Return an object of same shape as caller.

### Notes

The where method is an application of the if-then idiom. For each
element in the caller, if `cond` is `True` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with False.

The signature for [`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where) or
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `where` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.add.html.md

# dask.dataframe.Series.add

#### Series.add(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.align.html.md

# dask.dataframe.Series.align

#### Series.align(other, join='outer', axis=None, fill_value=None)

Align two objects on their axes with the specified join method.

This docstring was copied from pandas.DataFrame.align.

Some inconsistencies with the Dask version may exist.

Join method is specified for each axis Index.

* **Parameters:**
  **other**
  : The object to align with.

  **join**
  : Type of alignment to be performed.
    * left: use only keys from left frame, preserve key order.
    * right: use only keys from right frame, preserve key order.
    * outer: use union of keys from both frames, sort keys lexicographically.
    * inner: use intersection of keys from both frames,
      preserve the order of the left keys.

  **axis**
  : Align on index (0), columns (1), or both (None).

  **level**
  : Broadcast across a level, matching Index values on the
    passed MultiIndex level.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **fill_value**
  : Value to use for missing values. Defaults to NaN, but can be any
    “compatible” value.
* **Returns:**
  tuple of (Series/DataFrame, type of other)
  : Aligned objects.

#### SEE ALSO
[`Series.align`](#dask.dataframe.Series.align)
: Align two objects on their axes with specified join method.

[`DataFrame.align`](dask.dataframe.DataFrame.align.md#dask.dataframe.DataFrame.align)
: Align two objects on their axes with specified join method.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [[1, 2, 3, 4], [6, 7, 8, 9]], columns=["D", "B", "E", "A"], index=[1, 2]
... )
>>> other = pd.DataFrame(
...     [[10, 20, 30, 40], [60, 70, 80, 90], [600, 700, 800, 900]],
...     columns=["A", "B", "C", "D"],
...     index=[2, 3, 4],
... )
>>> df
   D  B  E  A
1  1  2  3  4
2  6  7  8  9
>>> other
    A    B    C    D
2   10   20   30   40
3   60   70   80   90
4  600  700  800  900
```

Align on columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=1)
>>> left
   A  B   C  D  E
1  4  2 NaN  1  3
2  9  7 NaN  6  8
>>> right
    A    B    C    D   E
2   10   20   30   40 NaN
3   60   70   80   90 NaN
4  600  700  800  900 NaN
```

We can also align on the index:

```pycon
>>> left, right = df.align(other, join="outer", axis=0)
>>> left
    D    B    E    A
1  1.0  2.0  3.0  4.0
2  6.0  7.0  8.0  9.0
3  NaN  NaN  NaN  NaN
4  NaN  NaN  NaN  NaN
>>> right
    A      B      C      D
1    NaN    NaN    NaN    NaN
2   10.0   20.0   30.0   40.0
3   60.0   70.0   80.0   90.0
4  600.0  700.0  800.0  900.0
```

Finally, the default axis=None will align on both index and columns:

```pycon
>>> left, right = df.align(other, join="outer", axis=None)
>>> left
     A    B   C    D    E
1  4.0  2.0 NaN  1.0  3.0
2  9.0  7.0 NaN  6.0  8.0
3  NaN  NaN NaN  NaN  NaN
4  NaN  NaN NaN  NaN  NaN
>>> right
       A      B      C      D   E
1    NaN    NaN    NaN    NaN NaN
2   10.0   20.0   30.0   40.0 NaN
3   60.0   70.0   80.0   90.0 NaN
4  600.0  700.0  800.0  900.0 NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.all.html.md

# dask.dataframe.Series.all

#### Series.all(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether all elements are True, potentially over an axis.

This docstring was copied from pandas.DataFrame.all.

Some inconsistencies with the Dask version may exist.

Returns True unless there at least one element within a series or
along a Dataframe axis that is False or equivalent (e.g. zero or
empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be True, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`Series.all`](#dask.dataframe.Series.all)
: Return True if all elements are True.

[`DataFrame.any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)
: Return True if one (or more) elements are True.

### Examples

**Series**

```pycon
>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([], dtype="float64").all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True
```

**DataFrames**

Create a DataFrame from a dictionary.

```pycon
>>> df = pd.DataFrame({"col1": [True, True], "col2": [True, False]})
>>> df
   col1   col2
0  True   True
1  True  False
```

Default behaviour checks if values in each column all return True.

```pycon
>>> df.all()
col1     True
col2    False
dtype: bool
```

Specify `axis='columns'` to check if values in each row all return True.

```pycon
>>> df.all(axis="columns")
0     True
1    False
dtype: bool
```

Or `axis=None` for whether every value is True.

```pycon
>>> df.all(axis=None)
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.any.html.md

# dask.dataframe.Series.any

#### Series.any(axis=0, skipna=True, split_every=False, \*\*kwargs)

Return whether any element is True, potentially over an axis.

This docstring was copied from pandas.DataFrame.any.

Some inconsistencies with the Dask version may exist.

Returns False unless there is at least one element within a series or
along a Dataframe axis that is True or equivalent (e.g. non-zero or
non-empty).

* **Parameters:**
  **axis**
  : Indicate which axis or axes should be reduced. For Series this parameter
    is unused and defaults to 0.
    * 0 / ‘index’ : reduce the index, return a Series whose index is the
      original column labels.
    * 1 / ‘columns’ : reduce the columns, return a Series whose index is the
      original index.
    * None : reduce all axes, return a scalar.

  **bool_only**
  : Include only boolean columns. Not implemented for Series.

  **skipna**
  : Exclude NA/null values. If the entire row/column is NA and skipna is
    True, then the result will be False, as for an empty row/column.
    If skipna is False, then NA are treated as True, because these are not
    equal to zero.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or scalar
  : If axis=None, then a scalar boolean is returned.
    Otherwise a Series is returned with index matching the index argument.

#### SEE ALSO
[`numpy.any`](https://numpy.org/doc/stable/reference/generated/numpy.any.html#numpy.any)
: Numpy version of this method.

[`Series.any`](#dask.dataframe.Series.any)
: Return whether any element is True.

[`Series.all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)
: Return whether all elements are True.

[`DataFrame.any`](dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)
: Return whether any element is True over requested axis.

[`DataFrame.all`](dask.dataframe.DataFrame.all.md#dask.dataframe.DataFrame.all)
: Return whether all elements are True over requested axis.

### Examples

**Series**

For Series input, the output is a scalar indicating whether any element
is True.

```pycon
>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([], dtype="float64").any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True
```

**DataFrame**

Whether each column contains at least one True element (the default).

```pycon
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
   A  B  C
0  1  0  0
1  2  2  0
```

```pycon
>>> df.any()
A     True
B     True
C    False
dtype: bool
```

Aggregating over the columns.

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
       A  B
0   True  1
1  False  2
```

```pycon
>>> df.any(axis="columns")
0    True
1    True
dtype: bool
```

```pycon
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
       A  B
0   True  1
1  False  0
```

```pycon
>>> df.any(axis="columns")
0    True
1    False
dtype: bool
```

Aggregating over the entire DataFrame with `axis=None`.

```pycon
>>> df.any(axis=None)
True
```

any for an empty DataFrame is an empty Series.

```pycon
>>> pd.DataFrame([]).any()
Series([], dtype: bool)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.apply.html.md

# dask.dataframe.Series.apply

#### Series.apply(function, \*args, meta=<no_default>, axis=0, \*\*kwargs)

Parallel version of pandas.Series.apply

* **Parameters:**
  **func**
  : Function to apply

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

  **args**
  : Positional arguments to pass to function in addition to the value.

  **Additional keyword arguments will be passed as keywords to the function.**
* **Returns:**
  **applied**

#### SEE ALSO
[`Series.map_partitions`](dask.dataframe.Series.map_partitions.md#dask.dataframe.Series.map_partitions)

### Examples

```pycon
>>> import dask.dataframe as dd
>>> s = pd.Series(range(5), name='x')
>>> ds = dd.from_pandas(s, npartitions=2)
```

Apply a function elementwise across the Series, passing in extra
arguments in `args` and `kwargs`:

```pycon
>>> def myadd(x, a, b=1):
...     return x + a + b
>>> res = ds.apply(myadd, args=(2,), b=1.5)
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with name `'x'`, and dtype
`float64`:

```pycon
>>> res = ds.apply(myadd, args=(2,), b=1.5, meta=('x', 'f8'))
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ds.apply(lambda x: x + 1, meta=ds)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.astype.html.md

# dask.dataframe.Series.astype

#### Series.astype(dtypes)

Cast a pandas object to a specified dtype `dtype`.

This docstring was copied from pandas.DataFrame.astype.

Some inconsistencies with the Dask version may exist.

This method allows the conversion of the data types of pandas objects,
including DataFrames and Series, to the specified dtype. It supports casting
entire objects to a single data type or applying different data types to
individual columns using a mapping.

* **Parameters:**
  **dtype**
  : Use a str, numpy.dtype, pandas.ExtensionDtype or Python type to
    cast entire pandas object to the same type. Alternatively, use a
    mapping, e.g. {col: dtype, …}, where col is a column label and dtype is
    a numpy.dtype or Python type to cast one or more of the DataFrame’s
    columns to column-specific types.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **errors**
  : Control raising of exceptions on invalid data for provided dtype.
    - `raise` : allow exceptions to be raised
    - `ignore` : suppress exceptions. On error return original object.
* **Returns:**
  same type as caller
  : The pandas object casted to the specified `dtype`.

#### SEE ALSO
[`to_datetime`](dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)
: Convert argument to datetime.

[`to_timedelta`](dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)
: Convert argument to timedelta.

[`to_numeric`](dask.dataframe.to_numeric.md#dask.dataframe.to_numeric)
: Convert argument to a numeric type.

[`numpy.ndarray.astype`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html#numpy.ndarray.astype)
: Cast a numpy array to a specified type.

### Notes

#### Versionchanged
Changed in version 2.0.0: Using `astype` to convert from timezone-naive dtype to
timezone-aware dtype will raise an exception.
Use `Series.dt.tz_localize()` instead.

### Examples

Create a DataFrame:

```pycon
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object
```

Cast all columns to int32:

```pycon
>>> df.astype("int32").dtypes
col1    int32
col2    int32
dtype: object
```

Cast col1 to int32 using a dictionary:

```pycon
>>> df.astype({"col1": "int32"}).dtypes
col1    int32
col2    int64
dtype: object
```

Create a series:

```pycon
>>> ser = pd.Series([1, 2], dtype="int32")
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype("int64")
0    1
1    2
dtype: int64
```

Convert to categorical type:

```pycon
>>> ser.astype("category")
0    1
1    2
dtype: category
Categories (2, int32): [1, 2]
```

Convert to ordered categorical type with custom ordering:

```pycon
>>> from pandas.api.types import CategoricalDtype
>>> cat_dtype = CategoricalDtype(categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]
```

Create a series of dates:

```pycon
>>> ser_date = pd.Series(pd.date_range("20200101", periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[us]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.autocorr.html.md

# dask.dataframe.Series.autocorr

#### Series.autocorr(lag=1, split_every=False)

Compute the lag-N autocorrelation.

This docstring was copied from pandas.Series.autocorr.

Some inconsistencies with the Dask version may exist.

This method computes the Pearson correlation between
the Series and its shifted self.

* **Parameters:**
  **lag**
  : Number of lags to apply before performing autocorrelation.
* **Returns:**
  float
  : The Pearson correlation between self and self.shift(lag).

#### SEE ALSO
[`Series.corr`](dask.dataframe.Series.corr.md#dask.dataframe.Series.corr)
: Compute the correlation between two Series.

[`Series.shift`](dask.dataframe.Series.shift.md#dask.dataframe.Series.shift)
: Shift index by desired number of periods.

[`DataFrame.corr`](dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)
: Compute pairwise correlation of columns.

`DataFrame.corrwith`
: Compute pairwise correlation between rows or columns of two DataFrame objects.

### Notes

If the Pearson correlation is not well defined return ‘NaN’.

### Examples

```pycon
>>> s = pd.Series([0.25, 0.5, 0.2, -0.05])
>>> s.autocorr()
0.10355...
>>> s.autocorr(lag=2)
-0.99999...
```

If the Pearson correlation is not well defined, then ‘NaN’ is returned.

```pycon
>>> s = pd.Series([1, 0, 0, 0])
>>> s.autocorr()
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.between.html.md

# dask.dataframe.Series.between

#### Series.between(left, right, inclusive='both')

Return boolean Series equivalent to left <= series <= right.

This docstring was copied from pandas.Series.between.

Some inconsistencies with the Dask version may exist.

This function returns a boolean vector containing True wherever the
corresponding Series element is between the boundary values left and
right. NA values are treated as False.

* **Parameters:**
  **left**
  : Left boundary.

  **right**
  : Right boundary.

  **inclusive**
  : Include boundaries. Whether to set each bound as closed or open.
* **Returns:**
  Series
  : Series representing whether each element is between left and
    right (inclusive).

#### SEE ALSO
[`Series.gt`](dask.dataframe.Series.gt.md#dask.dataframe.Series.gt)
: Greater than of series and other.

[`Series.lt`](dask.dataframe.Series.lt.md#dask.dataframe.Series.lt)
: Less than of series and other.

### Notes

This function is equivalent to `(left <= ser) & (ser <= right)`

### Examples

```pycon
>>> s = pd.Series([2, 0, 4, 8, np.nan])
```

Boundary values are included by default:

```pycon
>>> s.between(1, 4)
0     True
1    False
2     True
3    False
4    False
dtype: bool
```

With inclusive set to `"neither"` boundary values are excluded:

```pycon
>>> s.between(1, 4, inclusive="neither")
0     True
1    False
2    False
3    False
4    False
dtype: bool
```

left and right can be any scalar value:

```pycon
>>> s = pd.Series(["Alice", "Bob", "Carol", "Eve"])
>>> s.between("Anna", "Daniel")
0    False
1     True
2     True
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.bfill.html.md

# dask.dataframe.Series.bfill

#### Series.bfill(axis=0, limit=None)

Fill NA/NaN values by using the next valid observation to fill the gap.

This docstring was copied from pandas.DataFrame.bfill.

Some inconsistencies with the Dask version may exist.

This method fills missing values in a backward direction along the
specified axis, propagating non-null values from later positions to
earlier positions containing NaN.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.ffill`](dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)
: Fill NA/NaN values by propagating the last valid observation to next valid.

### Examples

For Series:

```pycon
>>> s = pd.Series([1, None, None, 2])
>>> s.bfill()
0    1.0
1    2.0
2    2.0
3    2.0
dtype: float64
>>> s.bfill(limit=1)
0    1.0
1    NaN
2    2.0
3    2.0
dtype: float64
```

With DataFrame:

```pycon
>>> df = pd.DataFrame({"A": [1, None, None, 4], "B": [None, 5, None, 7]})
>>> df
      A     B
0   1.0   NaN
1   NaN   5.0
2   NaN   NaN
3   4.0   7.0
>>> df.bfill()
      A     B
0   1.0   5.0
1   4.0   5.0
2   4.0   7.0
3   4.0   7.0
>>> df.bfill(limit=1)
      A     B
0   1.0   5.0
1   NaN   5.0
2   4.0   7.0
3   4.0   7.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.add_categories.html.md

# dask.dataframe.Series.cat.add_categories

#### dataframe.Series.cat.add_categories(new_categories) → Self

Add new categories.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.add_categories.

Some inconsistencies with the Dask version may exist.

new_categories will be included at the last/highest place in the
categories and will be unused directly after this call.

* **Parameters:**
  **new_categories**
  : The new categories to be included.
* **Returns:**
  Categorical
  : Categorical with new categories added.
* **Raises:**
  ValueError
  : If the new categories include old categories or do not validate as
    categories

#### SEE ALSO
[`rename_categories`](dask.dataframe.Series.cat.rename_categories.md#dask.dataframe.Series.cat.rename_categories)
: Rename categories.

[`reorder_categories`](dask.dataframe.Series.cat.reorder_categories.md#dask.dataframe.Series.cat.reorder_categories)
: Reorder categories.

[`remove_categories`](dask.dataframe.Series.cat.remove_categories.md#dask.dataframe.Series.cat.remove_categories)
: Remove the specified categories.

[`remove_unused_categories`](dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)
: Remove categories which are not used.

[`set_categories`](dask.dataframe.Series.cat.set_categories.md#dask.dataframe.Series.cat.set_categories)
: Set the categories to the specified ones.

### Examples

```pycon
>>> c = pd.Categorical(["c", "b", "c"])
>>> c
['c', 'b', 'c']
Categories (2, str): ['b', 'c']
```

```pycon
>>> c.add_categories(["d", "a"])
['c', 'b', 'c']
Categories (4, str): ['b', 'c', 'd', 'a']
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.as_known.html.md

# dask.dataframe.Series.cat.as_known

#### dataframe.Series.cat.as_known(\*\*kwargs)

Ensure the categories in this series are known.

If the categories are known, this is a no-op. If unknown, the
categories are computed, and a new series with known categories is
returned.

* **Parameters:**
  **kwargs**
  : Keywords to pass on to the call to compute.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.as_ordered.html.md

# dask.dataframe.Series.cat.as_ordered

#### dataframe.Series.cat.as_ordered() → Self

Set the Categorical to be ordered.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.as_ordered.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Categorical
  : Ordered Categorical.

#### SEE ALSO
[`as_unordered`](dask.dataframe.Series.cat.as_unordered.md#dask.dataframe.Series.cat.as_unordered)
: Set the Categorical to be unordered.

### Examples

For [`pandas.Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series):

```pycon
>>> ser = pd.Series(["a", "b", "c", "a"], dtype="category")
>>> ser.cat.ordered
False
>>> ser = ser.cat.as_ordered()
>>> ser.cat.ordered
True
```

For [`pandas.CategoricalIndex`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.CategoricalIndex.html#pandas.CategoricalIndex):

```pycon
>>> ci = pd.CategoricalIndex(["a", "b", "c", "a"])
>>> ci.ordered
False
>>> ci = ci.as_ordered()
>>> ci.ordered
True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.as_unknown.html.md

# dask.dataframe.Series.cat.as_unknown

#### dataframe.Series.cat.as_unknown()

Ensure the categories in this series are unknown

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.as_unordered.html.md

# dask.dataframe.Series.cat.as_unordered

#### dataframe.Series.cat.as_unordered() → Self

Set the Categorical to be unordered.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.as_unordered.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Categorical
  : Unordered Categorical.

#### SEE ALSO
[`as_ordered`](dask.dataframe.Series.cat.as_ordered.md#dask.dataframe.Series.cat.as_ordered)
: Set the Categorical to be ordered.

### Examples

For [`pandas.Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series):

```pycon
>>> raw_cat = pd.Categorical(["a", "b", "c", "a"], ordered=True)
>>> ser = pd.Series(raw_cat)
>>> ser.cat.ordered
True
>>> ser = ser.cat.as_unordered()
>>> ser.cat.ordered
False
```

For [`pandas.CategoricalIndex`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.CategoricalIndex.html#pandas.CategoricalIndex):

```pycon
>>> ci = pd.CategoricalIndex(["a", "b", "c", "a"], ordered=True)
>>> ci.ordered
True
>>> ci = ci.as_unordered()
>>> ci.ordered
False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.categories.html.md

# dask.dataframe.Series.cat.categories

#### dataframe.Series.cat.categories

The categories of this categorical.

If categories are unknown, an error is raised

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.codes.html.md

# dask.dataframe.Series.cat.codes

#### dataframe.Series.cat.codes

The codes of this categorical.

If categories are unknown, an error is raised

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.known.html.md

# dask.dataframe.Series.cat.known

#### dataframe.Series.cat.known

Whether the categories are fully known

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.ordered.html.md

# dask.dataframe.Series.cat.ordered

#### dataframe.Series.cat.ordered

Whether the categories have an ordered relationship

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.remove_categories.html.md

# dask.dataframe.Series.cat.remove_categories

#### dataframe.Series.cat.remove_categories(removals) → Self

Remove the specified categories.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.remove_categories.

Some inconsistencies with the Dask version may exist.

The `removals` argument must be a subset of the current categories.
Any values that were part of the removed categories will be set to NaN.

* **Parameters:**
  **removals**
  : The categories which should be removed.
* **Returns:**
  Categorical
  : Categorical with removed categories.
* **Raises:**
  ValueError
  : If the removals are not contained in the categories

#### SEE ALSO
[`rename_categories`](dask.dataframe.Series.cat.rename_categories.md#dask.dataframe.Series.cat.rename_categories)
: Rename categories.

[`reorder_categories`](dask.dataframe.Series.cat.reorder_categories.md#dask.dataframe.Series.cat.reorder_categories)
: Reorder categories.

[`add_categories`](dask.dataframe.Series.cat.add_categories.md#dask.dataframe.Series.cat.add_categories)
: Add new categories.

[`remove_unused_categories`](dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)
: Remove categories which are not used.

[`set_categories`](dask.dataframe.Series.cat.set_categories.md#dask.dataframe.Series.cat.set_categories)
: Set the categories to the specified ones.

### Examples

```pycon
>>> c = pd.Categorical(["a", "c", "b", "c", "d"])
>>> c
['a', 'c', 'b', 'c', 'd']
Categories (4, str): ['a', 'b', 'c', 'd']
```

```pycon
>>> c.remove_categories(["d", "a"])
[NaN, 'c', 'b', 'c', NaN]
Categories (2, str): ['b', 'c']
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.remove_unused_categories.html.md

# dask.dataframe.Series.cat.remove_unused_categories

#### dataframe.Series.cat.remove_unused_categories()

Removes categories which are not used

### Notes

This method requires a full scan of the data to compute the
unique values, which can be expensive.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.rename_categories.html.md

# dask.dataframe.Series.cat.rename_categories

#### dataframe.Series.cat.rename_categories(new_categories) → Self

Rename categories.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.rename_categories.

Some inconsistencies with the Dask version may exist.

This method is commonly used to re-label or adjust the
category names in categorical data without changing the
underlying data. It is useful in situations where you want
to modify the labels used for clarity, consistency,
or readability.

* **Parameters:**
  **new_categories**
  : New categories which will replace old categories.
    * list-like: all items must be unique and the number of items in
      the new categories must match the existing number of categories.
    * dict-like: specifies a mapping from
      old categories to new. Categories not contained in the mapping
      are passed through and extra categories in the mapping are
      ignored.
    * callable : a callable that is called on all items in the old
      categories and whose return values comprise the new categories.
* **Returns:**
  Categorical
  : Categorical with renamed categories.
* **Raises:**
  ValueError
  : If new categories are list-like and do not have the same number of
    items than the current categories or do not validate as categories

#### SEE ALSO
[`reorder_categories`](dask.dataframe.Series.cat.reorder_categories.md#dask.dataframe.Series.cat.reorder_categories)
: Reorder categories.

[`add_categories`](dask.dataframe.Series.cat.add_categories.md#dask.dataframe.Series.cat.add_categories)
: Add new categories.

[`remove_categories`](dask.dataframe.Series.cat.remove_categories.md#dask.dataframe.Series.cat.remove_categories)
: Remove the specified categories.

[`remove_unused_categories`](dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)
: Remove categories which are not used.

[`set_categories`](dask.dataframe.Series.cat.set_categories.md#dask.dataframe.Series.cat.set_categories)
: Set the categories to the specified ones.

### Examples

```pycon
>>> c = pd.Categorical(["a", "a", "b"])
>>> c.rename_categories([0, 1])
[0, 0, 1]
Categories (2, int64): [0, 1]
```

For dict-like `new_categories`, extra keys are ignored and
categories not in the dictionary are passed through

```pycon
>>> c.rename_categories({"a": "A", "c": "C"})
['A', 'A', 'b']
Categories (2, str): ['A', 'b']
```

You may also provide a callable to create the new categories

```pycon
>>> c.rename_categories(lambda x: x.upper())
['A', 'A', 'B']
Categories (2, str): ['A', 'B']
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.reorder_categories.html.md

# dask.dataframe.Series.cat.reorder_categories

#### dataframe.Series.cat.reorder_categories(new_categories, ordered=None) → Self

Reorder categories as specified in new_categories.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.reorder_categories.

Some inconsistencies with the Dask version may exist.

`new_categories` need to include all old categories and no new category
items.

* **Parameters:**
  **new_categories**
  : The categories in new order.

  **ordered**
  : Whether or not the categorical is treated as an ordered categorical.
    If not given, do not change the ordered information.
* **Returns:**
  Categorical
  : Categorical with reordered categories.
* **Raises:**
  ValueError
  : If the new categories do not contain all old category items or any
    new ones

#### SEE ALSO
[`rename_categories`](dask.dataframe.Series.cat.rename_categories.md#dask.dataframe.Series.cat.rename_categories)
: Rename categories.

[`add_categories`](dask.dataframe.Series.cat.add_categories.md#dask.dataframe.Series.cat.add_categories)
: Add new categories.

[`remove_categories`](dask.dataframe.Series.cat.remove_categories.md#dask.dataframe.Series.cat.remove_categories)
: Remove the specified categories.

[`remove_unused_categories`](dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)
: Remove categories which are not used.

[`set_categories`](dask.dataframe.Series.cat.set_categories.md#dask.dataframe.Series.cat.set_categories)
: Set the categories to the specified ones.

### Examples

For [`pandas.Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series):

```pycon
>>> ser = pd.Series(["a", "b", "c", "a"], dtype="category")
>>> ser = ser.cat.reorder_categories(["c", "b", "a"], ordered=True)
>>> ser
0   a
1   b
2   c
3   a
dtype: category
Categories (3, str): ['c' < 'b' < 'a']
```

```pycon
>>> ser.sort_values()
2   c
1   b
0   a
3   a
dtype: category
Categories (3, str): ['c' < 'b' < 'a']
```

For [`pandas.CategoricalIndex`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.CategoricalIndex.html#pandas.CategoricalIndex):

```pycon
>>> ci = pd.CategoricalIndex(["a", "b", "c", "a"])
>>> ci
CategoricalIndex(['a', 'b', 'c', 'a'], categories=['a', 'b', 'c'],
                 ordered=False, dtype='category')
>>> ci.reorder_categories(["c", "b", "a"], ordered=True)
CategoricalIndex(['a', 'b', 'c', 'a'], categories=['c', 'b', 'a'],
                 ordered=True, dtype='category')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cat.set_categories.html.md

# dask.dataframe.Series.cat.set_categories

#### dataframe.Series.cat.set_categories(new_categories, ordered=None, rename: [bool](https://docs.python.org/3/library/functions.html#bool) = False) → Self

Set the categories to the specified new categories.

This docstring was copied from pandas.core.arrays.categorical.CategoricalAccessor.set_categories.

Some inconsistencies with the Dask version may exist.

`new_categories` can include new categories (which will result in
unused categories) or remove old categories (which results in values
set to `NaN`). If `rename=True`, the categories will simply be renamed
(less or more items than in old categories will result in values set to
`NaN` or in unused categories respectively).

This method can be used to perform more than one action of adding,
removing, and reordering simultaneously and is therefore faster than
performing the individual steps via the more specialised methods.

On the other hand this methods does not do checks (e.g., whether the
old categories are included in the new categories on a reorder), which
can result in surprising changes, for example when using special string
dtypes, which do not consider a S1 string equal to a single char
python string.

* **Parameters:**
  **new_categories**
  : The categories in new order.

  **ordered**
  : Whether or not the categorical is treated as an ordered categorical.
    If not given, do not change the ordered information.

  **rename**
  : Whether or not the new_categories should be considered as a rename
    of the old categories or as reordered categories.
* **Returns:**
  Categorical
  : New categories to be used, with optional ordering changes.
* **Raises:**
  ValueError
  : If new_categories does not validate as categories

#### SEE ALSO
[`rename_categories`](dask.dataframe.Series.cat.rename_categories.md#dask.dataframe.Series.cat.rename_categories)
: Rename categories.

[`reorder_categories`](dask.dataframe.Series.cat.reorder_categories.md#dask.dataframe.Series.cat.reorder_categories)
: Reorder categories.

[`add_categories`](dask.dataframe.Series.cat.add_categories.md#dask.dataframe.Series.cat.add_categories)
: Add new categories.

[`remove_categories`](dask.dataframe.Series.cat.remove_categories.md#dask.dataframe.Series.cat.remove_categories)
: Remove the specified categories.

[`remove_unused_categories`](dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)
: Remove categories which are not used.

### Examples

For [`pandas.Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series):

```pycon
>>> raw_cat = pd.Categorical(
...     ["a", "b", "c", None], categories=["a", "b", "c"], ordered=True
... )
>>> ser = pd.Series(raw_cat)
>>> ser
0   a
1   b
2   c
3   NaN
dtype: category
Categories (3, str): ['a' < 'b' < 'c']
```

```pycon
>>> ser.cat.set_categories(["A", "B", "C"], rename=True)
0   A
1   B
2   C
3   NaN
dtype: category
Categories (3, str): ['A' < 'B' < 'C']
```

For [`pandas.CategoricalIndex`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.CategoricalIndex.html#pandas.CategoricalIndex):

```pycon
>>> ci = pd.CategoricalIndex(
...     ["a", "b", "c", None], categories=["a", "b", "c"], ordered=True
... )
>>> ci
CategoricalIndex(['a', 'b', 'c', nan], categories=['a', 'b', 'c'],
                 ordered=True, dtype='category')
```

```pycon
>>> ci.set_categories(["A", "b", "c"])
CategoricalIndex([nan, 'b', 'c', nan], categories=['A', 'b', 'c'],
                 ordered=True, dtype='category')
>>> ci.set_categories(["A", "b", "c"], rename=True)
CategoricalIndex(['A', 'b', 'c', nan], categories=['A', 'b', 'c'],
                 ordered=True, dtype='category')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.clear_divisions.html.md

# dask.dataframe.Series.clear_divisions

#### Series.clear_divisions()

Forget division information.

This is useful if the divisions are no longer meaningful.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.clip.html.md

# dask.dataframe.Series.clip

#### Series.clip(lower=None, upper=None, axis=None, \*\*kwargs)

Trim values at input threshold(s).

This docstring was copied from pandas.Series.clip.

Some inconsistencies with the Dask version may exist.

Assigns values outside boundary to boundary values. Thresholds
can be singular values or array like, and in the latter case
the clipping is performed element-wise in the specified axis.

* **Parameters:**
  **lower**
  : Minimum threshold value. All values below this
    threshold will be set to it. A missing
    threshold (e.g NA) will not clip the value.

  **upper**
  : Maximum threshold value. All values above this
    threshold will be set to it. A missing
    threshold (e.g NA) will not clip the value.

  **axis**
  : Align object with lower and upper along the given axis.
    For Series this parameter is unused and defaults to None.

  **inplace**
  : Whether to perform the operation in place on the data.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted
    for compatibility with numpy.
* **Returns:**
  Series or DataFrame
  : Same type as calling object with the values outside the
    clip boundaries replaced.

#### SEE ALSO
[`Series.clip`](#dask.dataframe.Series.clip)
: Trim values at input threshold in series.

`DataFrame.clip`
: Trim values at input threshold in DataFrame.

[`numpy.clip`](https://numpy.org/doc/stable/reference/generated/numpy.clip.html#numpy.clip)
: Clip (limit) the values in an array.

### Examples

```pycon
>>> data = {"col_0": [9, -3, 0, -1, 5], "col_1": [-2, -7, 6, 8, -5]}
>>> df = pd.DataFrame(data)
>>> df
   col_0  col_1
0      9     -2
1     -3     -7
2      0      6
3     -1      8
4      5     -5
```

Clips per column using lower and upper thresholds:

```pycon
>>> df.clip(-4, 6)
   col_0  col_1
0      6     -2
1     -3     -4
2      0      6
3     -1      6
4      5     -4
```

Clips using specific lower and upper thresholds per column:

```pycon
>>> df.clip([-2, -1], [4, 5])
    col_0  col_1
0      4     -1
1     -2     -1
2      0      5
3     -1      5
4      4     -1
```

Clips using specific lower and upper thresholds per column element:

```pycon
>>> t = pd.Series([2, -4, -1, 6, 3])
>>> t
0    2
1   -4
2   -1
3    6
4    3
dtype: int64
```

```pycon
>>> df.clip(t, t + 4, axis=0)
   col_0  col_1
0      6      2
1     -3     -4
2      0      3
3      6      8
4      5      3
```

Clips using specific lower threshold per column element, with missing values:

```pycon
>>> t = pd.Series([2, -4, np.nan, 6, 3])
>>> t
0    2.0
1   -4.0
2    NaN
3    6.0
4    3.0
dtype: float64
```

```pycon
>>> df.clip(t, axis=0)
col_0  col_1
0      9.0    2.0
1     -3.0   -4.0
2      0.0    6.0
3      6.0    8.0
4      5.0    3.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.compute.html.md

# dask.dataframe.Series.compute

#### Series.compute(\*\*kwargs)

Compute this dask collection

This turns a lazy Dask collection into its in-memory equivalent.
For example a Dask array turns into a NumPy array and a Dask dataframe
turns into a Pandas dataframe.  The entire dataset must fit into memory
before calling this operation.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

#### SEE ALSO
[`dask.compute`](../api.md#dask.compute)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.copy.html.md

# dask.dataframe.Series.copy

#### Series.copy(deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Make a copy of the dataframe

This is strictly a shallow copy of the underlying computational graph.
It does not affect the underlying data

* **Parameters:**
  **deep**
  : The deep value must be False and it is declared as a parameter just for
    compatibility with third-party libraries like cuDF and pandas

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.corr.html.md

# dask.dataframe.Series.corr

#### Series.corr(other, method='pearson', min_periods=None, split_every=False)

Compute correlation with other Series, excluding missing values.

This docstring was copied from pandas.Series.corr.

Some inconsistencies with the Dask version may exist.

The two Series objects are not required to be the same length and will be
aligned internally before the correlation function is applied.

* **Parameters:**
  **other**
  : Series with which to compute the correlation.

  **method**
  : Method used to compute correlation:
    - pearson : Standard correlation coefficient
    - kendall : Kendall Tau correlation coefficient
    - spearman : Spearman rank correlation
    - callable: Callable with input two 1d ndarrays and returning a float.
    <br/>
    #### WARNING
    Note that the returned matrix from corr will have 1 along the
    diagonals and will be symmetric regardless of the callable’s
    behavior.

  **min_periods**
  : Minimum number of observations needed to have a valid result.
* **Returns:**
  float
  : Correlation with other.

#### SEE ALSO
[`DataFrame.corr`](dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)
: Compute pairwise correlation between columns.

`DataFrame.corrwith`
: Compute pairwise correlation with another DataFrame or Series.

### Notes

Pearson, Kendall and Spearman correlation are currently computed using pairwise complete observations.

* [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)
* [Kendall rank correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)
* [Spearman’s rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient)

Automatic data alignment: as with all pandas operations, automatic data alignment is performed for this method.
`corr()` automatically considers values with matching indices.

### Examples

```pycon
>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> s1 = pd.Series([0.2, 0.0, 0.6, 0.2])
>>> s2 = pd.Series([0.3, 0.6, 0.0, 0.1])
>>> s1.corr(s2, method=histogram_intersection)
0.3
```

Pandas auto-aligns the values with matching indices

```pycon
>>> s1 = pd.Series([1, 2, 3], index=[0, 1, 2])
>>> s2 = pd.Series([1, 2, 3], index=[2, 1, 0])
>>> s1.corr(s2)
-1.0
```

If the input is a constant array, the correlation is not defined in this case,
and `np.nan` is returned.

```pycon
>>> s1 = pd.Series([0.45, 0.45])
>>> s1.corr(s1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.count.html.md

# dask.dataframe.Series.count

#### Series.count(axis=0, numeric_only=False, split_every=False)

Count non-NA cells for each column or row.

This docstring was copied from pandas.DataFrame.count.

Some inconsistencies with the Dask version may exist.

The values None, NaN, NaT, `pandas.NA` are considered NA.

* **Parameters:**
  **axis**
  : If 0 or ‘index’ counts are generated for each column.
    If 1 or ‘columns’ counts are generated for each row.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : For each column/row the number of non-NA/null entries.

#### SEE ALSO
[`Series.count`](#dask.dataframe.Series.count)
: Number of non-NA elements in a Series.

`DataFrame.value_counts`
: Count unique combinations of columns.

[`DataFrame.shape`](dask.dataframe.DataFrame.shape.md#dask.dataframe.DataFrame.shape)
: Number of DataFrame rows and columns (including NA elements).

[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Boolean same-sized DataFrame showing places of NA elements.

### Examples

Constructing DataFrame from a dictionary:

```pycon
>>> df = pd.DataFrame(
...     {
...         "Person": ["John", "Myla", "Lewis", "John", "Myla"],
...         "Age": [24.0, np.nan, 21.0, 33, 26],
...         "Single": [False, True, True, True, False],
...     }
... )
>>> df
   Person   Age  Single
0    John  24.0   False
1    Myla   NaN    True
2   Lewis  21.0    True
3    John  33.0    True
4    Myla  26.0   False
```

Notice the uncounted NA values:

```pycon
>>> df.count()
Person    5
Age       4
Single    5
dtype: int64
```

Counts for each **row**:

```pycon
>>> df.count(axis="columns")
0    3
1    2
2    3
3    3
4    3
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cov.html.md

# dask.dataframe.Series.cov

#### Series.cov(other, min_periods=None, split_every=False)

Compute covariance with Series, excluding missing values.

This docstring was copied from pandas.Series.cov.

Some inconsistencies with the Dask version may exist.

The two Series objects are not required to be the same length and
will be aligned internally before the covariance is calculated.

* **Parameters:**
  **other**
  : Series with which to compute the covariance.

  **min_periods**
  : Minimum number of observations needed to have a valid result.

  **ddof**
  : Delta degrees of freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.
* **Returns:**
  float
  : Covariance between Series and other normalized by N-1
    (unbiased estimator).

#### SEE ALSO
[`DataFrame.cov`](dask.dataframe.DataFrame.cov.md#dask.dataframe.DataFrame.cov)
: Compute pairwise covariance of columns.

### Examples

```pycon
>>> s1 = pd.Series([0.90010907, 0.13484424, 0.62036035])
>>> s2 = pd.Series([0.12528585, 0.26962463, 0.51111198])
>>> s1.cov(s2)
-0.01685762652715874
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cummax.html.md

# dask.dataframe.Series.cummax

#### Series.cummax(axis=0, skipna=True)

Return cumulative maximum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummax.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
maximum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative maximum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.max`
: Similar functionality but ignores `NaN` values.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummax()
0    2.0
1    NaN
2    5.0
3    5.0
4    5.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummax(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the maximum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummax()
     A    B
0  2.0  1.0
1  3.0  NaN
2  3.0  1.0
```

To iterate over columns and find the maximum in each row,
use `axis=1`

```pycon
>>> df.cummax(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cummin.html.md

# dask.dataframe.Series.cummin

#### Series.cummin(axis=0, skipna=True)

Return cumulative minimum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cummin.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
minimum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative minimum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.min`
: Similar functionality but ignores `NaN` values.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cummin()
0    2.0
1    NaN
2    2.0
3   -1.0
4   -1.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cummin(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the minimum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cummin()
     A    B
0  2.0  1.0
1  2.0  NaN
2  1.0  0.0
```

To iterate over columns and find the minimum in each row,
use `axis=1`

```pycon
>>> df.cummin(axis=1)
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cumprod.html.md

# dask.dataframe.Series.cumprod

#### Series.cumprod(axis=0, skipna=True, \*\*kwargs)

Return cumulative product over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumprod.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
product.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative product of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.prod`
: Similar functionality but ignores `NaN` values.

[`DataFrame.prod`](dask.dataframe.DataFrame.prod.md#dask.dataframe.DataFrame.prod)
: Return the product over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumprod()
0     2.0
1     NaN
2    10.0
3   -10.0
4    -0.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumprod(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the product
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumprod()
     A    B
0  2.0  1.0
1  6.0  NaN
2  6.0  0.0
```

To iterate over columns and find the product in each row,
use `axis=1`

```pycon
>>> df.cumprod(axis=1)
     A    B
0  2.0  2.0
1  3.0  NaN
2  1.0  0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.cumsum.html.md

# dask.dataframe.Series.cumsum

#### Series.cumsum(axis=0, skipna=True, \*\*kwargs)

Return cumulative sum over a DataFrame or Series axis.

This docstring was copied from pandas.DataFrame.cumsum.

Some inconsistencies with the Dask version may exist.

Returns a DataFrame or Series of the same size containing the cumulative
sum.

* **Parameters:**
  **axis**
  : The index or the name of the axis. 0 is equivalent to None or ‘index’.
    For Series this parameter is unused and defaults to 0.

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args, \*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with NumPy.
* **Returns:**
  Series or DataFrame
  : Return cumulative sum of Series or DataFrame.

#### SEE ALSO
`core.window.expanding.Expanding.sum`
: Similar functionality but ignores `NaN` values.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over DataFrame axis.

[`DataFrame.cummax`](dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)
: Return cumulative maximum over DataFrame axis.

[`DataFrame.cummin`](dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)
: Return cumulative minimum over DataFrame axis.

[`DataFrame.cumsum`](dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)
: Return cumulative sum over DataFrame axis.

[`DataFrame.cumprod`](dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)
: Return cumulative product over DataFrame axis.

### Examples

**Series**

```pycon
>>> s = pd.Series([2, np.nan, 5, -1, 0])
>>> s
0    2.0
1    NaN
2    5.0
3   -1.0
4    0.0
dtype: float64
```

By default, NA values are ignored.

```pycon
>>> s.cumsum()
0    2.0
1    NaN
2    7.0
3    6.0
4    6.0
dtype: float64
```

To include NA values in the operation, use `skipna=False`

```pycon
>>> s.cumsum(skipna=False)
0    2.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

**DataFrame**

```pycon
>>> df = pd.DataFrame(
...     [[2.0, 1.0], [3.0, np.nan], [1.0, 0.0]], columns=list("AB")
... )
>>> df
     A    B
0  2.0  1.0
1  3.0  NaN
2  1.0  0.0
```

By default, iterates over rows and finds the sum
in each column. This is equivalent to `axis=None` or `axis='index'`.

```pycon
>>> df.cumsum()
     A    B
0  2.0  1.0
1  5.0  NaN
2  6.0  1.0
```

To iterate over columns and find the sum in each row,
use `axis=1`

```pycon
>>> df.cumsum(axis=1)
     A    B
0  2.0  3.0
1  3.0  NaN
2  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.describe.html.md

# dask.dataframe.Series.describe

#### Series.describe(split_every=False, percentiles=None, percentiles_method='default', include=None, exclude=None)

Generate descriptive statistics.

This docstring was copied from pandas.Series.describe.

Some inconsistencies with the Dask version may exist.

Generate descriptive statistics.

Dask computes percentiles (used for the `25%`, `50%`, and
`75%` statistics) using an **approximate algorithm** by default.
Results may therefore differ slightly from pandas.  Use
`percentiles_method="dask"` for the built-in Dask algorithm or
`percentiles_method="tdigest"` for the t-digest algorithm.
See [`dask.dataframe.Series.quantile()`](dask.dataframe.Series.quantile.md#dask.dataframe.Series.quantile) for details.

* **Parameters:**
  **split_every**
  : Number of partitions to aggregate at once. Defaults to `False`
    which uses a single-pass reduction over all partitions.

  **percentiles**
  : The percentiles to include in the output. All should fall
    between 0 and 1. By default, `[0.25, 0.5, 0.75]` is used.

  **percentiles_method**
  : Method for computing percentiles. `"default"` uses the internal
    Dask algorithm. `"tdigest"` uses the t-digest algorithm for
    floats and ints and falls back to `"dask"` otherwise.

  **Descriptive statistics include those that summarize the central**

  **tendency, dispersion and shape of a**

  **dataset’s distribution, excluding \`\`NaN\`\` values.**

  **Analyzes both numeric and object series, as well**

  **as \`\`DataFrame\`\` column sets of mixed data types. The output**

  **will vary depending on what is provided. Refer to the notes**

  **below for more detail.**
* **Returns:**
  Series or DataFrame
  : Summary statistics of the Series or Dataframe provided.

#### SEE ALSO
[`DataFrame.count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)
: Count number of non-NA/null observations.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Maximum of the values in the object.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Minimum of the values in the object.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Mean of the values.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Standard deviation of the observations.

[`DataFrame.select_dtypes`](dask.dataframe.DataFrame.select_dtypes.md#dask.dataframe.DataFrame.select_dtypes)
: Subset of a DataFrame including/excluding columns based on their dtype.

### Notes

For numeric data, the result’s index will include `count`,
`mean`, `std`, `min`, `max` as well as lower, `50` and
upper percentiles. By default the lower percentile is `25` and the
upper percentile is `75`. The `50` percentile is the
same as the median.

For object data (e.g. strings), the result’s index
will include `count`, `unique`, `top`, and `freq`. The `top`
is the most common value. The `freq` is the most common value’s
frequency.

If multiple object values have the highest count, then the
`count` and `top` results will be arbitrarily chosen from
among those with the highest count.

For mixed data types provided via a `DataFrame`, the default is to
return only an analysis of numeric columns. If the DataFrame consists
only of object and categorical data without any numeric columns, the
default is to return an analysis of both the object and categorical
columns. If `include='all'` is provided as an option, the result
will include a union of attributes of each type.

The include and exclude parameters can be used to limit
which columns in a `DataFrame` are analyzed for the output.
The parameters are ignored when analyzing a `Series`.

### Examples

Describing a numeric `Series`.

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
dtype: float64
```

Describing a categorical `Series`.

```pycon
>>> s = pd.Series(["a", "a", "b", "c"])
>>> s.describe()
count     4
unique    3
top       a
freq      2
dtype: object
```

Describing a timestamp `Series`.

```pycon
>>> s = pd.Series(
...     [
...         np.datetime64("2000-01-01"),
...         np.datetime64("2010-01-01"),
...         np.datetime64("2010-01-01"),
...     ]
... )
>>> s.describe()
count                      3
mean     2006-09-01 08:00:00
min      2000-01-01 00:00:00
25%      2004-12-31 12:00:00
50%      2010-01-01 00:00:00
75%      2010-01-01 00:00:00
max      2010-01-01 00:00:00
dtype: object
```

Describing a `DataFrame`. By default only numeric fields
are returned.

```pycon
>>> df = pd.DataFrame(
...     {
...         "categorical": pd.Categorical(["d", "e", "f"]),
...         "numeric": [1, 2, 3],
...         "object": ["a", "b", "c"],
...     }
... )
>>> df.describe()
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Describing all columns of a `DataFrame` regardless of data type.

```pycon
>>> df.describe(include="all")
       categorical  numeric object
count            3      3.0      3
unique           3      NaN      3
top              f      NaN      a
freq             1      NaN      1
mean           NaN      2.0    NaN
std            NaN      1.0    NaN
min            NaN      1.0    NaN
25%            NaN      1.5    NaN
50%            NaN      2.0    NaN
75%            NaN      2.5    NaN
max            NaN      3.0    NaN
```

Describing a column from a `DataFrame` by accessing it as
an attribute.

```pycon
>>> df.numeric.describe()
count    3.0
mean     2.0
std      1.0
min      1.0
25%      1.5
50%      2.0
75%      2.5
max      3.0
Name: numeric, dtype: float64
```

Including only numeric columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[np.number])
       numeric
count      3.0
mean       2.0
std        1.0
min        1.0
25%        1.5
50%        2.0
75%        2.5
max        3.0
```

Including only string columns in a `DataFrame` description.

```pycon
>>> df.describe(include=[object])
       object
count       3
unique      3
top         a
freq        1
```

Including only categorical columns from a `DataFrame` description.

```pycon
>>> df.describe(include=["category"])
       categorical
count            3
unique           3
top              d
freq             1
```

Excluding numeric columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[np.number])
       categorical object
count            3      3
unique           3      3
top              f      a
freq             1      1
```

Excluding object columns from a `DataFrame` description.

```pycon
>>> df.describe(exclude=[object])
       categorical  numeric
count            3      3.0
unique           3      NaN
top              f      NaN
freq             1      NaN
mean           NaN      2.0
std            NaN      1.0
min            NaN      1.0
25%            NaN      1.5
50%            NaN      2.0
75%            NaN      2.5
max            NaN      3.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.diff.html.md

# dask.dataframe.Series.diff

#### Series.diff(periods=1, axis=0)

First discrete difference of element.

This docstring was copied from pandas.DataFrame.diff.

Some inconsistencies with the Dask version may exist.

#### NOTE
Pandas currently uses an `object`-dtype column to represent
boolean data with missing values. This can cause issues for
boolean-specific operations, like `|`. To enable boolean-
specific operations, at the cost of metadata that doesn’t match
pandas, use `.astype(bool)` after the `shift`.

Calculates the difference of a DataFrame element compared with another
element in the DataFrame (default is element in previous row).

* **Parameters:**
  **periods**
  : Periods to shift for calculating difference, accepts negative
    values.

  **axis**
  : Take difference over rows (0) or columns (1).
* **Returns:**
  DataFrame
  : First differences of the Series.

#### SEE ALSO
`DataFrame.pct_change`
: Percent change over given number of periods.

`DataFrame.shift`
: Shift index by desired number of periods with an optional time freq.

[`Series.diff`](#dask.dataframe.Series.diff)
: First discrete difference of object.

### Notes

For boolean dtypes, this uses `operator.xor()` rather than
`operator.sub()`.
The result is calculated according to current dtype in DataFrame,
however dtype of the result is always float64.

### Examples

Difference with previous row

```pycon
>>> df = pd.DataFrame(
...     {
...         "a": [1, 2, 3, 4, 5, 6],
...         "b": [1, 1, 2, 3, 5, 8],
...         "c": [1, 4, 9, 16, 25, 36],
...     }
... )
>>> df
   a  b   c
0  1  1   1
1  2  1   4
2  3  2   9
3  4  3  16
4  5  5  25
5  6  8  36
>>> df.diff()
     a    b     c
0  NaN  NaN   NaN
1  1.0  0.0   3.0
2  1.0  1.0   5.0
3  1.0  1.0   7.0
4  1.0  2.0   9.0
5  1.0  3.0  11.0
```

Difference with previous column

```pycon
>>> df.diff(axis=1)
    a  b   c
0 NaN  0   0
1 NaN -1   3
2 NaN -1   7
3 NaN -1  13
4 NaN  0  20
5 NaN  2  28
```

Difference with 3rd previous row

```pycon
>>> df.diff(periods=3)
     a    b     c
0  NaN  NaN   NaN
1  NaN  NaN   NaN
2  NaN  NaN   NaN
3  3.0  2.0  15.0
4  3.0  4.0  21.0
5  3.0  6.0  27.0
```

Difference with following row

```pycon
>>> df.diff(periods=-1)
     a    b     c
0 -1.0  0.0  -3.0
1 -1.0 -1.0  -5.0
2 -1.0 -1.0  -7.0
3 -1.0 -2.0  -9.0
4 -1.0 -3.0 -11.0
5  NaN  NaN   NaN
```

Overflow in input dtype

```pycon
>>> df = pd.DataFrame({"a": [1, 0]}, dtype=np.uint8)
>>> df.diff()
       a
0    NaN
1  255.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.div.html.md

# dask.dataframe.Series.div

#### Series.div(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.drop_duplicates.html.md

# dask.dataframe.Series.drop_duplicates

#### Series.drop_duplicates(ignore_index=False, split_every=None, split_out=True, shuffle_method=None, keep='first')

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dropna.html.md

# dask.dataframe.Series.dropna

#### Series.dropna()

Return a new Series with missing values removed.

This docstring was copied from pandas.Series.dropna.

Some inconsistencies with the Dask version may exist.

See the [User Guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data) for more on which values are
considered missing, and how to work with missing data.

* **Parameters:**
  **axis**
  : Unused. Parameter needed for compatibility with DataFrame.

  **inplace**
  : If True, do operation inplace and return None.

  **how**
  : Not in use. Kept for compatibility.

  **ignore_index**
  : If `True`, the resulting axis will be labeled 0, 1, …, n - 1.
    <br/>
    #### Versionadded
    Added in version 2.0.0.
* **Returns:**
  Series or None
  : Series with NA entries dropped from it or None if `inplace=True`.

#### SEE ALSO
[`Series.isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)
: Indicate missing values.

`Series.notna`
: Indicate existing (non-missing) values.

[`Series.fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)
: Replace missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Drop rows or columns which contain NA values.

[`Index.dropna`](dask.dataframe.Index.dropna.md#dask.dataframe.Index.dropna)
: Drop missing indices.

### Examples

```pycon
>>> ser = pd.Series([1.0, 2.0, np.nan])
>>> ser
0    1.0
1    2.0
2    NaN
dtype: float64
```

Drop NA values from a Series.

```pycon
>>> ser.dropna()
0    1.0
1    2.0
dtype: float64
```

Empty strings are not considered NA values. `None` is considered an
NA value.

```pycon
>>> ser = pd.Series([np.nan, 2, pd.NaT, "", None, "I stay"])
>>> ser
0       NaN
1         2
2       NaT
3
4      None
5    I stay
dtype: object
>>> ser.dropna()
1         2
3
5    I stay
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.ceil.html.md

# dask.dataframe.Series.dt.ceil

#### dataframe.Series.dt.ceil(freq, ambiguous: TimeAmbiguous = 'raise', nonexistent: TimeNonexistent = 'raise') → Self

Perform ceil operation on the data to the specified freq.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.ceil.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **freq**
  : The frequency level to ceil the index to. Must be a fixed
    frequency like ‘s’ (second) not ‘ME’ (month end). See
    [frequency aliases](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases) for
    a list of possible freq values.

  **ambiguous**
  : Only relevant for DatetimeIndex:
    - ‘infer’ will attempt to infer fall dst-transition hours based on
      order
    - bool-ndarray where True signifies a DST time, False designates
      a non-DST time (note that this flag is only applicable for
      ambiguous times)
    - ‘NaT’ will return NaT where there are ambiguous times
    - ‘raise’ will raise a ValueError if there are ambiguous
      times.

  **nonexistent**
  : A nonexistent time does not exist in a particular timezone
    where clocks moved forward due to DST.
    - ‘shift_forward’ will shift the nonexistent time forward to the
      closest existing time
    - ‘shift_backward’ will shift the nonexistent time backward to the
      closest existing time
    - ‘NaT’ will return NaT where there are nonexistent times
    - timedelta objects will shift nonexistent times by the timedelta
    - ‘raise’ will raise a ValueError if there are
      nonexistent times.
* **Returns:**
  DatetimeIndex, TimedeltaIndex, or Series
  : Index of the same type for a DatetimeIndex or TimedeltaIndex,
    or a Series with the same index for a Series.
* **Raises:**
  ValueError if the freq cannot be converted.

#### SEE ALSO
`DatetimeIndex.floor`
: Perform floor operation on the data to the specified freq.

`DatetimeIndex.snap`
: Snap time stamps to nearest occurring frequency.

### Notes

If the timestamps have a timezone, ceiling will take place relative to the
local (“wall”) time and re-localized to the same timezone. When ceiling
near daylight savings time, use `nonexistent` and `ambiguous` to
control the re-localization behavior.

### Examples

**DatetimeIndex**

```pycon
>>> rng = pd.date_range("1/1/2018 11:59:00", periods=3, freq="min")
>>> rng
DatetimeIndex(['2018-01-01 11:59:00', '2018-01-01 12:00:00',
               '2018-01-01 12:01:00'],
              dtype='datetime64[us]', freq='min')
```

```pycon
>>> rng.ceil('h')
DatetimeIndex(['2018-01-01 12:00:00', '2018-01-01 12:00:00',
               '2018-01-01 13:00:00'],
              dtype='datetime64[us]', freq=None)
```

**Series**

```pycon
>>> pd.Series(rng).dt.ceil("h")
0   2018-01-01 12:00:00
1   2018-01-01 12:00:00
2   2018-01-01 13:00:00
dtype: datetime64[us]
```

When rounding near a daylight savings time transition, use `ambiguous` or
`nonexistent` to control how the timestamp should be re-localized.

```pycon
>>> rng_tz = pd.DatetimeIndex(["2021-10-31 01:30:00"], tz="Europe/Amsterdam")
```

```pycon
>>> rng_tz.ceil("h", ambiguous=False)
DatetimeIndex(['2021-10-31 02:00:00+01:00'],
              dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

```pycon
>>> rng_tz.ceil("h", ambiguous=True)
DatetimeIndex(['2021-10-31 02:00:00+02:00'],
              dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.date.html.md

# dask.dataframe.Series.dt.date

#### dataframe.Series.dt.date

Returns numpy array of python [`datetime.date`](https://docs.python.org/3/library/datetime.html#datetime.date) objects.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.date.

Some inconsistencies with the Dask version may exist.

Namely, the date part of Timestamps without time and
timezone information.

#### SEE ALSO
`DatetimeIndex.time`
: Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects. The time part of the Timestamps.

`DatetimeIndex.year`
: The year of the datetime.

`DatetimeIndex.month`
: The month as January=1, December=12.

`DatetimeIndex.day`
: The day of the datetime.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.date
0    2020-01-01
1    2020-02-01
dtype: object
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(
...     ["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"]
... )
>>> idx.date
array([datetime.date(2020, 1, 1), datetime.date(2020, 2, 1)], dtype=object)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.day.html.md

# dask.dataframe.Series.dt.day

#### dataframe.Series.dt.day

The day of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.day.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.year`
: The year of the datetime.

`DatetimeIndex.month`
: The month as January=1, December=12.

`DatetimeIndex.hour`
: The hours of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="D")
... )
>>> datetime_series
0   2000-01-01
1   2000-01-02
2   2000-01-03
dtype: datetime64[us]
>>> datetime_series.dt.day
0    1
1    2
2    3
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.day_of_week.html.md

# dask.dataframe.Series.dt.day_of_week

#### dataframe.Series.dt.day_of_week

The day of the week with Monday=0, Sunday=6.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.day_of_week.

Some inconsistencies with the Dask version may exist.

Return the day of the week. It is assumed the week starts on
Monday, which is denoted by 0 and ends on Sunday which is denoted
by 6. This method is available on both Series with datetime
values (using the dt accessor) or DatetimeIndex.

* **Returns:**
  Series or Index
  : Containing integers indicating the day number.

#### SEE ALSO
`Series.dt.dayofweek`
: Alias.

`Series.dt.weekday`
: Alias.

`Series.dt.day_name`
: Returns the name of the day of the week.

### Examples

```pycon
>>> s = pd.date_range('2016-12-31', '2017-01-08', freq='D').to_series()
>>> s.dt.dayofweek
2016-12-31    5
2017-01-01    6
2017-01-02    0
2017-01-03    1
2017-01-04    2
2017-01-05    3
2017-01-06    4
2017-01-07    5
2017-01-08    6
Freq: D, dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.day_of_year.html.md

# dask.dataframe.Series.dt.day_of_year

#### dataframe.Series.dt.day_of_year

The ordinal day of the year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.day_of_year.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.dayofweek`
: The day of the week with Monday=0, Sunday=6.

`DatetimeIndex.day`
: The day of the datetime.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.dayofyear
0    1
1   32
dtype: int32
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(["1/1/2020 10:00:00+00:00",
...                         "2/1/2020 11:00:00+00:00"])
>>> idx.dayofyear
Index([1, 32], dtype='int32')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.dayofweek.html.md

# dask.dataframe.Series.dt.dayofweek

#### dataframe.Series.dt.dayofweek

The day of the week with Monday=0, Sunday=6.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.dayofweek.

Some inconsistencies with the Dask version may exist.

Return the day of the week. It is assumed the week starts on
Monday, which is denoted by 0 and ends on Sunday which is denoted
by 6. This method is available on both Series with datetime
values (using the dt accessor) or DatetimeIndex.

* **Returns:**
  Series or Index
  : Containing integers indicating the day number.

#### SEE ALSO
`Series.dt.dayofweek`
: Alias.

`Series.dt.weekday`
: Alias.

`Series.dt.day_name`
: Returns the name of the day of the week.

### Examples

```pycon
>>> s = pd.date_range('2016-12-31', '2017-01-08', freq='D').to_series()
>>> s.dt.dayofweek
2016-12-31    5
2017-01-01    6
2017-01-02    0
2017-01-03    1
2017-01-04    2
2017-01-05    3
2017-01-06    4
2017-01-07    5
2017-01-08    6
Freq: D, dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.dayofyear.html.md

# dask.dataframe.Series.dt.dayofyear

#### dataframe.Series.dt.dayofyear

The ordinal day of the year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.dayofyear.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.dayofweek`
: The day of the week with Monday=0, Sunday=6.

`DatetimeIndex.day`
: The day of the datetime.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.dayofyear
0    1
1   32
dtype: int32
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(["1/1/2020 10:00:00+00:00",
...                         "2/1/2020 11:00:00+00:00"])
>>> idx.dayofyear
Index([1, 32], dtype='int32')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.days_in_month.html.md

# dask.dataframe.Series.dt.days_in_month

#### dataframe.Series.dt.days_in_month

The number of days in the month.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.days_in_month.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`Series.dt.day`
: Return the day of the month.

`Series.dt.is_month_end`
: Return a boolean indicating if the date is the last day of the month.

`Series.dt.is_month_start`
: Return a boolean indicating if the date is the first day of the month.

`Series.dt.month`
: Return the month as January=1 through December=12.

### Examples

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.daysinmonth
0    31
1    29
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.daysinmonth.html.md

# dask.dataframe.Series.dt.daysinmonth

#### dataframe.Series.dt.daysinmonth

The number of days in the month.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.daysinmonth.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`Series.dt.day`
: Return the day of the month.

`Series.dt.is_month_end`
: Return a boolean indicating if the date is the last day of the month.

`Series.dt.is_month_start`
: Return a boolean indicating if the date is the first day of the month.

`Series.dt.month`
: Return the month as January=1 through December=12.

### Examples

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.daysinmonth
0    31
1    29
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.floor.html.md

# dask.dataframe.Series.dt.floor

#### dataframe.Series.dt.floor(freq, ambiguous: TimeAmbiguous = 'raise', nonexistent: TimeNonexistent = 'raise') → Self

Perform floor operation on the data to the specified freq.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.floor.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **freq**
  : The frequency level to floor the index to. Must be a fixed
    frequency like ‘s’ (second) not ‘ME’ (month end). See
    [frequency aliases](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases) for
    a list of possible freq values.

  **ambiguous**
  : Only relevant for DatetimeIndex:
    - ‘infer’ will attempt to infer fall dst-transition hours based on
      order
    - bool-ndarray where True signifies a DST time, False designates
      a non-DST time (note that this flag is only applicable for
      ambiguous times)
    - ‘NaT’ will return NaT where there are ambiguous times
    - ‘raise’ will raise a ValueError if there are ambiguous
      times.

  **nonexistent**
  : A nonexistent time does not exist in a particular timezone
    where clocks moved forward due to DST.
    - ‘shift_forward’ will shift the nonexistent time forward to the
      closest existing time
    - ‘shift_backward’ will shift the nonexistent time backward to the
      closest existing time
    - ‘NaT’ will return NaT where there are nonexistent times
    - timedelta objects will shift nonexistent times by the timedelta
    - ‘raise’ will raise a ValueError if there are
      nonexistent times.
* **Returns:**
  DatetimeIndex, TimedeltaIndex, or Series
  : Index of the same type for a DatetimeIndex or TimedeltaIndex,
    or a Series with the same index for a Series.
* **Raises:**
  ValueError if the freq cannot be converted.

#### SEE ALSO
`DatetimeIndex.floor`
: Perform floor operation on the data to the specified freq.

`DatetimeIndex.snap`
: Snap time stamps to nearest occurring frequency.

### Notes

If the timestamps have a timezone, flooring will take place relative to the
local (“wall”) time and re-localized to the same timezone. When flooring
near daylight savings time, use `nonexistent` and `ambiguous` to
control the re-localization behavior.

### Examples

**DatetimeIndex**

```pycon
>>> rng = pd.date_range("1/1/2018 11:59:00", periods=3, freq="min")
>>> rng
DatetimeIndex(['2018-01-01 11:59:00', '2018-01-01 12:00:00',
               '2018-01-01 12:01:00'],
              dtype='datetime64[us]', freq='min')
```

```pycon
>>> rng.floor('h')
DatetimeIndex(['2018-01-01 11:00:00', '2018-01-01 12:00:00',
               '2018-01-01 12:00:00'],
              dtype='datetime64[us]', freq=None)
```

**Series**

```pycon
>>> pd.Series(rng).dt.floor("h")
0   2018-01-01 11:00:00
1   2018-01-01 12:00:00
2   2018-01-01 12:00:00
dtype: datetime64[us]
```

When rounding near a daylight savings time transition, use `ambiguous` or
`nonexistent` to control how the timestamp should be re-localized.

```pycon
>>> rng_tz = pd.DatetimeIndex(["2021-10-31 03:30:00"], tz="Europe/Amsterdam")
```

```pycon
>>> rng_tz.floor("2h", ambiguous=False)
DatetimeIndex(['2021-10-31 02:00:00+01:00'],
             dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

```pycon
>>> rng_tz.floor("2h", ambiguous=True)
DatetimeIndex(['2021-10-31 02:00:00+02:00'],
              dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.freq.html.md

# dask.dataframe.Series.dt.freq

#### dataframe.Series.dt.freq

Tries to return a string representing a frequency generated by infer_freq.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.freq.

Some inconsistencies with the Dask version may exist.

Returns None if it can’t autodetect the frequency.

#### SEE ALSO
`Series.dt.to_period`
: Cast to PeriodArray/PeriodIndex at a particular frequency.

### Examples

```pycon
>>> ser = pd.Series(["2024-01-01", "2024-01-02", "2024-01-03", "2024-01-04"])
>>> ser = pd.to_datetime(ser)
>>> ser.dt.freq
'D'
```

```pycon
>>> ser = pd.Series(["2022-01-01", "2024-01-01", "2026-01-01", "2028-01-01"])
>>> ser = pd.to_datetime(ser)
>>> ser.dt.freq
'2YS-JAN'
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.hour.html.md

# dask.dataframe.Series.dt.hour

#### dataframe.Series.dt.hour

The hours of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.hour.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.day`
: The day of the datetime.

`DatetimeIndex.minute`
: The minutes of the datetime.

`DatetimeIndex.second`
: The seconds of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="h")
... )
>>> datetime_series
0   2000-01-01 00:00:00
1   2000-01-01 01:00:00
2   2000-01-01 02:00:00
dtype: datetime64[us]
>>> datetime_series.dt.hour
0    0
1    1
2    2
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_leap_year.html.md

# dask.dataframe.Series.dt.is_leap_year

#### dataframe.Series.dt.is_leap_year

Boolean indicator if the date belongs to a leap year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_leap_year.

Some inconsistencies with the Dask version may exist.

A leap year is a year, which has 366 days (instead of 365) including
29th of February as an intercalary day.
Leap years are years which are multiples of four with the exception
of years divisible by 100 but not by 400.

* **Returns:**
  Series or ndarray
  : Booleans indicating if dates belong to a leap year.

#### SEE ALSO
`DatetimeIndex.is_year_end`
: Indicate whether the date is the last day of the year.

`DatetimeIndex.is_year_start`
: Indicate whether the date is the first day of a year.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> idx = pd.date_range("2012-01-01", "2015-01-01", freq="YE")
>>> idx
DatetimeIndex(['2012-12-31', '2013-12-31', '2014-12-31'],
              dtype='datetime64[us]', freq='YE-DEC')
>>> idx.is_leap_year
array([ True, False, False])
```

```pycon
>>> dates_series = pd.Series(idx)
>>> dates_series
0   2012-12-31
1   2013-12-31
2   2014-12-31
dtype: datetime64[us]
>>> dates_series.dt.is_leap_year
0     True
1    False
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_month_end.html.md

# dask.dataframe.Series.dt.is_month_end

#### dataframe.Series.dt.is_month_end

Indicates whether the date is the last day of the month.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_month_end.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or array
  : For Series, returns a Series with boolean values.
    For DatetimeIndex, returns a boolean array.

#### SEE ALSO
[`is_month_start`](dask.dataframe.Series.dt.is_month_start.md#dask.dataframe.Series.dt.is_month_start)
: Return a boolean indicating whether the date is the first day of the month.

[`is_month_end`](#dask.dataframe.Series.dt.is_month_end)
: Return a boolean indicating whether the date is the last day of the month.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> s = pd.Series(pd.date_range("2018-02-27", periods=3))
>>> s
0   2018-02-27
1   2018-02-28
2   2018-03-01
dtype: datetime64[us]
>>> s.dt.is_month_start
0    False
1    False
2    True
dtype: bool
>>> s.dt.is_month_end
0    False
1    True
2    False
dtype: bool
```

```pycon
>>> idx = pd.date_range("2018-02-27", periods=3)
>>> idx.is_month_start
array([False, False, True])
>>> idx.is_month_end
array([False, True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_month_start.html.md

# dask.dataframe.Series.dt.is_month_start

#### dataframe.Series.dt.is_month_start

Indicates whether the date is the first day of the month.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_month_start.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or array
  : For Series, returns a Series with boolean values.
    For DatetimeIndex, returns a boolean array.

#### SEE ALSO
[`is_month_start`](#dask.dataframe.Series.dt.is_month_start)
: Return a boolean indicating whether the date is the first day of the month.

[`is_month_end`](dask.dataframe.Series.dt.is_month_end.md#dask.dataframe.Series.dt.is_month_end)
: Return a boolean indicating whether the date is the last day of the month.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> s = pd.Series(pd.date_range("2018-02-27", periods=3))
>>> s
0   2018-02-27
1   2018-02-28
2   2018-03-01
dtype: datetime64[us]
>>> s.dt.is_month_start
0    False
1    False
2    True
dtype: bool
>>> s.dt.is_month_end
0    False
1    True
2    False
dtype: bool
```

```pycon
>>> idx = pd.date_range("2018-02-27", periods=3)
>>> idx.is_month_start
array([False, False, True])
>>> idx.is_month_end
array([False, True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_quarter_end.html.md

# dask.dataframe.Series.dt.is_quarter_end

#### dataframe.Series.dt.is_quarter_end

Indicator for whether the date is the last day of a quarter.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_quarter_end.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  **is_quarter_end**
  : The same type as the original data with boolean values. Series will
    have the same name and index. DatetimeIndex will have the same
    name.

#### SEE ALSO
[`quarter`](dask.dataframe.Series.dt.quarter.md#dask.dataframe.Series.dt.quarter)
: Return the quarter of the date.

[`is_quarter_start`](dask.dataframe.Series.dt.is_quarter_start.md#dask.dataframe.Series.dt.is_quarter_start)
: Similar property indicating the quarter start.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> df = pd.DataFrame({'dates': pd.date_range("2017-03-30",
...                    periods=4)})
>>> df.assign(quarter=df.dates.dt.quarter,
...           is_quarter_end=df.dates.dt.is_quarter_end)
       dates  quarter    is_quarter_end
0 2017-03-30        1             False
1 2017-03-31        1              True
2 2017-04-01        2             False
3 2017-04-02        2             False
```

```pycon
>>> idx = pd.date_range('2017-03-30', periods=4)
>>> idx
DatetimeIndex(['2017-03-30', '2017-03-31', '2017-04-01', '2017-04-02'],
              dtype='datetime64[us]', freq='D')
```

```pycon
>>> idx.is_quarter_end
array([False,  True, False, False])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_quarter_start.html.md

# dask.dataframe.Series.dt.is_quarter_start

#### dataframe.Series.dt.is_quarter_start

Indicator for whether the date is the first day of a quarter.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_quarter_start.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  **is_quarter_start**
  : The same type as the original data with boolean values. Series will
    have the same name and index. DatetimeIndex will have the same
    name.

#### SEE ALSO
[`quarter`](dask.dataframe.Series.dt.quarter.md#dask.dataframe.Series.dt.quarter)
: Return the quarter of the date.

[`is_quarter_end`](dask.dataframe.Series.dt.is_quarter_end.md#dask.dataframe.Series.dt.is_quarter_end)
: Similar property for indicating the quarter end.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> df = pd.DataFrame({'dates': pd.date_range("2017-03-30",
...                   periods=4)})
>>> df.assign(quarter=df.dates.dt.quarter,
...           is_quarter_start=df.dates.dt.is_quarter_start)
       dates  quarter  is_quarter_start
0 2017-03-30        1             False
1 2017-03-31        1             False
2 2017-04-01        2              True
3 2017-04-02        2             False
```

```pycon
>>> idx = pd.date_range('2017-03-30', periods=4)
>>> idx
DatetimeIndex(['2017-03-30', '2017-03-31', '2017-04-01', '2017-04-02'],
              dtype='datetime64[us]', freq='D')
```

```pycon
>>> idx.is_quarter_start
array([False, False,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_year_end.html.md

# dask.dataframe.Series.dt.is_year_end

#### dataframe.Series.dt.is_year_end

Indicate whether the date is the last day of the year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_year_end.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or DatetimeIndex
  : The same type as the original data with boolean values. Series will
    have the same name and index. DatetimeIndex will have the same
    name.

#### SEE ALSO
[`is_year_start`](dask.dataframe.Series.dt.is_year_start.md#dask.dataframe.Series.dt.is_year_start)
: Similar property indicating the start of the year.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> dates = pd.Series(pd.date_range("2017-12-30", periods=3))
>>> dates
0   2017-12-30
1   2017-12-31
2   2018-01-01
dtype: datetime64[us]
```

```pycon
>>> dates.dt.is_year_end
0    False
1     True
2    False
dtype: bool
```

```pycon
>>> idx = pd.date_range("2017-12-30", periods=3)
>>> idx
DatetimeIndex(['2017-12-30', '2017-12-31', '2018-01-01'],
              dtype='datetime64[us]', freq='D')
```

```pycon
>>> idx.is_year_end
array([False,  True, False])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.is_year_start.html.md

# dask.dataframe.Series.dt.is_year_start

#### dataframe.Series.dt.is_year_start

Indicate whether the date is the first day of a year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.is_year_start.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or DatetimeIndex
  : The same type as the original data with boolean values. Series will
    have the same name and index. DatetimeIndex will have the same
    name.

#### SEE ALSO
[`is_year_end`](dask.dataframe.Series.dt.is_year_end.md#dask.dataframe.Series.dt.is_year_end)
: Similar property indicating the last day of the year.

### Examples

This method is available on Series with datetime values under
the `.dt` accessor, and directly on DatetimeIndex.

```pycon
>>> dates = pd.Series(pd.date_range("2017-12-30", periods=3))
>>> dates
0   2017-12-30
1   2017-12-31
2   2018-01-01
dtype: datetime64[us]
```

```pycon
>>> dates.dt.is_year_start
0    False
1    False
2    True
dtype: bool
```

```pycon
>>> idx = pd.date_range("2017-12-30", periods=3)
>>> idx
DatetimeIndex(['2017-12-30', '2017-12-31', '2018-01-01'],
              dtype='datetime64[us]', freq='D')
```

```pycon
>>> idx.is_year_start
array([False, False,  True])
```

This method, when applied to Series with datetime values under
the `.dt` accessor, will lose information about Business offsets.

```pycon
>>> dates = pd.Series(pd.date_range("2020-10-30", periods=4, freq="BYS"))
>>> dates
0   2021-01-01
1   2022-01-03
2   2023-01-02
3   2024-01-01
dtype: datetime64[us]
```

```pycon
>>> dates.dt.is_year_start
0    True
1    False
2    False
3    True
dtype: bool
```

```pycon
>>> idx = pd.date_range("2020-10-30", periods=4, freq="BYS")
>>> idx
DatetimeIndex(['2021-01-01', '2022-01-03', '2023-01-02', '2024-01-01'],
              dtype='datetime64[us]', freq='BYS-JAN')
```

```pycon
>>> idx.is_year_start
array([ True,  True,  True,  True])
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.isocalendar.html.md

# dask.dataframe.Series.dt.isocalendar

#### dataframe.Series.dt.isocalendar() → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)

Calculate year, week, and day according to the ISO 8601 standard.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.isocalendar.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  DataFrame
  : With columns year, week and day.

#### SEE ALSO
`Timestamp.isocalendar`
: Function return a 3-tuple containing ISO year, week number, and weekday for the given Timestamp object.

[`datetime.date.isocalendar`](https://docs.python.org/3/library/datetime.html#datetime.date.isocalendar)
: Return a named tuple object with three components: year, week and weekday.

### Examples

```pycon
>>> ser = pd.to_datetime(pd.Series(["2010-01-01", pd.NaT]))
>>> ser.dt.isocalendar()
   year  week  day
0  2009    53     5
1  <NA>  <NA>  <NA>
>>> ser.dt.isocalendar().week
0      53
1    <NA>
Name: week, dtype: UInt32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.microsecond.html.md

# dask.dataframe.Series.dt.microsecond

#### dataframe.Series.dt.microsecond

The microseconds of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.microsecond.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.second`
: The seconds of the datetime.

`DatetimeIndex.nanosecond`
: The nanoseconds of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="us")
... )
>>> datetime_series
0   2000-01-01 00:00:00.000000
1   2000-01-01 00:00:00.000001
2   2000-01-01 00:00:00.000002
dtype: datetime64[us]
>>> datetime_series.dt.microsecond
0       0
1       1
2       2
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.minute.html.md

# dask.dataframe.Series.dt.minute

#### dataframe.Series.dt.minute

The minutes of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.minute.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.hour`
: The hours of the datetime.

`DatetimeIndex.second`
: The seconds of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="min")
... )
>>> datetime_series
0   2000-01-01 00:00:00
1   2000-01-01 00:01:00
2   2000-01-01 00:02:00
dtype: datetime64[us]
>>> datetime_series.dt.minute
0    0
1    1
2    2
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.month.html.md

# dask.dataframe.Series.dt.month

#### dataframe.Series.dt.month

The month as January=1, December=12.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.month.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.year`
: The year of the datetime.

`DatetimeIndex.day`
: The day of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="ME")
... )
>>> datetime_series
0   2000-01-31
1   2000-02-29
2   2000-03-31
dtype: datetime64[us]
>>> datetime_series.dt.month
0    1
1    2
2    3
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.nanosecond.html.md

# dask.dataframe.Series.dt.nanosecond

#### dataframe.Series.dt.nanosecond

The nanoseconds of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.nanosecond.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.second`
: The seconds of the datetime.

`DatetimeIndex.microsecond`
: The microseconds of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="ns")
... )
>>> datetime_series
0   2000-01-01 00:00:00.000000000
1   2000-01-01 00:00:00.000000001
2   2000-01-01 00:00:00.000000002
dtype: datetime64[ns]
>>> datetime_series.dt.nanosecond
0       0
1       1
2       2
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.normalize.html.md

# dask.dataframe.Series.dt.normalize

#### dataframe.Series.dt.normalize() → Self

Convert times to midnight.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.normalize.

Some inconsistencies with the Dask version may exist.

The time component of the date-time is converted to midnight i.e.
00:00:00. This is useful in cases, when the time does not matter.
Length is unaltered. The timezones are unaffected.

This method is available on Series with datetime values under
the `.dt` accessor, and directly on Datetime Array/Index.

* **Returns:**
  DatetimeArray, DatetimeIndex or Series
  : The same type as the original data. Series will have the same
    name and index. DatetimeIndex will have the same name.

#### SEE ALSO
[`floor`](dask.dataframe.Series.dt.floor.md#dask.dataframe.Series.dt.floor)
: Floor the datetimes to the specified freq.

[`ceil`](dask.dataframe.Series.dt.ceil.md#dask.dataframe.Series.dt.ceil)
: Ceil the datetimes to the specified freq.

[`round`](dask.dataframe.Series.dt.round.md#dask.dataframe.Series.dt.round)
: Round the datetimes to the specified freq.

### Examples

```pycon
>>> idx = pd.date_range(
...     start="2014-08-01 10:00", freq="h", periods=3, tz="Asia/Calcutta"
... )
>>> idx
DatetimeIndex(['2014-08-01 10:00:00+05:30',
               '2014-08-01 11:00:00+05:30',
               '2014-08-01 12:00:00+05:30'],
                dtype='datetime64[us, Asia/Calcutta]', freq='h')
>>> idx.normalize()
DatetimeIndex(['2014-08-01 00:00:00+05:30',
               '2014-08-01 00:00:00+05:30',
               '2014-08-01 00:00:00+05:30'],
               dtype='datetime64[us, Asia/Calcutta]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.quarter.html.md

# dask.dataframe.Series.dt.quarter

#### dataframe.Series.dt.quarter

The quarter of the date.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.quarter.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.snap`
: Snap time stamps to nearest occurring frequency.

`DatetimeIndex.time`
: Returns numpy array of datetime.time objects. The time part of the Timestamps.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "4/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-04-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.quarter
0    1
1    2
dtype: int32
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(["1/1/2020 10:00:00+00:00",
...                         "2/1/2020 11:00:00+00:00"])
>>> idx.quarter
Index([1, 1], dtype='int32')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.round.html.md

# dask.dataframe.Series.dt.round

#### dataframe.Series.dt.round(freq, ambiguous: TimeAmbiguous = 'raise', nonexistent: TimeNonexistent = 'raise') → Self

Perform round operation on the data to the specified freq.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.round.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **freq**
  : The frequency level to round the index to. Must be a fixed
    frequency like ‘s’ (second) not ‘ME’ (month end). See
    [frequency aliases](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases) for
    a list of possible freq values.

  **ambiguous**
  : Only relevant for DatetimeIndex:
    - ‘infer’ will attempt to infer fall dst-transition hours based on
      order
    - bool-ndarray where True signifies a DST time, False designates
      a non-DST time (note that this flag is only applicable for
      ambiguous times)
    - ‘NaT’ will return NaT where there are ambiguous times
    - ‘raise’ will raise a ValueError if there are ambiguous
      times.

  **nonexistent**
  : A nonexistent time does not exist in a particular timezone
    where clocks moved forward due to DST.
    - ‘shift_forward’ will shift the nonexistent time forward to the
      closest existing time
    - ‘shift_backward’ will shift the nonexistent time backward to the
      closest existing time
    - ‘NaT’ will return NaT where there are nonexistent times
    - timedelta objects will shift nonexistent times by the timedelta
    - ‘raise’ will raise a ValueError if there are
      nonexistent times.
* **Returns:**
  DatetimeIndex, TimedeltaIndex, or Series
  : Index of the same type for a DatetimeIndex or TimedeltaIndex,
    or a Series with the same index for a Series.
* **Raises:**
  ValueError if the freq cannot be converted.

#### SEE ALSO
`DatetimeIndex.floor`
: Perform floor operation on the data to the specified freq.

`DatetimeIndex.snap`
: Snap time stamps to nearest occurring frequency.

### Notes

If the timestamps have a timezone, rounding will take place relative to the
local (“wall”) time and re-localized to the same timezone. When rounding
near daylight savings time, use `nonexistent` and `ambiguous` to
control the re-localization behavior.

### Examples

**DatetimeIndex**

```pycon
>>> rng = pd.date_range("1/1/2018 11:59:00", periods=3, freq="min")
>>> rng
DatetimeIndex(['2018-01-01 11:59:00', '2018-01-01 12:00:00',
               '2018-01-01 12:01:00'],
              dtype='datetime64[us]', freq='min')
```

```pycon
>>> rng.round('h')
DatetimeIndex(['2018-01-01 12:00:00', '2018-01-01 12:00:00',
               '2018-01-01 12:00:00'],
              dtype='datetime64[us]', freq=None)
```

**Series**

```pycon
>>> pd.Series(rng).dt.round("h")
0   2018-01-01 12:00:00
1   2018-01-01 12:00:00
2   2018-01-01 12:00:00
dtype: datetime64[us]
```

When rounding near a daylight savings time transition, use `ambiguous` or
`nonexistent` to control how the timestamp should be re-localized.

```pycon
>>> rng_tz = pd.DatetimeIndex(["2021-10-31 03:30:00"], tz="Europe/Amsterdam")
```

```pycon
>>> rng_tz.floor("2h", ambiguous=False)
DatetimeIndex(['2021-10-31 02:00:00+01:00'],
              dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

```pycon
>>> rng_tz.floor("2h", ambiguous=True)
DatetimeIndex(['2021-10-31 02:00:00+02:00'],
              dtype='datetime64[us, Europe/Amsterdam]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.second.html.md

# dask.dataframe.Series.dt.second

#### dataframe.Series.dt.second

The seconds of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.second.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.minute`
: The minutes of the datetime.

`DatetimeIndex.microsecond`
: The microseconds of the datetime.

`DatetimeIndex.nanosecond`
: The nanoseconds of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="s")
... )
>>> datetime_series
0   2000-01-01 00:00:00
1   2000-01-01 00:00:01
2   2000-01-01 00:00:02
dtype: datetime64[us]
>>> datetime_series.dt.second
0    0
1    1
2    2
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.strftime.html.md

# dask.dataframe.Series.dt.strftime

#### dataframe.Series.dt.strftime(date_format: [str](https://docs.python.org/3/library/stdtypes.html#str)) → npt.NDArray[np.object_]

Convert to Index using specified date_format.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.strftime.

Some inconsistencies with the Dask version may exist.

Return an Index of formatted strings specified by date_format, which
supports the same string format as the python standard library. Details
of the string format can be found in [python string format
doc](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior).

Formats supported by the C strftime API but not by the python string format
doc (such as “%R”, “%r”) are not officially supported and should be
preferably replaced with their supported equivalents (such as “%H:%M”,
“%I:%M:%S %p”).

Note that PeriodIndex support additional directives, detailed in
Period.strftime.

* **Parameters:**
  **date_format**
  : Date format string (e.g. “%%Y-%%m-%%d”).
* **Returns:**
  ndarray[object]
  : NumPy ndarray of formatted strings.

#### SEE ALSO
`to_datetime`
: Convert the given argument to datetime.

`DatetimeIndex.normalize`
: Return DatetimeIndex with times to midnight.

`DatetimeIndex.round`
: Round the DatetimeIndex to the specified freq.

`DatetimeIndex.floor`
: Floor the DatetimeIndex to the specified freq.

`Timestamp.strftime`
: Format a single Timestamp.

`Period.strftime`
: Format a single Period.

### Examples

```pycon
>>> rng = pd.date_range(pd.Timestamp("2018-03-10 09:00"), periods=3, freq="s")
>>> rng.strftime("%B %d, %Y, %r")
Index(['March 10, 2018, 09:00:00 AM', 'March 10, 2018, 09:00:01 AM',
       'March 10, 2018, 09:00:02 AM'],
      dtype='str')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.time.html.md

# dask.dataframe.Series.dt.time

#### dataframe.Series.dt.time

Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.time.

Some inconsistencies with the Dask version may exist.

The time part of the Timestamps.

#### SEE ALSO
`DatetimeIndex.timetz`
: Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects with timezones. The time part of the Timestamps.

`DatetimeIndex.date`
: Returns numpy array of python [`datetime.date`](https://docs.python.org/3/library/datetime.html#datetime.date) objects. Namely, the date part of Timestamps without time and timezone information.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.time
0    10:00:00
1    11:00:00
dtype: object
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(
...     ["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"]
... )
>>> idx.time
array([datetime.time(10, 0), datetime.time(11, 0)], dtype=object)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.timetz.html.md

# dask.dataframe.Series.dt.timetz

#### dataframe.Series.dt.timetz

Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects with timezones.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.timetz.

Some inconsistencies with the Dask version may exist.

The time part of the Timestamps.

#### SEE ALSO
`DatetimeIndex.time`
: Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects. The time part of the Timestamps.

`DatetimeIndex.tz`
: Return the timezone.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.timetz
0    10:00:00+00:00
1    11:00:00+00:00
dtype: object
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(
...     ["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"]
... )
>>> idx.timetz
array([datetime.time(10, 0, tzinfo=datetime.timezone.utc),
datetime.time(11, 0, tzinfo=datetime.timezone.utc)], dtype=object)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.tz.html.md

# dask.dataframe.Series.dt.tz

#### dataframe.Series.dt.tz

Return the timezone.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.tz.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  zoneinfo.ZoneInfo,, datetime.tzinfo, pytz.tzinfo.BaseTZInfo, dateutil.tz.tz.tzfile, or None
  : Returns None when the array is tz-naive.

#### SEE ALSO
`DatetimeIndex.tz_localize`
: Localize tz-naive DatetimeIndex to a given time zone, or remove timezone from a tz-aware DatetimeIndex.

`DatetimeIndex.tz_convert`
: Convert tz-aware DatetimeIndex from one time zone to another.

### Examples

For Series:

```pycon
>>> s = pd.Series(["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"])
>>> s = pd.to_datetime(s)
>>> s
0   2020-01-01 10:00:00+00:00
1   2020-02-01 11:00:00+00:00
dtype: datetime64[us, UTC]
>>> s.dt.tz
datetime.timezone.utc
```

For DatetimeIndex:

```pycon
>>> idx = pd.DatetimeIndex(
...     ["1/1/2020 10:00:00+00:00", "2/1/2020 11:00:00+00:00"]
... )
>>> idx.tz
datetime.timezone.utc
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.week.html.md

# dask.dataframe.Series.dt.week

#### dataframe.Series.dt.week

The week ordinal of the year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.week.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`PeriodIndex.day_of_week`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.dayofweek`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.week`
: The week ordinal of the year.

`PeriodIndex.weekday`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.year`
: The year of the period.

### Examples

```pycon
>>> idx = pd.PeriodIndex(["2023-01", "2023-02", "2023-03"], freq="M")
>>> idx.week  # It can be written `weekofyear`
Index([5, 9, 13], dtype='int64')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.weekday.html.md

# dask.dataframe.Series.dt.weekday

#### dataframe.Series.dt.weekday

The day of the week with Monday=0, Sunday=6.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.weekday.

Some inconsistencies with the Dask version may exist.

Return the day of the week. It is assumed the week starts on
Monday, which is denoted by 0 and ends on Sunday which is denoted
by 6. This method is available on both Series with datetime
values (using the dt accessor) or DatetimeIndex.

* **Returns:**
  Series or Index
  : Containing integers indicating the day number.

#### SEE ALSO
`Series.dt.dayofweek`
: Alias.

`Series.dt.weekday`
: Alias.

`Series.dt.day_name`
: Returns the name of the day of the week.

### Examples

```pycon
>>> s = pd.date_range('2016-12-31', '2017-01-08', freq='D').to_series()
>>> s.dt.dayofweek
2016-12-31    5
2017-01-01    6
2017-01-02    0
2017-01-03    1
2017-01-04    2
2017-01-05    3
2017-01-06    4
2017-01-07    5
2017-01-08    6
Freq: D, dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.weekofyear.html.md

# dask.dataframe.Series.dt.weekofyear

#### dataframe.Series.dt.weekofyear

The week ordinal of the year.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.weekofyear.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`PeriodIndex.day_of_week`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.dayofweek`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.week`
: The week ordinal of the year.

`PeriodIndex.weekday`
: The day of the week with Monday=0, Sunday=6.

`PeriodIndex.year`
: The year of the period.

### Examples

```pycon
>>> idx = pd.PeriodIndex(["2023-01", "2023-02", "2023-03"], freq="M")
>>> idx.week  # It can be written `weekofyear`
Index([5, 9, 13], dtype='int64')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dt.year.html.md

# dask.dataframe.Series.dt.year

#### dataframe.Series.dt.year

The year of the datetime.

This docstring was copied from pandas.core.indexes.accessors.CombinedDatetimelikeProperties.year.

Some inconsistencies with the Dask version may exist.

#### SEE ALSO
`DatetimeIndex.month`
: The month as January=1, December=12.

`DatetimeIndex.day`
: The day of the datetime.

### Examples

```pycon
>>> datetime_series = pd.Series(
...     pd.date_range("2000-01-01", periods=3, freq="YE")
... )
>>> datetime_series
0   2000-12-31
1   2001-12-31
2   2002-12-31
dtype: datetime64[us]
>>> datetime_series.dt.year
0    2000
1    2001
2    2002
dtype: int32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.dtype.html.md

# dask.dataframe.Series.dtype

#### *property* Series.dtype

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.eq.html.md

# dask.dataframe.Series.eq

#### Series.eq(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.explode.html.md

# dask.dataframe.Series.explode

#### Series.explode()

Transform each element of a list-like to a row.

This docstring was copied from pandas.Series.explode.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ignore_index**
  : If True, the resulting index will be labeled 0, 1, …, n - 1.
* **Returns:**
  Series
  : Exploded lists to rows; index will be duplicated for these rows.

#### SEE ALSO
[`Series.str.split`](dask.dataframe.Series.str.split.md#dask.dataframe.Series.str.split)
: Split string values on specified separator.

`Series.unstack`
: Unstack, a.k.a. pivot, Series with MultiIndex to produce DataFrame.

[`DataFrame.melt`](dask.dataframe.DataFrame.melt.md#dask.dataframe.DataFrame.melt)
: Unpivot a DataFrame from wide format to long format.

[`DataFrame.explode`](dask.dataframe.DataFrame.explode.md#dask.dataframe.DataFrame.explode)
: Explode a DataFrame from list-like columns to long format.

### Notes

This routine will explode list-likes including lists, tuples, sets,
Series, and np.ndarray. The result dtype of the subset rows will
be object. Scalars will be returned unchanged, and empty list-likes will
result in an np.nan for that row. In addition, the ordering of elements in
the output will be non-deterministic when exploding sets.

Reference [the user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/reshaping.html#reshaping-explode) for more examples.

### Examples

```pycon
>>> s = pd.Series([[1, 2, 3], "foo", [], [3, 4]])
>>> s
0    [1, 2, 3]
1          foo
2           []
3       [3, 4]
dtype: object
```

```pycon
>>> s.explode()
0      1
0      2
0      3
1    foo
2    NaN
3      3
3      4
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.ffill.html.md

# dask.dataframe.Series.ffill

#### Series.ffill(axis=0, limit=None)

Fill NA/NaN values by propagating the last valid observation to next valid.

This docstring was copied from pandas.DataFrame.ffill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : If method is specified, this is the maximum number of consecutive
    NaN values to forward/backward fill. In other words, if there is
    a gap with more than this number of consecutive NaNs, it will only
    be partially filled. If method is not specified, this is the
    maximum number of entries along the entire axis where NaNs will be
    filled. Must be greater than 0 if not None.

  **limit_area**
  : If limit is specified, consecutive NaNs will be filled with this
    restriction.
    * `None`: No fill restriction.
    * ‘inside’: Only fill NaNs surrounded by valid values
      (interpolate).
    * ‘outside’: Only fill NaNs outside valid values (extrapolate).
    <br/>
    #### Versionadded
    Added in version 2.2.0.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`DataFrame.bfill`](dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)
: Fill NA/NaN values by using the next valid observation to fill the gap.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

```pycon
>>> df.ffill()
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  3.0  4.0 NaN  1.0
3  3.0  3.0 NaN  4.0
```

```pycon
>>> ser = pd.Series([1, np.nan, 2, 3])
>>> ser.ffill()
0   1.0
1   1.0
2   2.0
3   3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.fillna.html.md

# dask.dataframe.Series.fillna

#### Series.fillna(value=None, axis=None)

Fill NA/NaN values with value.

This docstring was copied from pandas.DataFrame.fillna.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **value**
  : Value to use to fill holes (e.g. 0), alternately a
    dict/Series/DataFrame of values specifying which value to use for
    each index (for a Series) or column (for a DataFrame).  Values not
    in the dict/Series/DataFrame will not be filled. This value cannot
    be a list.

  **axis**
  : Axis along which to fill missing values. For Series
    this parameter is unused and defaults to 0.

  **inplace**
  : If True, fill in-place. Note: this will modify any
    other views on this object (e.g., a no-copy slice for a column in a
    DataFrame).

  **limit**
  : This is the maximum number of entries along the entire axis
    where NaNs will be filled. Must be greater than 0 if not None.
* **Returns:**
  Series/DataFrame
  : Object with missing values filled.

#### SEE ALSO
[`ffill`](dask.dataframe.Series.ffill.md#dask.dataframe.Series.ffill)
: Fill values by propagating the last valid observation to next valid.

[`bfill`](dask.dataframe.Series.bfill.md#dask.dataframe.Series.bfill)
: Fill values by using the next valid observation to fill the gap.

`interpolate`
: Fill NaN values using interpolation.

`reindex`
: Conform object to new index.

`asfreq`
: Convert TimeSeries to specified frequency.

### Notes

For non-object dtype, `value=None` will use the NA value of the dtype.
See more details in the [Filling missing data](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data-fillna)
section.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [
...         [np.nan, 2, np.nan, 0],
...         [3, 4, np.nan, 1],
...         [np.nan, np.nan, np.nan, np.nan],
...         [np.nan, 3, np.nan, 4],
...     ],
...     columns=list("ABCD"),
... )
>>> df
     A    B   C    D
0  NaN  2.0 NaN  0.0
1  3.0  4.0 NaN  1.0
2  NaN  NaN NaN  NaN
3  NaN  3.0 NaN  4.0
```

Replace all NaN elements with 0s.

```pycon
>>> df.fillna(0)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  0.0
3  0.0  3.0  0.0  4.0
```

Replace all NaN elements in column ‘A’, ‘B’, ‘C’, and ‘D’, with 0, 1,
2, and 3 respectively.

```pycon
>>> values = {"A": 0, "B": 1, "C": 2, "D": 3}
>>> df.fillna(value=values)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  2.0  1.0
2  0.0  1.0  2.0  3.0
3  0.0  3.0  2.0  4.0
```

Only replace the first NaN element.

```pycon
>>> df.fillna(value=values, limit=1)
     A    B    C    D
0  0.0  2.0  2.0  0.0
1  3.0  4.0  NaN  1.0
2  NaN  1.0  NaN  3.0
3  NaN  3.0  NaN  4.0
```

When filling using a DataFrame, replacement happens along
the same column names and same indices

```pycon
>>> df2 = pd.DataFrame(np.zeros((4, 4)), columns=list("ABCE"))
>>> df.fillna(df2)
     A    B    C    D
0  0.0  2.0  0.0  0.0
1  3.0  4.0  0.0  1.0
2  0.0  0.0  0.0  NaN
3  0.0  3.0  0.0  4.0
```

Note that column D is not affected since it is not present in df2.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.floordiv.html.md

# dask.dataframe.Series.floordiv

#### Series.floordiv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.ge.html.md

# dask.dataframe.Series.ge

#### Series.ge(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.get_partition.html.md

# dask.dataframe.Series.get_partition

#### Series.get_partition(n)

Get a dask DataFrame/Series representing the nth partition.

* **Parameters:**
  **n**
  : The 0-indexed partition number to select.
* **Returns:**
  Dask DataFrame or Series
  : The same type as the original object.

#### SEE ALSO
[`DataFrame.partitions`](dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.groupby.html.md

# dask.dataframe.Series.groupby

#### Series.groupby(by, \*\*kwargs)

Group Series using a mapper or by a Series of columns.

This docstring was copied from pandas.Series.groupby.

Some inconsistencies with the Dask version may exist.

A groupby operation involves some combination of splitting the
object, applying a function, and combining the results. This can be
used to group large amounts of data and compute operations on these
groups.

* **Parameters:**
  **by**
  : Used to determine the groups for the groupby.
    If `by` is a function, it’s called on each value of the object’s
    index. If a dict or Series is passed, the Series or dict VALUES
    will be used to determine the groups (the Series’ values are first
    aligned; see `.align()` method). If a list or ndarray of length
    equal to the selected axis is passed (see the [groupby user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#splitting-an-object-into-groups)),
    the values are used as-is to determine the groups. A label or list
    of labels may be passed to group by the columns in `self`.
    Notice that a tuple is interpreted as a (single) key.

  **level**
  : If the axis is a MultiIndex (hierarchical), group by a particular
    level or levels. Do not specify both `by` and `level`.

  **as_index**
  : Return object with group labels as the
    index. Only relevant for DataFrame input. as_index=False is
    effectively “SQL-style” grouped output. This argument has no effect
    on filtrations (see the [filtrations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).

  **sort**
  : Sort group keys. Get better performance by turning this off.
    Note this does not influence the order of observations within each
    group. Groupby preserves the order of rows within each group. If False,
    the groups will appear in the same order as they did in the original DataFrame.
    This argument has no effect on filtrations (see the [filtrations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#filtration)),
    such as `head()`, `tail()`, `nth()` and in transformations
    (see the [transformations in the user guide](https://pandas.pydata.org/docs/dev/user_guide/groupby.html#transformation)).
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: Specifying `sort=False` with an ordered categorical grouper will no
    longer sort the values.

  **group_keys**
  : When calling apply and the `by` argument produces a like-indexed
    (i.e. [a transform](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html#groupby-transform)) result, add group keys to
    index to identify pieces. By default group keys are not included
    when the result’s index (and column) labels match the inputs, and
    are included otherwise.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `True`.

  **observed**
  : This only applies if any of the groupers are Categoricals.
    If True: only show observed values for categorical groupers.
    If False: show all values for categorical groupers.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0: The default value is now `True`.

  **dropna**
  : If True, and if group keys contain NA values, NA values together
    with row/column will be dropped.
    If False, NA values will also be treated as the key in groups.
* **Returns:**
  pandas.api.typing.SeriesGroupBy
  : Returns a groupby object that contains information about the groups.

#### SEE ALSO
[`resample`](dask.dataframe.Series.resample.md#dask.dataframe.Series.resample)
: Convenience method for frequency conversion and resampling of time series.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/groupby.html) for more
detailed usage and examples, including splitting an object into groups,
iterating through groups, selecting a group, aggregation, and more.

The implementation of groupby is hash-based, meaning in particular that
objects that compare as equal will be considered to be in the same group.
An exception to this is that pandas has special handling of NA values:
any NA values will be collapsed to a single group, regardless of how
they compare. See the user guide linked above for more details.

### Examples

```pycon
>>> ser = pd.Series([390., 350., 30., 20.],
...                 index=['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...                 name="Max Speed")
>>> ser
Falcon    390.0
Falcon    350.0
Parrot     30.0
Parrot     20.0
Name: Max Speed, dtype: float64
```

We can pass a list of values to group the Series data by custom labels:

```pycon
>>> ser.groupby(["a", "b", "a", "b"]).mean()
a    210.0
b    185.0
Name: Max Speed, dtype: float64
```

Grouping by numeric labels yields similar results:

```pycon
>>> ser.groupby([0, 1, 0, 1]).mean()
0    210.0
1    185.0
Name: Max Speed, dtype: float64
```

We can group by a level of the index:

```pycon
>>> ser.groupby(level=0).mean()
Falcon    370.0
Parrot     25.0
Name: Max Speed, dtype: float64
```

We can group by a condition applied to the Series values:

```pycon
>>> ser.groupby(ser > 100).mean()
Max Speed
False     25.0
True     370.0
Name: Max Speed, dtype: float64
```

**Grouping by Indexes**

We can groupby different levels of a hierarchical index
using the level parameter:

```pycon
>>> arrays = [['Falcon', 'Falcon', 'Parrot', 'Parrot'],
...           ['Captive', 'Wild', 'Captive', 'Wild']]
>>> index = pd.MultiIndex.from_arrays(arrays, names=('Animal', 'Type'))
>>> ser = pd.Series([390., 350., 30., 20.], index=index, name="Max Speed")
>>> ser
Animal  Type
Falcon  Captive    390.0
        Wild       350.0
Parrot  Captive     30.0
        Wild        20.0
Name: Max Speed, dtype: float64
```

```pycon
>>> ser.groupby(level=0).mean()
Animal
Falcon    370.0
Parrot     25.0
Name: Max Speed, dtype: float64
```

We can also group by the ‘Type’ level of the hierarchical index
to get the mean speed for each type:

```pycon
>>> ser.groupby(level="Type").mean()
Type
Captive    210.0
Wild       185.0
Name: Max Speed, dtype: float64
```

We can also choose to include NA in group keys or not by defining
dropna parameter, the default setting is True.

```pycon
>>> ser = pd.Series([1, 2, 3, 3], index=["a", 'a', 'b', np.nan])
>>> ser.groupby(level=0).sum()
a    3
b    3
dtype: int64
```

To include NA values in the group keys, set dropna=False:

```pycon
>>> ser.groupby(level=0, dropna=False).sum()
a    3
b    3
NaN  3
dtype: int64
```

We can also group by a custom list with NaN values to handle
missing group labels:

```pycon
>>> arrays = ['Falcon', 'Falcon', 'Parrot', 'Parrot']
>>> ser = pd.Series([390., 350., 30., 20.], index=arrays, name="Max Speed")
>>> ser.groupby(["a", "b", "a", np.nan]).mean()
a    210.0
b    350.0
Name: Max Speed, dtype: float64
```

```pycon
>>> ser.groupby(["a", "b", "a", np.nan], dropna=False).mean()
a    210.0
b    350.0
NaN   20.0
Name: Max Speed, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.gt.html.md

# dask.dataframe.Series.gt

#### Series.gt(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.head.html.md

# dask.dataframe.Series.head

#### Series.head(n: [int](https://docs.python.org/3/library/functions.html#int) = 5, npartitions=1, compute: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

First n rows of the dataset

* **Parameters:**
  **n**
  : The number of rows to return. Default is 5.

  **npartitions**
  : Elements are only taken from the first `npartitions`, with a
    default of 1. If there are fewer than `n` rows in the first
    `npartitions` a warning will be raised and any found rows
    returned. Pass -1 to use all partitions.

  **compute**
  : Whether to compute the result, default is True.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.html.md

# dask.dataframe.Series

### *class* dask.dataframe.Series(expr)

Series-like Expr Collection.

The constructor takes the expression that represents the query as input. The class
is not meant to be instantiated directly. Instead, use one of the IO connectors from
Dask.

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(expr)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.dataframe.Series.__init__)(expr)                                                                                                 |                                                                              |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| `abs`()                                                                                                                                             | Return a Series/DataFrame with absolute numeric value of each element.       |
| [`add`](dask.dataframe.Series.add.md#dask.dataframe.Series.add)(other[, level, fill_value, axis])                                                   |                                                                              |
| `add_prefix`(prefix)                                                                                                                                | Prefix labels with string prefix.                                            |
| `add_suffix`(suffix)                                                                                                                                | Suffix labels with string suffix.                                            |
| [`align`](dask.dataframe.Series.align.md#dask.dataframe.Series.align)(other[, join, axis, fill_value])                                              | Align two objects on their axes with the specified join method.              |
| [`all`](dask.dataframe.Series.all.md#dask.dataframe.Series.all)([axis, skipna, split_every])                                                        | Return whether all elements are True, potentially over an axis.              |
| `analyze`([filename, format])                                                                                                                       | Outputs statistics about every node in the expression.                       |
| [`any`](dask.dataframe.Series.any.md#dask.dataframe.Series.any)([axis, skipna, split_every])                                                        | Return whether any element is True, potentially over an axis.                |
| [`apply`](dask.dataframe.Series.apply.md#dask.dataframe.Series.apply)(function, \*args[, meta, axis])                                               | Parallel version of pandas.Series.apply                                      |
| [`astype`](dask.dataframe.Series.astype.md#dask.dataframe.Series.astype)(dtypes)                                                                    | Cast a pandas object to a specified dtype `dtype`.                           |
| [`autocorr`](dask.dataframe.Series.autocorr.md#dask.dataframe.Series.autocorr)([lag, split_every])                                                  | Compute the lag-N autocorrelation.                                           |
| [`between`](dask.dataframe.Series.between.md#dask.dataframe.Series.between)(left, right[, inclusive])                                               | Return boolean Series equivalent to left <= series <= right.                 |
| [`bfill`](dask.dataframe.Series.bfill.md#dask.dataframe.Series.bfill)([axis, limit])                                                                | Fill NA/NaN values by using the next valid observation to fill the gap.      |
| `case_when`(caselist)                                                                                                                               | Replace values where the conditions are True.                                |
| [`clear_divisions`](dask.dataframe.Series.clear_divisions.md#dask.dataframe.Series.clear_divisions)()                                               | Forget division information.                                                 |
| [`clip`](dask.dataframe.Series.clip.md#dask.dataframe.Series.clip)([lower, upper, axis])                                                            | Trim values at input threshold(s).                                           |
| `combine`(other, func[, fill_value])                                                                                                                | Combine the Series with a Series or scalar according to func.                |
| `combine_first`(other)                                                                                                                              | Update null elements with value in the same location in other.               |
| [`compute`](dask.dataframe.Series.compute.md#dask.dataframe.Series.compute)(\*\*kwargs)                                                             | Compute this dask collection                                                 |
| `compute_current_divisions`([col, set_divisions])                                                                                                   | Compute the current divisions of the DataFrame.                              |
| [`copy`](dask.dataframe.Series.copy.md#dask.dataframe.Series.copy)([deep])                                                                          | Make a copy of the dataframe                                                 |
| [`corr`](dask.dataframe.Series.corr.md#dask.dataframe.Series.corr)(other[, method, min_periods, split_every])                                       | Compute correlation with other Series, excluding missing values.             |
| [`count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)([axis, numeric_only, split_every])                                            | Count non-NA cells for each column or row.                                   |
| [`cov`](dask.dataframe.Series.cov.md#dask.dataframe.Series.cov)(other[, min_periods, split_every])                                                  | Compute covariance with Series, excluding missing values.                    |
| [`cummax`](dask.dataframe.Series.cummax.md#dask.dataframe.Series.cummax)([axis, skipna])                                                            | Return cumulative maximum over a DataFrame or Series axis.                   |
| [`cummin`](dask.dataframe.Series.cummin.md#dask.dataframe.Series.cummin)([axis, skipna])                                                            | Return cumulative minimum over a DataFrame or Series axis.                   |
| [`cumprod`](dask.dataframe.Series.cumprod.md#dask.dataframe.Series.cumprod)([axis, skipna])                                                         | Return cumulative product over a DataFrame or Series axis.                   |
| [`cumsum`](dask.dataframe.Series.cumsum.md#dask.dataframe.Series.cumsum)([axis, skipna])                                                            | Return cumulative sum over a DataFrame or Series axis.                       |
| [`describe`](dask.dataframe.Series.describe.md#dask.dataframe.Series.describe)([split_every, percentiles, ...])                                     | Generate descriptive statistics.                                             |
| [`diff`](dask.dataframe.Series.diff.md#dask.dataframe.Series.diff)([periods, axis])                                                                 | First discrete difference of element.                                        |
| [`div`](dask.dataframe.Series.div.md#dask.dataframe.Series.div)(other[, level, fill_value, axis])                                                   |                                                                              |
| `divide`(other[, level, fill_value, axis])                                                                                                          |                                                                              |
| `dot`(other[, meta])                                                                                                                                | Compute the dot product between the Series and the columns of other.         |
| [`drop_duplicates`](dask.dataframe.Series.drop_duplicates.md#dask.dataframe.Series.drop_duplicates)([ignore_index, split_every, ...])               |                                                                              |
| [`dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)()                                                                          | Return a new Series with missing values removed.                             |
| `enforce_runtime_divisions`()                                                                                                                       | Enforce the current divisions at runtime.                                    |
| [`eq`](dask.dataframe.Series.eq.md#dask.dataframe.Series.eq)(other[, level, fill_value, axis])                                                      |                                                                              |
| `explain`([stage, format])                                                                                                                          | Create a graph representation of the Expression.                             |
| [`explode`](dask.dataframe.Series.explode.md#dask.dataframe.Series.explode)()                                                                       | Transform each element of a list-like to a row.                              |
| [`ffill`](dask.dataframe.Series.ffill.md#dask.dataframe.Series.ffill)([axis, limit])                                                                | Fill NA/NaN values by propagating the last valid observation to next valid.  |
| [`fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)([value, axis])                                                             | Fill NA/NaN values with value.                                               |
| [`floordiv`](dask.dataframe.Series.floordiv.md#dask.dataframe.Series.floordiv)(other[, level, fill_value, axis])                                    |                                                                              |
| `from_dict`(data, \*[, npartitions, orient, ...])                                                                                                   | Construct a Dask DataFrame from a Python Dictionary                          |
| [`ge`](dask.dataframe.Series.ge.md#dask.dataframe.Series.ge)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`get_partition`](dask.dataframe.Series.get_partition.md#dask.dataframe.Series.get_partition)(n)                                                    | Get a dask DataFrame/Series representing the nth partition.                  |
| [`groupby`](dask.dataframe.Series.groupby.md#dask.dataframe.Series.groupby)(by, \*\*kwargs)                                                         | Group Series using a mapper or by a Series of columns.                       |
| [`gt`](dask.dataframe.Series.gt.md#dask.dataframe.Series.gt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`head`](dask.dataframe.Series.head.md#dask.dataframe.Series.head)([n, npartitions, compute])                                                       | First n rows of the dataset                                                  |
| [`idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)([axis, skipna, numeric_only, split_every])                                 | Return index of first occurrence of maximum over requested axis.             |
| [`idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)([axis, skipna, numeric_only, split_every])                                 | Return index of first occurrence of minimum over requested axis.             |
| [`isin`](dask.dataframe.Series.isin.md#dask.dataframe.Series.isin)(values)                                                                          | Whether each element in the DataFrame is contained in values.                |
| [`isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)()                                                                                | Detect missing values.                                                       |
| [`isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)()                                                                          | DataFrame.isnull is an alias for DataFrame.isna.                             |
| `kurt`([axis, fisher, bias, nan_policy, ...])                                                                                                       | Return unbiased kurtosis over requested axis.                                |
| `kurtosis`([axis, fisher, bias, nan_policy, ...])                                                                                                   | Return unbiased kurtosis over requested axis.                                |
| [`le`](dask.dataframe.Series.le.md#dask.dataframe.Series.le)(other[, level, fill_value, axis])                                                      |                                                                              |
| `lower_once`()                                                                                                                                      |                                                                              |
| [`lt`](dask.dataframe.Series.lt.md#dask.dataframe.Series.lt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)(arg[, na_action, meta])                                                             | Map values of Series according to an input mapping or function.              |
| [`map_overlap`](dask.dataframe.Series.map_overlap.md#dask.dataframe.Series.map_overlap)(func, before, after, \*args[, ...])                         | Apply a function to each partition, sharing rows with adjacent partitions.   |
| [`map_partitions`](dask.dataframe.Series.map_partitions.md#dask.dataframe.Series.map_partitions)(func, \*args[, meta, ...])                         | Apply a Python function to each partition                                    |
| [`mask`](dask.dataframe.Series.mask.md#dask.dataframe.Series.mask)(cond[, other])                                                                   | Replace values where the condition is True.                                  |
| [`max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)([axis, skipna, numeric_only, split_every])                                          | Return the maximum of the values over the requested axis.                    |
| [`mean`](dask.dataframe.Series.mean.md#dask.dataframe.Series.mean)([axis, skipna, numeric_only, split_every])                                       | Return the mean of the values over the requested axis.                       |
| [`median`](dask.dataframe.Series.median.md#dask.dataframe.Series.median)()                                                                          | Return the median of the values over the requested axis.                     |
| [`median_approximate`](dask.dataframe.Series.median_approximate.md#dask.dataframe.Series.median_approximate)([method])                              | Return the approximate median of the values over the requested axis.         |
| [`memory_usage`](dask.dataframe.Series.memory_usage.md#dask.dataframe.Series.memory_usage)([deep, index])                                           | Return the memory usage of the Series.                                       |
| [`memory_usage_per_partition`](dask.dataframe.Series.memory_usage_per_partition.md#dask.dataframe.Series.memory_usage_per_partition)([index, deep]) | Return the memory usage of each partition                                    |
| [`min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)([axis, skipna, numeric_only, split_every])                                          | Return the minimum of the values over the requested axis.                    |
| [`mod`](dask.dataframe.Series.mod.md#dask.dataframe.Series.mod)(other[, level, fill_value, axis])                                                   |                                                                              |
| `mode`([dropna, split_every])                                                                                                                       | Return the mode(s) of the Series.                                            |
| [`mul`](dask.dataframe.Series.mul.md#dask.dataframe.Series.mul)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`ne`](dask.dataframe.Series.ne.md#dask.dataframe.Series.ne)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`nlargest`](dask.dataframe.Series.nlargest.md#dask.dataframe.Series.nlargest)([n, split_every])                                                    | Return the largest n elements.                                               |
| [`notnull`](dask.dataframe.Series.notnull.md#dask.dataframe.Series.notnull)()                                                                       | DataFrame.notnull is an alias for DataFrame.notna.                           |
| [`nsmallest`](dask.dataframe.Series.nsmallest.md#dask.dataframe.Series.nsmallest)([n, split_every])                                                 | Return the smallest n elements.                                              |
| [`nunique`](dask.dataframe.Series.nunique.md#dask.dataframe.Series.nunique)([dropna, split_every, split_out])                                       | Return number of unique elements in the object.                              |
| [`nunique_approx`](dask.dataframe.Series.nunique_approx.md#dask.dataframe.Series.nunique_approx)([split_every])                                     | Approximate number of unique rows.                                           |
| `optimize`([fuse])                                                                                                                                  | Optimizes the DataFrame.                                                     |
| [`persist`](dask.dataframe.Series.persist.md#dask.dataframe.Series.persist)([fuse])                                                                 | Persist this dask collection into memory                                     |
| [`pipe`](dask.dataframe.Series.pipe.md#dask.dataframe.Series.pipe)(func, \*args, \*\*kwargs)                                                        | Apply chainable functions that expect Series or DataFrames.                  |
| [`pow`](dask.dataframe.Series.pow.md#dask.dataframe.Series.pow)(other[, level, fill_value, axis])                                                   |                                                                              |
| `pprint`()                                                                                                                                          | Outputs a string representation of the DataFrame.                            |
| [`prod`](dask.dataframe.Series.prod.md#dask.dataframe.Series.prod)([axis, skipna, numeric_only, ...])                                               | Return the product of the values over the requested axis.                    |
| `product`([axis, skipna, numeric_only, ...])                                                                                                        | Return the product of the values over the requested axis.                    |
| [`quantile`](dask.dataframe.Series.quantile.md#dask.dataframe.Series.quantile)([q, method])                                                         | Approximate quantiles of Series                                              |
| [`radd`](dask.dataframe.Series.radd.md#dask.dataframe.Series.radd)(other[, level, fill_value, axis])                                                |                                                                              |
| [`random_split`](dask.dataframe.Series.random_split.md#dask.dataframe.Series.random_split)(frac[, random_state, shuffle])                           | Pseudorandomly split dataframe into different pieces row-wise                |
| [`rdiv`](dask.dataframe.Series.rdiv.md#dask.dataframe.Series.rdiv)(other[, level, fill_value, axis])                                                |                                                                              |
| `reduction`(chunk[, aggregate, combine, meta, ...])                                                                                                 | Generic row-wise reductions.                                                 |
| [`rename`](dask.dataframe.Series.rename.md#dask.dataframe.Series.rename)(index[, sorted_index])                                                     | Alter Series index labels or name                                            |
| `rename_axis`([mapper, index, columns, axis])                                                                                                       | Set the name of the axis for the index or columns.                           |
| [`repartition`](dask.dataframe.Series.repartition.md#dask.dataframe.Series.repartition)([divisions, npartitions, ...])                              | Repartition a collection                                                     |
| [`replace`](dask.dataframe.Series.replace.md#dask.dataframe.Series.replace)([to_replace, value, regex])                                             | Replace values given in to_replace with value.                               |
| [`resample`](dask.dataframe.Series.resample.md#dask.dataframe.Series.resample)(rule[, closed, label])                                               | Resample time-series data.                                                   |
| [`reset_index`](dask.dataframe.Series.reset_index.md#dask.dataframe.Series.reset_index)([drop])                                                     | Reset the index to the default index.                                        |
| `rfloordiv`(other[, level, fill_value, axis])                                                                                                       |                                                                              |
| `rmod`(other[, level, fill_value, axis])                                                                                                            |                                                                              |
| `rmul`(other[, level, fill_value, axis])                                                                                                            |                                                                              |
| [`rolling`](dask.dataframe.Series.rolling.md#dask.dataframe.Series.rolling)(window, \*\*kwargs)                                                     | Provides rolling transformations.                                            |
| [`round`](dask.dataframe.Series.round.md#dask.dataframe.Series.round)([decimals])                                                                   | Round numeric columns in a DataFrame to a variable number of decimal places. |
| `rpow`(other[, level, fill_value, axis])                                                                                                            |                                                                              |
| `rsub`(other[, level, fill_value, axis])                                                                                                            |                                                                              |
| `rtruediv`(other[, level, fill_value, axis])                                                                                                        |                                                                              |
| [`sample`](dask.dataframe.Series.sample.md#dask.dataframe.Series.sample)([n, frac, replace, random_state])                                          | Random sample of items                                                       |
| [`sem`](dask.dataframe.Series.sem.md#dask.dataframe.Series.sem)([axis, skipna, ddof, split_every, ...])                                             | Return unbiased standard error of the mean over requested axis.              |
| [`shift`](dask.dataframe.Series.shift.md#dask.dataframe.Series.shift)([periods, freq, axis])                                                        | Shift index by desired number of periods with an optional time freq.         |
| `shuffle`([on, ignore_index, npartitions, ...])                                                                                                     | Rearrange DataFrame into new partitions                                      |
| `simplify`()                                                                                                                                        |                                                                              |
| `skew`([axis, bias, nan_policy, numeric_only])                                                                                                      | Return unbiased skew over requested axis.                                    |
| `squeeze`()                                                                                                                                         | Squeeze 1 dimensional axis objects into scalars.                             |
| [`std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)([axis, skipna, ddof, numeric_only, ...])                                            | Return sample standard deviation over requested axis.                        |
| [`sub`](dask.dataframe.Series.sub.md#dask.dataframe.Series.sub)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)([axis, skipna, numeric_only, min_count, ...])                                       | Return the sum of the values over the requested axis.                        |
| `tail`([n, compute])                                                                                                                                | Last n rows of the dataset                                                   |
| [`to_backend`](dask.dataframe.Series.to_backend.md#dask.dataframe.Series.to_backend)([backend])                                                     | Move to a new DataFrame backend                                              |
| [`to_bag`](dask.dataframe.Series.to_bag.md#dask.dataframe.Series.to_bag)([index, format])                                                           | Create a Dask Bag from a Series                                              |
| [`to_csv`](dask.dataframe.Series.to_csv.md#dask.dataframe.Series.to_csv)(filename, \*\*kwargs)                                                      | See dd.to_csv docstring for more information                                 |
| [`to_dask_array`](dask.dataframe.Series.to_dask_array.md#dask.dataframe.Series.to_dask_array)([lengths, meta, optimize])                            | Convert a dask DataFrame to a dask array.                                    |
| [`to_delayed`](dask.dataframe.Series.to_delayed.md#dask.dataframe.Series.to_delayed)([optimize_graph])                                              | Convert into a list of `dask.delayed` objects, one per partition.            |
| [`to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)([name])                                                              | Convert Series to DataFrame.                                                 |
| [`to_hdf`](dask.dataframe.Series.to_hdf.md#dask.dataframe.Series.to_hdf)(path_or_buf, key[, mode, append])                                          | See dd.to_hdf docstring for more information                                 |
| `to_json`(filename, \*args, \*\*kwargs)                                                                                                             | See dd.to_json docstring for more information                                |
| `to_orc`(path, \*args, \*\*kwargs)                                                                                                                  | See dd.to_orc docstring for more information                                 |
| `to_records`([index, lengths])                                                                                                                      |                                                                              |
| `to_sql`(name, uri[, schema, if_exists, ...])                                                                                                       |                                                                              |
| [`to_string`](dask.dataframe.Series.to_string.md#dask.dataframe.Series.to_string)([max_rows])                                                       | Render a string representation of the Series.                                |
| [`to_timestamp`](dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)([freq, how])                                             | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.   |
| [`truediv`](dask.dataframe.Series.truediv.md#dask.dataframe.Series.truediv)(other[, level, fill_value, axis])                                       |                                                                              |
| [`unique`](dask.dataframe.Series.unique.md#dask.dataframe.Series.unique)([split_every, split_out, shuffle_method])                                  | Return Series of unique values in the object.                                |
| [`value_counts`](dask.dataframe.Series.value_counts.md#dask.dataframe.Series.value_counts)([sort, ascending, dropna, ...])                          | Return a Series containing counts of unique values.                          |
| [`var`](dask.dataframe.Series.var.md#dask.dataframe.Series.var)([axis, skipna, ddof, numeric_only, ...])                                            | Return unbiased variance over requested axis.                                |
| [`visualize`](dask.dataframe.Series.visualize.md#dask.dataframe.Series.visualize)([tasks])                                                          | Visualize the expression or task graph                                       |
| [`where`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)(cond[, other])                                                                | Replace values where the condition is False.                                 |

### Attributes

| `axes`                                                                                              |                                                                                                                  |
|-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| `columns`                                                                                           |                                                                                                                  |
| `dask`                                                                                              |                                                                                                                  |
| `divisions`                                                                                         | Tuple of `npartitions + 1` values, in ascending order, marking the lower/upper bounds of each partition's index. |
| [`dtype`](dask.dataframe.Series.dtype.md#dask.dataframe.Series.dtype)                               |                                                                                                                  |
| `dtypes`                                                                                            | Return data types                                                                                                |
| `expr`                                                                                              |                                                                                                                  |
| `index`                                                                                             | Return dask Index instance                                                                                       |
| `is_monotonic_decreasing`                                                                           | Return True if values in the object are monotonically decreasing.                                                |
| `is_monotonic_increasing`                                                                           | Return True if values in the object are monotonically increasing.                                                |
| [`known_divisions`](dask.dataframe.Series.known_divisions.md#dask.dataframe.Series.known_divisions) | Whether the divisions are known.                                                                                 |
| [`loc`](dask.dataframe.Series.loc.md#dask.dataframe.Series.loc)                                     | Purely label-location based indexer for selection by label.                                                      |
| `name`                                                                                              |                                                                                                                  |
| [`nbytes`](dask.dataframe.Series.nbytes.md#dask.dataframe.Series.nbytes)                            | Number of bytes                                                                                                  |
| [`ndim`](dask.dataframe.Series.ndim.md#dask.dataframe.Series.ndim)                                  | Return dimensionality                                                                                            |
| `npartitions`                                                                                       | Return number of partitions                                                                                      |
| `partitions`                                                                                        | Slice dataframe by partitions                                                                                    |
| [`shape`](dask.dataframe.Series.shape.md#dask.dataframe.Series.shape)                               | Return a tuple representing the dimensionality of the DataFrame.                                                 |
| [`size`](dask.dataframe.Series.size.md#dask.dataframe.Series.size)                                  | Size of the Series or DataFrame as a Delayed object.                                                             |
| [`values`](dask.dataframe.Series.values.md#dask.dataframe.Series.values)                            | Return a dask.array of the values of this dataframe                                                              |


# dask.dataframe.Series.idxmax.html.md

# dask.dataframe.Series.idxmax

#### Series.idxmax(axis=0, skipna=True, numeric_only=False, split_every=False)

Return index of first occurrence of maximum over requested axis.

This docstring was copied from pandas.DataFrame.idxmax.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of maxima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
[`Series.idxmax`](#dask.dataframe.Series.idxmax)
: Return index of the maximum element.

### Notes

This method is the DataFrame version of `ndarray.argmax`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the maximum value in each column.

```pycon
>>> df.idxmax()
consumption      Wheat Products
co2_emissions              Beef
dtype: str
```

To return the index for the maximum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmax(axis="columns")
Pork              co2_emissions
Wheat Products     consumption
Beef              co2_emissions
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.idxmin.html.md

# dask.dataframe.Series.idxmin

#### Series.idxmin(axis=0, skipna=True, numeric_only=False, split_every=False)

Return index of first occurrence of minimum over requested axis.

This docstring was copied from pandas.DataFrame.idxmin.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of minima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
[`Series.idxmin`](#dask.dataframe.Series.idxmin)
: Return index of the minimum element.

### Notes

This method is the DataFrame version of `ndarray.argmin`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the minimum value in each column.

```pycon
>>> df.idxmin()
consumption                Pork
co2_emissions    Wheat Products
dtype: str
```

To return the index for the minimum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmin(axis="columns")
Pork                consumption
Wheat Products    co2_emissions
Beef                consumption
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.isin.html.md

# dask.dataframe.Series.isin

#### Series.isin(values)

Whether each element in the DataFrame is contained in values.

This docstring was copied from pandas.DataFrame.isin.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **values**
  : The result will only be true at a location if all the
    labels match. If values is a Series, that’s the index. If
    values is a dict, the keys must be the column names,
    which must match. If values is a DataFrame,
    then both the index and column labels must match.
* **Returns:**
  DataFrame
  : DataFrame of booleans showing whether each element in the DataFrame
    is contained in values.

#### SEE ALSO
[`DataFrame.eq`](dask.dataframe.DataFrame.eq.md#dask.dataframe.DataFrame.eq)
: Equality test for DataFrame.

[`Series.isin`](#dask.dataframe.Series.isin)
: Equivalent method on Series.

[`Series.str.contains`](dask.dataframe.Series.str.contains.md#dask.dataframe.Series.str.contains)
: Test if pattern or regex is contained within a string of a Series or Index.

### Notes

`__iter__` is used (and not `__contains__`) to iterate over values
when checking if it contains the elements in DataFrame.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"num_legs": [2, 4], "num_wings": [2, 0]}, index=["falcon", "dog"]
... )
>>> df
        num_legs  num_wings
falcon         2          2
dog            4          0
```

When `values` is a list check whether every value in the DataFrame
is present in the list (which animals have 0 or 2 legs or wings)

```pycon
>>> df.isin([0, 2])
        num_legs  num_wings
falcon      True       True
dog        False       True
```

To check if `values` is *not* in the DataFrame, use the `~` operator:

```pycon
>>> ~df.isin([0, 2])
        num_legs  num_wings
falcon     False      False
dog         True      False
```

When `values` is a dict, we can pass values to check for each
column separately:

```pycon
>>> df.isin({"num_wings": [0, 3]})
        num_legs  num_wings
falcon     False      False
dog        False       True
```

When `values` is a Series or DataFrame the index and column must
match. Note that ‘falcon’ does not match based on the number of legs
in other.

```pycon
>>> other = pd.DataFrame(
...     {"num_legs": [8, 3], "num_wings": [0, 2]}, index=["spider", "falcon"]
... )
>>> df.isin(other)
        num_legs  num_wings
falcon     False       True
dog        False      False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.isna.html.md

# dask.dataframe.Series.isna

#### Series.isna()

Detect missing values.

This docstring was copied from pandas.DataFrame.isna.

Some inconsistencies with the Dask version may exist.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](#dask.dataframe.Series.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.isnull.html.md

# dask.dataframe.Series.isnull

#### Series.isnull()

DataFrame.isnull is an alias for DataFrame.isna.

This docstring was copied from pandas.DataFrame.isnull.

Some inconsistencies with the Dask version may exist.

Detect missing values.

Return a boolean same-sized object indicating if the values are NA.
NA values, such as None or `numpy.NaN`, gets mapped to True
values.
Everything else gets mapped to False values. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is an NA value.

#### SEE ALSO
[`Series.isnull`](#dask.dataframe.Series.isnull)
: Alias of isna.

[`DataFrame.isnull`](dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)
: Alias of isna.

`Series.notna`
: Boolean inverse of isna.

`DataFrame.notna`
: Boolean inverse of isna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

[`isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)
: Top-level isna.

### Examples

Show which entries in a DataFrame are NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.isna()
     age   born   name    toy
0  False   True  False   True
1  False  False  False  False
2   True  False  False  False
```

Show which entries in a Series are NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.isna()
0    False
1    False
2     True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.known_divisions.html.md

# dask.dataframe.Series.known_divisions

#### *property* Series.known_divisions

Whether the divisions are known.

This check can be expensive if the division calculation is expensive.
DataFrame.set_index is a good example where the calculation needs an
inspection of the data.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.le.html.md

# dask.dataframe.Series.le

#### Series.le(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.loc.html.md

# dask.dataframe.Series.loc

#### *property* Series.loc

Purely label-location based indexer for selection by label.

```pycon
>>> df.loc["b"]
>>> df.loc["b":"d"]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.lt.html.md

# dask.dataframe.Series.lt

#### Series.lt(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.map.html.md

# dask.dataframe.Series.map

#### Series.map(arg, na_action=None, meta=None)

Map values of Series according to an input mapping or function.

This docstring was copied from pandas.Series.map.

Some inconsistencies with the Dask version may exist.

Used for substituting each value in a Series with another value,
that may be derived from a function, a `dict` or
a [`Series`](dask.dataframe.Series.md#dask.dataframe.Series).

* **Parameters:**
  **func**
  : Function or mapping correspondence.

  **na_action**
  : If ‘ignore’, propagate NaN values, without passing them to the
    mapping correspondence.

  **engine**
  : Choose the execution engine to use to run the function. Only used for
    functions. If `map` is called with a mapping or `Series`, an
    exception will be raised. If `engine` is not provided the function will
    be executed by the regular Python interpreter.
    <br/>
    Options include JIT compilers such as Numba, Bodo or Blosc2, which in some
    cases can speed up the execution. To use an executor you can provide the
    decorators `numba.jit`, `numba.njit`, `bodo.jit` or `blosc2.jit`.
    You can also provide the decorator with parameters, like
    `numba.jit(nogit=True)`.
    <br/>
    Not all functions can be executed with all execution engines. In general,
    JIT compilers will require type stability in the function (no variable
    should change data type during the execution). And not all pandas and
    NumPy APIs are supported. Check the engine documentation for limitations.
    <br/>
    #### Versionadded
    Added in version 3.0.0.

  **\*\*kwargs**
  : Additional keyword arguments to pass as keywords arguments to
    arg.
    <br/>
    #### Versionadded
    Added in version 3.0.0.
* **Returns:**
  Series
  : Same index as caller.

#### SEE ALSO
[`Series.apply`](dask.dataframe.Series.apply.md#dask.dataframe.Series.apply)
: For applying more complex functions on a Series.

[`Series.replace`](dask.dataframe.Series.replace.md#dask.dataframe.Series.replace)
: Replace values given in to_replace with value.

[`DataFrame.apply`](dask.dataframe.DataFrame.apply.md#dask.dataframe.DataFrame.apply)
: Apply a function row-/column-wise.

`DataFrame.map`
: Apply a function elementwise on a whole DataFrame.

### Notes

When `arg` is a dictionary, values in Series that are not in the
dictionary (as keys) are converted to `NaN`. However, if the
dictionary is a `dict` subclass that defines `__missing__` (i.e.
provides a method for default values), then this default is used
rather than `NaN`.

### Examples

```pycon
>>> s = pd.Series(["cat", "dog", np.nan, "rabbit"])
>>> s
0      cat
1      dog
2      NaN
3   rabbit
dtype: str
```

`map` accepts a `dict` or a `Series`. Values that are not found
in the `dict` are converted to `NaN`, unless the dict has a default
value (e.g. `defaultdict`):

```pycon
>>> s.map({"cat": "kitten", "dog": "puppy"})
0   kitten
1    puppy
2      NaN
3      NaN
dtype: str
```

It also accepts a function:

```pycon
>>> s.map("I am a {}".format)
0       I am a cat
1       I am a dog
2       I am a nan
3    I am a rabbit
dtype: str
```

To avoid applying the function to missing values (and keep them as
`NaN`) `na_action='ignore'` can be used:

```pycon
>>> s.map("I am a {}".format, na_action="ignore")
0     I am a cat
1     I am a dog
2            NaN
3  I am a rabbit
dtype: str
```

For categorical data, the function is only applied to the categories:

```pycon
>>> s = pd.Series(list("cabaa"))
>>> s.map(print)
c
a
b
a
a
0    None
1    None
2    None
3    None
4    None
dtype: object
```

```pycon
>>> s_cat = s.astype("category")
>>> s_cat.map(print)  # function called once per unique category
a
b
c
0    None
1    None
2    None
3    None
4    None
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.map_overlap.html.md

# dask.dataframe.Series.map_overlap

#### Series.map_overlap(func, before, after, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, \*\*kwargs)

Apply a function to each partition, sharing rows with adjacent partitions.

This can be useful for implementing windowing functions such as
`df.rolling(...).mean()` or `df.diff()`.

* **Parameters:**
  **func**
  : Function applied to each partition.

  **before**
  : The rows to prepend to partition `i` from the end of
    partition `i - 1`.

  **after**
  : The rows to append to partition `i` from the beginning
    of partition `i + 1`.

  **args, kwargs**
  : Positional and keyword arguments to pass to the function.
    Positional arguments are computed on a per-partition basis, while
    keyword arguments are shared across all partitions. The partition
    itself will be the first positional argument, with all other
    arguments passed *after*. Arguments can be `Scalar`, `Delayed`,
    or regular Python objects. DataFrame-like args (both dask and
    pandas) will be repartitioned to align (if necessary) before
    applying the function; see `align_dataframes` to control this
    behavior.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition,
    and will raise an error if this doesn’t work,
    but it won’t raise if dtypes don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **align_dataframes**
  : Whether to repartition DataFrame- or Series-like args
    (both dask and pandas) so their divisions align before applying
    the function. This requires all inputs to have known divisions.
    Single-partition inputs will be split into multiple partitions.
    <br/>
    If False, all inputs must have either the same number of partitions
    or a single partition. Single-partition inputs will be broadcast to
    every partition of multi-partition inputs.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

### Notes

Given positive integers `before` and `after`, and a function
`func`, `map_overlap` does the following:

1. Prepend `before` rows to each partition `i` from the end of
   partition `i - 1`. The first partition has no rows prepended.
2. Append `after` rows to each partition `i` from the beginning of
   partition `i + 1`. The last partition has no rows appended.
3. Apply `func` to each partition, passing in any extra `args` and
   `kwargs` if provided.
4. Trim `before` rows from the beginning of all but the first
   partition.
5. Trim `after` rows from the end of all but the last partition.

### Examples

Given a DataFrame, Series, or Index, such as:

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 4, 7, 11],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

A rolling sum with a trailing moving window of size 2 can be computed by
overlapping 2 rows before each partition, and then mapping calls to
`df.rolling(2).sum()`:

```pycon
>>> ddf.compute()
    x    y
0   1  1.0
1   2  2.0
2   4  3.0
3   7  4.0
4  11  5.0
>>> ddf.map_overlap(lambda df: df.rolling(2).sum(), 2, 0).compute()
      x    y
0   NaN  NaN
1   3.0  3.0
2   6.0  5.0
3  11.0  7.0
4  18.0  9.0
```

The pandas `diff` method computes a discrete difference shifted by a
number of periods (can be positive or negative). This can be
implemented by mapping calls to `df.diff` to each partition after
prepending/appending that many rows, depending on sign:

```pycon
>>> def diff(df, periods=1):
...     before, after = (periods, 0) if periods > 0 else (0, -periods)
...     return df.map_overlap(lambda df, periods=1: df.diff(periods),
...                           periods, 0, periods=periods)
>>> diff(ddf, 1).compute()
     x    y
0  NaN  NaN
1  1.0  1.0
2  2.0  1.0
3  3.0  1.0
4  4.0  1.0
```

If you have a `DatetimeIndex`, you can use a `pd.Timedelta` for time-
based windows or any `pd.Timedelta` convertible string:

```pycon
>>> ts = pd.Series(range(10), index=pd.date_range('2017', periods=10))
>>> dts = dd.from_pandas(ts, npartitions=2)
>>> dts.map_overlap(lambda df: df.rolling('2D').sum(),
...                 pd.Timedelta('2D'), 0).compute()
2017-01-01     0.0
2017-01-02     1.0
2017-01-03     3.0
2017-01-04     5.0
2017-01-05     7.0
2017-01-06     9.0
2017-01-07    11.0
2017-01-08    13.0
2017-01-09    15.0
2017-01-10    17.0
Freq: D, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.map_partitions.html.md

# dask.dataframe.Series.map_partitions

#### Series.map_partitions(func, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, parent_meta=None, required_columns=None, \*\*kwargs)

Apply a Python function to each partition

* **Parameters:**
  **func**
  : Function applied to each partition.

  **args, kwargs**
  : Arguments and keywords to pass to the function. Arguments and
    keywords may contain `FrameBase` or regular python objects.
    DataFrame-like args (both dask and pandas) must have the same
    number of partitions as `self` or comprise a single partition.
    Key-word arguments, Single-partition arguments, and general
    python-object arguments will be broadcasted to all partitions.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition, and will
    raise an error if this doesn’t work, but it won’t raise if dtypes
    don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **clear_divisions**
  : Whether divisions should be cleared. If True, transform_divisions
    will be ignored.

  **required_columns**
  : List of columns that `func` requires for execution. These columns
    must belong to the first DataFrame argument (in `args`). If None
    is specified (the default), the query optimizer will assume that
    all input columns are required.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.

### Examples

Given a DataFrame, Series, or Index, such as:

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame({'x': [1, 2, 3, 4, 5],
...                    'y': [1., 2., 3., 4., 5.]})
>>> ddf = dd.from_pandas(df, npartitions=2)
```

One can use `map_partitions` to apply a function on each partition.
Extra arguments and keywords can optionally be provided, and will be
passed to the function after the partition.

Here we apply a function with arguments and keywords to a DataFrame,
resulting in a Series:

```pycon
>>> def myadd(df, a, b=1):
...     return df.x + df.y + a + b
>>> res = ddf.map_partitions(myadd, 1, b=2)
>>> res.dtype
dtype('float64')
```

Here we apply a function to a Series resulting in a Series:

```pycon
>>> res = ddf.x.map_partitions(lambda x: len(x)) # ddf.x is a Dask Series Structure
>>> res.dtype
dtype('int64')
```

By default, dask tries to infer the output metadata by running your
provided function on some fake data. This works well in many cases, but
can sometimes be expensive, or even fail. To avoid this, you can
manually specify the output metadata with the `meta` keyword. This
can be specified in many forms, for more information see
`dask.dataframe.utils.make_meta`.

Here we specify the output is a Series with no name, and dtype
`float64`:

```pycon
>>> res = ddf.map_partitions(myadd, 1, b=2, meta=(None, 'f8'))
```

Here we map a function that takes in a DataFrame, and returns a
DataFrame with a new column:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y))
>>> res.dtypes
x      int64
y    float64
z    float64
dtype: object
```

As before, the output metadata can also be specified manually. This
time we pass in a `dict`, as the output is a DataFrame:

```pycon
>>> res = ddf.map_partitions(lambda df: df.assign(z=df.x * df.y),
...                          meta={'x': 'i8', 'y': 'f8', 'z': 'f8'})
```

In the case where the metadata doesn’t change, you can also pass in
the object itself directly:

```pycon
>>> res = ddf.map_partitions(lambda df: df.head(), meta=ddf)
```

Also note that the index and divisions are assumed to remain unchanged.
If the function you’re mapping changes the index/divisions, you’ll need
to pass `clear_divisions=True`.

```pycon
>>> ddf.map_partitions(func, clear_divisions=True)
```

Your map function gets information about where it is in the dataframe by
accepting a special `partition_info` keyword argument.

```pycon
>>> def func(partition, partition_info=None):
...     pass
```

This will receive the following information:

```pycon
>>> partition_info
{'number': 1, 'division': 3}
```

For each argument and keyword arguments that are dask dataframes you will
receive the number (n) which represents the nth partition of the dataframe
and the division (the first index value in the partition). If divisions
are not known (for instance if the index is not sorted) then you will get
None as the division.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.mask.html.md

# dask.dataframe.Series.mask

#### Series.mask(cond, other=nan)

Replace values where the condition is True.

This docstring was copied from pandas.DataFrame.mask.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **cond**
  : Where cond is False, keep the original value. Where
    True, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is True are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Return an object of same shape as caller.

[`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Return an object of same shape as caller.

### Notes

The mask method is an application of the if-then idiom. For each
element in the caller, if `cond` is `False` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with True.

The signature for [`Series.where()`](dask.dataframe.Series.where.md#dask.dataframe.Series.where) or
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `mask` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.max.html.md

# dask.dataframe.Series.max

#### Series.max(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the maximum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.max.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the maximum, use `idxmax`.
This is the equivalent of the `numpy.ndarray` method `argmax`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.max()
8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.mean.html.md

# dask.dataframe.Series.mean

#### Series.mean(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the mean of the values over the requested axis.

This docstring was copied from pandas.DataFrame.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.mean()
2.0
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.mean()
a   1.5
b   2.5
dtype: float64
```

Using axis=1

```pycon
>>> df.mean(axis=1)
tiger   1.5
zebra   2.5
dtype: float64
```

In this case, numeric_only should be set to True to avoid
getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.mean(numeric_only=True)
a   1.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.median.html.md

# dask.dataframe.Series.median

#### Series.median()

Return the median of the values over the requested axis.

This docstring was copied from pandas.Series.median.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  scalar or Series (if level specified)
  : Median of the values for the requested axis.

#### SEE ALSO
[`numpy.median`](https://numpy.org/doc/stable/reference/generated/numpy.median.html#numpy.median)
: Equivalent numpy function for computing median.

[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Sum of the values.

[`Series.median`](#dask.dataframe.Series.median)
: Median of the values.

[`Series.std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)
: Standard deviation of the values.

[`Series.var`](dask.dataframe.Series.var.md#dask.dataframe.Series.var)
: Variance of the values.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Minimum value.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Maximum value.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> s.median()
2.0
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.median()
a   1.5
b   2.5
dtype: float64
```

Using axis=1

```pycon
>>> df.median(axis=1)
tiger   1.5
zebra   2.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.median(numeric_only=True)
a   1.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.median_approximate.html.md

# dask.dataframe.Series.median_approximate

#### Series.median_approximate(method='default')

Return the approximate median of the values over the requested axis.

* **Parameters:**
  **method**
  : What method to use. By default will use Dask’s internal custom
    algorithm (`"dask"`).  If set to `"tdigest"` will use tdigest
    for floats and ints and fallback to the `"dask"` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.memory_usage.html.md

# dask.dataframe.Series.memory_usage

#### Series.memory_usage(deep=False, index=True)

Return the memory usage of the Series.

This docstring was copied from pandas.Series.memory_usage.

Some inconsistencies with the Dask version may exist.

The memory usage can optionally include the contribution of
the index and of elements of object dtype.

* **Parameters:**
  **index**
  : Specifies whether to include the memory usage of the Series index.

  **deep**
  : If True, introspect the data deeply by interrogating
    object dtypes for system-level memory consumption, and include
    it in the returned value.
* **Returns:**
  int
  : Bytes of memory consumed.

#### SEE ALSO
[`numpy.ndarray.nbytes`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.nbytes.html#numpy.ndarray.nbytes)
: Total bytes consumed by the elements of the array.

[`DataFrame.memory_usage`](dask.dataframe.DataFrame.memory_usage.md#dask.dataframe.DataFrame.memory_usage)
: Bytes consumed by a DataFrame.

### Examples

```pycon
>>> s = pd.Series(range(3))
>>> s.memory_usage()
156
```

Not including the index gives the size of the rest of the data, which
is necessarily smaller:

```pycon
>>> s.memory_usage(index=False)
24
```

The memory footprint of object values is ignored by default:

```pycon
>>> s = pd.Series(["a", "b"])
>>> s.values
<ArrowStringArray>
['a', 'b']
Length: 2, dtype: str
>>> s.memory_usage()
150
>>> s.memory_usage(deep=True)
150
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.memory_usage_per_partition.html.md

# dask.dataframe.Series.memory_usage_per_partition

#### Series.memory_usage_per_partition(index: [bool](https://docs.python.org/3/library/functions.html#bool) = True, deep: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Return the memory usage of each partition

* **Parameters:**
  **index**
  : Specifies whether to include the memory usage of the index in
    returned Series.

  **deep**
  : If True, introspect the data deeply by interrogating
    `object` dtypes for system-level memory consumption, and include
    it in the returned values.
* **Returns:**
  Series
  : A Series whose index is the partition number and whose values
    are the memory usage of each partition in bytes.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.min.html.md

# dask.dataframe.Series.min

#### Series.min(axis=0, skipna=True, numeric_only=False, split_every=False, \*\*kwargs)

Return the minimum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.min.

Some inconsistencies with the Dask version may exist.

If you want the *index* of the minimum, use `idxmin`.
This is the equivalent of the `numpy.ndarray` method `argmin`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    For DataFrames, specifying `axis=None` will apply the aggregation
    across both axes.
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Value containing the calculation referenced in the description.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.min()
0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.mod.html.md

# dask.dataframe.Series.mod

#### Series.mod(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.mul.html.md

# dask.dataframe.Series.mul

#### Series.mul(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.nbytes.html.md

# dask.dataframe.Series.nbytes

#### *property* Series.nbytes

Number of bytes

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.ndim.html.md

# dask.dataframe.Series.ndim

#### *property* Series.ndim

Return dimensionality

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.ne.html.md

# dask.dataframe.Series.ne

#### Series.ne(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.nlargest.html.md

# dask.dataframe.Series.nlargest

#### Series.nlargest(n=5, split_every=None)

Return the largest n elements.

This docstring was copied from pandas.Series.nlargest.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : Return this many descending sorted values.

  **keep**
  : When there are duplicate values that cannot all fit in a
    Series of n elements:
    - `first` : return the first n occurrences in order
      of appearance.
    - `last` : return the last n occurrences in reverse
      order of appearance.
    - `all` : keep all occurrences. This can result in a Series of
      size larger than n.
* **Returns:**
  Series
  : The n largest values in the Series, sorted in decreasing order.

#### SEE ALSO
[`Series.nsmallest`](dask.dataframe.Series.nsmallest.md#dask.dataframe.Series.nsmallest)
: Get the n smallest elements.

`Series.sort_values`
: Sort Series by values.

[`Series.head`](dask.dataframe.Series.head.md#dask.dataframe.Series.head)
: Return the first n rows.

### Notes

Faster than `.sort_values(ascending=False).head(n)` for small n
relative to the size of the `Series` object.

### Examples

```pycon
>>> countries_population = {
...     "Italy": 59000000,
...     "France": 65000000,
...     "Malta": 434000,
...     "Maldives": 434000,
...     "Brunei": 434000,
...     "Iceland": 337000,
...     "Nauru": 11300,
...     "Tuvalu": 11300,
...     "Anguilla": 11300,
...     "Montserrat": 5200,
... }
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Malta         434000
Maldives      434000
Brunei        434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64
```

The n largest elements where `n=5` by default.

```pycon
>>> s.nlargest()
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64
```

The n largest elements where `n=3`. Default keep value is ‘first’
so Malta will be kept.

```pycon
>>> s.nlargest(3)
France    65000000
Italy     59000000
Malta       434000
dtype: int64
```

The n largest elements where `n=3` and keeping the last duplicates.
Brunei will be kept since it is the last with value 434000 based on
the index order.

```pycon
>>> s.nlargest(3, keep="last")
France      65000000
Italy       59000000
Brunei        434000
dtype: int64
```

The n largest elements where `n=3` with all duplicates kept. Note
that the returned Series has five elements due to the three duplicates.

```pycon
>>> s.nlargest(3, keep="all")
France      65000000
Italy       59000000
Malta         434000
Maldives      434000
Brunei        434000
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.notnull.html.md

# dask.dataframe.Series.notnull

#### Series.notnull()

DataFrame.notnull is an alias for DataFrame.notna.

This docstring was copied from pandas.DataFrame.notnull.

Some inconsistencies with the Dask version may exist.

Detect existing (non-missing) values.

Return a boolean same-sized object indicating if the values are not NA.
Non-missing values get mapped to True. Characters such as empty
strings `''` or `numpy.inf` are not considered NA values.
NA values, such as None or `numpy.NaN`, get mapped to False
values.

* **Returns:**
  Series/DataFrame
  : Mask of bool values for each element in Series/DataFrame
    that indicates whether an element is not an NA value.

#### SEE ALSO
[`Series.notnull`](#dask.dataframe.Series.notnull)
: Alias of notna.

`DataFrame.notnull`
: Alias of notna.

[`Series.isna`](dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)
: Boolean inverse of notna.

[`DataFrame.isna`](dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)
: Boolean inverse of notna.

[`Series.dropna`](dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)
: Omit axes labels with missing values.

[`DataFrame.dropna`](dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)
: Omit axes labels with missing values.

`notna`
: Top-level notna.

### Examples

Show which entries in a DataFrame are not NA.

```pycon
>>> df = pd.DataFrame(
...     dict(
...         age=[5, 6, np.nan],
...         born=[
...             pd.NaT,
...             pd.Timestamp("1939-05-27"),
...             pd.Timestamp("1940-04-25"),
...         ],
...         name=["Alfred", "Batman", ""],
...         toy=[None, "Batmobile", "Joker"],
...     )
... )
>>> df
   age       born    name        toy
0  5.0        NaT  Alfred        NaN
1  6.0 1939-05-27  Batman  Batmobile
2  NaN 1940-04-25              Joker
```

```pycon
>>> df.notnull()
     age   born  name    toy
0   True  False  True  False
1   True   True  True   True
2  False   True  True   True
```

Show which entries in a Series are not NA.

```pycon
>>> ser = pd.Series([5, 6, np.nan])
>>> ser
0    5.0
1    6.0
2    NaN
dtype: float64
```

```pycon
>>> ser.notnull()
0     True
1     True
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.nsmallest.html.md

# dask.dataframe.Series.nsmallest

#### Series.nsmallest(n=5, split_every=None)

Return the smallest n elements.

This docstring was copied from pandas.Series.nsmallest.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **n**
  : Return this many ascending sorted values.

  **keep**
  : When there are duplicate values that cannot all fit in a
    Series of n elements:
    - `first` : return the first n occurrences in order
      of appearance.
    - `last` : return the last n occurrences in reverse
      order of appearance.
    - `all` : keep all occurrences. This can result in a Series of
      size larger than n.
* **Returns:**
  Series
  : The n smallest values in the Series, sorted in increasing order.

#### SEE ALSO
[`Series.nlargest`](dask.dataframe.Series.nlargest.md#dask.dataframe.Series.nlargest)
: Get the n largest elements.

`Series.sort_values`
: Sort Series by values.

[`Series.head`](dask.dataframe.Series.head.md#dask.dataframe.Series.head)
: Return the first n rows.

### Notes

Faster than `.sort_values().head(n)` for small n relative to
the size of the `Series` object.

### Examples

```pycon
>>> countries_population = {
...     "Italy": 59000000,
...     "France": 65000000,
...     "Brunei": 434000,
...     "Malta": 434000,
...     "Maldives": 434000,
...     "Iceland": 337000,
...     "Nauru": 11300,
...     "Tuvalu": 11300,
...     "Anguilla": 11300,
...     "Montserrat": 5200,
... }
>>> s = pd.Series(countries_population)
>>> s
Italy       59000000
France      65000000
Brunei        434000
Malta         434000
Maldives      434000
Iceland       337000
Nauru          11300
Tuvalu         11300
Anguilla       11300
Montserrat      5200
dtype: int64
```

The n smallest elements where `n=5` by default.

```pycon
>>> s.nsmallest()
Montserrat    5200
Nauru        11300
Tuvalu       11300
Anguilla     11300
Iceland     337000
dtype: int64
```

The n smallest elements where `n=3`. Default keep value is
‘first’ so Nauru and Tuvalu will be kept.

```pycon
>>> s.nsmallest(3)
Montserrat   5200
Nauru       11300
Tuvalu      11300
dtype: int64
```

The n smallest elements where `n=3` and keeping the last
duplicates. Anguilla and Tuvalu will be kept since they are the last
with value 11300 based on the index order.

```pycon
>>> s.nsmallest(3, keep="last")
Montserrat   5200
Anguilla    11300
Tuvalu      11300
dtype: int64
```

The n smallest elements where `n=3` with all duplicates kept. Note
that the returned Series has four elements due to the three duplicates.

```pycon
>>> s.nsmallest(3, keep="all")
Montserrat   5200
Nauru       11300
Tuvalu      11300
Anguilla    11300
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.nunique.html.md

# dask.dataframe.Series.nunique

#### Series.nunique(dropna=True, split_every=False, split_out=True)

Return number of unique elements in the object.

This docstring was copied from pandas.Series.nunique.

Some inconsistencies with the Dask version may exist.

Excludes NA values by default.

* **Parameters:**
  **dropna**
  : Don’t include NaN in the count.
* **Returns:**
  int
  : An integer indicating the number of unique elements in the object.

#### SEE ALSO
`DataFrame.nunique`
: Method nunique for DataFrame.

[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Count non-NA/null observations in the Series.

### Examples

```pycon
>>> s = pd.Series([1, 3, 5, 7, 7])
>>> s
0    1
1    3
2    5
3    7
4    7
dtype: int64
```

```pycon
>>> s.nunique()
4
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.nunique_approx.html.md

# dask.dataframe.Series.nunique_approx

#### Series.nunique_approx(split_every=None)

Approximate number of unique rows.

This method uses the HyperLogLog algorithm for cardinality
estimation to compute the approximate number of unique rows.
The approximate error is 0.406%.

* **Parameters:**
  **split_every**
  : Group partitions into groups of this size while performing a
    tree-reduction. If set to False, no tree-reduction will be used.
    Default is 8.
* **Returns:**
  a float representing the approximate number of elements

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.persist.html.md

# dask.dataframe.Series.persist

#### Series.persist(fuse=True, \*\*kwargs)

Persist this dask collection into memory

This turns a lazy Dask collection into a Dask collection with the same
metadata, but now with the results fully computed or actively computing
in the background.

The action of function differs significantly depending on the active
task scheduler.  If the task scheduler supports asynchronous computing,
such as is the case of the dask.distributed scheduler, then persist
will return *immediately* and the return value’s task graph will
contain Dask Future objects.  However if the task scheduler only
supports blocking computation then the call to persist will *block*
and the return value’s task graph will contain concrete Python results.

This function is particularly useful when using distributed systems,
because the results will be kept in distributed memory, rather than
returned to the local process as with compute.

* **Parameters:**
  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the graph is optimized before computation.
    Otherwise the graph is run as is. This can be useful for debugging.

  **\*\*kwargs**
  : Extra keywords to forward to the scheduler function.
* **Returns:**
  New dask collections backed by in-memory data

#### SEE ALSO
[`dask.persist`](../api.md#dask.persist)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.pipe.html.md

# dask.dataframe.Series.pipe

#### Series.pipe(func, \*args, \*\*kwargs)

Apply chainable functions that expect Series or DataFrames.

This docstring was copied from pandas.DataFrame.pipe.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **func**
  : Function to apply to the Series/DataFrame.
    `args`, and `kwargs` are passed into `func`.
    Alternatively a `(callable, data_keyword)` tuple where
    `data_keyword` is a string indicating the keyword of
    `callable` that expects the Series/DataFrame.

  **\*args**
  : Positional arguments passed into `func`.

  **\*\*kwargs**
  : A dictionary of keyword arguments passed into `func`.
* **Returns:**
  The return type of `func`.
  : The result of applying `func` to the Series or DataFrame.

#### SEE ALSO
[`DataFrame.apply`](dask.dataframe.DataFrame.apply.md#dask.dataframe.DataFrame.apply)
: Apply a function along input axis of DataFrame.

`DataFrame.map`
: Apply a function elementwise on a whole DataFrame.

[`Series.map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)
: Apply a mapping correspondence on a [`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series).

### Notes

Use `.pipe` when chaining together functions that expect
Series, DataFrames or GroupBy objects.

### Examples

Constructing an income DataFrame from a dictionary.

```pycon
>>> data = [[8000, 1000], [9500, np.nan], [5000, 2000]]
>>> df = pd.DataFrame(data, columns=["Salary", "Others"])
>>> df
   Salary  Others
0    8000  1000.0
1    9500     NaN
2    5000  2000.0
```

Functions that perform tax reductions on an income DataFrame.

```pycon
>>> def subtract_federal_tax(df):
...     return df * 0.9
>>> def subtract_state_tax(df, rate):
...     return df * (1 - rate)
>>> def subtract_national_insurance(df, rate, rate_increase):
...     new_rate = rate + rate_increase
...     return df * (1 - new_rate)
```

Instead of writing

```pycon
>>> subtract_national_insurance(
...     subtract_state_tax(subtract_federal_tax(df), rate=0.12),
...     rate=0.05,
...     rate_increase=0.02,
... )
```

You can write

```pycon
>>> (
...     df.pipe(subtract_federal_tax)
...     .pipe(subtract_state_tax, rate=0.12)
...     .pipe(subtract_national_insurance, rate=0.05, rate_increase=0.02)
... )
    Salary   Others
0  5892.48   736.56
1  6997.32      NaN
2  3682.80  1473.12
```

If you have a function that takes the data as (say) the second
argument, pass a tuple indicating which keyword expects the
data. For example, suppose `national_insurance` takes its data as `df`
in the second argument:

```pycon
>>> def subtract_national_insurance(rate, df, rate_increase):
...     new_rate = rate + rate_increase
...     return df * (1 - new_rate)
>>> (
...     df.pipe(subtract_federal_tax)
...     .pipe(subtract_state_tax, rate=0.12)
...     .pipe(
...         (subtract_national_insurance, "df"), rate=0.05, rate_increase=0.02
...     )
... )
    Salary   Others
0  5892.48   736.56
1  6997.32      NaN
2  3682.80  1473.12
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.pow.html.md

# dask.dataframe.Series.pow

#### Series.pow(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.prod.html.md

# dask.dataframe.Series.prod

#### Series.prod(axis=0, skipna=True, numeric_only=False, min_count=0, split_every=False, \*\*kwargs)

Return the product of the values over the requested axis.

This docstring was copied from pandas.DataFrame.prod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.prod with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **min_count**
  : The required number of valid values to perform the operation. If fewer than
    `min_count` non-NA values are present the result will be NA.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : The product of the values over the requested axis.

#### SEE ALSO
[`Series.sum`](dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)
: Return the sum.

[`Series.min`](dask.dataframe.Series.min.md#dask.dataframe.Series.min)
: Return the minimum.

[`Series.max`](dask.dataframe.Series.max.md#dask.dataframe.Series.max)
: Return the maximum.

[`Series.idxmin`](dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)
: Return the index of the minimum.

[`Series.idxmax`](dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)
: Return the index of the maximum.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum over the requested axis.

[`DataFrame.min`](dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)
: Return the minimum over the requested axis.

[`DataFrame.max`](dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)
: Return the maximum over the requested axis.

[`DataFrame.idxmin`](dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)
: Return the index of the minimum over the requested axis.

[`DataFrame.idxmax`](dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)
: Return the index of the maximum over the requested axis.

### Examples

By default, the product of an empty or all-NA Series is `1`

```pycon
>>> pd.Series([], dtype="float64").prod()
1.0
```

This can be controlled with the `min_count` parameter

```pycon
>>> pd.Series([], dtype="float64").prod(min_count=1)
nan
```

Thanks to the `skipna` parameter, `min_count` handles all-NA and
empty series identically.

```pycon
>>> pd.Series([np.nan]).prod()
1.0
```

```pycon
>>> pd.Series([np.nan]).prod(min_count=1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.quantile.html.md

# dask.dataframe.Series.quantile

#### Series.quantile(q=0.5, method='default')

Approximate quantiles of Series

* **Parameters:**
  **q**
  : Iterable of numbers ranging from 0 to 1 for the desired quantiles

  **method**
  : What method to use. By default will use dask’s internal custom
    algorithm (`'dask'`).  If set to `'tdigest'` will use tdigest
    for floats and ints and fallback to the `'dask'` otherwise.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.radd.html.md

# dask.dataframe.Series.radd

#### Series.radd(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.random_split.html.md

# dask.dataframe.Series.random_split

#### Series.random_split(frac, random_state=None, shuffle=False)

Pseudorandomly split dataframe into different pieces row-wise

* **Parameters:**
  **frac**
  : List of floats that should sum to one.

  **random_state**
  : If int or None create a new RandomState with this as the seed.
    Otherwise draw from the passed RandomState.

  **shuffle**
  : If set to True, the dataframe is shuffled (within partition)
    before the split.

#### SEE ALSO
`dask.DataFrame.sample`

### Examples

50/50 split

```pycon
>>> a, b = df.random_split([0.5, 0.5])
```

80/10/10 split, consistent random_state

```pycon
>>> a, b, c = df.random_split([0.8, 0.1, 0.1], random_state=123)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.rdiv.html.md

# dask.dataframe.Series.rdiv

#### Series.rdiv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.rename.html.md

# dask.dataframe.Series.rename

#### Series.rename(index, sorted_index=False)

Alter Series index labels or name

Function / dict values must be unique (1-to-1). Labels not contained in
a dict / Series will be left as-is. Extra labels listed don’t throw an
error.

Alternatively, change `Series.name` with a scalar value.

* **Parameters:**
  **index**
  : If dict-like or callable, the transformation is applied to the
    index. Scalar or hashable sequence-like will alter the
    `Series.name` attribute.

  **inplace**
  : Whether to return a new Series or modify this one inplace.

  **sorted_index**
  : If true, the output `Series` will have known divisions inferred
    from the input series and the transformation. Ignored for
    non-callable/dict-like `index` or when the input series has
    unknown divisions. Note that this may only be set to `True` if
    you know that the transformed index is monotonically increasing. Dask
    will check that transformed divisions are monotonic, but cannot
    check all the values between divisions, so incorrectly setting this
    can result in bugs.
* **Returns:**
  **renamed**

#### SEE ALSO
[`pandas.Series.rename`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.rename.html#pandas.Series.rename)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.repartition.html.md

# dask.dataframe.Series.repartition

#### Series.repartition(divisions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) | [None](https://docs.python.org/3/library/constants.html#None) = None, npartitions: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, partition_size: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, freq=None, force: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Repartition a collection

Exactly one of divisions, npartitions or partition_size should be
specified. A `ValueError` will be raised when that is not the case.

* **Parameters:**
  **divisions**
  : The “dividing lines” used to split the dataframe into partitions.
    For `divisions=[0, 10, 50, 100]`, there would be three output partitions,
    where the new index contained [0, 10), [10, 50), and [50, 100), respectively.
    See [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions).

  **npartitions**
  : Approximate number of partitions of output. The number of
    partitions used may be slightly lower than npartitions depending
    on data distribution, but will never be higher.
    The Callable gets the number of partitions of the input as an argument
    and should return an int.

  **partition_size**
  : Max number of bytes of memory for each partition. Use numbers or strings
    like 5MB. If specified npartitions and divisions will be ignored. Note that
    the size reflects the number of bytes used as computed by
    pandas.DataFrame.memory_usage, which will not necessarily match the size
    when storing to disk.
    <br/>
    #### WARNING
    This keyword argument triggers computation to determine
    the memory size of each partition, which may be expensive.

  **force**
  : Allows the expansion of the existing divisions.
    If False then the new divisions’ lower and upper bounds must be
    the same as the old divisions’.

  **freq**
  : A period on which to partition timeseries data like `'7D'` or
    `'12h'` or `pd.Timedelta(hours=12)`.  Assumes a datetime index.

#### SEE ALSO
[`DataFrame.memory_usage_per_partition`](dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition)

[`pandas.DataFrame.memory_usage`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.memory_usage.html#pandas.DataFrame.memory_usage)

### Notes

Exactly one of divisions, npartitions, partition_size, or freq
should be specified. A `ValueError` will be raised when that is
not the case.

Also note that `len(divisions)` is equal to `npartitions + 1`. This is because `divisions`
represents the upper and lower bounds of each partition. The first item is the
lower bound of the first partition, the second item is the lower bound of the
second partition and the upper bound of the first partition, and so on.
The second-to-last item is the lower bound of the last partition, and the last
(extra) item is the upper bound of the last partition.

### Examples

```pycon
>>> df = df.repartition(npartitions=10)
>>> df = df.repartition(divisions=[0, 5, 10, 20])
>>> df = df.repartition(freq='7d')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.replace.html.md

# dask.dataframe.Series.replace

#### Series.replace(to_replace=None, value=<no_default>, regex=False)

Replace values given in to_replace with value.

This docstring was copied from pandas.DataFrame.replace.

Some inconsistencies with the Dask version may exist.

Values of the Series/DataFrame are replaced with other values dynamically.
This differs from updating with `.loc` or `.iloc`, which require
you to specify a location to update with some value.

* **Parameters:**
  **to_replace**
  : How to find the values that will be replaced.
    * numeric, str or regex:
      > - numeric: numeric values equal to to_replace will be
      >   replaced with value
      > - str: string exactly matching to_replace will be replaced
      >   with value
      > - regex: regexes matching to_replace will be replaced with
      >   value
    * list of str, regex, or numeric:
      > - First, if to_replace and value are both lists, they
      >   **must** be the same length.
      > - Second, if `regex=True` then all of the strings in **both**
      >   lists will be interpreted as regexes otherwise they will match
      >   directly. This doesn’t matter much for value since there
      >   are only a few possible substitution regexes you can use.
      > - str, regex and numeric rules apply as above.
    * dict:
      > - Dicts can be used to specify different replacement values
      >   for different existing values. For example,
      >   `{'a': 'b', 'y': 'z'}` replaces the value ‘a’ with ‘b’ and
      >   ‘y’ with ‘z’. To use a dict in this way, the optional value
      >   parameter should not be given.
      > - For a DataFrame a dict can specify that different values
      >   should be replaced in different columns. For example,
      >   `{'a': 1, 'b': 'z'}` looks for the value 1 in column ‘a’
      >   and the value ‘z’ in column ‘b’ and replaces these values
      >   with whatever is specified in value. The value parameter
      >   should not be `None` in this case. You can treat this as a
      >   special case of passing two lists except that you are
      >   specifying the column to search in.
      > - For a DataFrame nested dictionaries, e.g.,
      >   `{'a': {'b': np.nan}}`, are read as follows: look in column
      >   ‘a’ for the value ‘b’ and replace it with NaN. The optional value
      >   parameter should not be specified to use a nested dict in this
      >   way. You can nest regular expressions as well. Note that
      >   column names (the top-level dictionary keys in a nested
      >   dictionary) **cannot** be regular expressions.
    * None:
      > - This means that the regex argument must be a string,
      >   compiled regular expression, or list, dict, ndarray or
      >   Series of such elements. If value is also `None` then
      >   this **must** be a nested dictionary or Series.
    <br/>
    See the examples section for examples of each of these.

  **value**
  : Value to replace any values matching to_replace with.
    For a DataFrame a dict of values can be used to specify which
    value to use for each column (columns not in the dict will not be
    filled). Regular expressions, strings and lists or dicts of such
    objects are also allowed.

  **inplace**
  : If True, performs operation inplace.

  **regex**
  : Whether to interpret to_replace and/or value as regular
    expressions. Alternatively, this could be a regular expression or a
    list, dict, or array of regular expressions in which case
    to_replace must be `None`.
* **Returns:**
  Series/DataFrame
  : Object after replacement.
* **Raises:**
  AssertionError
  : * If regex is not a `bool` and to_replace is not
      `None`.

  TypeError
  : * If to_replace is not a scalar, array-like, `dict`, or `None`
    * If to_replace is a `dict` and value is not a `list`,
      `dict`, `ndarray`, or `Series`
    * If to_replace is `None` and regex is not compilable
      into a regular expression or is a list, dict, ndarray, or
      Series.
    * When replacing multiple `bool` or `datetime64` objects and
      the arguments to to_replace does not match the type of the
      value being replaced

  ValueError
  : * If a `list` or an `ndarray` is passed to to_replace and
      value but they are not the same length.

#### SEE ALSO
[`Series.fillna`](dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)
: Fill NA values.

[`DataFrame.fillna`](dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)
: Fill NA values.

[`Series.where`](dask.dataframe.Series.where.md#dask.dataframe.Series.where)
: Replace values based on boolean condition.

[`DataFrame.where`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)
: Replace values based on boolean condition.

`DataFrame.map`
: Apply a function to a Dataframe elementwise.

[`Series.map`](dask.dataframe.Series.map.md#dask.dataframe.Series.map)
: Map values of Series according to an input mapping or function.

[`Series.str.replace`](dask.dataframe.Series.str.replace.md#dask.dataframe.Series.str.replace)
: Simple string replacement.

### Notes

* Regex substitution is performed under the hood with `re.sub`. The
  rules for substitution for `re.sub` are the same.
* Regular expressions will only substitute on strings, meaning you
  cannot provide, for example, a regular expression matching floating
  point numbers and expect the columns in your frame that have a
  numeric dtype to be matched. However, if those floating point
  numbers *are* strings, then you can do this.
* This method has *a lot* of options. You are encouraged to experiment
  and play with this method to gain intuition about how it works.
* When dict is used as the to_replace value, it is like
  key(s) in the dict are the to_replace part and
  value(s) in the dict are the value parameter.

### Examples

**Scalar \`to_replace\` and \`value\`**

```pycon
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> s.replace(1, 5)
0    5
1    2
2    3
3    4
4    5
dtype: int64
```

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": [5, 6, 7, 8, 9],
...         "C": ["a", "b", "c", "d", "e"],
...     }
... )
>>> df.replace(0, 5)
    A  B  C
0  5  5  a
1  1  6  b
2  2  7  c
3  3  8  d
4  4  9  e
```

**List-like \`to_replace\`**

```pycon
>>> df.replace([0, 1, 2, 3], 4)
    A  B  C
0  4  5  a
1  4  6  b
2  4  7  c
3  4  8  d
4  4  9  e
```

```pycon
>>> df.replace([0, 1, 2, 3], [4, 3, 2, 1])
    A  B  C
0  4  5  a
1  3  6  b
2  2  7  c
3  1  8  d
4  4  9  e
```

**dict-like \`to_replace\`**

```pycon
>>> df.replace({0: 10, 1: 100})
        A  B  C
0   10  5  a
1  100  6  b
2    2  7  c
3    3  8  d
4    4  9  e
```

```pycon
>>> df.replace({"A": 0, "B": 5}, 100)
        A    B  C
0  100  100  a
1    1    6  b
2    2    7  c
3    3    8  d
4    4    9  e
```

```pycon
>>> df.replace({"A": {0: 100, 4: 400}})
        A  B  C
0  100  5  a
1    1  6  b
2    2  7  c
3    3  8  d
4  400  9  e
```

**Regular expression \`to_replace\`**

```pycon
>>> df = pd.DataFrame({"A": ["bat", "foo", "bait"], "B": ["abc", "bar", "xyz"]})
>>> df.replace(to_replace=r"^ba.$", value="new", regex=True)
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace({"A": r"^ba.$"}, {"A": "new"}, regex=True)
        A    B
0   new  abc
1   foo  bar
2  bait  xyz
```

```pycon
>>> df.replace(regex=r"^ba.$", value="new")
        A    B
0   new  abc
1   foo  new
2  bait  xyz
```

```pycon
>>> df.replace(regex={r"^ba.$": "new", "foo": "xyz"})
        A    B
0   new  abc
1   xyz  new
2  bait  xyz
```

```pycon
>>> df.replace(regex=[r"^ba.$", "foo"], value="new")
        A    B
0   new  abc
1   new  new
2  bait  xyz
```

Compare the behavior of `s.replace({'a': None})` and
`s.replace('a', None)` to understand the peculiarities
of the to_replace parameter:

```pycon
>>> s = pd.Series([10, "a", "a", "b", "a"])
```

When one uses a dict as the to_replace value, it is like the
value(s) in the dict are equal to the value parameter.
`s.replace({'a': None})` is equivalent to
`s.replace(to_replace={'a': None}, value=None)`:

```pycon
>>> s.replace({"a": None})
0      10
1    None
2    None
3       b
4    None
dtype: object
```

If `None` is explicitly passed for `value`, it will be respected:

```pycon
>>> s.replace("a", None)
0      10
1    None
2    None
3       b
4    None
dtype: object
```

When `regex=True`, `value` is not `None` and to_replace is a string,
the replacement will be applied in all columns of the DataFrame.

```pycon
>>> df = pd.DataFrame(
...     {
...         "A": [0, 1, 2, 3, 4],
...         "B": ["a", "b", "c", "d", "e"],
...         "C": ["f", "g", "h", "i", "j"],
...     }
... )
```

```pycon
>>> df.replace(to_replace="^[a-g]", value="e", regex=True)
    A  B  C
0  0  e  e
1  1  e  e
2  2  e  h
3  3  e  i
4  4  e  j
```

If `value` is not `None` and to_replace is a dictionary, the dictionary
keys will be the DataFrame columns that the replacement will be applied.

```pycon
>>> df.replace(to_replace={"B": "^[a-c]", "C": "^[h-j]"}, value="e", regex=True)
    A  B  C
0  0  e  f
1  1  e  g
2  2  e  e
3  3  d  e
4  4  e  e
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.resample.html.md

# dask.dataframe.Series.resample

#### Series.resample(rule, closed=None, label=None)

Resample time-series data.

This docstring was copied from pandas.DataFrame.resample.

Some inconsistencies with the Dask version may exist.

Convenience method for frequency conversion and resampling of time series.
The object must have a datetime-like index (DatetimeIndex, PeriodIndex,
or TimedeltaIndex), or the caller must pass the label of a datetime-like
series/index to the `on`/`level` keyword parameter.

* **Parameters:**
  **rule**
  : The offset string or object representing target conversion.

  **closed**
  : Which side of bin interval is closed. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **label**
  : Which bin edge label to label bucket with. The default is ‘left’
    for all frequency offsets except for ‘ME’, ‘YE’, ‘QE’, ‘BME’,
    ‘BA’, ‘BQE’, and ‘W’ which all have a default of ‘right’.

  **convention**
  : For PeriodIndex only, controls whether to use the start or
    end of rule.

  **on**
  : For a DataFrame, column to use instead of index for resampling.
    Column must be datetime-like.

  **level**
  : For a MultiIndex, level (name or number) to use for
    resampling. level must be datetime-like.

  **origin**
  : The timestamp on which to adjust the grouping. The timezone of origin
    must match the timezone of the index.
    If string, must be Timestamp convertible or one of the following:
    - ‘epoch’: origin is 1970-01-01
    - ‘start’: origin is the first value of the timeseries
    - ‘start_day’: origin is the first day at midnight of the timeseries
    - ‘end’: origin is the last value of the timeseries
    - ‘end_day’: origin is the ceiling midnight of the last day
    <br/>
    #### NOTE
    Only takes effect for Tick-frequencies (i.e. fixed frequencies like
    days, hours, and minutes, rather than months or quarters).

  **offset**
  : An offset timedelta added to the origin.

  **group_keys**
  : Whether to include the group keys in the result index when using
    `.apply()` on the resampled object.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: `group_keys` now defaults to `False`.
* **Returns:**
  pandas.api.typing.Resampler
  : `Resampler` object.

#### SEE ALSO
[`Series.resample`](#dask.dataframe.Series.resample)
: Resample a Series.

[`DataFrame.resample`](dask.dataframe.DataFrame.resample.md#dask.dataframe.DataFrame.resample)
: Resample a DataFrame.

[`groupby`](dask.dataframe.Series.groupby.md#dask.dataframe.Series.groupby)
: Group Series/DataFrame by mapping, function, label, or list of labels.

`asfreq`
: Reindex a Series/DataFrame with the given frequency without grouping.

### Notes

See the [user guide](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#resampling)
for more.

To learn more about the offset strings, please see [this link](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects).

### Examples

Start by creating a series with 9 one minute timestamps.

```pycon
>>> index = pd.date_range("1/1/2000", periods=9, freq="min")
>>> series = pd.Series(range(9), index=index)
>>> series
2000-01-01 00:00:00    0
2000-01-01 00:01:00    1
2000-01-01 00:02:00    2
2000-01-01 00:03:00    3
2000-01-01 00:04:00    4
2000-01-01 00:05:00    5
2000-01-01 00:06:00    6
2000-01-01 00:07:00    7
2000-01-01 00:08:00    8
Freq: min, dtype: int64
```

Downsample the series into 3 minute bins and sum the values
of the timestamps falling into a bin.

```pycon
>>> series.resample("3min").sum()
2000-01-01 00:00:00     3
2000-01-01 00:03:00    12
2000-01-01 00:06:00    21
Freq: 3min, dtype: int64
```

Downsample the series into 3 minute bins as above, but label each
bin using the right edge instead of the left. Please note that the
value in the bucket used as the label is not included in the bucket,
which it labels. For example, in the original series the
bucket `2000-01-01 00:03:00` contains the value 3, but the summed
value in the resampled bucket with the label `2000-01-01 00:03:00`
does not include 3 (if it did, the summed value would be 6, not 3).

```pycon
>>> series.resample("3min", label="right").sum()
2000-01-01 00:03:00     3
2000-01-01 00:06:00    12
2000-01-01 00:09:00    21
Freq: 3min, dtype: int64
```

To include this value close the right side of the bin interval,
as shown below.

```pycon
>>> series.resample("3min", label="right", closed="right").sum()
2000-01-01 00:00:00     0
2000-01-01 00:03:00     6
2000-01-01 00:06:00    15
2000-01-01 00:09:00    15
Freq: 3min, dtype: int64
```

Upsample the series into 30 second bins.

```pycon
>>> series.resample("30s").asfreq()[0:5]  # Select first 5 rows
2000-01-01 00:00:00   0.0
2000-01-01 00:00:30   NaN
2000-01-01 00:01:00   1.0
2000-01-01 00:01:30   NaN
2000-01-01 00:02:00   2.0
Freq: 30s, dtype: float64
```

Upsample the series into 30 second bins and fill the `NaN`
values using the `ffill` method.

```pycon
>>> series.resample("30s").ffill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    0
2000-01-01 00:01:00    1
2000-01-01 00:01:30    1
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Upsample the series into 30 second bins and fill the
`NaN` values using the `bfill` method.

```pycon
>>> series.resample("30s").bfill()[0:5]
2000-01-01 00:00:00    0
2000-01-01 00:00:30    1
2000-01-01 00:01:00    1
2000-01-01 00:01:30    2
2000-01-01 00:02:00    2
Freq: 30s, dtype: int64
```

Pass a custom function via `apply`

```pycon
>>> def custom_resampler(arraylike):
...     return np.sum(arraylike) + 5
>>> series.resample("3min").apply(custom_resampler)
2000-01-01 00:00:00     8
2000-01-01 00:03:00    17
2000-01-01 00:06:00    26
Freq: 3min, dtype: int64
```

For a Series with a PeriodIndex, the keyword convention can be
used to control whether to use the start or end of rule.

Resample a year by quarter using ‘start’ convention. Values are
assigned to the first quarter of the period.

```pycon
>>> s = pd.Series(
...     [1, 2], index=pd.period_range("2012-01-01", freq="Y", periods=2)
... )
>>> s
2012    1
2013    2
Freq: Y-DEC, dtype: int64
>>> s.resample("Q", convention="start").asfreq()
2012Q1    1.0
2012Q2    NaN
2012Q3    NaN
2012Q4    NaN
2013Q1    2.0
2013Q2    NaN
2013Q3    NaN
2013Q4    NaN
Freq: Q-DEC, dtype: float64
```

Resample quarters by month using ‘end’ convention. Values are
assigned to the last month of the period.

```pycon
>>> q = pd.Series(
...     [1, 2, 3, 4], index=pd.period_range("2018-01-01", freq="Q", periods=4)
... )
>>> q
2018Q1    1
2018Q2    2
2018Q3    3
2018Q4    4
Freq: Q-DEC, dtype: int64
>>> q.resample("M", convention="end").asfreq()
2018-03    1.0
2018-04    NaN
2018-05    NaN
2018-06    2.0
2018-07    NaN
2018-08    NaN
2018-09    3.0
2018-10    NaN
2018-11    NaN
2018-12    4.0
Freq: M, dtype: float64
```

For DataFrame objects, the keyword on can be used to specify the
column instead of the index for resampling.

```pycon
>>> df = pd.DataFrame([10, 11, 9, 13, 14, 18, 17, 19], columns=["price"])
>>> df["volume"] = [50, 60, 40, 100, 50, 100, 40, 50]
>>> df["week_starting"] = pd.date_range("01/01/2018", periods=8, freq="W")
>>> df
   price  volume week_starting
0     10      50    2018-01-07
1     11      60    2018-01-14
2      9      40    2018-01-21
3     13     100    2018-01-28
4     14      50    2018-02-04
5     18     100    2018-02-11
6     17      40    2018-02-18
7     19      50    2018-02-25
>>> df.resample("ME", on="week_starting").mean()
               price  volume
week_starting
2018-01-31     10.75    62.5
2018-02-28     17.00    60.0
```

For a DataFrame with MultiIndex, the keyword level can be used to
specify on which level the resampling needs to take place.

```pycon
>>> days = pd.date_range("1/1/2000", periods=4, freq="D")
>>> df2 = pd.DataFrame(
...     [
...         [10, 50],
...         [11, 60],
...         [9, 40],
...         [13, 100],
...         [14, 50],
...         [18, 100],
...         [17, 40],
...         [19, 50],
...     ],
...     columns=["price", "volume"],
...     index=pd.MultiIndex.from_product([days, ["morning", "afternoon"]]),
... )
>>> df2
                      price  volume
2000-01-01 morning       10      50
           afternoon     11      60
2000-01-02 morning        9      40
           afternoon     13     100
2000-01-03 morning       14      50
           afternoon     18     100
2000-01-04 morning       17      40
           afternoon     19      50
>>> df2.resample("D", level=0).sum()
            price  volume
2000-01-01     21     110
2000-01-02     22     140
2000-01-03     32     150
2000-01-04     36      90
```

If you want to adjust the start of the bins based on a fixed timestamp:

```pycon
>>> start, end = "2000-10-01 23:30:00", "2000-10-02 00:30:00"
>>> rng = pd.date_range(start, end, freq="7min")
>>> ts = pd.Series(np.arange(len(rng)) * 3, index=rng)
>>> ts
2000-10-01 23:30:00     0
2000-10-01 23:37:00     3
2000-10-01 23:44:00     6
2000-10-01 23:51:00     9
2000-10-01 23:58:00    12
2000-10-02 00:05:00    15
2000-10-02 00:12:00    18
2000-10-02 00:19:00    21
2000-10-02 00:26:00    24
Freq: 7min, dtype: int64
```

```pycon
>>> ts.resample("17min").sum()
2000-10-01 23:14:00     0
2000-10-01 23:31:00     9
2000-10-01 23:48:00    21
2000-10-02 00:05:00    54
2000-10-02 00:22:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="epoch").sum()
2000-10-01 23:18:00     0
2000-10-01 23:35:00    18
2000-10-01 23:52:00    27
2000-10-02 00:09:00    39
2000-10-02 00:26:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", origin="2000-01-01").sum()
2000-10-01 23:24:00     3
2000-10-01 23:41:00    15
2000-10-01 23:58:00    45
2000-10-02 00:15:00    45
Freq: 17min, dtype: int64
```

If you want to adjust the start of the bins with an offset Timedelta, the two
following lines are equivalent:

```pycon
>>> ts.resample("17min", origin="start").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

```pycon
>>> ts.resample("17min", offset="23h30min").sum()
2000-10-01 23:30:00     9
2000-10-01 23:47:00    21
2000-10-02 00:04:00    54
2000-10-02 00:21:00    24
Freq: 17min, dtype: int64
```

If you want to take the largest Timestamp as the end of the bins:

```pycon
>>> ts.resample("17min", origin="end").sum()
2000-10-01 23:35:00     0
2000-10-01 23:52:00    18
2000-10-02 00:09:00    27
2000-10-02 00:26:00    63
Freq: 17min, dtype: int64
```

In contrast with the start_day, you can use end_day to take the ceiling
midnight of the largest Timestamp as the end of the bins and drop the bins
not containing data:

```pycon
>>> ts.resample("17min", origin="end_day").sum()
2000-10-01 23:38:00     3
2000-10-01 23:55:00    15
2000-10-02 00:12:00    45
2000-10-02 00:29:00    45
Freq: 17min, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.reset_index.html.md

# dask.dataframe.Series.reset_index

#### Series.reset_index(drop: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Reset the index to the default index.

Note that unlike in `pandas`, the reset index for a Dask DataFrame will
not be monotonically increasing from 0. Instead, it will restart at 0
for each partition (e.g. `index1 = [0, ..., 10], index2 = [0, ...]`).
This is due to the inability to statically know the full length of the
index.

For DataFrame with multi-level index, returns a new DataFrame with
labeling information in the columns under the index names, defaulting
to ‘level_0’, ‘level_1’, etc. if any are None. For a standard index,
the index name will be used (if set), otherwise a default ‘index’ or
‘level_0’ (if ‘index’ is already taken) will be used.

* **Parameters:**
  **drop**
  : Do not try to insert index into dataframe columns.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.rolling.html.md

# dask.dataframe.Series.rolling

#### Series.rolling(window, \*\*kwargs)

Provides rolling transformations.

* **Parameters:**
  **window**
  : Size of the moving window. This is the number of observations used
    for calculating the statistic. When not using a `DatetimeIndex`,
    the window size must not be so large as to span more than one
    adjacent partition. If using an offset or offset alias like ‘5D’,
    the data must have a `DatetimeIndex`

  **min_periods**
  : Minimum number of observations in window required to have a value
    (otherwise result is NA).

  **center**
  : Set the labels at the center of the window.

  **win_type**
  : Provide a window type. The recognized window types are identical
    to pandas.

  **axis**
  : This parameter is deprecated with `pandas>=2.1`.
* **Returns:**
  a Rolling object on which to call a method to compute a statistic

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.round.html.md

# dask.dataframe.Series.round

#### Series.round(decimals=0)

Round numeric columns in a DataFrame to a variable number of decimal places.

This docstring was copied from pandas.DataFrame.round.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **decimals**
  : Number of decimal places to round each column to. If an int is
    given, round each column to the same number of places.
    Otherwise dict and Series round to variable numbers of places.
    Column names should be in the keys if decimals is a
    dict-like, or in the index if decimals is a Series. Any
    columns not included in decimals will be left as is. Elements
    of decimals which are not columns of the input will be
    ignored.

  **\*args**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.

  **\*\*kwargs**
  : Additional keywords have no effect but might be accepted for
    compatibility with numpy.
* **Returns:**
  DataFrame
  : A DataFrame with the affected columns rounded to the specified
    number of decimal places.

#### SEE ALSO
[`numpy.around`](https://numpy.org/doc/stable/reference/generated/numpy.around.html#numpy.around)
: Round a numpy array to the given number of decimals.

[`Series.round`](#dask.dataframe.Series.round)
: Round a Series to the given number of decimals.

### Notes

For values exactly halfway between rounded decimal values, pandas rounds
to the nearest even value (e.g. -0.5 and 0.5 round to 0.0, 1.5 and 2.5
round to 2.0, etc.).

### Examples

```pycon
>>> df = pd.DataFrame(
...     [(0.21, 0.32), (0.01, 0.67), (0.66, 0.03), (0.21, 0.18)],
...     columns=["dogs", "cats"],
... )
>>> df
    dogs  cats
0  0.21  0.32
1  0.01  0.67
2  0.66  0.03
3  0.21  0.18
```

By providing an integer each column is rounded to the same number
of decimal places

```pycon
>>> df.round(1)
    dogs  cats
0   0.2   0.3
1   0.0   0.7
2   0.7   0.0
3   0.2   0.2
```

With a dict, the number of places for specific columns can be
specified with the column names as key and the number of decimal
places as value

```pycon
>>> df.round({"dogs": 1, "cats": 0})
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

Using a Series, the number of places for specific columns can be
specified with the column names as index and the number of
decimal places as value

```pycon
>>> decimals = pd.Series([0, 1], index=["cats", "dogs"])
>>> df.round(decimals)
    dogs  cats
0   0.2   0.0
1   0.0   1.0
2   0.7   0.0
3   0.2   0.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.sample.html.md

# dask.dataframe.Series.sample

#### Series.sample(n=None, frac=None, replace=False, random_state=None)

Random sample of items

* **Parameters:**
  **n**
  : Number of items to return is not supported by dask. Use frac
    instead.

  **frac**
  : Approximate fraction of items to return. This sampling fraction is
    applied to all partitions equally. Note that this is an
    **approximate fraction**. You should not expect exactly `len(df) * frac`
    items to be returned, as the exact number of elements selected will
    depend on how your data is partitioned (but should be pretty close
    in practice).

  **replace**
  : Sample with or without replacement. Default = False.

  **random_state**
  : If an int, we create a new RandomState with this as the seed;
    Otherwise we draw from the passed RandomState.

#### SEE ALSO
[`DataFrame.random_split`](dask.dataframe.DataFrame.random_split.md#dask.dataframe.DataFrame.random_split)

[`pandas.DataFrame.sample`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sample.html#pandas.DataFrame.sample)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.sem.html.md

# dask.dataframe.Series.sem

#### Series.sem(axis=None, skipna=True, ddof=1, split_every=False, numeric_only=False)

Return unbiased standard error of the mean over requested axis.

This docstring was copied from pandas.DataFrame.sem.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.sem with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keywords passed.
* **Returns:**
  Series or DataFrame (if level specified)
  : Unbiased standard error of the mean over requested axis.

#### SEE ALSO
[`DataFrame.var`](dask.dataframe.DataFrame.var.md#dask.dataframe.DataFrame.var)
: Return unbiased variance over requested axis.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Returns sample standard deviation over requested axis.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3])
>>> round(s.sem(), 6)
0.57735
```

With a DataFrame

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": [2, 3]}, index=["tiger", "zebra"])
>>> df
       a   b
tiger  1   2
zebra  2   3
>>> df.sem()
a   0.5
b   0.5
dtype: float64
```

Using axis=1

```pycon
>>> df.sem(axis=1)
tiger   0.5
zebra   0.5
dtype: float64
```

In this case, numeric_only should be set to True
to avoid getting an error.

```pycon
>>> df = pd.DataFrame({"a": [1, 2], "b": ["T", "Z"]}, index=["tiger", "zebra"])
>>> df.sem(numeric_only=True)
a   0.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.shape.html.md

# dask.dataframe.Series.shape

#### *property* Series.shape

Return a tuple representing the dimensionality of the DataFrame.

The number of rows is a Delayed result. The number of columns
is a concrete integer.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.shift.html.md

# dask.dataframe.Series.shift

#### Series.shift(periods=1, freq=None, axis=0)

Shift index by desired number of periods with an optional time freq.

This docstring was copied from pandas.DataFrame.shift.

Some inconsistencies with the Dask version may exist.

When freq is not passed, shift the index without realigning the data.
If freq is passed (in this case, the index must be date or datetime,
or it will raise a NotImplementedError), the index will be
increased using the periods and the freq. freq can be inferred
when specified as “infer” as long as either freq or inferred_freq
attribute is set in the index.

* **Parameters:**
  **periods**
  : Number of periods to shift. Can be positive or negative.
    If an iterable of ints, the data will be shifted once by each int.
    This is equivalent to shifting by one value at a time and
    concatenating all resulting frames. The resulting columns will have
    the shift suffixed to their column names. For multiple periods,
    axis must not be 1.

  **freq**
  : Offset to use from the tseries module or time rule (e.g. ‘EOM’).
    If freq is specified then the index values are shifted but the
    data is not realigned. That is, use freq if you would like to
    extend the index when shifting and preserve the original data.
    If freq is specified as “infer” then it will be inferred from
    the freq or inferred_freq attributes of the index. If neither of
    those attributes exist, a ValueError is thrown.

  **axis**
  : Shift direction. For Series this parameter is unused and defaults to 0.

  **fill_value**
  : The scalar value to use for newly introduced missing values.
    the default depends on the dtype of self.
    For Boolean and numeric NumPy data types, `np.nan` is used.
    For datetime, timedelta, or period data, etc. `NaT` is used.
    For extension dtypes, `self.dtype.na_value` is used.

  **suffix**
  : If str and periods is an iterable, this is added after the column
    name and before the shift value for each shifted column name.
    For Series this parameter is unused and defaults to None.
* **Returns:**
  DataFrame
  : Copy of input object, shifted.

#### SEE ALSO
[`Index.shift`](dask.dataframe.Index.shift.md#dask.dataframe.Index.shift)
: Shift values of Index.

`DatetimeIndex.shift`
: Shift values of DatetimeIndex.

`PeriodIndex.shift`
: Shift values of PeriodIndex.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [[10, 13, 17], [20, 23, 27], [15, 18, 22], [30, 33, 37], [45, 48, 52]],
...     columns=["Col1", "Col2", "Col3"],
...     index=pd.date_range("2020-01-01", "2020-01-05"),
... )
>>> df
            Col1  Col2  Col3
2020-01-01    10    13    17
2020-01-02    20    23    27
2020-01-03    15    18    22
2020-01-04    30    33    37
2020-01-05    45    48    52
```

```pycon
>>> df.shift(periods=3)
            Col1  Col2  Col3
2020-01-01   NaN   NaN   NaN
2020-01-02   NaN   NaN   NaN
2020-01-03   NaN   NaN   NaN
2020-01-04  10.0  13.0  17.0
2020-01-05  20.0  23.0  27.0
```

```pycon
>>> df.shift(periods=1, axis="columns")
            Col1  Col2  Col3
2020-01-01   NaN    10    13
2020-01-02   NaN    20    23
2020-01-03   NaN    15    18
2020-01-04   NaN    30    33
2020-01-05   NaN    45    48
```

```pycon
>>> df.shift(periods=3, fill_value=0)
            Col1  Col2  Col3
2020-01-01     0     0     0
2020-01-02     0     0     0
2020-01-03     0     0     0
2020-01-04    10    13    17
2020-01-05    20    23    27
```

```pycon
>>> df.shift(periods=3, freq="D")
            Col1  Col2  Col3
2020-01-04    10    13    17
2020-01-05    20    23    27
2020-01-06    15    18    22
2020-01-07    30    33    37
2020-01-08    45    48    52
```

```pycon
>>> df.shift(periods=3, freq="infer")
            Col1  Col2  Col3
2020-01-04    10    13    17
2020-01-05    20    23    27
2020-01-06    15    18    22
2020-01-07    30    33    37
2020-01-08    45    48    52
```

```pycon
>>> df["Col1"].shift(periods=[0, 1, 2])
            Col1_0  Col1_1  Col1_2
2020-01-01      10     NaN     NaN
2020-01-02      20    10.0     NaN
2020-01-03      15    20.0    10.0
2020-01-04      30    15.0    20.0
2020-01-05      45    30.0    15.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.size.html.md

# dask.dataframe.Series.size

#### *property* Series.size

Size of the Series or DataFrame as a Delayed object.

### Examples

```pycon
>>> series.size
<dask_expr.expr.Scalar: expr=df.size(), dtype=int64>
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.std.html.md

# dask.dataframe.Series.std

#### Series.std(axis=0, skipna=True, ddof=1, numeric_only=False, split_every=False, \*\*kwargs)

Return sample standard deviation over requested axis.

This docstring was copied from pandas.DataFrame.std.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument.

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.std with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Standard deviation over requested axis.

#### SEE ALSO
[`Series.std`](#dask.dataframe.Series.std)
: Return standard deviation over Series values.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Return the mean of the values over the requested axis.

[`DataFrame.median`](dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)
: Return the median of the values over the requested axis.

[`DataFrame.mode`](dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)
: Get the mode(s) of each element along the requested axis.

[`DataFrame.sum`](dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)
: Return the sum of the values over the requested axis.

### Notes

To have the same behaviour as numpy.std, use ddof=0 (instead of the
default ddof=1)

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "person_id": [0, 1, 2, 3],
...         "age": [21, 25, 62, 43],
...         "height": [1.61, 1.87, 1.49, 2.01],
...     }
... ).set_index("person_id")
>>> df
           age  height
person_id
0           21    1.61
1           25    1.87
2           62    1.49
3           43    2.01
```

The standard deviation of the columns can be found as follows:

```pycon
>>> df.std()
age       18.786076
height     0.237417
dtype: float64
```

Alternatively, ddof=0 can be set to normalize by N instead of N-1:

```pycon
>>> df.std(ddof=0)
age       16.269219
height     0.205609
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.capitalize.html.md

# dask.dataframe.Series.str.capitalize

#### dataframe.Series.str.capitalize()

Convert strings in the Series/Index to be capitalized.

This docstring was copied from pandas.core.strings.accessor.StringMethods.capitalize.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.capitalize()`](https://docs.python.org/3/library/stdtypes.html#str.capitalize).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.capitalize()`](https://docs.python.org/3/library/stdtypes.html#str.capitalize).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.casefold.html.md

# dask.dataframe.Series.str.casefold

#### dataframe.Series.str.casefold()

Convert strings in the Series/Index to be casefolded.

This docstring was copied from pandas.core.strings.accessor.StringMethods.casefold.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.casefold()`](https://docs.python.org/3/library/stdtypes.html#str.casefold).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.casefold()`](https://docs.python.org/3/library/stdtypes.html#str.casefold).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.cat.html.md

# dask.dataframe.Series.str.cat

#### dataframe.Series.str.cat(others=None, sep=None, na_rep=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.center.html.md

# dask.dataframe.Series.str.center

#### dataframe.Series.str.center(width: [int](https://docs.python.org/3/library/functions.html#int), fillchar: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ')

Pad left and right side of strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.center.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.center()`](https://docs.python.org/3/library/stdtypes.html#str.center).

* **Parameters:**
  **width**
  : Minimum width of resulting string; additional characters will be filled
    with `fillchar`.

  **fillchar**
  : Additional character for filling, default is whitespace.
* **Returns:**
  Series/Index of objects.
  : A Series or Index where the strings are modified by [`str.center()`](https://docs.python.org/3/library/stdtypes.html#str.center).

#### SEE ALSO
`Series.str.rjust`
: Fills the left side of strings with an arbitrary character.

`Series.str.ljust`
: Fills the right side of strings with an arbitrary character.

`Series.str.center`
: Fills both sides of strings with an arbitrary character.

`Series.str.zfill`
: Pad strings in the Series/Index by prepending ‘0’ character.

### Examples

For Series.str.center:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.center(8, fillchar=".")
0   ..dog...
1   ..bird..
2   .mouse..
dtype: str
```

For Series.str.ljust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.ljust(8, fillchar=".")
0   dog.....
1   bird....
2   mouse...
dtype: str
```

For Series.str.rjust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.rjust(8, fillchar=".")
0   .....dog
1   ....bird
2   ...mouse
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.contains.html.md

# dask.dataframe.Series.str.contains

#### dataframe.Series.str.contains(pat, case: bool = True, flags: int = 0, na=<no_default>, regex: bool = True)

Test if pattern or regex is contained within a string of a Series or Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.contains.

Some inconsistencies with the Dask version may exist.

Return boolean Series or Index based on whether a given pattern or regex is
contained within a string of a Series or Index.

* **Parameters:**
  **pat**
  : Character sequence or regular expression.

  **case**
  : If True, case sensitive.

  **flags**
  : Flags to pass through to the re module, e.g. re.IGNORECASE.

  **na**
  : Fill value for missing values. The default depends on dtype of the
    array. For the `"str"` dtype, `False` is used. For object
    dtype, `numpy.nan` is used. For the nullable `StringDtype`,
    `pandas.NA` is used.

  **regex**
  : If True, assumes the pat is a regular expression.
    <br/>
    If False, treats the pat as a literal string.
* **Returns:**
  Series or Index of boolean values
  : A Series or Index of boolean values indicating whether the
    given pattern is contained within the string of each element
    of the Series or Index.

#### SEE ALSO
[`match`](dask.dataframe.Series.str.match.md#dask.dataframe.Series.str.match)
: Analogous, but stricter, relying on re.match instead of re.search.

`Series.str.startswith`
: Test if the start of each string element matches a pattern.

`Series.str.endswith`
: Same as startswith, but tests the end of string.

### Examples

Returning a Series of booleans using only a literal pattern.

```pycon
>>> s1 = pd.Series(["Mouse", "dog", "house and parrot", "23", np.nan])
>>> s1.str.contains("og", regex=False)
0    False
1     True
2    False
3    False
4    False
dtype: bool
```

Returning an Index of booleans using only a literal pattern.

```pycon
>>> ind = pd.Index(["Mouse", "dog", "house and parrot", "23.0", np.nan])
>>> ind.str.contains("23", regex=False)
array([False, False, False,  True, False])
```

Specifying case sensitivity using case.

```pycon
>>> s1.str.contains("oG", case=True, regex=True)
0    False
1    False
2    False
3    False
4    False
dtype: bool
```

Returning ‘house’ or ‘dog’ when either expression occurs in a string.

```pycon
>>> s1.str.contains("house|dog", regex=True)
0    False
1     True
2     True
3    False
4    False
dtype: bool
```

Ignoring case sensitivity using flags with regex.

```pycon
>>> import re
>>> s1.str.contains("PARROT", flags=re.IGNORECASE, regex=True)
0    False
1    False
2     True
3    False
4    False
dtype: bool
```

Returning any digit using regular expression.

```pycon
>>> s1.str.contains("\\d", regex=True)
0    False
1    False
2    False
3     True
4    False
dtype: bool
```

Ensure pat is a not a literal pattern when regex is set to True.
Note in the following example one might expect only s2[1] and s2[3] to
return True. However, ‘.0’ as a regex matches any character
followed by a 0.

```pycon
>>> s2 = pd.Series(["40", "40.0", "41", "41.0", "35"])
>>> s2.str.contains(".0", regex=True)
0     True
1     True
2    False
3     True
4    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.count.html.md

# dask.dataframe.Series.str.count

#### dataframe.Series.str.count(pat, flags: [int](https://docs.python.org/3/library/functions.html#int) = 0)

Count occurrences of pattern in each string of the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.count.

Some inconsistencies with the Dask version may exist.

This function is used to count the number of times a particular regex
pattern is repeated in each of the string elements of the
[`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series).

* **Parameters:**
  **pat**
  : Valid regular expression.

  **flags**
  : Flags for the re module. For a complete list, [see here](https://docs.python.org/3/howto/regex.html#compilation-flags).
* **Returns:**
  Series or Index
  : Same type as the calling object containing the integer counts.

#### SEE ALSO
[`re`](https://docs.python.org/3/library/re.html#module-re)
: Standard library module for regular expressions.

[`str.count`](https://docs.python.org/3/library/stdtypes.html#str.count)
: Standard library version, without regular expression support.

### Notes

Some characters need to be escaped when passing in pat.
eg. `'$'` has a special meaning in regex and must be escaped when
finding this literal character.

### Examples

```pycon
>>> s = pd.Series(["A", "B", "Aaba", "Baca", np.nan, "CABA", "cat"])
>>> s.str.count("a")
0    0.0
1    0.0
2    2.0
3    2.0
4    NaN
5    0.0
6    1.0
dtype: float64
```

Escape `'$'` to find the literal dollar sign.

```pycon
>>> s = pd.Series(["$", "B", "Aab$", "$$ca", "C$B$", "cat"])
>>> s.str.count("\\$")
0    1
1    0
2    1
3    2
4    2
5    0
dtype: int64
```

This is also available on Index

```pycon
>>> pd.Index(["A", "A", "Aaba", "cat"]).str.count("a")
Index([0, 0, 2, 1], dtype='int64')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.decode.html.md

# dask.dataframe.Series.str.decode

#### dataframe.Series.str.decode(encoding, errors: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'strict', dtype: [str](https://docs.python.org/3/library/stdtypes.html#str) | DtypeObj | [None](https://docs.python.org/3/library/constants.html#None) = None)

Decode character string in the Series/Index using indicated encoding.

This docstring was copied from pandas.core.strings.accessor.StringMethods.decode.

Some inconsistencies with the Dask version may exist.

Equivalent to `str.decode()` in python2 and [`bytes.decode()`](https://docs.python.org/3/library/stdtypes.html#bytes.decode) in
python3.

* **Parameters:**
  **encoding**
  : Specifies the encoding to be used.

  **errors**
  : Specifies the error handling scheme.
    Possible values are those supported by [`bytes.decode()`](https://docs.python.org/3/library/stdtypes.html#bytes.decode).

  **dtype**
  : The dtype of the result. When not `None`, must be either a string or
    object dtype. When `None`, the dtype of the result is determined by
    `pd.options.future.infer_string`.
    <br/>
    #### Versionadded
    Added in version 2.3.0.
* **Returns:**
  Series or Index
  : A Series or Index with decoded strings.

#### SEE ALSO
`Series.str.encode`
: Encodes strings into bytes in a Series/Index.

### Examples

For Series:

```pycon
>>> ser = pd.Series([b"cow", b"123", b"()"])
>>> ser.str.decode("ascii")
0   cow
1   123
2   ()
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.encode.html.md

# dask.dataframe.Series.str.encode

#### dataframe.Series.str.encode(encoding, errors: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'strict')

Encode character string in the Series/Index using indicated encoding.

This docstring was copied from pandas.core.strings.accessor.StringMethods.encode.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.encode()`](https://docs.python.org/3/library/stdtypes.html#str.encode).

* **Parameters:**
  **encoding**
  : Specifies the encoding to be used.

  **errors**
  : Specifies the error handling scheme.
    Possible values are those supported by [`str.encode()`](https://docs.python.org/3/library/stdtypes.html#str.encode).
* **Returns:**
  Series/Index of objects
  : A Series or Index with strings encoded into bytes.

#### SEE ALSO
`Series.str.decode`
: Decodes bytes into strings in a Series/Index.

### Examples

```pycon
>>> ser = pd.Series(["cow", "123", "()"])
>>> ser.str.encode(encoding="ascii")
0     b'cow'
1     b'123'
2      b'()'
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.endswith.html.md

# dask.dataframe.Series.str.endswith

#### dataframe.Series.str.endswith(pat: str | tuple[str, ...], na: Scalar | lib.NoDefault = <no_default>) → [Series](dask.dataframe.Series.md#dask.dataframe.Series) | [Index](dask.dataframe.Index.md#dask.dataframe.Index)

Test if the end of each string element matches a pattern.

This docstring was copied from pandas.core.strings.accessor.StringMethods.endswith.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.endswith()`](https://docs.python.org/3/library/stdtypes.html#str.endswith).

* **Parameters:**
  **pat**
  : Character sequence or tuple of strings. Regular expressions are not
    accepted.

  **na**
  : Object shown if element tested is not a string. The default depends
    on dtype of the array. For the `"str"` dtype, `False` is used.
    For object dtype, `numpy.nan` is used. For the nullable
    `StringDtype`, `pandas.NA` is used.
* **Returns:**
  Series or Index of bool
  : A Series of booleans indicating whether the given pattern matches
    the end of each string element.

#### SEE ALSO
[`str.endswith`](https://docs.python.org/3/library/stdtypes.html#str.endswith)
: Python standard library string method.

`Series.str.startswith`
: Same as endswith, but tests the start of string.

`Series.str.contains`
: Tests if string element contains a pattern.

### Examples

```pycon
>>> s = pd.Series(["bat", "bear", "caT", np.nan])
>>> s
0     bat
1    bear
2     caT
3     NaN
dtype: str
```

```pycon
>>> s.str.endswith("t")
0     True
1    False
2    False
3    False
dtype: bool
```

```pycon
>>> s.str.endswith(("t", "T"))
0     True
1    False
2     True
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.extract.html.md

# dask.dataframe.Series.str.extract

#### dataframe.Series.str.extract(pat: [str](https://docs.python.org/3/library/stdtypes.html#str), flags: [int](https://docs.python.org/3/library/functions.html#int) = 0, expand: [bool](https://docs.python.org/3/library/functions.html#bool) = True) → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame) | [Series](dask.dataframe.Series.md#dask.dataframe.Series) | [Index](dask.dataframe.Index.md#dask.dataframe.Index)

Extract capture groups in the regex pat as columns in a DataFrame.

This docstring was copied from pandas.core.strings.accessor.StringMethods.extract.

Some inconsistencies with the Dask version may exist.

For each subject string in the Series, extract groups from the
first match of regular expression pat.

* **Parameters:**
  **pat**
  : Regular expression pattern with capturing groups.

  **flags**
  : Flags from the `re` module, e.g. `re.IGNORECASE`, that
    modify regular expression matching for things like case,
    spaces, etc. For more details, see [`re`](https://docs.python.org/3/library/re.html#module-re).

  **expand**
  : If True, return DataFrame with one column per capture group.
    If False, return a Series/Index if there is one capture group
    or DataFrame if there are multiple capture groups.
* **Returns:**
  DataFrame or Series or Index
  : A DataFrame with one row for each subject string, and one
    column for each group. Any capture group names in regular
    expression pat will be used for column names; otherwise
    capture group numbers will be used. The dtype of each result
    column is always object, even when no match is found. If
    `expand=False` and pat has only one capture group, then
    return a Series (if subject is a Series) or Index (if subject
    is an Index).

#### SEE ALSO
[`extractall`](dask.dataframe.Series.str.extractall.md#dask.dataframe.Series.str.extractall)
: Returns all matches (not just the first match).

### Examples

A pattern with two groups will return a DataFrame with two columns.
Non-matches will be NaN.

```pycon
>>> s = pd.Series(["a1", "b2", "c3"])
>>> s.str.extract(r"([ab])(\d)")
    0    1
0    a    1
1    b    2
2  NaN  NaN
```

A pattern may contain optional groups.

```pycon
>>> s.str.extract(r"([ab])?(\d)")
    0  1
0    a  1
1    b  2
2  NaN  3
```

Named groups will become column names in the result.

```pycon
>>> s.str.extract(r"(?P<letter>[ab])(?P<digit>\d)")
letter digit
0      a     1
1      b     2
2    NaN   NaN
```

A pattern with one group will return a DataFrame with one column
if expand=True.

```pycon
>>> s.str.extract(r"[ab](\d)", expand=True)
     0
0    1
1    2
2  NaN
```

A pattern with one group will return a Series if expand=False.

```pycon
>>> s.str.extract(r"[ab](\d)", expand=False)
0      1
1      2
2    NaN
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.extractall.html.md

# dask.dataframe.Series.str.extractall

#### dataframe.Series.str.extractall(pat, flags: [int](https://docs.python.org/3/library/functions.html#int) = 0) → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)

Extract capture groups in the regex pat as columns in DataFrame.

This docstring was copied from pandas.core.strings.accessor.StringMethods.extractall.

Some inconsistencies with the Dask version may exist.

For each subject string in the Series, extract groups from all
matches of regular expression pat. When each subject string in the
Series has exactly one match, extractall(pat).xs(0, level=’match’)
is the same as extract(pat).

* **Parameters:**
  **pat**
  : Regular expression pattern with capturing groups.

  **flags**
  : A `re` module flag, for example `re.IGNORECASE`. These allow
    to modify regular expression matching for things like case, spaces,
    etc. Multiple flags can be combined with the bitwise OR operator,
    for example `re.IGNORECASE | re.MULTILINE`.
* **Returns:**
  DataFrame
  : A `DataFrame` with one row for each match, and one column for each
    group. Its rows have a `MultiIndex` with first levels that come from
    the subject `Series`. The last level is named ‘match’ and indexes the
    matches in each item of the `Series`. Any capture group names in
    regular expression pat will be used for column names; otherwise capture
    group numbers will be used.

#### SEE ALSO
[`extract`](dask.dataframe.Series.str.extract.md#dask.dataframe.Series.str.extract)
: Returns first match only (not all matches).

### Examples

A pattern with one group will return a DataFrame with one column.
Indices with no matches will not appear in the result.

```pycon
>>> s = pd.Series(["a1a2", "b1", "c1"], index=["A", "B", "C"])
>>> s.str.extractall(r"[ab](\d)")
        0
match
A 0      1
  1      2
B 0      1
```

Capture group names are used for column names of the result.

```pycon
>>> s.str.extractall(r"[ab](?P<digit>\d)")
        digit
match
A 0         1
  1         2
B 0         1
```

A pattern with two groups will return a DataFrame with two columns.

```pycon
>>> s.str.extractall(r"(?P<letter>[ab])(?P<digit>\d)")
        letter digit
match
A 0          a     1
  1          a     2
B 0          b     1
```

Optional groups that do not match are NaN in the result.

```pycon
>>> s.str.extractall(r"(?P<letter>[ab])?(?P<digit>\d)")
        letter digit
match
A 0          a     1
  1          a     2
B 0          b     1
C 0        NaN     1
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.find.html.md

# dask.dataframe.Series.str.find

#### dataframe.Series.str.find(sub, start: [int](https://docs.python.org/3/library/functions.html#int) = 0, end=None)

Return lowest indexes in each strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.find.

Some inconsistencies with the Dask version may exist.

Each of returned indexes corresponds to the position where the
substring is fully contained between [start:end]. Return -1 on
failure. Equivalent to standard [`str.find()`](https://docs.python.org/3/library/stdtypes.html#str.find).

* **Parameters:**
  **sub**
  : Substring being searched.

  **start**
  : Left edge index.

  **end**
  : Right edge index.
* **Returns:**
  Series or Index of int.
  : A Series (if the input is a Series) or an Index (if the input is an
    Index) of the lowest indexes corresponding to the positions where the
    substring is found in each string of the input.

#### SEE ALSO
[`rfind`](dask.dataframe.Series.str.rfind.md#dask.dataframe.Series.str.rfind)
: Return highest indexes in each strings.

### Examples

For Series.str.find:

```pycon
>>> ser = pd.Series(["_cow_", "duck_", "do_v_e"])
>>> ser.str.find("_")
0   0
1   4
2   2
dtype: int64
```

For Series.str.rfind:

```pycon
>>> ser = pd.Series(["_cow_", "duck_", "do_v_e"])
>>> ser.str.rfind("_")
0   4
1   4
2   4
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.findall.html.md

# dask.dataframe.Series.str.findall

#### dataframe.Series.str.findall(pat, flags: [int](https://docs.python.org/3/library/functions.html#int) = 0)

Find all occurrences of pattern or regular expression in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.findall.

Some inconsistencies with the Dask version may exist.

Equivalent to applying [`re.findall()`](https://docs.python.org/3/library/re.html#re.findall) to all the elements in the
Series/Index.

* **Parameters:**
  **pat**
  : Pattern or regular expression.

  **flags**
  : Flags from `re` module, e.g. re.IGNORECASE (default is 0, which
    means no flags).
* **Returns:**
  Series/Index of lists of strings
  : All non-overlapping matches of pattern or regular expression in each
    string of this Series/Index.

#### SEE ALSO
[`count`](dask.dataframe.Series.str.count.md#dask.dataframe.Series.str.count)
: Count occurrences of pattern or regular expression in each string of the Series/Index.

[`extractall`](dask.dataframe.Series.str.extractall.md#dask.dataframe.Series.str.extractall)
: For each string in the Series, extract groups from all matches of regular expression and return a DataFrame with one row for each match and one column for each group.

[`re.findall`](https://docs.python.org/3/library/re.html#re.findall)
: The equivalent `re` function to all non-overlapping matches of pattern or regular expression in string, as a list of strings.

### Examples

```pycon
>>> s = pd.Series(["Lion", "Monkey", "Rabbit"])
```

The search for the pattern ‘Monkey’ returns one match:

```pycon
>>> s.str.findall("Monkey")
0          []
1    [Monkey]
2          []
dtype: object
```

On the other hand, the search for the pattern ‘MONKEY’ doesn’t return any
match:

```pycon
>>> s.str.findall("MONKEY")
0    []
1    []
2    []
dtype: object
```

Flags can be added to the pattern or regular expression. For instance,
to find the pattern ‘MONKEY’ ignoring the case:

```pycon
>>> import re
>>> s.str.findall("MONKEY", flags=re.IGNORECASE)
0          []
1    [Monkey]
2          []
dtype: object
```

When the pattern matches more than one string in the Series, all matches
are returned:

```pycon
>>> s.str.findall("on")
0    [on]
1    [on]
2      []
dtype: object
```

Regular expressions are supported too. For instance, the search for all the
strings ending with the word ‘on’ is shown next:

```pycon
>>> s.str.findall("on$")
0    [on]
1      []
2      []
dtype: object
```

If the pattern is found more than once in the same string, then a list of
multiple strings is returned:

```pycon
>>> s.str.findall("b")
0        []
1        []
2    [b, b]
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.fullmatch.html.md

# dask.dataframe.Series.str.fullmatch

#### dataframe.Series.str.fullmatch(pat, case: bool = True, flags: int = 0, na=<no_default>)

Determine if each string entirely matches a regular expression.

This docstring was copied from pandas.core.strings.accessor.StringMethods.fullmatch.

Some inconsistencies with the Dask version may exist.

Checks if each string in the Series or Index fully matches the
specified regular expression pattern. This function is useful when the
requirement is for an entire string to conform to a pattern, such as
validating formats like phone numbers or email addresses.

* **Parameters:**
  **pat**
  : Character sequence or regular expression.

  **case**
  : If True, case sensitive.

  **flags**
  : Regex module flags, e.g. re.IGNORECASE.

  **na**
  : Fill value for missing values. The default depends on dtype of the
    array. For the `"str"` dtype, `False` is used. For object
    dtype, `numpy.nan` is used. For the nullable `StringDtype`,
    `pandas.NA` is used.
* **Returns:**
  Series/Index/array of boolean values
  : The function returns a Series, Index, or array of boolean values,
    where True indicates that the entire string matches the regular
    expression pattern and False indicates that it does not.

#### SEE ALSO
[`match`](dask.dataframe.Series.str.match.md#dask.dataframe.Series.str.match)
: Similar, but also returns True when only a *prefix* of the string matches the regular expression.

[`extract`](dask.dataframe.Series.str.extract.md#dask.dataframe.Series.str.extract)
: Extract matched groups.

### Examples

```pycon
>>> ser = pd.Series(["cat", "duck", "dove"])
>>> ser.str.fullmatch(r"d.+")
0   False
1    True
2    True
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.get.html.md

# dask.dataframe.Series.str.get

#### dataframe.Series.str.get(i)

Extract element from each component at specified position or with specified key.

This docstring was copied from pandas.core.strings.accessor.StringMethods.get.

Some inconsistencies with the Dask version may exist.

Extract element from lists, tuples, dict, or strings in each element in the
Series/Index.

* **Parameters:**
  **i**
  : Position or key of element to extract.
* **Returns:**
  Series or Index
  : Series or Index where each value is the extracted element from
    the corresponding input component.

#### SEE ALSO
`Series.str.extract`
: Extract capture groups in the regex as columns in a DataFrame.

### Examples

```pycon
>>> s = pd.Series(
...     [
...         "String",
...         (1, 2, 3),
...         ["a", "b", "c"],
...         123,
...         -456,
...         {1: "Hello", "2": "World"},
...     ]
... )
>>> s
0                        String
1                     (1, 2, 3)
2                     [a, b, c]
3                           123
4                          -456
5    {1: 'Hello', '2': 'World'}
dtype: object
```

```pycon
>>> s.str.get(1)
0        t
1        2
2        b
3      NaN
4      NaN
5    Hello
dtype: object
```

```pycon
>>> s.str.get(-1)
0      g
1      3
2      c
3    NaN
4    NaN
5    None
dtype: object
```

Return element with given key

```pycon
>>> s = pd.Series(
...     [
...         {"name": "Hello", "value": "World"},
...         {"name": "Goodbye", "value": "Planet"},
...     ]
... )
>>> s.str.get("name")
0      Hello
1    Goodbye
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.index.html.md

# dask.dataframe.Series.str.index

#### dataframe.Series.str.index(sub, start: [int](https://docs.python.org/3/library/functions.html#int) = 0, end=None)

Return lowest indexes in each string in Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.index.

Some inconsistencies with the Dask version may exist.

Each of the returned indexes corresponds to the position where the
substring is fully contained between [start:end]. This is the same
as `str.find` except instead of returning -1, it raises a
ValueError when the substring is not found. Equivalent to standard
`str.index`.

* **Parameters:**
  **sub**
  : Substring being searched.

  **start**
  : Left edge index.

  **end**
  : Right edge index.
* **Returns:**
  Series or Index of object
  : Returns a Series or an Index of the lowest indexes
    in each string of the input.

#### SEE ALSO
[`rindex`](dask.dataframe.Series.str.rindex.md#dask.dataframe.Series.str.rindex)
: Return highest indexes in each strings.

### Examples

For Series.str.index:

```pycon
>>> ser = pd.Series(["horse", "eagle", "donkey"])
>>> ser.str.index("e")
0   4
1   0
2   4
dtype: int64
```

For Series.str.rindex:

```pycon
>>> ser = pd.Series(["Deer", "eagle", "Sheep"])
>>> ser.str.rindex("e")
0   2
1   4
2   3
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isalnum.html.md

# dask.dataframe.Series.str.isalnum

#### dataframe.Series.str.isalnum()

Check whether all characters in each string are alphanumeric.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isalnum.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isalnum()`](https://docs.python.org/3/library/stdtypes.html#str.isalnum) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

```pycon
>>> s1 = pd.Series(["one", "one1", "1", ""])
>>> s1.str.isalnum()
0     True
1     True
2     True
3    False
dtype: bool
```

Note that checks against characters mixed with any additional punctuation
or whitespace will evaluate to false for an alphanumeric check.

```pycon
>>> s2 = pd.Series(["A B", "1.5", "3,000"])
>>> s2.str.isalnum()
0    False
1    False
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isalpha.html.md

# dask.dataframe.Series.str.isalpha

#### dataframe.Series.str.isalpha()

Check whether all characters in each string are alphabetic.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isalpha.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isalpha()`](https://docs.python.org/3/library/stdtypes.html#str.isalpha) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

```pycon
>>> s1 = pd.Series(["one", "one1", "1", ""])
>>> s1.str.isalpha()
0     True
1    False
2    False
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isdecimal.html.md

# dask.dataframe.Series.str.isdecimal

#### dataframe.Series.str.isdecimal()

Check whether all characters in each string are decimal.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isdecimal.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isdecimal()`](https://docs.python.org/3/library/stdtypes.html#str.isdecimal) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

The `s3.str.isdecimal` method checks for characters used to form
numbers in base 10.

```pycon
>>> s3 = pd.Series(["23", "³", "⅕", ""])
>>> s3.str.isdecimal()
0     True
1    False
2    False
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isdigit.html.md

# dask.dataframe.Series.str.isdigit

#### dataframe.Series.str.isdigit()

Check whether all characters in each string are digits.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isdigit.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isdigit()`](https://docs.python.org/3/library/stdtypes.html#str.isdigit) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Notes

Similar to `str.isdecimal` but also includes special digits, like
superscripted and subscripted digits in unicode.

The exact behavior of this method, i.e. which unicode characters are
considered as digits, depends on the backend used for string operations,
and there can be small differences.
For example, Python considers the ³ superscript character as a digit, but
not the ⅕ fraction character, while PyArrow considers both as digits. For
simple (ascii) decimal numbers, the behaviour is consistent.

### Examples

```pycon
>>> s3 = pd.Series(["23", "³", "⅕", ""])
>>> s3.str.isdigit()
0     True
1     True
2     True
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.islower.html.md

# dask.dataframe.Series.str.islower

#### dataframe.Series.str.islower()

Check whether all characters in each string are lowercase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.islower.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.islower()`](https://docs.python.org/3/library/stdtypes.html#str.islower) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

```pycon
>>> s5 = pd.Series(["leopard", "Golden Eagle", "SNAKE", ""])
>>> s5.str.islower()
0     True
1    False
2    False
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isnumeric.html.md

# dask.dataframe.Series.str.isnumeric

#### dataframe.Series.str.isnumeric()

Check whether all characters in each string are numeric.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isnumeric.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isnumeric()`](https://docs.python.org/3/library/stdtypes.html#str.isnumeric) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

The `s.str.isnumeric` method is the same as `s3.str.isdigit` but
also includes other characters that can represent quantities such as
unicode fractions.

```pycon
>>> s1 = pd.Series(["one", "one1", "1", "", "³", "⅕"])
>>> s1.str.isnumeric()
0    False
1    False
2     True
3    False
4     True
5     True
dtype: bool
```

For a string to be considered numeric, all its characters must have a Unicode
numeric property matching `str.is_numeric()`. As a consequence,
the following cases are **not** recognized as numeric:

- **Decimal numbers** (e.g., “1.1”): due to period `"."`
- **Negative numbers** (e.g., “-5”):  due to minus sign `"-"`
- **Scientific notation** (e.g., “1e3”): due to characters like `"e"`

```pycon
>>> s2 = pd.Series(["1.1", "-5", "1e3"])
>>> s2.str.isnumeric()
0    False
1    False
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isspace.html.md

# dask.dataframe.Series.str.isspace

#### dataframe.Series.str.isspace()

Check whether all characters in each string are whitespace.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isspace.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isspace()`](https://docs.python.org/3/library/stdtypes.html#str.isspace) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

```pycon
>>> s4 = pd.Series([" ", "\t\r\n ", ""])
>>> s4.str.isspace()
0     True
1     True
2    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.istitle.html.md

# dask.dataframe.Series.str.istitle

#### dataframe.Series.str.istitle()

Check whether all characters in each string are titlecase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.istitle.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.istitle()`](https://docs.python.org/3/library/stdtypes.html#str.istitle) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.isupper`
: Check whether all characters are uppercase.

### Examples

The `s5.str.istitle` method checks for whether all words are in title
case (whether only the first letter of each word is capitalized). Words are
assumed to be as any sequence of non-numeric characters separated by
whitespace characters.

```pycon
>>> s5 = pd.Series(["leopard", "Golden Eagle", "SNAKE", ""])
>>> s5.str.istitle()
0    False
1     True
2    False
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.isupper.html.md

# dask.dataframe.Series.str.isupper

#### dataframe.Series.str.isupper()

Check whether all characters in each string are uppercase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.isupper.

Some inconsistencies with the Dask version may exist.

This is equivalent to running the Python string method
[`str.isupper()`](https://docs.python.org/3/library/stdtypes.html#str.isupper) for each element of the Series/Index. If a string
has zero characters, `False` is returned for that check.

* **Returns:**
  Series or Index of bool
  : Series or Index of boolean values with the same length as the original
    Series/Index.

#### SEE ALSO
`Series.str.isalpha`
: Check whether all characters are alphabetic.

`Series.str.isnumeric`
: Check whether all characters are numeric.

`Series.str.isalnum`
: Check whether all characters are alphanumeric.

`Series.str.isdigit`
: Check whether all characters are digits.

`Series.str.isdecimal`
: Check whether all characters are decimal.

`Series.str.isspace`
: Check whether all characters are whitespace.

`Series.str.islower`
: Check whether all characters are lowercase.

`Series.str.isascii`
: Check whether all characters are ascii.

`Series.str.istitle`
: Check whether all characters are titlecase.

### Examples

```pycon
>>> s5 = pd.Series(["leopard", "Golden Eagle", "SNAKE", ""])
>>> s5.str.isupper()
0    False
1    False
2     True
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.join.html.md

# dask.dataframe.Series.str.join

#### dataframe.Series.str.join(sep: [str](https://docs.python.org/3/library/stdtypes.html#str))

Join lists contained as elements in the Series/Index with passed delimiter.

This docstring was copied from pandas.core.strings.accessor.StringMethods.join.

Some inconsistencies with the Dask version may exist.

If the elements of a Series are lists themselves, join the content of these
lists using the delimiter passed to the function.
This function is an equivalent to [`str.join()`](https://docs.python.org/3/library/stdtypes.html#str.join).

* **Parameters:**
  **sep**
  : Delimiter to use between list entries.
* **Returns:**
  Series/Index: object
  : The list entries concatenated by intervening occurrences of the
    delimiter.
* **Raises:**
  AttributeError
  : If the supplied Series contains neither strings nor lists.

#### SEE ALSO
[`str.join`](https://docs.python.org/3/library/stdtypes.html#str.join)
: Standard library version of this method.

`Series.str.split`
: Split strings around given separator/delimiter.

### Notes

If any of the list items is not a string object, the result of the join
will be NaN.

### Examples

Example with a list that contains non-string elements.

```pycon
>>> s = pd.Series(
...     [
...         ["lion", "elephant", "zebra"],
...         [1.1, 2.2, 3.3],
...         ["cat", np.nan, "dog"],
...         ["cow", 4.5, "goat"],
...         ["duck", ["swan", "fish"], "guppy"],
...     ]
... )
>>> s
0        [lion, elephant, zebra]
1                [1.1, 2.2, 3.3]
2                [cat, nan, dog]
3               [cow, 4.5, goat]
4    [duck, [swan, fish], guppy]
dtype: object
```

Join all lists using a ‘-’. The lists containing object(s) of types other
than str will produce a NaN.

```pycon
>>> s.str.join("-")
0    lion-elephant-zebra
1                    NaN
2                    NaN
3                    NaN
4                    NaN
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.len.html.md

# dask.dataframe.Series.str.len

#### dataframe.Series.str.len()

Compute the length of each element in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.len.

Some inconsistencies with the Dask version may exist.

The element may be a sequence (such as a string, tuple or list) or a collection
(such as a dictionary).

* **Returns:**
  Series or Index of int
  : A Series or Index of integer values indicating the length of each
    element in the Series or Index.

#### SEE ALSO
`str.len`
: Python built-in function returning the length of an object.

`Series.size`
: Returns the length of the Series.

### Examples

Returns the length (number of characters) in a string. Returns the
number of entries for dictionaries, lists or tuples.

```pycon
>>> s = pd.Series(
...     ["dog", "", 5, {"foo": "bar"}, [2, 3, 5, 7], ("one", "two", "three")]
... )
>>> s
0                  dog
1
2                    5
3       {'foo': 'bar'}
4         [2, 3, 5, 7]
5    (one, two, three)
dtype: object
>>> s.str.len()
0    3.0
1    0.0
2    NaN
3    1.0
4    4.0
5    3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.ljust.html.md

# dask.dataframe.Series.str.ljust

#### dataframe.Series.str.ljust(width: [int](https://docs.python.org/3/library/functions.html#int), fillchar: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ')

Pad right side of strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.ljust.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.ljust()`](https://docs.python.org/3/library/stdtypes.html#str.ljust).

* **Parameters:**
  **width**
  : Minimum width of resulting string; additional characters will be filled
    with `fillchar`.

  **fillchar**
  : Additional character for filling, default is whitespace.
* **Returns:**
  Series/Index of objects.
  : A Series or Index where the strings are modified by [`str.ljust()`](https://docs.python.org/3/library/stdtypes.html#str.ljust).

#### SEE ALSO
`Series.str.rjust`
: Fills the left side of strings with an arbitrary character.

`Series.str.ljust`
: Fills the right side of strings with an arbitrary character.

`Series.str.center`
: Fills both sides of strings with an arbitrary character.

`Series.str.zfill`
: Pad strings in the Series/Index by prepending ‘0’ character.

### Examples

For Series.str.center:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.center(8, fillchar=".")
0   ..dog...
1   ..bird..
2   .mouse..
dtype: str
```

For Series.str.ljust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.ljust(8, fillchar=".")
0   dog.....
1   bird....
2   mouse...
dtype: str
```

For Series.str.rjust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.rjust(8, fillchar=".")
0   .....dog
1   ....bird
2   ...mouse
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.lower.html.md

# dask.dataframe.Series.str.lower

#### dataframe.Series.str.lower()

Convert strings in the Series/Index to lowercase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.lower.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.lower()`](https://docs.python.org/3/library/stdtypes.html#str.lower).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.lower()`](https://docs.python.org/3/library/stdtypes.html#str.lower).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.lstrip.html.md

# dask.dataframe.Series.str.lstrip

#### dataframe.Series.str.lstrip(to_strip=None)

Remove leading characters.

This docstring was copied from pandas.core.strings.accessor.StringMethods.lstrip.

Some inconsistencies with the Dask version may exist.

Strip whitespaces (including newlines) or a set of specified characters
from each string in the Series/Index from left side.
Replaces any non-strings in Series with NaNs.
Equivalent to [`str.lstrip()`](https://docs.python.org/3/library/stdtypes.html#str.lstrip).

* **Parameters:**
  **to_strip**
  : Specifying the set of characters to be removed.
    All combinations of this set of characters will be stripped.
    If None then whitespaces are removed.
* **Returns:**
  Series or Index of object
  : Series or Index with the strings being stripped from the left side.

#### SEE ALSO
`Series.str.strip`
: Remove leading and trailing characters in Series/Index.

`Series.str.lstrip`
: Remove leading characters in Series/Index.

`Series.str.rstrip`
: Remove trailing characters in Series/Index.

### Examples

```pycon
>>> s = pd.Series(["1. Ant.  ", "2. Bee!\n", "3. Cat?\t", np.nan, 10, True])
>>> s
0    1. Ant.
1    2. Bee!\n
2    3. Cat?\t
3          NaN
4           10
5         True
dtype: object
```

```pycon
>>> s.str.strip()
0    1. Ant.
1    2. Bee!
2    3. Cat?
3        NaN
4        NaN
5        NaN
dtype: object
```

```pycon
>>> s.str.lstrip("123.")
0    Ant.
1    Bee!\n
2    Cat?\t
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.rstrip(".!? \n\t")
0    1. Ant
1    2. Bee
2    3. Cat
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.strip("123.!? \n\t")
0    Ant
1    Bee
2    Cat
3    NaN
4    NaN
5    NaN
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.match.html.md

# dask.dataframe.Series.str.match

#### dataframe.Series.str.match(pat: str | re.Pattern, case: bool | lib.NoDefault = <no_default>, flags: int | lib.NoDefault = <no_default>, na=<no_default>)

Determine if each string starts with a match of a regular expression.

This docstring was copied from pandas.core.strings.accessor.StringMethods.match.

Some inconsistencies with the Dask version may exist.

Determines whether each string in the Series or Index starts with a
match to a specified regular expression. This function is especially
useful for validating prefixes, such as ensuring that codes, tags, or
identifiers begin with a specific pattern.

* **Parameters:**
  **pat**
  : Character sequence or regular expression.

  **case**
  : If True, case sensitive.

  **flags**
  : Regex module flags, e.g. re.IGNORECASE.

  **na**
  : Fill value for missing values. The default depends on dtype of the
    array. For the `"str"` dtype, `False` is used. For object
    dtype, `numpy.nan` is used. For the nullable `StringDtype`,
    `pandas.NA` is used.
* **Returns:**
  Series/Index/array of boolean values
  : A Series, Index, or array of boolean values indicating whether the start
    of each string matches the pattern. The result will be of the same type
    as the input.

#### SEE ALSO
[`fullmatch`](dask.dataframe.Series.str.fullmatch.md#dask.dataframe.Series.str.fullmatch)
: Stricter matching that requires the entire string to match.

[`contains`](dask.dataframe.Series.str.contains.md#dask.dataframe.Series.str.contains)
: Analogous, but less strict, relying on re.search instead of re.match.

[`extract`](dask.dataframe.Series.str.extract.md#dask.dataframe.Series.str.extract)
: Extract matched groups.

### Examples

```pycon
>>> ser = pd.Series(["horse", "eagle", "donkey"])
>>> ser.str.match("e")
0   False
1   True
2   False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.normalize.html.md

# dask.dataframe.Series.str.normalize

#### dataframe.Series.str.normalize(form)

Return the Unicode normal form for the strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.normalize.

Some inconsistencies with the Dask version may exist.

For more information on the forms, see the
[`unicodedata.normalize()`](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize).

* **Parameters:**
  **form**
  : Unicode form.
* **Returns:**
  Series/Index of objects
  : A Series or Index of strings in the same Unicode form specified by form.
    The returned object retains the same type as the input (Series or Index),
    and contains the normalized strings.

#### SEE ALSO
`Series.str.upper`
: Convert all characters in each string to uppercase.

`Series.str.lower`
: Convert all characters in each string to lowercase.

`Series.str.title`
: Convert each string to title case (capitalizing the first letter of each word).

`Series.str.strip`
: Remove leading and trailing whitespace from each string.

`Series.str.replace`
: Replace occurrences of a substring with another substring in each string.

### Examples

```pycon
>>> ser = pd.Series(["ñ"])
>>> ser.str.normalize("NFC") == ser.str.normalize("NFD")
0   False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.pad.html.md

# dask.dataframe.Series.str.pad

#### dataframe.Series.str.pad(width: [int](https://docs.python.org/3/library/functions.html#int), side: Literal['left', 'right', 'both'] = 'left', fillchar: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ')

Pad strings in the Series/Index up to width.

This docstring was copied from pandas.core.strings.accessor.StringMethods.pad.

Some inconsistencies with the Dask version may exist.

This function pads strings in a Series or Index to a specified width,
filling the extra space with a character of your choice. It provides
flexibility in positioning the padding, allowing it to be added to the
left, right, or both sides. This is useful for formatting strings to
align text or ensure consistent string lengths in data processing.

* **Parameters:**
  **width**
  : Minimum width of resulting string; additional characters will be filled
    with character defined in fillchar.

  **side**
  : Side from which to fill resulting string.

  **fillchar**
  : Additional character for filling, default is whitespace.
* **Returns:**
  Series or Index of object
  : Returns Series or Index with minimum number of char in object.

#### SEE ALSO
`Series.str.rjust`
: Fills the left side of strings with an arbitrary character. Equivalent to `Series.str.pad(side='left')`.

`Series.str.ljust`
: Fills the right side of strings with an arbitrary character. Equivalent to `Series.str.pad(side='right')`.

`Series.str.center`
: Fills both sides of strings with an arbitrary character. Equivalent to `Series.str.pad(side='both')`.

`Series.str.zfill`
: Pad strings in the Series/Index by prepending ‘0’ character. Equivalent to `Series.str.pad(side='left', fillchar='0')`.

### Examples

```pycon
>>> s = pd.Series(["caribou", "tiger"])
>>> s
0    caribou
1      tiger
dtype: str
```

```pycon
>>> s.str.pad(width=10)
0       caribou
1         tiger
dtype: str
```

```pycon
>>> s.str.pad(width=10, side="right", fillchar="-")
0    caribou---
1    tiger-----
dtype: str
```

```pycon
>>> s.str.pad(width=10, side="both", fillchar="-")
0    -caribou--
1    --tiger---
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.partition.html.md

# dask.dataframe.Series.str.partition

#### dataframe.Series.str.partition(sep: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ', expand: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Split the string at the first occurrence of sep.

This docstring was copied from pandas.core.strings.accessor.StringMethods.partition.

Some inconsistencies with the Dask version may exist.

This method splits the string at the first occurrence of sep,
and returns 3 elements containing the part before the separator,
the separator itself, and the part after the separator.
If the separator is not found, return 3 elements containing the string itself,
followed by two empty strings.

* **Parameters:**
  **sep**
  : String to split on.

  **expand**
  : If True, return DataFrame/MultiIndex expanding dimensionality.
    If False, return Series/Index.
* **Returns:**
  DataFrame/MultiIndex or Series/Index of objects
  : Returns appropriate type based on expand parameter with strings
    split based on the sep parameter.

#### SEE ALSO
[`rpartition`](dask.dataframe.Series.str.rpartition.md#dask.dataframe.Series.str.rpartition)
: Split the string at the last occurrence of sep.

`Series.str.split`
: Split strings around given separators.

[`str.partition`](https://docs.python.org/3/library/stdtypes.html#str.partition)
: Standard library version.

### Examples

```pycon
>>> s = pd.Series(["Linda van der Berg", "George Pitt-Rivers"])
>>> s
0    Linda van der Berg
1    George Pitt-Rivers
dtype: str
```

```pycon
>>> s.str.partition()
        0  1             2
0   Linda     van der Berg
1  George      Pitt-Rivers
```

To partition by the last space instead of the first one:

```pycon
>>> s.str.rpartition()
               0  1            2
0  Linda van der            Berg
1         George     Pitt-Rivers
```

To partition by something different than a space:

```pycon
>>> s.str.partition("-")
                    0  1       2
0  Linda van der Berg
1         George Pitt  -  Rivers
```

To return a Series containing tuples instead of a DataFrame:

```pycon
>>> s.str.partition("-", expand=False)
0    (Linda van der Berg, , )
1    (George Pitt, -, Rivers)
dtype: object
```

Also available on indices:

```pycon
>>> idx = pd.Index(["X 123", "Y 999"])
>>> idx
Index(['X 123', 'Y 999'], dtype='str')
```

Which will create a MultiIndex:

```pycon
>>> idx.str.partition()
MultiIndex([('X', ' ', '123'),
            ('Y', ' ', '999')],
           )
```

Or an index with tuples with `expand=False`:

```pycon
>>> idx.str.partition(expand=False)
Index([('X', ' ', '123'), ('Y', ' ', '999')], dtype='object')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.repeat.html.md

# dask.dataframe.Series.str.repeat

#### dataframe.Series.str.repeat(repeats)

Duplicate each string in the Series or Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.repeat.

Some inconsistencies with the Dask version may exist.

Duplicates each string in the Series or Index, either by applying the
same repeat count to all elements or by using different repeat values
for each element.

* **Parameters:**
  **repeats**
  : Same value for all (int) or different value per (sequence).
* **Returns:**
  Series or pandas.Index
  : Series or Index of repeated string objects specified by
    input parameter repeats.

#### SEE ALSO
`Series.str.lower`
: Convert all characters in each string to lowercase.

`Series.str.upper`
: Convert all characters in each string to uppercase.

`Series.str.title`
: Convert each string to title case (capitalizing the first letter of each word).

`Series.str.strip`
: Remove leading and trailing whitespace from each string.

`Series.str.replace`
: Replace occurrences of a substring with another substring in each string.

`Series.str.ljust`
: Left-justify each string in the Series/Index by padding with a specified character.

`Series.str.rjust`
: Right-justify each string in the Series/Index by padding with a specified character.

### Examples

```pycon
>>> s = pd.Series(["a", "b", "c"])
>>> s
0    a
1    b
2    c
dtype: str
```

Single int repeats string in Series

```pycon
>>> s.str.repeat(repeats=2)
0    aa
1    bb
2    cc
dtype: str
```

Sequence of int repeats corresponding string in Series

```pycon
>>> s.str.repeat(repeats=[1, 2, 3])
0      a
1     bb
2    ccc
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.replace.html.md

# dask.dataframe.Series.str.replace

#### dataframe.Series.str.replace(pat: [str](https://docs.python.org/3/library/stdtypes.html#str) | [re.Pattern](https://docs.python.org/3/library/re.html#re.Pattern) | [dict](https://docs.python.org/3/library/stdtypes.html#dict), repl: [str](https://docs.python.org/3/library/stdtypes.html#str) | Callable | [None](https://docs.python.org/3/library/constants.html#None) = None, n: [int](https://docs.python.org/3/library/functions.html#int) = -1, case: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, flags: [int](https://docs.python.org/3/library/functions.html#int) = 0, regex: [bool](https://docs.python.org/3/library/functions.html#bool) = False)

Replace each occurrence of pattern/regex in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.replace.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.replace()`](https://docs.python.org/3/library/stdtypes.html#str.replace) or [`re.sub()`](https://docs.python.org/3/library/re.html#re.sub), depending on
the regex value.

* **Parameters:**
  **pat**
  : String can be a character sequence or regular expression.
    Dictionary contains <key : value> pairs of strings to be replaced
    along with the updated value.

  **repl**
  : Replacement string or a callable. The callable is passed the regex
    match object and must return a replacement string to be used.
    Must have a value of None if pat is a dict
    See [`re.sub()`](https://docs.python.org/3/library/re.html#re.sub).

  **n**
  : Number of replacements to make from start.

  **case**
  : Determines if replace is case sensitive:
    - If True, case sensitive (the default if pat is a string)
    - Set to False for case insensitive
    - Cannot be set if pat is a compiled regex.

  **flags**
  : Regex module flags, e.g. re.IGNORECASE. Cannot be set if pat is a compiled
    regex.

  **regex**
  : Determines if the passed-in pattern is a regular expression:
    - If True, assumes the passed-in pattern is a regular expression.
    - If False, treats the pattern as a literal string
    - Cannot be set to False if pat is a compiled regex or repl is
      a callable.
* **Returns:**
  Series or Index of object
  : A copy of the object with all matching occurrences of pat replaced by
    repl.
* **Raises:**
  ValueError
  : * if regex is False and repl is a callable or pat is a compiled
      regex
    * if pat is a compiled regex and case or flags is set
    * if pat is a dictionary and repl is not None.

#### SEE ALSO
`Series.str.replace`
: Method to replace occurrences of a substring with another substring.

`Series.str.extract`
: Extract substrings using a regular expression.

`Series.str.findall`
: Find all occurrences of a pattern or regex in each string.

`Series.str.split`
: Split each string by a specified delimiter or pattern.

### Notes

When pat is a compiled regex, all flags should be included in the
compiled regex. Use of case, flags, or regex=False with a compiled
regex will raise an error.

### Examples

When pat is a dictionary, every key in pat is replaced
with its corresponding value:

```pycon
>>> pd.Series(["A", "B", np.nan]).str.replace(pat={"A": "a", "B": "b"})
0    a
1    b
2    NaN
dtype: str
```

When pat is a string and regex is True, the given pat
is compiled as a regex. When repl is a string, it replaces matching
regex patterns as with `re.sub()`. NaN value(s) in the Series are
left as is:

```pycon
>>> pd.Series(["foo", "fuz", np.nan]).str.replace("f.", "ba", regex=True)
0    bao
1    baz
2    NaN
dtype: str
```

When pat is a string and regex is False, every pat is replaced with
repl as with [`str.replace()`](https://docs.python.org/3/library/stdtypes.html#str.replace):

```pycon
>>> pd.Series(["f.o", "fuz", np.nan]).str.replace("f.", "ba", regex=False)
0    bao
1    fuz
2    NaN
dtype: str
```

When repl is a callable, it is called on every pat using
[`re.sub()`](https://docs.python.org/3/library/re.html#re.sub). The callable should expect one positional argument
(a regex object) and return a string.

To get the idea:

```pycon
>>> pd.Series(["foo", "fuz", np.nan]).str.replace("f", repr, regex=True)
0    <re.Match object; span=(0, 1), match='f'>oo
1    <re.Match object; span=(0, 1), match='f'>uz
2                                            NaN
dtype: str
```

Reverse every lowercase alphabetic word:

```pycon
>>> repl = lambda m: m.group(0)[::-1]
>>> ser = pd.Series(["foo 123", "bar baz", np.nan])
>>> ser.str.replace(r"[a-z]+", repl, regex=True)
0    oof 123
1    rab zab
2        NaN
dtype: str
```

Using regex groups (extract second group and swap case):

```pycon
>>> pat = r"(?P<one>\w+) (?P<two>\w+) (?P<three>\w+)"
>>> repl = lambda m: m.group("two").swapcase()
>>> ser = pd.Series(["One Two Three", "Foo Bar Baz"])
>>> ser.str.replace(pat, repl, regex=True)
0    tWO
1    bAR
dtype: str
```

Using a compiled regex with flags

```pycon
>>> import re
>>> regex_pat = re.compile(r"FUZ", flags=re.IGNORECASE)
>>> pd.Series(["foo", "fuz", np.nan]).str.replace(regex_pat, "bar", regex=True)
0    foo
1    bar
2    NaN
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rfind.html.md

# dask.dataframe.Series.str.rfind

#### dataframe.Series.str.rfind(sub, start: [int](https://docs.python.org/3/library/functions.html#int) = 0, end=None)

Return highest indexes in each strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.rfind.

Some inconsistencies with the Dask version may exist.

Each of returned indexes corresponds to the position where the
substring is fully contained between [start:end]. Return -1 on
failure. Equivalent to standard [`str.rfind()`](https://docs.python.org/3/library/stdtypes.html#str.rfind).

* **Parameters:**
  **sub**
  : Substring being searched.

  **start**
  : Left edge index.

  **end**
  : Right edge index.
* **Returns:**
  Series or Index of int.
  : A Series (if the input is a Series) or an Index (if the input is an
    Index) of the highest indexes corresponding to the positions where the
    substring is found in each string of the input.

#### SEE ALSO
[`find`](dask.dataframe.Series.str.find.md#dask.dataframe.Series.str.find)
: Return lowest indexes in each strings.

### Examples

For Series.str.find:

```pycon
>>> ser = pd.Series(["_cow_", "duck_", "do_v_e"])
>>> ser.str.find("_")
0   0
1   4
2   2
dtype: int64
```

For Series.str.rfind:

```pycon
>>> ser = pd.Series(["_cow_", "duck_", "do_v_e"])
>>> ser.str.rfind("_")
0   4
1   4
2   4
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rindex.html.md

# dask.dataframe.Series.str.rindex

#### dataframe.Series.str.rindex(sub, start: [int](https://docs.python.org/3/library/functions.html#int) = 0, end=None)

Return highest indexes in each string in Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.rindex.

Some inconsistencies with the Dask version may exist.

Each of the returned indexes corresponds to the position where the
substring is fully contained between [start:end]. This is the same
as `str.rfind` except instead of returning -1, it raises a
ValueError when the substring is not found. Equivalent to standard
`str.rindex`.

* **Parameters:**
  **sub**
  : Substring being searched.

  **start**
  : Left edge index.

  **end**
  : Right edge index.
* **Returns:**
  Series or Index of object
  : Returns a Series or an Index of the highest indexes
    in each string of the input.

#### SEE ALSO
[`index`](dask.dataframe.Series.str.index.md#dask.dataframe.Series.str.index)
: Return lowest indexes in each strings.

### Examples

For Series.str.index:

```pycon
>>> ser = pd.Series(["horse", "eagle", "donkey"])
>>> ser.str.index("e")
0   4
1   0
2   4
dtype: int64
```

For Series.str.rindex:

```pycon
>>> ser = pd.Series(["Deer", "eagle", "Sheep"])
>>> ser.str.rindex("e")
0   2
1   4
2   3
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rjust.html.md

# dask.dataframe.Series.str.rjust

#### dataframe.Series.str.rjust(width: [int](https://docs.python.org/3/library/functions.html#int), fillchar: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ')

Pad left side of strings in the Series/Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.rjust.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.rjust()`](https://docs.python.org/3/library/stdtypes.html#str.rjust).

* **Parameters:**
  **width**
  : Minimum width of resulting string; additional characters will be filled
    with `fillchar`.

  **fillchar**
  : Additional character for filling, default is whitespace.
* **Returns:**
  Series/Index of objects.
  : A Series or Index where the strings are modified by [`str.rjust()`](https://docs.python.org/3/library/stdtypes.html#str.rjust).

#### SEE ALSO
`Series.str.rjust`
: Fills the left side of strings with an arbitrary character.

`Series.str.ljust`
: Fills the right side of strings with an arbitrary character.

`Series.str.center`
: Fills both sides of strings with an arbitrary character.

`Series.str.zfill`
: Pad strings in the Series/Index by prepending ‘0’ character.

### Examples

For Series.str.center:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.center(8, fillchar=".")
0   ..dog...
1   ..bird..
2   .mouse..
dtype: str
```

For Series.str.ljust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.ljust(8, fillchar=".")
0   dog.....
1   bird....
2   mouse...
dtype: str
```

For Series.str.rjust:

```pycon
>>> ser = pd.Series(["dog", "bird", "mouse"])
>>> ser.str.rjust(8, fillchar=".")
0   .....dog
1   ....bird
2   ...mouse
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rpartition.html.md

# dask.dataframe.Series.str.rpartition

#### dataframe.Series.str.rpartition(sep: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' ', expand: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Split the string at the last occurrence of sep.

This docstring was copied from pandas.core.strings.accessor.StringMethods.rpartition.

Some inconsistencies with the Dask version may exist.

This method splits the string at the last occurrence of sep,
and returns 3 elements containing the part before the separator,
the separator itself, and the part after the separator.
If the separator is not found, return 3 elements containing two empty strings,
followed by the string itself.

* **Parameters:**
  **sep**
  : String to split on.

  **expand**
  : If True, return DataFrame/MultiIndex expanding dimensionality.
    If False, return Series/Index.
* **Returns:**
  DataFrame/MultiIndex or Series/Index of objects
  : Returns appropriate type based on expand parameter with strings
    split based on the sep parameter.

#### SEE ALSO
[`partition`](dask.dataframe.Series.str.partition.md#dask.dataframe.Series.str.partition)
: Split the string at the first occurrence of sep.

`Series.str.split`
: Split strings around given separators.

[`str.partition`](https://docs.python.org/3/library/stdtypes.html#str.partition)
: Standard library version.

### Examples

```pycon
>>> s = pd.Series(["Linda van der Berg", "George Pitt-Rivers"])
>>> s
0    Linda van der Berg
1    George Pitt-Rivers
dtype: str
```

```pycon
>>> s.str.partition()
        0  1             2
0   Linda     van der Berg
1  George      Pitt-Rivers
```

To partition by the last space instead of the first one:

```pycon
>>> s.str.rpartition()
               0  1            2
0  Linda van der            Berg
1         George     Pitt-Rivers
```

To partition by something different than a space:

```pycon
>>> s.str.partition("-")
                    0  1       2
0  Linda van der Berg
1         George Pitt  -  Rivers
```

To return a Series containing tuples instead of a DataFrame:

```pycon
>>> s.str.partition("-", expand=False)
0    (Linda van der Berg, , )
1    (George Pitt, -, Rivers)
dtype: object
```

Also available on indices:

```pycon
>>> idx = pd.Index(["X 123", "Y 999"])
>>> idx
Index(['X 123', 'Y 999'], dtype='str')
```

Which will create a MultiIndex:

```pycon
>>> idx.str.partition()
MultiIndex([('X', ' ', '123'),
            ('Y', ' ', '999')],
           )
```

Or an index with tuples with `expand=False`:

```pycon
>>> idx.str.partition(expand=False)
Index([('X', ' ', '123'), ('Y', ' ', '999')], dtype='object')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rsplit.html.md

# dask.dataframe.Series.str.rsplit

#### dataframe.Series.str.rsplit(pat=None, n=-1, expand=False)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.rstrip.html.md

# dask.dataframe.Series.str.rstrip

#### dataframe.Series.str.rstrip(to_strip=None)

Remove trailing characters.

This docstring was copied from pandas.core.strings.accessor.StringMethods.rstrip.

Some inconsistencies with the Dask version may exist.

Strip whitespaces (including newlines) or a set of specified characters
from each string in the Series/Index from right side.
Replaces any non-strings in Series with NaNs.
Equivalent to [`str.rstrip()`](https://docs.python.org/3/library/stdtypes.html#str.rstrip).

* **Parameters:**
  **to_strip**
  : Specifying the set of characters to be removed.
    All combinations of this set of characters will be stripped.
    If None then whitespaces are removed.
* **Returns:**
  Series or Index of object
  : Series or Index with the strings being stripped from the right side.

#### SEE ALSO
`Series.str.strip`
: Remove leading and trailing characters in Series/Index.

`Series.str.lstrip`
: Remove leading characters in Series/Index.

`Series.str.rstrip`
: Remove trailing characters in Series/Index.

### Examples

```pycon
>>> s = pd.Series(["1. Ant.  ", "2. Bee!\n", "3. Cat?\t", np.nan, 10, True])
>>> s
0    1. Ant.
1    2. Bee!\n
2    3. Cat?\t
3          NaN
4           10
5         True
dtype: object
```

```pycon
>>> s.str.strip()
0    1. Ant.
1    2. Bee!
2    3. Cat?
3        NaN
4        NaN
5        NaN
dtype: object
```

```pycon
>>> s.str.lstrip("123.")
0    Ant.
1    Bee!\n
2    Cat?\t
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.rstrip(".!? \n\t")
0    1. Ant
1    2. Bee
2    3. Cat
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.strip("123.!? \n\t")
0    Ant
1    Bee
2    Cat
3    NaN
4    NaN
5    NaN
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.slice.html.md

# dask.dataframe.Series.str.slice

#### dataframe.Series.str.slice(start=None, stop=None, step=None)

Slice substrings from each element in the Series or Index.

This docstring was copied from pandas.core.strings.accessor.StringMethods.slice.

Some inconsistencies with the Dask version may exist.

Slicing substrings from strings in a Series or Index helps extract
specific portions of data, making it easier to analyze or manipulate
text. This is useful for tasks like parsing structured text fields or
isolating parts of strings with a consistent format.

* **Parameters:**
  **start**
  : Start position for slice operation.

  **stop**
  : Stop position for slice operation.

  **step**
  : Step size for slice operation.
* **Returns:**
  Series or Index of object
  : Series or Index from sliced substring from original string object.

#### SEE ALSO
`Series.str.slice_replace`
: Replace a slice with a string.

`Series.str.get`
: Return element at position. Equivalent to Series.str.slice(start=i, stop=i+1) with i being the position.

### Examples

```pycon
>>> s = pd.Series(["koala", "dog", "chameleon"])
>>> s
0        koala
1          dog
2    chameleon
dtype: str
```

```pycon
>>> s.str.slice(start=1)
0        oala
1          og
2    hameleon
dtype: str
```

```pycon
>>> s.str.slice(start=-1)
0           a
1           g
2           n
dtype: str
```

```pycon
>>> s.str.slice(stop=2)
0    ko
1    do
2    ch
dtype: str
```

```pycon
>>> s.str.slice(step=2)
0      kaa
1       dg
2    caeen
dtype: str
```

```pycon
>>> s.str.slice(start=0, stop=5, step=3)
0    kl
1     d
2    cm
dtype: str
```

Equivalent behaviour to:

```pycon
>>> s.str[0:5:3]
0    kl
1     d
2    cm
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.split.html.md

# dask.dataframe.Series.str.split

#### dataframe.Series.str.split(pat=None, n=-1, expand=False)

Known inconsistencies: `expand=True` with unknown `n` will raise a `NotImplementedError`.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.startswith.html.md

# dask.dataframe.Series.str.startswith

#### dataframe.Series.str.startswith(pat: str | tuple[str, ...], na: Scalar | lib.NoDefault = <no_default>) → [Series](dask.dataframe.Series.md#dask.dataframe.Series) | [Index](dask.dataframe.Index.md#dask.dataframe.Index)

Test if the start of each string element matches a pattern.

This docstring was copied from pandas.core.strings.accessor.StringMethods.startswith.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.startswith()`](https://docs.python.org/3/library/stdtypes.html#str.startswith).

* **Parameters:**
  **pat**
  : Character sequence or tuple of strings. Regular expressions are not
    accepted.

  **na**
  : Object shown if element tested is not a string. The default depends
    on dtype of the array. For the `"str"` dtype, `False` is used.
    For object dtype, `numpy.nan` is used. For the nullable
    `StringDtype`, `pandas.NA` is used.
* **Returns:**
  Series or Index of bool
  : A Series of booleans indicating whether the given pattern matches
    the start of each string element.

#### SEE ALSO
[`str.startswith`](https://docs.python.org/3/library/stdtypes.html#str.startswith)
: Python standard library string method.

`Series.str.endswith`
: Same as startswith, but tests the end of string.

`Series.str.contains`
: Tests if string element contains a pattern.

### Examples

```pycon
>>> s = pd.Series(["bat", "Bear", "cat", np.nan])
>>> s
0     bat
1    Bear
2     cat
3     NaN
dtype: str
```

```pycon
>>> s.str.startswith("b")
0     True
1    False
2    False
3    False
dtype: bool
```

```pycon
>>> s.str.startswith(("b", "B"))
0     True
1     True
2    False
3    False
dtype: bool
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.strip.html.md

# dask.dataframe.Series.str.strip

#### dataframe.Series.str.strip(to_strip=None)

Remove leading and trailing characters.

This docstring was copied from pandas.core.strings.accessor.StringMethods.strip.

Some inconsistencies with the Dask version may exist.

Strip whitespaces (including newlines) or a set of specified characters
from each string in the Series/Index from left and right sides.
Replaces any non-strings in Series with NaNs.
Equivalent to [`str.strip()`](https://docs.python.org/3/library/stdtypes.html#str.strip).

* **Parameters:**
  **to_strip**
  : Specifying the set of characters to be removed.
    All combinations of this set of characters will be stripped.
    If None then whitespaces are removed.
* **Returns:**
  Series or Index of object
  : Series or Index with the strings being stripped from the left and
    right sides.

#### SEE ALSO
`Series.str.strip`
: Remove leading and trailing characters in Series/Index.

`Series.str.lstrip`
: Remove leading characters in Series/Index.

`Series.str.rstrip`
: Remove trailing characters in Series/Index.

### Examples

```pycon
>>> s = pd.Series(["1. Ant.  ", "2. Bee!\n", "3. Cat?\t", np.nan, 10, True])
>>> s
0    1. Ant.
1    2. Bee!\n
2    3. Cat?\t
3          NaN
4           10
5         True
dtype: object
```

```pycon
>>> s.str.strip()
0    1. Ant.
1    2. Bee!
2    3. Cat?
3        NaN
4        NaN
5        NaN
dtype: object
```

```pycon
>>> s.str.lstrip("123.")
0    Ant.
1    Bee!\n
2    Cat?\t
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.rstrip(".!? \n\t")
0    1. Ant
1    2. Bee
2    3. Cat
3       NaN
4       NaN
5       NaN
dtype: object
```

```pycon
>>> s.str.strip("123.!? \n\t")
0    Ant
1    Bee
2    Cat
3    NaN
4    NaN
5    NaN
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.swapcase.html.md

# dask.dataframe.Series.str.swapcase

#### dataframe.Series.str.swapcase()

Convert strings in the Series/Index to be swapcased.

This docstring was copied from pandas.core.strings.accessor.StringMethods.swapcase.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.swapcase()`](https://docs.python.org/3/library/stdtypes.html#str.swapcase).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.swapcase()`](https://docs.python.org/3/library/stdtypes.html#str.swapcase).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.title.html.md

# dask.dataframe.Series.str.title

#### dataframe.Series.str.title()

Convert strings in the Series/Index to titlecase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.title.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.title()`](https://docs.python.org/3/library/stdtypes.html#str.title).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.title()`](https://docs.python.org/3/library/stdtypes.html#str.title).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.translate.html.md

# dask.dataframe.Series.str.translate

#### dataframe.Series.str.translate(table)

Map all characters in the string through the given mapping table.

This docstring was copied from pandas.core.strings.accessor.StringMethods.translate.

Some inconsistencies with the Dask version may exist.

This method is equivalent to the standard [`str.translate()`](https://docs.python.org/3/library/stdtypes.html#str.translate)
method for strings. It maps each character in the string to a new
character according to the translation table provided. Unmapped
characters are left unchanged, while characters mapped to None
are removed.

* **Parameters:**
  **table**
  : Table is a mapping of Unicode ordinals to Unicode ordinals, strings, or
    None. Unmapped characters are left untouched.
    Characters mapped to None are deleted. [`str.maketrans()`](https://docs.python.org/3/library/stdtypes.html#str.maketrans) is a
    helper function for making translation tables.
* **Returns:**
  Series or Index
  : A new Series or Index with translated strings.

#### SEE ALSO
`Series.str.replace`
: Replace occurrences of pattern/regex in the Series with some other string.

`Index.str.replace`
: Replace occurrences of pattern/regex in the Index with some other string.

### Examples

```pycon
>>> ser = pd.Series(["El niño", "Françoise"])
>>> mytable = str.maketrans({"ñ": "n", "ç": "c"})
>>> ser.str.translate(mytable)
0   El nino
1   Francoise
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.upper.html.md

# dask.dataframe.Series.str.upper

#### dataframe.Series.str.upper()

Convert strings in the Series/Index to uppercase.

This docstring was copied from pandas.core.strings.accessor.StringMethods.upper.

Some inconsistencies with the Dask version may exist.

Equivalent to [`str.upper()`](https://docs.python.org/3/library/stdtypes.html#str.upper).

* **Returns:**
  Series or Index of objects
  : A Series or Index where the strings are modified by [`str.upper()`](https://docs.python.org/3/library/stdtypes.html#str.upper).

#### SEE ALSO
`Series.str.lower`
: Converts all characters to lowercase.

`Series.str.upper`
: Converts all characters to uppercase.

`Series.str.title`
: Converts first character of each word to uppercase and remaining to lowercase.

`Series.str.capitalize`
: Converts first character to uppercase and remaining to lowercase.

`Series.str.swapcase`
: Converts uppercase to lowercase and lowercase to uppercase.

`Series.str.casefold`
: Removes all case distinctions in the string.

### Examples

```pycon
>>> s = pd.Series(["lower", "CAPITALS", "this is a sentence", "SwApCaSe"])
>>> s
0                 lower
1              CAPITALS
2    this is a sentence
3              SwApCaSe
dtype: str
```

```pycon
>>> s.str.lower()
0                 lower
1              capitals
2    this is a sentence
3              swapcase
dtype: str
```

```pycon
>>> s.str.upper()
0                 LOWER
1              CAPITALS
2    THIS IS A SENTENCE
3              SWAPCASE
dtype: str
```

```pycon
>>> s.str.title()
0                 Lower
1              Capitals
2    This Is A Sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.capitalize()
0                 Lower
1              Capitals
2    This is a sentence
3              Swapcase
dtype: str
```

```pycon
>>> s.str.swapcase()
0                 LOWER
1              capitals
2    THIS IS A SENTENCE
3              sWaPcAsE
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.wrap.html.md

# dask.dataframe.Series.str.wrap

#### dataframe.Series.str.wrap(width: [int](https://docs.python.org/3/library/functions.html#int), expand_tabs: [bool](https://docs.python.org/3/library/functions.html#bool) = True, tabsize: [int](https://docs.python.org/3/library/functions.html#int) = 8, replace_whitespace: [bool](https://docs.python.org/3/library/functions.html#bool) = True, drop_whitespace: [bool](https://docs.python.org/3/library/functions.html#bool) = True, initial_indent: [str](https://docs.python.org/3/library/stdtypes.html#str) = '', subsequent_indent: [str](https://docs.python.org/3/library/stdtypes.html#str) = '', fix_sentence_endings: [bool](https://docs.python.org/3/library/functions.html#bool) = False, break_long_words: [bool](https://docs.python.org/3/library/functions.html#bool) = True, break_on_hyphens: [bool](https://docs.python.org/3/library/functions.html#bool) = True, max_lines: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, placeholder: [str](https://docs.python.org/3/library/stdtypes.html#str) = ' [...]')

Wrap strings in Series/Index at specified line width.

This docstring was copied from pandas.core.strings.accessor.StringMethods.wrap.

Some inconsistencies with the Dask version may exist.

This method has the same keyword parameters and defaults as
: [`textwrap.TextWrapper`](https://docs.python.org/3/library/textwrap.html#textwrap.TextWrapper).

* **Parameters:**
  **width**
  : Maximum line width.

  **expand_tabs**
  : If True, tab characters will be expanded to spaces (default: True).

  **tabsize**
  : If expand_tabs is true, then all tab characters in text will be
    expanded to zero or more spaces, depending on the current column
    and the given tab size (default: 8).

  **replace_whitespace**
  : If True, each whitespace character (as defined by string.whitespace)
    remaining after tab expansion will be replaced by a single space
    (default: True).

  **drop_whitespace**
  : If True, whitespace that, after wrapping, happens to end up at the
    beginning or end of a line is dropped (default: True).

  **initial_indent**
  : String that will be prepended to the first line of wrapped output.
    Counts towards the length of the first line. The empty string is
    not indented (default: ‘’).

  **subsequent_indent**
  : String that will be prepended to all lines of wrapped output except
    the first. Counts towards the length of each line except the first
    (default: ‘’).

  **fix_sentence_endings**
  : If true, TextWrapper attempts to detect sentence endings and ensure
    that sentences are always separated by exactly two spaces. This is
    generally desired for text in a monospaced font. However, the sentence
    detection algorithm is imperfect: it assumes that a sentence ending
    consists of a lowercase letter followed by one of ‘.’, ‘!’, or ‘?’,
    possibly followed by one of ‘”’ or “’”, followed by a space. One
    problem with this algorithm is that it is unable to detect the
    difference between “Dr.” in […] Dr. Frankenstein’s monster […]
    and “Spot.” in […] See Spot. See Spot run […]
    Since the sentence detection algorithm relies on string.lowercase
    for the definition of “lowercase letter”, and a convention of using
    two spaces after a period to separate sentences on the same line,
    it is specific to English-language texts (default: False).

  **break_long_words**
  : If True, then words longer than width will be broken in order to ensure
    that no lines are longer than width. If it is false, long words will
    not be broken, and some lines may be longer than width (default: True).

  **break_on_hyphens**
  : If True, wrapping will occur preferably on whitespace and right after
    hyphens in compound words, as it is customary in English. If false,
    only whitespaces will be considered as potentially good places for line
    breaks, but you need to set break_long_words to false if you want truly
    insecable words (default: True).

  **max_lines**
  : If not None, then the output will contain at most max_lines lines, with
    placeholder appearing at the end of the output (default: None).

  **placeholder**
  : String that will appear at the end of the output text if it has been
    truncated (default: ‘ […]’).
* **Returns:**
  Series or Index
  : A Series or Index where the strings are wrapped at the specified line width.

#### SEE ALSO
`Series.str.strip`
: Remove leading and trailing characters in Series/Index.

`Series.str.lstrip`
: Remove leading characters in Series/Index.

`Series.str.rstrip`
: Remove trailing characters in Series/Index.

### Notes

Internally, this method uses a [`textwrap.TextWrapper`](https://docs.python.org/3/library/textwrap.html#textwrap.TextWrapper) instance with
default settings. To achieve behavior matching R’s stringr library str_wrap
function, use the arguments:

- expand_tabs = False
- replace_whitespace = True
- drop_whitespace = True
- break_long_words = False
- break_on_hyphens = False

### Examples

```pycon
>>> s = pd.Series(["line to be wrapped", "another line to be wrapped"])
>>> s.str.wrap(12)
0             line to be\nwrapped
1    another line\nto be\nwrapped
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.str.zfill.html.md

# dask.dataframe.Series.str.zfill

#### dataframe.Series.str.zfill(width: [int](https://docs.python.org/3/library/functions.html#int))

Pad strings in the Series/Index by prepending ‘0’ characters.

This docstring was copied from pandas.core.strings.accessor.StringMethods.zfill.

Some inconsistencies with the Dask version may exist.

Strings in the Series/Index are padded with ‘0’ characters on the
left of the string to reach a total string length  width. Strings
in the Series/Index with length greater or equal to width are
unchanged.

* **Parameters:**
  **width**
  : Minimum length of resulting string; strings with length less
    than width be prepended with ‘0’ characters.
* **Returns:**
  Series/Index of objects.
  : A Series or Index where the strings are prepended with ‘0’ characters.

#### SEE ALSO
`Series.str.rjust`
: Fills the left side of strings with an arbitrary character.

`Series.str.ljust`
: Fills the right side of strings with an arbitrary character.

`Series.str.pad`
: Fills the specified sides of strings with an arbitrary character.

`Series.str.center`
: Fills both sides of strings with an arbitrary character.

### Notes

Differs from [`str.zfill()`](https://docs.python.org/3/library/stdtypes.html#str.zfill) which has special handling
for ‘+’/’-’ in the string.

### Examples

```pycon
>>> s = pd.Series(["-1", "1", "1000", 10, np.nan])
>>> s
0      -1
1       1
2    1000
3      10
4     NaN
dtype: object
```

Note that `10` and `NaN` are not strings, therefore they are
converted to `NaN`. The minus sign in `'-1'` is treated as a
special character and the zero is added to the right of it
([`str.zfill()`](https://docs.python.org/3/library/stdtypes.html#str.zfill) would have moved it to the left). `1000`
remains unchanged as it is longer than width.

```pycon
>>> s.str.zfill(3)
0     -01
1     001
2    1000
3     NaN
4     NaN
dtype: object
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.sub.html.md

# dask.dataframe.Series.sub

#### Series.sub(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.sum.html.md

# dask.dataframe.Series.sum

#### Series.sum(axis=0, skipna=True, numeric_only=False, min_count=0, split_every=False, \*\*kwargs)

Return the sum of the values over the requested axis.

This docstring was copied from pandas.DataFrame.sum.

Some inconsistencies with the Dask version may exist.

This is equivalent to the method `numpy.sum`.

* **Parameters:**
  **axis**
  : Axis for the function to be applied on.
    For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.sum with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).
    <br/>
    #### Versionadded
    Added in version 2.0.0.

  **skipna**
  : Exclude NA/null values when computing the result.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **min_count**
  : The required number of valid values to perform the operation. If fewer than
    `min_count` non-NA values are present the result will be NA.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  Series or scalar
  : Sum over requested axis.

#### SEE ALSO
[`Series.sum`](#dask.dataframe.Series.sum)
: Return the sum over Series values.

[`DataFrame.mean`](dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)
: Return the mean of the values over the requested axis.

[`DataFrame.median`](dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)
: Return the median of the values over the requested axis.

[`DataFrame.mode`](dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)
: Get the mode(s) of each element along the requested axis.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Return the standard deviation of the values over the requested axis.

### Examples

```pycon
>>> idx = pd.MultiIndex.from_arrays(
...     [["warm", "warm", "cold", "cold"], ["dog", "falcon", "fish", "spider"]],
...     names=["blooded", "animal"],
... )
>>> s = pd.Series([4, 2, 0, 8], name="legs", index=idx)
>>> s
blooded  animal
warm     dog       4
         falcon    2
cold     fish      0
         spider    8
Name: legs, dtype: int64
```

```pycon
>>> s.sum()
14
```

By default, the sum of an empty or all-NA Series is `0`.

```pycon
>>> pd.Series([], dtype="float64").sum()  # min_count=0 is the default
0.0
```

This can be controlled with the `min_count` parameter. For example, if
you’d like the sum of an empty series to be NaN, pass `min_count=1`.

```pycon
>>> pd.Series([], dtype="float64").sum(min_count=1)
nan
```

Thanks to the `skipna` parameter, `min_count` handles all-NA and
empty series identically.

```pycon
>>> pd.Series([np.nan]).sum()
0.0
```

```pycon
>>> pd.Series([np.nan]).sum(min_count=1)
nan
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_backend.html.md

# dask.dataframe.Series.to_backend

#### Series.to_backend(backend: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs)

Move to a new DataFrame backend

* **Parameters:**
  **backend**
  : The name of the new backend to move to. The default
    is the current “dataframe.backend” configuration.
* **Returns:**
  DataFrame, Series or Index

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_bag.html.md

# dask.dataframe.Series.to_bag

#### Series.to_bag(index=False, format='tuple')

Create a Dask Bag from a Series

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_csv.html.md

# dask.dataframe.Series.to_csv

#### Series.to_csv(filename, \*\*kwargs)

See dd.to_csv docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_dask_array.html.md

# dask.dataframe.Series.to_dask_array

#### Series.to_dask_array(lengths=None, meta=None, optimize: [bool](https://docs.python.org/3/library/functions.html#bool) = True, \*\*optimize_kwargs) → [Array](dask.array.Array.md#dask.array.Array)

Convert a dask DataFrame to a dask array.

* **Parameters:**
  **lengths**
  : How to determine the chunks sizes for the output array.
    By default, the output array will have unknown chunk lengths
    along the first axis, which can cause some later operations
    to fail.
    * True : immediately compute the length of each partition
    * Sequence : a sequence of integers to use for the chunk sizes
      on the first axis. These values are *not* validated for
      correctness, beyond ensuring that the number of items
      matches the number of partitions.

  **meta**
  : An optional meta parameter can be passed for dask to override the
    default metadata on the underlying dask array.

  **optimize**
  : Whether to optimize the expression before converting to an Array.
* **Returns:**
  A Dask Array

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_delayed.html.md

# dask.dataframe.Series.to_delayed

#### Series.to_delayed(optimize_graph=True)

Convert into a list of `dask.delayed` objects, one per partition.

* **Parameters:**
  **optimize_graph**
  : If True [default], the graph is optimized before converting into
    `dask.delayed` objects.

#### SEE ALSO
`dask_expr.from_delayed`

### Examples

```pycon
>>> partitions = df.to_delayed()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_frame.html.md

# dask.dataframe.Series.to_frame

#### Series.to_frame(name=<no_default>)

Convert Series to DataFrame.

This docstring was copied from pandas.Series.to_frame.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **name**
  : The passed name should substitute for the series name (if it has
    one).
* **Returns:**
  DataFrame
  : DataFrame representation of Series.

#### SEE ALSO
`Series.to_dict`
: Convert Series to dict object.

### Examples

```pycon
>>> s = pd.Series(["a", "b", "c"], name="vals")
>>> s.to_frame()
  vals
0    a
1    b
2    c
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_hdf.html.md

# dask.dataframe.Series.to_hdf

#### Series.to_hdf(path_or_buf, key, mode='a', append=False, \*\*kwargs)

See dd.to_hdf docstring for more information

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_string.html.md

# dask.dataframe.Series.to_string

#### Series.to_string(max_rows=5)

Render a string representation of the Series.

This docstring was copied from pandas.Series.to_string.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **buf**
  : Buffer to write to.

  **na_rep**
  : String representation of NaN to use, default ‘NaN’.

  **float_format**
  : Formatter function to apply to columns’ elements if they are
    floats, default None.

  **header**
  : Add the Series header (index name).

  **index**
  : Add index (row) labels, default True.

  **length**
  : Add the Series length.

  **dtype**
  : Add the Series dtype.

  **name**
  : Add the Series name if not None.

  **max_rows**
  : Maximum number of rows to show before truncating. If None, show
    all.

  **min_rows**
  : The number of rows to display in a truncated repr (when number
    of rows is above max_rows).
* **Returns:**
  str or None
  : String representation of Series if `buf=None`, otherwise None.

#### SEE ALSO
`Series.to_dict`
: Convert Series to dict object.

[`Series.to_frame`](dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)
: Convert Series to DataFrame object.

`Series.to_markdown`
: Print Series in Markdown-friendly format.

[`Series.to_timestamp`](dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)
: Cast to DatetimeIndex of Timestamps.

### Examples

```pycon
>>> ser = pd.Series([1, 2, 3]).to_string()
>>> ser
'0    1\n1    2\n2    3'
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.to_timestamp.html.md

# dask.dataframe.Series.to_timestamp

#### Series.to_timestamp(freq=None, how='start')

Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.

This docstring was copied from pandas.DataFrame.to_timestamp.

Some inconsistencies with the Dask version may exist.

This can be changed to the *end* of the period, by specifying how=”e”.

* **Parameters:**
  **freq**
  : Desired frequency.

  **how**
  : Convention for converting period to timestamp; start of period
    vs. end.

  **axis**
  : The axis to convert (the index by default).

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.
* **Returns:**
  DataFrame with DatetimeIndex
  : DataFrame with the PeriodIndex cast to DatetimeIndex.

#### SEE ALSO
`DataFrame.to_period`
: Inverse method to cast DatetimeIndex to PeriodIndex.

[`Series.to_timestamp`](#dask.dataframe.Series.to_timestamp)
: Equivalent method for Series.

### Examples

```pycon
>>> idx = pd.PeriodIndex(["2023", "2024"], freq="Y")
>>> d = {"col1": [1, 2], "col2": [3, 4]}
>>> df1 = pd.DataFrame(data=d, index=idx)
>>> df1
      col1   col2
2023     1      3
2024     2      4
```

The resulting timestamps will be at the beginning of the year in this case

```pycon
>>> df1 = df1.to_timestamp()
>>> df1
            col1   col2
2023-01-01     1      3
2024-01-01     2      4
>>> df1.index
DatetimeIndex(['2023-01-01', '2024-01-01'], dtype='datetime64[us]', freq=None)
```

Using freq which is the offset that the Timestamps will have

```pycon
>>> df2 = pd.DataFrame(data=d, index=idx)
>>> df2 = df2.to_timestamp(freq="M")
>>> df2
            col1   col2
2023-01-31     1      3
2024-01-31     2      4
>>> df2.index
DatetimeIndex(['2023-01-31', '2024-01-31'], dtype='datetime64[us]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.truediv.html.md

# dask.dataframe.Series.truediv

#### Series.truediv(other, level=None, fill_value=None, axis=0)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.unique.html.md

# dask.dataframe.Series.unique

#### Series.unique(split_every=None, split_out=True, shuffle_method=None)

Return Series of unique values in the object. Includes NA values.

* **Returns:**
  **uniques**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.value_counts.html.md

# dask.dataframe.Series.value_counts

#### Series.value_counts(sort=None, ascending=False, dropna=True, normalize=False, split_every=None, split_out=<no_default>)

Return a Series containing counts of unique values.

This docstring was copied from pandas.Series.value_counts.

Some inconsistencies with the Dask version may exist.

The resulting object will be in descending order so that the
first element is the most frequently-occurring element.
Excludes NA values by default.

* **Parameters:**
  **normalize**
  : If True then the object returned will contain the relative
    frequencies of the unique values.

  **sort**
  : Stable sort by frequencies when True. Preserve the order of the data
    when False.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0: Prior to 3.0.0, the sort was unstable.

  **ascending**
  : Sort in ascending order.

  **bins**
  : Rather than count values, group them into half-open bins,
    a convenience for `pd.cut`, only works with numeric data.

  **dropna**
  : Don’t include counts of NaN.
* **Returns:**
  Series
  : Series containing counts of unique values.

#### SEE ALSO
[`Series.count`](dask.dataframe.Series.count.md#dask.dataframe.Series.count)
: Number of non-NA elements in a Series.

[`DataFrame.count`](dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)
: Number of non-NA elements in a DataFrame.

`DataFrame.value_counts`
: Equivalent method on DataFrames.

### Examples

```pycon
>>> index = pd.Index([3, 1, 2, 3, 4, np.nan])
>>> index.value_counts()
3.0    2
1.0    1
2.0    1
4.0    1
Name: count, dtype: int64
```

With normalize set to True, returns the relative frequency by
dividing all values by the sum of values.

```pycon
>>> s = pd.Series([3, 1, 2, 3, 4, np.nan])
>>> s.value_counts(normalize=True)
3.0    0.4
1.0    0.2
2.0    0.2
4.0    0.2
Name: proportion, dtype: float64
```

**bins**

Bins can be useful for going from a continuous variable to a
categorical variable; instead of counting unique
apparitions of values, divide the index in the specified
number of half-open bins.

```pycon
>>> s.value_counts(bins=3)
(0.996, 2.0]    2
(2.0, 3.0]      2
(3.0, 4.0]      1
Name: count, dtype: int64
```

**dropna**

With dropna set to False we can also see NaN index values.

```pycon
>>> s.value_counts(dropna=False)
3.0    2
1.0    1
2.0    1
4.0    1
NaN    1
Name: count, dtype: int64
```

**Categorical Dtypes**

Rows with categorical type will be counted as one group
if they have same categories and order.
In the example below, even though `a`, `c`, and `d`
all have the same data types of `category`,
only `c` and `d` will be counted as one group
since `a` doesn’t have the same categories.

```pycon
>>> df = pd.DataFrame({"a": [1], "b": ["2"], "c": [3], "d": [3]})
>>> df = df.astype({"a": "category", "c": "category", "d": "category"})
>>> df
   a  b  c  d
0  1  2  3  3
```

```pycon
>>> df.dtypes
a    category
b      str
c    category
d    category
dtype: object
```

```pycon
>>> df.dtypes.value_counts()
category    2
category    1
str         1
Name: count, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.values.html.md

# dask.dataframe.Series.values

#### *property* Series.values

Return a dask.array of the values of this dataframe

Warning: This creates a dask.array without precise shape information.
Operations that depend on shape information, like slicing or reshaping,
will not work.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.var.html.md

# dask.dataframe.Series.var

#### Series.var(axis=0, skipna=True, ddof=1, numeric_only=False, split_every=False, \*\*kwargs)

Return unbiased variance over requested axis.

This docstring was copied from pandas.DataFrame.var.

Some inconsistencies with the Dask version may exist.

Normalized by N-1 by default. This can be changed using the ddof argument.

* **Parameters:**
  **axis**
  : For Series this parameter is unused and defaults to 0.
    <br/>
    #### WARNING
    The behavior of DataFrame.var with `axis=None` is deprecated,
    in a future version this will reduce over both axes and return a scalar
    To retain the old behavior, pass axis=0 (or do not pass axis).

  **skipna**
  : Exclude NA/null values. If an entire row/column is NA, the result
    will be NA.

  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is N - ddof,
    where N represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns. Not implemented for Series.

  **\*\*kwargs**
  : Additional keywords passed.
* **Returns:**
  Series or scalaer
  : Unbiased variance over requested axis.

#### SEE ALSO
[`numpy.var`](https://numpy.org/doc/stable/reference/generated/numpy.var.html#numpy.var)
: Equivalent function in NumPy.

[`Series.var`](#dask.dataframe.Series.var)
: Return unbiased variance over Series values.

[`Series.std`](dask.dataframe.Series.std.md#dask.dataframe.Series.std)
: Return standard deviation over Series values.

[`DataFrame.std`](dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)
: Return standard deviation of the values over the requested axis.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {
...         "person_id": [0, 1, 2, 3],
...         "age": [21, 25, 62, 43],
...         "height": [1.61, 1.87, 1.49, 2.01],
...     }
... ).set_index("person_id")
>>> df
           age  height
person_id
0           21    1.61
1           25    1.87
2           62    1.49
3           43    2.01
```

```pycon
>>> df.var()
age       352.916667
height      0.056367
dtype: float64
```

Alternatively, `ddof=0` can be set to normalize by N instead of N-1:

```pycon
>>> df.var(ddof=0)
age       264.687500
height      0.042275
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.visualize.html.md

# dask.dataframe.Series.visualize

#### Series.visualize(tasks: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs)

Visualize the expression or task graph

* **Parameters:**
  **tasks:**
  : Whether to visualize the task graph. By default
    the expression graph will be visualized instead.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.Series.where.html.md

# dask.dataframe.Series.where

#### Series.where(cond, other=nan)

Replace values where the condition is False.

This docstring was copied from pandas.DataFrame.where.

Some inconsistencies with the Dask version may exist.

This method allows conditional replacement of values. Where the
condition evaluates to True, the original values are retained; where
it evaluates to False, values are replaced with corresponding entries
from `other`.

* **Parameters:**
  **cond**
  : Where cond is True, keep the original value. Where
    False, replace with corresponding value from other.
    If cond is callable, it is computed on the Series/DataFrame and
    should return boolean Series/DataFrame or array. The callable must
    not change input Series/DataFrame (though pandas doesn’t check it).

  **other**
  : Entries where cond is False are replaced with
    corresponding value from other.
    If other is callable, it is computed on the Series/DataFrame and
    should return scalar or Series/DataFrame. The callable must not
    change input Series/DataFrame (though pandas doesn’t check it).
    If not specified, entries will be filled with the corresponding
    NULL value (`np.nan` for numpy dtypes, `pd.NA` for extension
    dtypes).

  **inplace**
  : Whether to perform the operation in place on the data.

  **axis**
  : Alignment axis if needed. For Series this parameter is
    unused and defaults to 0.

  **level**
  : Alignment level if needed.
* **Returns:**
  Series or DataFrame
  : When applied to a Series, the function will return a Series,
    and when applied to a DataFrame, it will return a DataFrame.

#### SEE ALSO
[`DataFrame.mask()`](dask.dataframe.DataFrame.mask.md#dask.dataframe.DataFrame.mask)
: Return an object of same shape as caller.

[`Series.mask()`](dask.dataframe.Series.mask.md#dask.dataframe.Series.mask)
: Return an object of same shape as caller.

### Notes

The where method is an application of the if-then idiom. For each
element in the caller, if `cond` is `True` the
element is used; otherwise the corresponding element from
`other` is used. If the axis of `other` does not align with axis of
`cond` Series/DataFrame, the values of `cond` on misaligned index positions
will be filled with False.

The signature for [`Series.where()`](#dask.dataframe.Series.where) or
[`DataFrame.where()`](dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where) differs from [`numpy.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html#numpy.where).
Roughly `df1.where(m, df2)` is equivalent to `np.where(m, df1, df2)`.

For further details and examples see the `where` documentation in
[indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-where-mask).

The dtype of the object takes precedence. The fill value is casted to
the object’s dtype, if this can be done losslessly.

### Examples

```pycon
>>> s = pd.Series(range(5))
>>> s.where(s > 0)
0    NaN
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64
>>> s.mask(s > 0)
0    0.0
1    NaN
2    NaN
3    NaN
4    NaN
dtype: float64
```

```pycon
>>> s = pd.Series(range(5))
>>> t = pd.Series([True, False])
>>> s.where(t, 99)
0     0
1    99
2    99
3    99
4    99
dtype: int64
>>> s.mask(t, 99)
0    99
1     1
2    99
3    99
4    99
dtype: int64
```

```pycon
>>> s.where(s > 1, 10)
0    10
1    10
2    2
3    3
4    4
dtype: int64
>>> s.mask(s > 1, 10)
0     0
1     1
2    10
3    10
4    10
dtype: int64
```

```pycon
>>> df = pd.DataFrame(np.arange(10).reshape(-1, 2), columns=["A", "B"])
>>> df
   A  B
0  0  1
1  2  3
2  4  5
3  6  7
4  8  9
>>> m = df % 3 == 0
>>> df.where(m, -df)
   A  B
0  0 -1
1 -2  3
2 -4 -5
3  6 -7
4 -8  9
>>> df.where(m, -df) == np.where(m, df, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
>>> df.where(m, -df) == df.mask(~m, -df)
      A     B
0  True  True
1  True  True
2  True  True
3  True  True
4  True  True
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.aggregate.html.md

# dask.dataframe.api.GroupBy.aggregate

#### GroupBy.aggregate(arg=None, split_every=8, split_out=None, shuffle_method=None, \*\*kwargs)

Aggregate using one or more specified operations

Based on pd.core.groupby.DataFrameGroupBy.agg

* **Parameters:**
  **arg**
  : Aggregation spec. Accepted combinations are:
    - callable function
    - string function name
    - list of functions and/or function names, e.g. `[np.sum, 'mean']`
    - dict of column names -> function, function name or list of such.
    - None only if named aggregation syntax is used

  **split_every**
  : Number of intermediate partitions that may be aggregated at once.
    This defaults to 8.
    Determines the depth of the recursive aggregation. If set to or more 
    than the number of input chunks, the aggregation will be performed in 
    two steps, one `chunk` function per input chunk and a single 
    `aggregate` function at the end. If set to less than that, an 
    intermediate `combine` function will be used, so that any one 
    `combine` or `aggregate` function has no more than `split_every` 
    inputs. The depth of the aggregation graph will be 
    $\log_\text{split_every}(\text{input chunks along reduced axes})$.
    Setting to a low value can reduce cache size and network transfers, at
    the cost of more CPU and a larger dask graph.

  **split_out**
  : Number of output results in group-by like aggregations (defaults to 1)

  **shuffle**
  : Whether a shuffle-based algorithm should be used. A specific
    algorithm name may also be specified (e.g. `"tasks"` or `"p2p"`).
    The shuffle-based algorithm is likely to be more efficient than
    `shuffle=False` when `split_out>1` and the number of unique
    groups is large (high cardinality). Default is `False` when
    `split_out = 1`. When `split_out > 1`, it chooses the algorithm
    set by the `shuffle` option in the dask config system, or `"tasks"`
    if nothing is set.

  **kwargs: tuple or pd.NamedAgg, optional**
  : Used for named aggregations where the keywords are the output column
    names and the values are tuples where the first element is the input
    column name and the second element is the aggregation function.
    `pandas.NamedAgg` can also be used as the value. To use the named
    aggregation syntax, arg must be set to None.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.apply.html.md

# dask.dataframe.api.GroupBy.apply

#### GroupBy.apply(func, \*args, meta=<no_default>, shuffle_method=None, \*\*kwargs)

Parallel version of pandas GroupBy.apply

This mimics the pandas version except for the following:

1. If the grouper does not align with the index then this causes a full
   shuffle.  The order of rows within each group may not be preserved.
2. Dask’s GroupBy.apply is not appropriate for aggregations. For custom
   aggregations, use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

#### WARNING
Pandas’ groupby-apply can be used to to apply arbitrary functions,
including aggregations that result in one row per group. Dask’s
groupby-apply will apply `func` once on each group, doing a shuffle
if needed, such that each group is contained in one partition.
When `func` is a reduction, e.g., you’ll end up with one row
per group. To apply a custom aggregation with Dask,
use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

* **Parameters:**
  **func: function**
  : Function to apply

  **args, kwargs**
  : Arguments and keywords to pass to the function.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.
* **Returns:**
  **applied**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.bfill.html.md

# dask.dataframe.api.GroupBy.bfill

#### GroupBy.bfill(limit=None, shuffle_method=None)

Backward fill the values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.bfill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **limit**
  : Limit of how many values to fill.
* **Returns:**
  Series or DataFrame
  : Object with missing values filled.

#### SEE ALSO
`Series.bfill`
: Backward fill the missing values in the dataset.

`DataFrame.bfill`
: Backward fill the missing values in the dataset.

`Series.fillna`
: Fill NaN values of a Series.

`DataFrame.fillna`
: Fill NaN values of a DataFrame.

### Examples

With Series:

```pycon
>>> index = ["Falcon", "Falcon", "Parrot", "Parrot", "Parrot"]
>>> s = pd.Series([None, 1, None, None, 3], index=index)
>>> s
Falcon    NaN
Falcon    1.0
Parrot    NaN
Parrot    NaN
Parrot    3.0
dtype: float64
>>> s.groupby(level=0).bfill()
Falcon    1.0
Falcon    1.0
Parrot    3.0
Parrot    3.0
Parrot    3.0
dtype: float64
>>> s.groupby(level=0).bfill(limit=1)
Falcon    1.0
Falcon    1.0
Parrot    NaN
Parrot    3.0
Parrot    3.0
dtype: float64
```

With DataFrame:

```pycon
>>> df = pd.DataFrame(
...     {"A": [1, None, None, None, 4], "B": [None, None, 5, None, 7]},
...     index=index,
... )
>>> df
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  NaN       5.0
Parrot  NaN       NaN
Parrot  4.0       7.0
>>> df.groupby(level=0).bfill()
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  4.0       5.0
Parrot  4.0       7.0
Parrot  4.0       7.0
>>> df.groupby(level=0).bfill(limit=1)
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  NaN       5.0
Parrot  4.0       7.0
Parrot  4.0       7.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.corr.html.md

# dask.dataframe.api.GroupBy.corr

#### GroupBy.corr(split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute pairwise correlation of columns, excluding NA/null values.

This docstring was copied from pandas.DataFrame.corr.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **method**
  : Method of correlation:
    * pearson : standard correlation coefficient
    * kendall : Kendall Tau correlation coefficient
    * spearman : Spearman rank correlation
    * callable: callable with input two 1d ndarrays
      : and returning a float. Note that the returned matrix from corr
        will have 1 along the diagonals and will be symmetric
        regardless of the callable’s behavior.

  **min_periods**
  : Minimum number of observations required per pair of columns
    to have a valid result. Currently only available for Pearson
    and Spearman correlation.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: The default value of `numeric_only` is now `False`.
* **Returns:**
  DataFrame
  : Correlation matrix.

#### SEE ALSO
`DataFrame.corrwith`
: Compute pairwise correlation with another DataFrame or Series.

`Series.corr`
: Compute the correlation between two Series.

### Notes

Pearson, Kendall and Spearman correlation are currently computed using pairwise complete observations.

* [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)
* [Kendall rank correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient)
* [Spearman’s rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient)

### Examples

```pycon
>>> def histogram_intersection(a, b):
...     v = np.minimum(a, b).sum().round(decimals=1)
...     return v
>>> df = pd.DataFrame(
...     [(0.2, 0.3), (0.0, 0.6), (0.6, 0.0), (0.2, 0.1)],
...     columns=["dogs", "cats"],
... )
>>> df.corr(method=histogram_intersection)
      dogs  cats
dogs   1.0   0.3
cats   0.3   1.0
```

```pycon
>>> df = pd.DataFrame(
...     [(1, 1), (2, np.nan), (np.nan, 3), (4, 4)], columns=["dogs", "cats"]
... )
>>> df.corr(min_periods=3)
      dogs  cats
dogs   1.0   NaN
cats   NaN   1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.count.html.md

# dask.dataframe.api.GroupBy.count

#### GroupBy.count(\*\*kwargs)

Compute count of group, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.count.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or DataFrame
  : Count of values within each group.

#### SEE ALSO
`Series.count`
: Apply function count to a Series.

`DataFrame.count`
: Apply function count to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, np.nan], index=lst)
>>> ser
a    1.0
a    2.0
b    NaN
dtype: float64
>>> ser.groupby(level=0).count()
a    2
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, np.nan, 3], [1, np.nan, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["cow", "horse", "bull"]
... )
>>> df
        a         b     c
cow     1       NaN     3
horse   1       NaN     6
bull    7       8.0     9
>>> df.groupby("a").count()
    b   c
a
1   0   2
7   1   1
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").count()
2023-01-01    2
2023-02-01    2
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.cov.html.md

# dask.dataframe.api.GroupBy.cov

#### GroupBy.cov(ddof=1, split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute pairwise covariance of columns, excluding NA/null values.

This docstring was copied from pandas.DataFrame.cov.

Some inconsistencies with the Dask version may exist.

Compute the pairwise covariance among the series of a DataFrame.
The returned data frame is the [covariance matrix](https://en.wikipedia.org/wiki/Covariance_matrix) of the columns
of the DataFrame.

Both NA and null values are automatically excluded from the
calculation. (See the note below about bias from missing values.)
A threshold can be set for the minimum number of
observations for each value created. Comparisons with observations
below this threshold will be returned as `NaN`.

This method is generally used for the analysis of time series data to
understand the relationship between different measures
across time.

* **Parameters:**
  **min_periods**
  : Minimum number of observations required per pair of columns
    to have a valid result.

  **ddof**
  : Delta degrees of freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.
    This argument is applicable only when no `nan` is in the dataframe.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: The default value of `numeric_only` is now `False`.
* **Returns:**
  DataFrame
  : The covariance matrix of the series of the DataFrame.

#### SEE ALSO
`Series.cov`
: Compute covariance with another Series.

`core.window.ewm.ExponentialMovingWindow.cov`
: Exponential weighted sample covariance.

`core.window.expanding.Expanding.cov`
: Expanding sample covariance.

`core.window.rolling.Rolling.cov`
: Rolling sample covariance.

### Notes

Returns the covariance matrix of the DataFrame’s time series.
The covariance is normalized by N-ddof.

For DataFrames that have Series that are missing data (assuming that
data is [missing at random](https://en.wikipedia.org/wiki/Missing_data#Missing_at_random))
the returned covariance matrix will be an unbiased estimate
of the variance and covariance between the member Series.

However, for many applications this estimate may not be acceptable
because the estimate covariance matrix is not guaranteed to be positive
semi-definite. This could lead to estimate correlations having
absolute values which are greater than one, and/or a non-invertible
covariance matrix. See [Estimation of covariance matrices](https://en.wikipedia.org/w/index.php?title=Estimation_of_covariance_matrices) for more details.

### Examples

```pycon
>>> df = pd.DataFrame(
...     [(1, 2), (0, 3), (2, 0), (1, 1)], columns=["dogs", "cats"]
... )
>>> df.cov()
          dogs      cats
dogs  0.666667 -1.000000
cats -1.000000  1.666667
```

```pycon
>>> np.random.seed(42)
>>> df = pd.DataFrame(
...     np.random.randn(1000, 5), columns=["a", "b", "c", "d", "e"]
... )
>>> df.cov()
          a         b         c         d         e
a  0.998438 -0.020161  0.059277 -0.008943  0.014144
b -0.020161  1.059352 -0.008543 -0.024738  0.009826
c  0.059277 -0.008543  1.010670 -0.001486 -0.000271
d -0.008943 -0.024738 -0.001486  0.921297 -0.013692
e  0.014144  0.009826 -0.000271 -0.013692  0.977795
```

**Minimum number of periods**

This method also supports an optional `min_periods` keyword
that specifies the required minimum number of non-NA observations for
each column pair in order to have a valid result:

```pycon
>>> np.random.seed(42)
>>> df = pd.DataFrame(np.random.randn(20, 3), columns=["a", "b", "c"])
>>> df.loc[df.index[:5], "a"] = np.nan
>>> df.loc[df.index[5:10], "b"] = np.nan
>>> df.cov(min_periods=12)
          a         b         c
a  0.316741       NaN -0.150812
b       NaN  1.248003  0.191417
c -0.150812  0.191417  0.895202
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.cumcount.html.md

# dask.dataframe.api.GroupBy.cumcount

#### GroupBy.cumcount()

Number each item in each group from 0 to the length of that group - 1.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumcount.

Some inconsistencies with the Dask version may exist.

Essentially this is equivalent to

```python
self.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
```

* **Parameters:**
  **ascending**
  : If False, number in reverse, from length of group - 1 to 0.
* **Returns:**
  Series
  : Sequence number of each element within each group.

#### SEE ALSO
`ngroup`
: Number the groups themselves.

### Examples

```pycon
>>> df = pd.DataFrame([["a"], ["a"], ["a"], ["b"], ["b"], ["a"]], columns=["A"])
>>> df
   A
0  a
1  a
2  a
3  b
4  b
5  a
>>> df.groupby("A").cumcount()
0    0
1    1
2    2
3    0
4    1
5    3
dtype: int64
>>> df.groupby("A").cumcount(ascending=False)
0    3
1    2
2    1
3    1
4    0
5    0
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.cumprod.html.md

# dask.dataframe.api.GroupBy.cumprod

#### GroupBy.cumprod(numeric_only=False)

Cumulative product for each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumprod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args**
  : Positional arguments to be passed to func.

  **\*\*kwargs**
  : Additional/specific keyword arguments to be passed to the function,
    such as numeric_only and skipna.
* **Returns:**
  Series or DataFrame
  : Cumulative product for each group. Same object type as the caller.

#### SEE ALSO
`Series.cumprod`
: Apply function cumprod to a Series.

`DataFrame.cumprod`
: Apply function cumprod to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([6, 2, 0], index=lst)
>>> ser
a    6
a    2
b    0
dtype: int64
>>> ser.groupby(level=0).cumprod()
a    6
a   12
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["cow", "horse", "bull"]
... )
>>> df
        a   b   c
cow     1   8   2
horse   1   2   5
bull    2   6   9
>>> df.groupby("a").groups
{1: ['cow', 'horse'], 2: ['bull']}
>>> df.groupby("a").cumprod()
        b   c
cow     8   2
horse  16  10
bull    6   9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.cumsum.html.md

# dask.dataframe.api.GroupBy.cumsum

#### GroupBy.cumsum(numeric_only=False)

Cumulative sum for each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumsum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args**
  : Positional arguments to be passed to func.

  **\*\*kwargs**
  : Additional/specific keyword arguments to be passed to the function,
    such as numeric_only and skipna.
* **Returns:**
  Series or DataFrame
  : Cumulative sum for each group. Same object type as the caller.

#### SEE ALSO
`Series.cumsum`
: Apply function cumsum to a Series.

`DataFrame.cumsum`
: Apply function cumsum to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([6, 2, 0], index=lst)
>>> ser
a    6
a    2
b    0
dtype: int64
>>> ser.groupby(level=0).cumsum()
a    6
a    8
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["fox", "gorilla", "lion"]
... )
>>> df
          a   b   c
fox       1   8   2
gorilla   1   2   5
lion      2   6   9
>>> df.groupby("a").groups
{1: ['fox', 'gorilla'], 2: ['lion']}
>>> df.groupby("a").cumsum()
          b   c
fox       8   2
gorilla  10   7
lion      6   9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.ffill.html.md

# dask.dataframe.api.GroupBy.ffill

#### GroupBy.ffill(limit=None, shuffle_method=None)

Forward fill the values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.ffill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **limit**
  : Limit of how many values to fill.
* **Returns:**
  Series or DataFrame
  : Object with missing values filled.

#### SEE ALSO
`Series.ffill`
: Returns Series with minimum number of char in object.

`DataFrame.ffill`
: Object with missing values filled or None if inplace=True.

`Series.fillna`
: Fill NaN values of a Series.

`DataFrame.fillna`
: Fill NaN values of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> key = [0, 0, 1, 1]
>>> ser = pd.Series([np.nan, 2, 3, np.nan], index=key)
>>> ser
0    NaN
0    2.0
1    3.0
1    NaN
dtype: float64
>>> ser.groupby(level=0).ffill()
0    NaN
0    2.0
1    3.0
1    3.0
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> df = pd.DataFrame(
...     {
...         "key": [0, 0, 1, 1, 1],
...         "A": [np.nan, 2, np.nan, 3, np.nan],
...         "B": [2, 3, np.nan, np.nan, np.nan],
...         "C": [np.nan, np.nan, 2, np.nan, np.nan],
...     }
... )
>>> df
   key    A    B   C
0    0  NaN  2.0 NaN
1    0  2.0  3.0 NaN
2    1  NaN  NaN 2.0
3    1  3.0  NaN NaN
4    1  NaN  NaN NaN
```

Propagate non-null values forward or backward within each group along columns.

```pycon
>>> df.groupby("key").ffill()
     A    B   C
0  NaN  2.0 NaN
1  2.0  3.0 NaN
2  NaN  NaN 2.0
3  3.0  NaN 2.0
4  3.0  NaN 2.0
```

Propagate non-null values forward or backward within each group along rows.

```pycon
>>> df.T.groupby(np.array([0, 0, 1, 1])).ffill().T
   key    A    B    C
0  0.0  0.0  2.0  2.0
1  0.0  2.0  3.0  3.0
2  1.0  1.0  NaN  2.0
3  1.0  3.0  NaN  NaN
4  1.0  1.0  NaN  NaN
```

Only replace the first NaN element within a group along columns.

```pycon
>>> df.groupby("key").ffill(limit=1)
     A    B    C
0  NaN  2.0  NaN
1  2.0  3.0  NaN
2  NaN  NaN  2.0
3  3.0  NaN  2.0
4  3.0  NaN  NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.first.html.md

# dask.dataframe.api.GroupBy.first

#### GroupBy.first(numeric_only=False, sort=None, \*\*kwargs)

Compute the first entry of each column within each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.first.

Some inconsistencies with the Dask version may exist.

Defaults to skipping NA elements.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` valid values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 2.2.1.
* **Returns:**
  Series or DataFrame
  : First values within each group.

#### SEE ALSO
`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

`core.groupby.DataFrameGroupBy.last`
: Compute the last non-null entry of each column.

`core.groupby.DataFrameGroupBy.nth`
: Take the nth row from each group.

### Examples

```pycon
>>> df = pd.DataFrame(
...     dict(
...         A=[1, 1, 3],
...         B=[None, 5, 6],
...         C=[1, 2, 3],
...         D=["3/11/2000", "3/12/2000", "3/13/2000"],
...     )
... )
>>> df["D"] = pd.to_datetime(df["D"])
>>> df.groupby("A").first()
     B  C          D
A
1  5.0  1 2000-03-11
3  6.0  3 2000-03-13
>>> df.groupby("A").first(min_count=2)
    B    C          D
A
1 NaN  1.0 2000-03-11
3 NaN  NaN        NaT
>>> df.groupby("A").first(numeric_only=True)
     B  C
A
1  5.0  1
3  6.0  3
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.get_group.html.md

# dask.dataframe.api.GroupBy.get_group

#### GroupBy.get_group(key)

Construct DataFrame from group with provided name.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.get_group.

Some inconsistencies with the Dask version may exist.

Known inconsistencies: 
: If the group is not present, Dask will return an empty Series/DataFrame.

* **Parameters:**
  **name**
  : The name of the group to get as a DataFrame.
* **Returns:**
  Series or DataFrame
  : Get the respective Series or DataFrame corresponding to the group provided.

#### SEE ALSO
`DataFrameGroupBy.groups`
: Dictionary representation of the groupings formed during a groupby operation.

`DataFrameGroupBy.indices`
: Provides a mapping of group rows to positions of the elements.

`SeriesGroupBy.groups`
: Dictionary representation of the groupings formed during a groupby operation.

`SeriesGroupBy.indices`
: Provides a mapping of group rows to positions of the elements.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, 3], index=lst)
>>> ser
a    1
a    2
b    3
dtype: int64
>>> ser.groupby(level=0).get_group("a")
a    1
a    2
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 2, 3], [1, 5, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["owl", "toucan", "eagle"]
... )
>>> df
        a  b  c
owl     1  2  3
toucan  1  5  6
eagle   7  8  9
>>> df.groupby(by=["a"]).get_group((1,))
        a  b  c
owl     1  2  3
toucan  1  5  6
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").get_group("2023-01-01")
2023-01-01    1
2023-01-15    2
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.idxmax.html.md

# dask.dataframe.api.GroupBy.idxmax

#### GroupBy.idxmax(split_every=None, split_out=None, skipna=True, numeric_only=False, shuffle_method=None)

Return index of first occurrence of maximum over requested axis.

This docstring was copied from pandas.DataFrame.idxmax.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of maxima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
`Series.idxmax`
: Return index of the maximum element.

### Notes

This method is the DataFrame version of `ndarray.argmax`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the maximum value in each column.

```pycon
>>> df.idxmax()
consumption      Wheat Products
co2_emissions              Beef
dtype: str
```

To return the index for the maximum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmax(axis="columns")
Pork              co2_emissions
Wheat Products     consumption
Beef              co2_emissions
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.idxmin.html.md

# dask.dataframe.api.GroupBy.idxmin

#### GroupBy.idxmin(split_every=None, split_out=None, skipna=True, numeric_only=False, shuffle_method=None)

Return index of first occurrence of minimum over requested axis.

This docstring was copied from pandas.DataFrame.idxmin.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of minima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
`Series.idxmin`
: Return index of the minimum element.

### Notes

This method is the DataFrame version of `ndarray.argmin`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the minimum value in each column.

```pycon
>>> df.idxmin()
consumption                Pork
co2_emissions    Wheat Products
dtype: str
```

To return the index for the minimum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmin(axis="columns")
Pork                consumption
Wheat Products    co2_emissions
Beef                consumption
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.last.html.md

# dask.dataframe.api.GroupBy.last

#### GroupBy.last(numeric_only=False, sort=None, \*\*kwargs)

Compute the last entry of each column within each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.last.

Some inconsistencies with the Dask version may exist.

Defaults to skipping NA elements.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns. If None, will attempt to use
    everything, then use only numeric data.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` valid values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 2.2.1.
* **Returns:**
  Series or DataFrame
  : Last of values within each group.

#### SEE ALSO
`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

`core.groupby.DataFrameGroupBy.first`
: Compute the first non-null entry of each column.

`core.groupby.DataFrameGroupBy.nth`
: Take the nth row from each group.

### Examples

```pycon
>>> df = pd.DataFrame(dict(A=[1, 1, 3], B=[5, None, 6], C=[1, 2, 3]))
>>> df.groupby("A").last()
     B  C
A
1  5.0  2
3  6.0  3
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.max.html.md

# dask.dataframe.api.GroupBy.max

#### GroupBy.max(numeric_only=False, \*\*kwargs)

Compute max of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.max.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed max of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).max()
a    2
b    4
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").max()
    b  c
a
1   8  5
2   6  9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.mean.html.md

# dask.dataframe.api.GroupBy.mean

#### GroupBy.mean(numeric_only=False, split_out=None, \*\*kwargs)

Compute mean of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None` and defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`
* **Returns:**
  pandas.Series or pandas.DataFrame
  : Mean of values within each group. Same object type as the caller.

#### SEE ALSO
`Series.mean`
: Apply function mean to a Series.

`DataFrame.mean`
: Apply function mean to each row or column of a DataFrame.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"A": [1, 1, 2, 1, 2], "B": [np.nan, 2, 3, 4, 5], "C": [1, 2, 1, 1, 2]},
...     columns=["A", "B", "C"],
... )
```

Groupby one column and return the mean of the remaining columns in
each group.

```pycon
>>> df.groupby("A").mean()
     B         C
A
1  3.0  1.333333
2  4.0  1.500000
```

Groupby two columns and return the mean of the remaining column.

```pycon
>>> df.groupby(["A", "B"]).mean()
         C
A B
1 2.0  2.0
  4.0  1.0
2 3.0  1.0
  5.0  2.0
```

Groupby one column and return the mean of only particular column in
the group.

```pycon
>>> df.groupby("A")["B"].mean()
A
1    3.0
2    4.0
Name: B, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.min.html.md

# dask.dataframe.api.GroupBy.min

#### GroupBy.min(numeric_only=False, \*\*kwargs)

Compute min of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.min.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed min of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).min()
a    1
b    3
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").min()
    b  c
a
1   2  2
2   5  8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.rolling.html.md

# dask.dataframe.api.GroupBy.rolling

#### GroupBy.rolling(window, min_periods=None, center=False, win_type=None, axis=0)

Provides rolling transformations.

#### NOTE
Since MultiIndexes are not well supported in Dask, this method returns a
dataframe with the same index as the original data. The groupby column is
not added as the first level of the index like pandas does.

This method works differently from other groupby methods. It does a groupby
on each partition (plus some overlap). This means that the output has the
same shape and number of partitions as the original.

* **Parameters:**
  **window**
  : Size of the moving window. This is the number of observations used
    for calculating the statistic. Data must have a `DatetimeIndex`

  **min_periods**
  : Minimum number of observations in window required to have a value
    (otherwise result is NA).

  **center**
  : Set the labels at the center of the window.

  **win_type**
  : Provide a window type. The recognized window types are identical
    to pandas.

  **axis**
* **Returns:**
  a Rolling object on which to call a method to compute a statistic

### Examples

```pycon
>>> import dask
>>> ddf = dask.datasets.timeseries(freq="1h")
>>> result = ddf.groupby("name").x.rolling('1D').max()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.size.html.md

# dask.dataframe.api.GroupBy.size

#### GroupBy.size(\*\*kwargs)

Compute group sizes.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.size.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  DataFrame or Series
  : Number of rows in each group as a Series if as_index is True
    or a DataFrame if as_index is False.

#### SEE ALSO
`Series.size`
: Apply function size to a Series.

`DataFrame.size`
: Apply function size to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, 3], index=lst)
>>> ser
a     1
a     2
b     3
dtype: int64
>>> ser.groupby(level=0).size()
a    2
b    1
dtype: int64
```

```pycon
>>> data = [[1, 2, 3], [1, 5, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["owl", "toucan", "eagle"]
... )
>>> df
        a  b  c
owl     1  2  3
toucan  1  5  6
eagle   7  8  9
>>> df.groupby("a").size()
a
1    2
7    1
dtype: int64
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3],
...     index=pd.DatetimeIndex(["2023-01-01", "2023-01-15", "2023-02-01"]),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
dtype: int64
>>> ser.resample("MS").size()
2023-01-01    2
2023-02-01    1
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.std.html.md

# dask.dataframe.api.GroupBy.std

#### GroupBy.std(ddof=1, split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute standard deviation of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.std.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex.

* **Parameters:**
  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is `N - ddof`,
    where `N` represents the number of elements.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 3.0.0.
* **Returns:**
  Series or DataFrame
  : Standard deviation of values within each group.

#### SEE ALSO
`Series.std`
: Apply function std to a Series.

`DataFrame.std`
: Apply function std to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "a", "b", "b", "b"]
>>> ser = pd.Series([7, 2, 8, 4, 3, 3], index=lst)
>>> ser
a     7
a     2
a     8
b     4
b     3
b     3
dtype: int64
>>> ser.groupby(level=0).std()
a    3.21455
b    0.57735
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> data = {"a": [1, 3, 5, 7, 7, 8, 3], "b": [1, 4, 8, 4, 4, 2, 1]}
>>> df = pd.DataFrame(
...     data, index=["dog", "dog", "dog", "mouse", "mouse", "mouse", "mouse"]
... )
>>> df
         a  b
  dog    1  1
  dog    3  4
  dog    5  8
mouse    7  4
mouse    7  4
mouse    8  2
mouse    3  1
>>> df.groupby(level=0).std()
              a         b
dog    2.000000  3.511885
mouse  2.217356  1.500000
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.sum.html.md

# dask.dataframe.api.GroupBy.sum

#### GroupBy.sum(numeric_only=False, min_count=None, \*\*kwargs)

Compute sum of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.sum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed sum of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).sum()
a    3
b    7
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").sum()
     b   c
a
1   10   7
2   11  17
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.transform.html.md

# dask.dataframe.api.GroupBy.transform

#### GroupBy.transform(func, meta=<no_default>, shuffle_method=None, \*args, \*\*kwargs)

Parallel version of pandas GroupBy.transform

This mimics the pandas version except for the following:

1. If the grouper does not align with the index then this causes a full
   shuffle.  The order of rows within each group may not be preserved.
2. Dask’s GroupBy.transform is not appropriate for aggregations. For custom
   aggregations, use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

#### WARNING
Pandas’ groupby-transform can be used to apply arbitrary functions,
including aggregations that result in one row per group. Dask’s
groupby-transform will apply `func` once on each group, doing a shuffle
if needed, such that each group is contained in one partition.
When `func` is a reduction, e.g., you’ll end up with one row
per group. To apply a custom aggregation with Dask,
use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

* **Parameters:**
  **func: function**
  : Function to apply

  **args, kwargs**
  : Arguments and keywords to pass to the function.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.
* **Returns:**
  **applied**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.GroupBy.var.html.md

# dask.dataframe.api.GroupBy.var

#### GroupBy.var(ddof=1, split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute variance of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.var.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex.

* **Parameters:**
  **ddof**
  : Degrees of freedom.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 3.0.0.
* **Returns:**
  Series or DataFrame
  : Variance of values within each group.

#### SEE ALSO
`Series.var`
: Apply function var to a Series.

`DataFrame.var`
: Apply function var to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "a", "b", "b", "b"]
>>> ser = pd.Series([7, 2, 8, 4, 3, 3], index=lst)
>>> ser
a     7
a     2
a     8
b     4
b     3
b     3
dtype: int64
>>> ser.groupby(level=0).var()
a    10.333333
b     0.333333
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> data = {"a": [1, 3, 5, 7, 7, 8, 3], "b": [1, 4, 8, 4, 4, 2, 1]}
>>> df = pd.DataFrame(
...     data, index=["dog", "dog", "dog", "mouse", "mouse", "mouse", "mouse"]
... )
>>> df
         a  b
  dog    1  1
  dog    3  4
  dog    5  8
mouse    7  4
mouse    7  4
mouse    8  2
mouse    3  1
>>> df.groupby(level=0).var()
              a          b
dog    4.000000  12.333333
mouse  4.916667   2.250000
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.apply.html.md

# dask.dataframe.api.Rolling.apply

#### Rolling.apply(func, \*args, \*\*kwargs)

Calculate the rolling custom aggregation function.

This docstring was copied from pandas.api.typing.Rolling.apply.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **func**
  : Must produce a single value from an ndarray input if `raw=True`
    or a single value from a Series if `raw=False`. Can also accept a
    Numba JIT function with `engine='numba'` specified.

  **raw**
  : * `False` : passes each row or column as a Series to the
      function.
    * `True` : the passed function will receive ndarray
      objects instead.
    <br/>
    If you are just applying a NumPy reduction function this will
    achieve much better performance.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'` : Runs rolling apply through JIT compiled code from numba.
      Only available when `raw` is set to `True`.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`.

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
    applied to both the `func` and the `apply` rolling aggregation.

  **args**
  : Positional arguments to be passed into func.

  **kwargs**
  : Keyword arguments to be passed into func.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.apply`
: Aggregating apply for Series.

`DataFrame.apply`
: Aggregating apply for DataFrame.

### Examples

```pycon
>>> ser = pd.Series([1, 6, 5, 4])
>>> ser.rolling(2).apply(lambda s: s.sum() - s.min())
0    NaN
1    6.0
2    6.0
3    5.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.count.html.md

# dask.dataframe.api.Rolling.count

#### Rolling.count(\*args, \*\*kwargs)

Calculate the rolling count of non NaN observations.

This docstring was copied from pandas.api.typing.Rolling.count.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.count`
: Aggregating count for Series.

`DataFrame.count`
: Aggregating count for DataFrame.

### Examples

```pycon
>>> s = pd.Series([2, 3, np.nan, 10])
>>> s.rolling(2).count()
0    NaN
1    2.0
2    1.0
3    1.0
dtype: float64
>>> s.rolling(3).count()
0    NaN
1    NaN
2    2.0
3    2.0
dtype: float64
>>> s.rolling(4).count()
0    NaN
1    NaN
2    NaN
3    3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.kurt.html.md

# dask.dataframe.api.Rolling.kurt

#### Rolling.kurt(\*args, \*\*kwargs)

Calculate the rolling Fisher’s definition of kurtosis without bias.

This docstring was copied from pandas.api.typing.Rolling.kurt.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
[`scipy.stats.kurtosis`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kurtosis.html#scipy.stats.kurtosis)
: Reference SciPy method.

`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.kurt`
: Aggregating kurt for Series.

`DataFrame.kurt`
: Aggregating kurt for DataFrame.

### Notes

A minimum of four periods is required for the calculation.

### Examples

The example below will show a rolling calculation with a window size of
four matching the equivalent function call using scipy.stats.

```pycon
>>> arr = [1, 2, 3, 4, 999]
>>> import scipy.stats
>>> print(f"{scipy.stats.kurtosis(arr[:-1], bias=False):.6f}")
-1.200000
>>> print(f"{scipy.stats.kurtosis(arr[1:], bias=False):.6f}")
3.999946
>>> s = pd.Series(arr)
>>> s.rolling(4).kurt()
0         NaN
1         NaN
2         NaN
3   -1.200000
4    3.999946
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.max.html.md

# dask.dataframe.api.Rolling.max

#### Rolling.max(\*args, \*\*kwargs)

Calculate the rolling maximum.

This docstring was copied from pandas.api.typing.Rolling.max.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args**
  : Positional arguments passed into `func`.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.

  **\*\*kwargs**
  : A dictionary of keyword arguments passed into `func`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.max`
: Aggregating max for Series.

`DataFrame.max`
: Aggregating max for DataFrame.

### Notes

See [Numba engine](https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-numba-engine) and [Numba (JIT compilation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-numba)
for extended documentation and performance considerations
for the Numba engine.

### Examples

```pycon
>>> ser = pd.Series([1, 2, 3, 4])
>>> ser.rolling(2).max()
0    NaN
1    2.0
2    3.0
3    4.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.mean.html.md

# dask.dataframe.api.Rolling.mean

#### Rolling.mean(\*args, \*\*kwargs)

Calculate the rolling mean.

This docstring was copied from pandas.api.typing.Rolling.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.mean`
: Aggregating mean for Series.

`DataFrame.mean`
: Aggregating mean for DataFrame.

### Notes

See [Numba engine](https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-numba-engine) and [Numba (JIT compilation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-numba)
for extended documentation and performance considerations
for the Numba engine.

### Examples

The below examples will show rolling mean calculations with window sizes of
two and three, respectively.

```pycon
>>> s = pd.Series([1, 2, 3, 4])
>>> s.rolling(2).mean()
0    NaN
1    1.5
2    2.5
3    3.5
dtype: float64
```

```pycon
>>> s.rolling(3).mean()
0    NaN
1    NaN
2    2.0
3    3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.median.html.md

# dask.dataframe.api.Rolling.median

#### Rolling.median(\*args, \*\*kwargs)

Calculate the rolling median.

This docstring was copied from pandas.api.typing.Rolling.median.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.median`
: Aggregating median for Series.

`DataFrame.median`
: Aggregating median for DataFrame.

### Notes

See [Numba engine](https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-numba-engine) and [Numba (JIT compilation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-numba)
for extended documentation and performance considerations
for the Numba engine.

### Examples

Compute the rolling median of a series with a window size of 3.

```pycon
>>> s = pd.Series([0, 1, 2, 3, 4])
>>> s.rolling(3).median()
0    NaN
1    NaN
2    1.0
3    2.0
4    3.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.min.html.md

# dask.dataframe.api.Rolling.min

#### Rolling.min(\*args, \*\*kwargs)

Calculate the rolling minimum.

This docstring was copied from pandas.api.typing.Rolling.min.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.min`
: Aggregating min for Series.

`DataFrame.min`
: Aggregating min for DataFrame.

### Notes

See [Numba engine](https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-numba-engine) and [Numba (JIT compilation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-numba)
for extended documentation and performance considerations
for the Numba engine.

### Examples

Performing a rolling minimum with a window size of 3.

```pycon
>>> s = pd.Series([4, 3, 5, 2, 6])
>>> s.rolling(3).min()
0    NaN
1    NaN
2    3.0
3    2.0
4    2.0
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.quantile.html.md

# dask.dataframe.api.Rolling.quantile

#### Rolling.quantile(q, \*args, \*\*kwargs)

Calculate the rolling quantile.

This docstring was copied from pandas.api.typing.Rolling.quantile.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **q**
  : Quantile to compute. 0 <= quantile <= 1.

  **interpolation**
  : This optional parameter specifies the interpolation method to use,
    when the desired quantile lies between two data points i and j:
    > * linear: i + (j - i) \* fraction, where fraction is the
    >   fractional part of the index surrounded by i and j.
    > * lower: i.
    > * higher: j.
    > * nearest: i or j whichever is nearest.
    > * midpoint: (i + j) / 2.

  **numeric_only**
  : Include only float, int, boolean columns.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.quantile`
: Aggregating quantile for Series.

`DataFrame.quantile`
: Aggregating quantile for DataFrame.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3, 4])
>>> s.rolling(2).quantile(0.4, interpolation="lower")
0    NaN
1    1.0
2    2.0
3    3.0
dtype: float64
```

```pycon
>>> s.rolling(2).quantile(0.4, interpolation="midpoint")
0    NaN
1    1.5
2    2.5
3    3.5
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.skew.html.md

# dask.dataframe.api.Rolling.skew

#### Rolling.skew(\*args, \*\*kwargs)

Calculate the rolling unbiased skewness.

This docstring was copied from pandas.api.typing.Rolling.skew.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
[`scipy.stats.skew`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skew.html#scipy.stats.skew)
: Third moment of a probability density.

`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.skew`
: Aggregating skew for Series.

`DataFrame.skew`
: Aggregating skew for DataFrame.

### Notes

A minimum of three periods is required for the rolling calculation.

### Examples

```pycon
>>> ser = pd.Series([1, 5, 2, 7, 15, 6])
>>> ser.rolling(3).skew().round(6)
0         NaN
1         NaN
2    1.293343
3   -0.585583
4    0.670284
5    1.652317
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.std.html.md

# dask.dataframe.api.Rolling.std

#### Rolling.std(\*args, \*\*kwargs)

Calculate the rolling standard deviation.

This docstring was copied from pandas.api.typing.Rolling.std.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ddof**
  : Delta Degrees of Freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
[`numpy.std`](https://numpy.org/doc/stable/reference/generated/numpy.std.html#numpy.std)
: Equivalent method for NumPy array.

`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.std`
: Aggregating std for Series.

`DataFrame.std`
: Aggregating std for DataFrame.

### Notes

The default `ddof` of 1 used in `Series.std()` is different
than the default `ddof` of 0 in [`numpy.std()`](https://numpy.org/doc/stable/reference/generated/numpy.std.html#numpy.std).

A minimum of one period is required for the rolling calculation.

### Examples

```pycon
>>> s = pd.Series([5, 5, 6, 7, 5, 5, 5])
>>> s.rolling(3).std()
0         NaN
1         NaN
2    0.577350
3    1.000000
4    1.000000
5    1.154701
6    0.000000
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.sum.html.md

# dask.dataframe.api.Rolling.sum

#### Rolling.sum(\*args, \*\*kwargs)

Calculate the rolling sum.

This docstring was copied from pandas.api.typing.Rolling.sum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.sum`
: Aggregating sum for Series.

`DataFrame.sum`
: Aggregating sum for DataFrame.

### Notes

See [Numba engine](https://pandas.pydata.org/pandas-docs/stable/user_guide/window.html#window-numba-engine) and [Numba (JIT compilation)](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#enhancingperf-numba)
for extended documentation and performance considerations
for the Numba engine.

### Examples

```pycon
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> s
0    1
1    2
2    3
3    4
4    5
dtype: int64
```

```pycon
>>> s.rolling(3).sum()
0     NaN
1     NaN
2     6.0
3     9.0
4    12.0
dtype: float64
```

```pycon
>>> s.rolling(3, center=True).sum()
0     NaN
1     6.0
2     9.0
3    12.0
4     NaN
dtype: float64
```

For DataFrame, each sum is computed column-wise.

```pycon
>>> df = pd.DataFrame({"A": s, "B": s**2})
>>> df
A   B
0  1   1
1  2   4
2  3   9
3  4  16
4  5  25
```

```pycon
>>> df.rolling(3).sum()
     A     B
0   NaN   NaN
1   NaN   NaN
2   6.0  14.0
3   9.0  29.0
4  12.0  50.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.Rolling.var.html.md

# dask.dataframe.api.Rolling.var

#### Rolling.var(\*args, \*\*kwargs)

Calculate the rolling variance.

This docstring was copied from pandas.api.typing.Rolling.var.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ddof**
  : Delta Degrees of Freedom.  The divisor used in calculations
    is `N - ddof`, where `N` represents the number of elements.

  **numeric_only**
  : Include only float, int, boolean columns.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or
      globally setting `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`.
    <br/>
    The default `engine_kwargs` for the `'numba'` engine is
    `{'nopython': True, 'nogil': False, 'parallel': False}`.
* **Returns:**
  Series or DataFrame
  : Return type is the same as the original object with `np.float64` dtype.

#### SEE ALSO
[`numpy.var`](https://numpy.org/doc/stable/reference/generated/numpy.var.html#numpy.var)
: Equivalent method for NumPy array.

`Series.rolling`
: Calling rolling with Series data.

`DataFrame.rolling`
: Calling rolling with DataFrames.

`Series.var`
: Aggregating var for Series.

`DataFrame.var`
: Aggregating var for DataFrame.

### Notes

The default `ddof` of 1 used in `Series.var()` is different
than the default `ddof` of 0 in [`numpy.var()`](https://numpy.org/doc/stable/reference/generated/numpy.var.html#numpy.var).

A minimum of one period is required for the rolling calculation.

### Examples

```pycon
>>> s = pd.Series([5, 5, 6, 7, 5, 5, 5])
>>> s.rolling(3).var()
0         NaN
1         NaN
2    0.333333
3    1.000000
4    1.000000
5    1.333333
6    0.000000
dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.aggregate.html.md

# dask.dataframe.api.SeriesGroupBy.aggregate

#### SeriesGroupBy.aggregate(arg=None, split_every=8, split_out=None, shuffle_method=None, \*\*kwargs)

Aggregate using one or more specified operations

Based on pd.core.groupby.DataFrameGroupBy.agg

* **Parameters:**
  **arg**
  : Aggregation spec. Accepted combinations are:
    - callable function
    - string function name
    - list of functions and/or function names, e.g. `[np.sum, 'mean']`
    - dict of column names -> function, function name or list of such.
    - None only if named aggregation syntax is used

  **split_every**
  : Number of intermediate partitions that may be aggregated at once.
    This defaults to 8.
    Determines the depth of the recursive aggregation. If set to or more 
    than the number of input chunks, the aggregation will be performed in 
    two steps, one `chunk` function per input chunk and a single 
    `aggregate` function at the end. If set to less than that, an 
    intermediate `combine` function will be used, so that any one 
    `combine` or `aggregate` function has no more than `split_every` 
    inputs. The depth of the aggregation graph will be 
    $\log_\text{split_every}(\text{input chunks along reduced axes})$.
    Setting to a low value can reduce cache size and network transfers, at
    the cost of more CPU and a larger dask graph.

  **split_out**
  : Number of output results in group-by like aggregations (defaults to 1)

  **shuffle**
  : Whether a shuffle-based algorithm should be used. A specific
    algorithm name may also be specified (e.g. `"tasks"` or `"p2p"`).
    The shuffle-based algorithm is likely to be more efficient than
    `shuffle=False` when `split_out>1` and the number of unique
    groups is large (high cardinality). Default is `False` when
    `split_out = 1`. When `split_out > 1`, it chooses the algorithm
    set by the `shuffle` option in the dask config system, or `"tasks"`
    if nothing is set.

  **kwargs: tuple or pd.NamedAgg, optional**
  : Used for named aggregations where the keywords are the output column
    names and the values are tuples where the first element is the input
    column name and the second element is the aggregation function.
    `pandas.NamedAgg` can also be used as the value. To use the named
    aggregation syntax, arg must be set to None.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.apply.html.md

# dask.dataframe.api.SeriesGroupBy.apply

#### SeriesGroupBy.apply(func, \*args, meta=<no_default>, shuffle_method=None, \*\*kwargs)

Parallel version of pandas GroupBy.apply

This mimics the pandas version except for the following:

1. If the grouper does not align with the index then this causes a full
   shuffle.  The order of rows within each group may not be preserved.
2. Dask’s GroupBy.apply is not appropriate for aggregations. For custom
   aggregations, use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

#### WARNING
Pandas’ groupby-apply can be used to to apply arbitrary functions,
including aggregations that result in one row per group. Dask’s
groupby-apply will apply `func` once on each group, doing a shuffle
if needed, such that each group is contained in one partition.
When `func` is a reduction, e.g., you’ll end up with one row
per group. To apply a custom aggregation with Dask,
use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

* **Parameters:**
  **func: function**
  : Function to apply

  **args, kwargs**
  : Arguments and keywords to pass to the function.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.
* **Returns:**
  **applied**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.bfill.html.md

# dask.dataframe.api.SeriesGroupBy.bfill

#### SeriesGroupBy.bfill(limit=None, shuffle_method=None)

Backward fill the values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.bfill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **limit**
  : Limit of how many values to fill.
* **Returns:**
  Series or DataFrame
  : Object with missing values filled.

#### SEE ALSO
`Series.bfill`
: Backward fill the missing values in the dataset.

`DataFrame.bfill`
: Backward fill the missing values in the dataset.

`Series.fillna`
: Fill NaN values of a Series.

`DataFrame.fillna`
: Fill NaN values of a DataFrame.

### Examples

With Series:

```pycon
>>> index = ["Falcon", "Falcon", "Parrot", "Parrot", "Parrot"]
>>> s = pd.Series([None, 1, None, None, 3], index=index)
>>> s
Falcon    NaN
Falcon    1.0
Parrot    NaN
Parrot    NaN
Parrot    3.0
dtype: float64
>>> s.groupby(level=0).bfill()
Falcon    1.0
Falcon    1.0
Parrot    3.0
Parrot    3.0
Parrot    3.0
dtype: float64
>>> s.groupby(level=0).bfill(limit=1)
Falcon    1.0
Falcon    1.0
Parrot    NaN
Parrot    3.0
Parrot    3.0
dtype: float64
```

With DataFrame:

```pycon
>>> df = pd.DataFrame(
...     {"A": [1, None, None, None, 4], "B": [None, None, 5, None, 7]},
...     index=index,
... )
>>> df
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  NaN       5.0
Parrot  NaN       NaN
Parrot  4.0       7.0
>>> df.groupby(level=0).bfill()
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  4.0       5.0
Parrot  4.0       7.0
Parrot  4.0       7.0
>>> df.groupby(level=0).bfill(limit=1)
          A         B
Falcon  1.0       NaN
Falcon  NaN       NaN
Parrot  NaN       5.0
Parrot  4.0       7.0
Parrot  4.0       7.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.count.html.md

# dask.dataframe.api.SeriesGroupBy.count

#### SeriesGroupBy.count(\*\*kwargs)

Compute count of group, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.count.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or DataFrame
  : Count of values within each group.

#### SEE ALSO
`Series.count`
: Apply function count to a Series.

`DataFrame.count`
: Apply function count to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, np.nan], index=lst)
>>> ser
a    1.0
a    2.0
b    NaN
dtype: float64
>>> ser.groupby(level=0).count()
a    2
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, np.nan, 3], [1, np.nan, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["cow", "horse", "bull"]
... )
>>> df
        a         b     c
cow     1       NaN     3
horse   1       NaN     6
bull    7       8.0     9
>>> df.groupby("a").count()
    b   c
a
1   0   2
7   1   1
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").count()
2023-01-01    2
2023-02-01    2
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.cumcount.html.md

# dask.dataframe.api.SeriesGroupBy.cumcount

#### SeriesGroupBy.cumcount()

Number each item in each group from 0 to the length of that group - 1.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumcount.

Some inconsistencies with the Dask version may exist.

Essentially this is equivalent to

```python
self.apply(lambda x: pd.Series(np.arange(len(x)), x.index))
```

* **Parameters:**
  **ascending**
  : If False, number in reverse, from length of group - 1 to 0.
* **Returns:**
  Series
  : Sequence number of each element within each group.

#### SEE ALSO
`ngroup`
: Number the groups themselves.

### Examples

```pycon
>>> df = pd.DataFrame([["a"], ["a"], ["a"], ["b"], ["b"], ["a"]], columns=["A"])
>>> df
   A
0  a
1  a
2  a
3  b
4  b
5  a
>>> df.groupby("A").cumcount()
0    0
1    1
2    2
3    0
4    1
5    3
dtype: int64
>>> df.groupby("A").cumcount(ascending=False)
0    3
1    2
2    1
3    1
4    0
5    0
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.cumprod.html.md

# dask.dataframe.api.SeriesGroupBy.cumprod

#### SeriesGroupBy.cumprod(numeric_only=False)

Cumulative product for each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumprod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args**
  : Positional arguments to be passed to func.

  **\*\*kwargs**
  : Additional/specific keyword arguments to be passed to the function,
    such as numeric_only and skipna.
* **Returns:**
  Series or DataFrame
  : Cumulative product for each group. Same object type as the caller.

#### SEE ALSO
`Series.cumprod`
: Apply function cumprod to a Series.

`DataFrame.cumprod`
: Apply function cumprod to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([6, 2, 0], index=lst)
>>> ser
a    6
a    2
b    0
dtype: int64
>>> ser.groupby(level=0).cumprod()
a    6
a   12
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["cow", "horse", "bull"]
... )
>>> df
        a   b   c
cow     1   8   2
horse   1   2   5
bull    2   6   9
>>> df.groupby("a").groups
{1: ['cow', 'horse'], 2: ['bull']}
>>> df.groupby("a").cumprod()
        b   c
cow     8   2
horse  16  10
bull    6   9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.cumsum.html.md

# dask.dataframe.api.SeriesGroupBy.cumsum

#### SeriesGroupBy.cumsum(numeric_only=False)

Cumulative sum for each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.cumsum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **\*args**
  : Positional arguments to be passed to func.

  **\*\*kwargs**
  : Additional/specific keyword arguments to be passed to the function,
    such as numeric_only and skipna.
* **Returns:**
  Series or DataFrame
  : Cumulative sum for each group. Same object type as the caller.

#### SEE ALSO
`Series.cumsum`
: Apply function cumsum to a Series.

`DataFrame.cumsum`
: Apply function cumsum to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([6, 2, 0], index=lst)
>>> ser
a    6
a    2
b    0
dtype: int64
>>> ser.groupby(level=0).cumsum()
a    6
a    8
b    0
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["fox", "gorilla", "lion"]
... )
>>> df
          a   b   c
fox       1   8   2
gorilla   1   2   5
lion      2   6   9
>>> df.groupby("a").groups
{1: ['fox', 'gorilla'], 2: ['lion']}
>>> df.groupby("a").cumsum()
          b   c
fox       8   2
gorilla  10   7
lion      6   9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.ffill.html.md

# dask.dataframe.api.SeriesGroupBy.ffill

#### SeriesGroupBy.ffill(limit=None, shuffle_method=None)

Forward fill the values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.ffill.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **limit**
  : Limit of how many values to fill.
* **Returns:**
  Series or DataFrame
  : Object with missing values filled.

#### SEE ALSO
`Series.ffill`
: Returns Series with minimum number of char in object.

`DataFrame.ffill`
: Object with missing values filled or None if inplace=True.

`Series.fillna`
: Fill NaN values of a Series.

`DataFrame.fillna`
: Fill NaN values of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> key = [0, 0, 1, 1]
>>> ser = pd.Series([np.nan, 2, 3, np.nan], index=key)
>>> ser
0    NaN
0    2.0
1    3.0
1    NaN
dtype: float64
>>> ser.groupby(level=0).ffill()
0    NaN
0    2.0
1    3.0
1    3.0
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> df = pd.DataFrame(
...     {
...         "key": [0, 0, 1, 1, 1],
...         "A": [np.nan, 2, np.nan, 3, np.nan],
...         "B": [2, 3, np.nan, np.nan, np.nan],
...         "C": [np.nan, np.nan, 2, np.nan, np.nan],
...     }
... )
>>> df
   key    A    B   C
0    0  NaN  2.0 NaN
1    0  2.0  3.0 NaN
2    1  NaN  NaN 2.0
3    1  3.0  NaN NaN
4    1  NaN  NaN NaN
```

Propagate non-null values forward or backward within each group along columns.

```pycon
>>> df.groupby("key").ffill()
     A    B   C
0  NaN  2.0 NaN
1  2.0  3.0 NaN
2  NaN  NaN 2.0
3  3.0  NaN 2.0
4  3.0  NaN 2.0
```

Propagate non-null values forward or backward within each group along rows.

```pycon
>>> df.T.groupby(np.array([0, 0, 1, 1])).ffill().T
   key    A    B    C
0  0.0  0.0  2.0  2.0
1  0.0  2.0  3.0  3.0
2  1.0  1.0  NaN  2.0
3  1.0  3.0  NaN  NaN
4  1.0  1.0  NaN  NaN
```

Only replace the first NaN element within a group along columns.

```pycon
>>> df.groupby("key").ffill(limit=1)
     A    B    C
0  NaN  2.0  NaN
1  2.0  3.0  NaN
2  NaN  NaN  2.0
3  3.0  NaN  2.0
4  3.0  NaN  NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.first.html.md

# dask.dataframe.api.SeriesGroupBy.first

#### SeriesGroupBy.first(numeric_only=False, sort=None, \*\*kwargs)

Compute the first entry of each column within each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.first.

Some inconsistencies with the Dask version may exist.

Defaults to skipping NA elements.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` valid values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 2.2.1.
* **Returns:**
  Series or DataFrame
  : First values within each group.

#### SEE ALSO
`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

`core.groupby.DataFrameGroupBy.last`
: Compute the last non-null entry of each column.

`core.groupby.DataFrameGroupBy.nth`
: Take the nth row from each group.

### Examples

```pycon
>>> df = pd.DataFrame(
...     dict(
...         A=[1, 1, 3],
...         B=[None, 5, 6],
...         C=[1, 2, 3],
...         D=["3/11/2000", "3/12/2000", "3/13/2000"],
...     )
... )
>>> df["D"] = pd.to_datetime(df["D"])
>>> df.groupby("A").first()
     B  C          D
A
1  5.0  1 2000-03-11
3  6.0  3 2000-03-13
>>> df.groupby("A").first(min_count=2)
    B    C          D
A
1 NaN  1.0 2000-03-11
3 NaN  NaN        NaT
>>> df.groupby("A").first(numeric_only=True)
     B  C
A
1  5.0  1
3  6.0  3
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.get_group.html.md

# dask.dataframe.api.SeriesGroupBy.get_group

#### SeriesGroupBy.get_group(key)

Construct DataFrame from group with provided name.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.get_group.

Some inconsistencies with the Dask version may exist.

Known inconsistencies: 
: If the group is not present, Dask will return an empty Series/DataFrame.

* **Parameters:**
  **name**
  : The name of the group to get as a DataFrame.
* **Returns:**
  Series or DataFrame
  : Get the respective Series or DataFrame corresponding to the group provided.

#### SEE ALSO
`DataFrameGroupBy.groups`
: Dictionary representation of the groupings formed during a groupby operation.

`DataFrameGroupBy.indices`
: Provides a mapping of group rows to positions of the elements.

`SeriesGroupBy.groups`
: Dictionary representation of the groupings formed during a groupby operation.

`SeriesGroupBy.indices`
: Provides a mapping of group rows to positions of the elements.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, 3], index=lst)
>>> ser
a    1
a    2
b    3
dtype: int64
>>> ser.groupby(level=0).get_group("a")
a    1
a    2
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 2, 3], [1, 5, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["owl", "toucan", "eagle"]
... )
>>> df
        a  b  c
owl     1  2  3
toucan  1  5  6
eagle   7  8  9
>>> df.groupby(by=["a"]).get_group((1,))
        a  b  c
owl     1  2  3
toucan  1  5  6
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").get_group("2023-01-01")
2023-01-01    1
2023-01-15    2
dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.idxmax.html.md

# dask.dataframe.api.SeriesGroupBy.idxmax

#### SeriesGroupBy.idxmax(split_every=None, split_out=None, skipna=True, numeric_only=False, \*\*kwargs)

Return index of first occurrence of maximum over requested axis.

This docstring was copied from pandas.DataFrame.idxmax.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of maxima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
`Series.idxmax`
: Return index of the maximum element.

### Notes

This method is the DataFrame version of `ndarray.argmax`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the maximum value in each column.

```pycon
>>> df.idxmax()
consumption      Wheat Products
co2_emissions              Beef
dtype: str
```

To return the index for the maximum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmax(axis="columns")
Pork              co2_emissions
Wheat Products     consumption
Beef              co2_emissions
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.idxmin.html.md

# dask.dataframe.api.SeriesGroupBy.idxmin

#### SeriesGroupBy.idxmin(split_every=None, split_out=None, skipna=True, numeric_only=False, \*\*kwargs)

Return index of first occurrence of minimum over requested axis.

This docstring was copied from pandas.DataFrame.idxmin.

Some inconsistencies with the Dask version may exist.

NA/null values are excluded.

* **Parameters:**
  **axis**
  : The axis to use. 0 or ‘index’ for row-wise, 1 or ‘columns’ for column-wise.

  **skipna**
  : Exclude NA/null values. If the entire DataFrame is NA,
    or if `skipna=False` and there is an NA value, this method
    will raise a `ValueError`.

  **numeric_only**
  : Include only float, int or boolean data.
* **Returns:**
  Series
  : Indexes of minima along the specified axis.
* **Raises:**
  ValueError
  : * If the row/column is empty

#### SEE ALSO
`Series.idxmin`
: Return index of the minimum element.

### Notes

This method is the DataFrame version of `ndarray.argmin`.

### Examples

Consider a dataset containing food consumption in Argentina.

```pycon
>>> df = pd.DataFrame(
...     {
...         "consumption": [10.51, 103.11, 55.48],
...         "co2_emissions": [37.2, 19.66, 1712],
...     },
...     index=["Pork", "Wheat Products", "Beef"],
... )
```

```pycon
>>> df
                consumption  co2_emissions
Pork                  10.51         37.20
Wheat Products       103.11         19.66
Beef                  55.48       1712.00
```

By default, it returns the index for the minimum value in each column.

```pycon
>>> df.idxmin()
consumption                Pork
co2_emissions    Wheat Products
dtype: str
```

To return the index for the minimum value in each row, use `axis="columns"`.

```pycon
>>> df.idxmin(axis="columns")
Pork                consumption
Wheat Products    co2_emissions
Beef                consumption
dtype: str
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.last.html.md

# dask.dataframe.api.SeriesGroupBy.last

#### SeriesGroupBy.last(numeric_only=False, sort=None, \*\*kwargs)

Compute the last entry of each column within each group.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.last.

Some inconsistencies with the Dask version may exist.

Defaults to skipping NA elements.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns. If None, will attempt to use
    everything, then use only numeric data.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` valid values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 2.2.1.
* **Returns:**
  Series or DataFrame
  : Last of values within each group.

#### SEE ALSO
`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

`core.groupby.DataFrameGroupBy.first`
: Compute the first non-null entry of each column.

`core.groupby.DataFrameGroupBy.nth`
: Take the nth row from each group.

### Examples

```pycon
>>> df = pd.DataFrame(dict(A=[1, 1, 3], B=[5, None, 6], C=[1, 2, 3]))
>>> df.groupby("A").last()
     B  C
A
1  5.0  2
3  6.0  3
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.max.html.md

# dask.dataframe.api.SeriesGroupBy.max

#### SeriesGroupBy.max(numeric_only=False, \*\*kwargs)

Compute max of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.max.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed max of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).max()
a    2
b    4
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").max()
    b  c
a
1   8  5
2   6  9
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.mean.html.md

# dask.dataframe.api.SeriesGroupBy.mean

#### SeriesGroupBy.mean(numeric_only=False, split_out=None, \*\*kwargs)

Compute mean of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None` and defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`
* **Returns:**
  pandas.Series or pandas.DataFrame
  : Mean of values within each group. Same object type as the caller.

#### SEE ALSO
`Series.mean`
: Apply function mean to a Series.

`DataFrame.mean`
: Apply function mean to each row or column of a DataFrame.

### Examples

```pycon
>>> df = pd.DataFrame(
...     {"A": [1, 1, 2, 1, 2], "B": [np.nan, 2, 3, 4, 5], "C": [1, 2, 1, 1, 2]},
...     columns=["A", "B", "C"],
... )
```

Groupby one column and return the mean of the remaining columns in
each group.

```pycon
>>> df.groupby("A").mean()
     B         C
A
1  3.0  1.333333
2  4.0  1.500000
```

Groupby two columns and return the mean of the remaining column.

```pycon
>>> df.groupby(["A", "B"]).mean()
         C
A B
1 2.0  2.0
  4.0  1.0
2 3.0  1.0
  5.0  2.0
```

Groupby one column and return the mean of only particular column in
the group.

```pycon
>>> df.groupby("A")["B"].mean()
A
1    3.0
2    4.0
Name: B, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.min.html.md

# dask.dataframe.api.SeriesGroupBy.min

#### SeriesGroupBy.min(numeric_only=False, \*\*kwargs)

Compute min of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.min.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed min of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).min()
a    1
b    3
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").min()
    b  c
a
1   2  2
2   5  8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.nunique.html.md

# dask.dataframe.api.SeriesGroupBy.nunique

#### SeriesGroupBy.nunique(split_every=None, split_out=True, shuffle_method=None)

Return number of unique elements in the group.

This docstring was copied from pandas.api.typing.SeriesGroupBy.nunique.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **dropna**
  : Don’t include NaN in the counts.
* **Returns:**
  Series
  : Number of unique values within each group.

#### SEE ALSO
`core.resample.Resampler.nunique`
: Method nunique for Resampler.

### Examples

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> d = {'col1': [1, 2, 3, 4], 'col2': [5, 6, 7, 8]}
>>> df = pd.DataFrame(data=d)
>>> ddf = dd.from_pandas(df, 2)
>>> ddf.groupby(['col1']).col2.nunique().compute()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.rolling.html.md

# dask.dataframe.api.SeriesGroupBy.rolling

#### SeriesGroupBy.rolling(window, min_periods=None, center=False, win_type=None, axis=0)

Provides rolling transformations.

#### NOTE
Since MultiIndexes are not well supported in Dask, this method returns a
dataframe with the same index as the original data. The groupby column is
not added as the first level of the index like pandas does.

This method works differently from other groupby methods. It does a groupby
on each partition (plus some overlap). This means that the output has the
same shape and number of partitions as the original.

* **Parameters:**
  **window**
  : Size of the moving window. This is the number of observations used
    for calculating the statistic. Data must have a `DatetimeIndex`

  **min_periods**
  : Minimum number of observations in window required to have a value
    (otherwise result is NA).

  **center**
  : Set the labels at the center of the window.

  **win_type**
  : Provide a window type. The recognized window types are identical
    to pandas.

  **axis**
* **Returns:**
  a Rolling object on which to call a method to compute a statistic

### Examples

```pycon
>>> import dask
>>> ddf = dask.datasets.timeseries(freq="1h")
>>> result = ddf.groupby("name").x.rolling('1D').max()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.size.html.md

# dask.dataframe.api.SeriesGroupBy.size

#### SeriesGroupBy.size(\*\*kwargs)

Compute group sizes.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.size.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  DataFrame or Series
  : Number of rows in each group as a Series if as_index is True
    or a DataFrame if as_index is False.

#### SEE ALSO
`Series.size`
: Apply function size to a Series.

`DataFrame.size`
: Apply function size to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b"]
>>> ser = pd.Series([1, 2, 3], index=lst)
>>> ser
a     1
a     2
b     3
dtype: int64
>>> ser.groupby(level=0).size()
a    2
b    1
dtype: int64
```

```pycon
>>> data = [[1, 2, 3], [1, 5, 6], [7, 8, 9]]
>>> df = pd.DataFrame(
...     data, columns=["a", "b", "c"], index=["owl", "toucan", "eagle"]
... )
>>> df
        a  b  c
owl     1  2  3
toucan  1  5  6
eagle   7  8  9
>>> df.groupby("a").size()
a
1    2
7    1
dtype: int64
```

For Resampler:

```pycon
>>> ser = pd.Series(
...     [1, 2, 3],
...     index=pd.DatetimeIndex(["2023-01-01", "2023-01-15", "2023-02-01"]),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
dtype: int64
>>> ser.resample("MS").size()
2023-01-01    2
2023-02-01    1
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.std.html.md

# dask.dataframe.api.SeriesGroupBy.std

#### SeriesGroupBy.std(ddof=1, split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute standard deviation of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.std.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex.

* **Parameters:**
  **ddof**
  : Delta Degrees of Freedom. The divisor used in calculations is `N - ddof`,
    where `N` represents the number of elements.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 3.0.0.
* **Returns:**
  Series or DataFrame
  : Standard deviation of values within each group.

#### SEE ALSO
`Series.std`
: Apply function std to a Series.

`DataFrame.std`
: Apply function std to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "a", "b", "b", "b"]
>>> ser = pd.Series([7, 2, 8, 4, 3, 3], index=lst)
>>> ser
a     7
a     2
a     8
b     4
b     3
b     3
dtype: int64
>>> ser.groupby(level=0).std()
a    3.21455
b    0.57735
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> data = {"a": [1, 3, 5, 7, 7, 8, 3], "b": [1, 4, 8, 4, 4, 2, 1]}
>>> df = pd.DataFrame(
...     data, index=["dog", "dog", "dog", "mouse", "mouse", "mouse", "mouse"]
... )
>>> df
         a  b
  dog    1  1
  dog    3  4
  dog    5  8
mouse    7  4
mouse    7  4
mouse    8  2
mouse    3  1
>>> df.groupby(level=0).std()
              a         b
dog    2.000000  3.511885
mouse  2.217356  1.500000
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.sum.html.md

# dask.dataframe.api.SeriesGroupBy.sum

#### SeriesGroupBy.sum(numeric_only=False, min_count=None, \*\*kwargs)

Compute sum of group values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.sum.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If the entire group is NA and `skipna` is
    `True`, the result will be NA.
    <br/>
    #### Versionchanged
    Changed in version 3.0.0.

  **engine**
  : * `'cython'` : Runs rolling apply through C-extensions from cython.
    * `'numba'`
      : Only available when `raw` is set to `True`.
    * `None`
      : `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      : and `parallel` dictionary keys. The values must either be `True` or
        `False`. The default `engine_kwargs` for the `'numba'` engine is
        `{'nopython': True, 'nogil': False, 'parallel': False}` and will be
        applied to both the `func` and the `apply` groupby aggregation.
* **Returns:**
  Series or DataFrame
  : Computed sum of values within each group.

#### SEE ALSO
[`SeriesGroupBy.min`](dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)
: Return the min of the group values.

`DataFrameGroupBy.min`
: Return the min of the group values.

[`SeriesGroupBy.max`](dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)
: Return the max of the group values.

`DataFrameGroupBy.max`
: Return the max of the group values.

[`SeriesGroupBy.sum`](#dask.dataframe.api.SeriesGroupBy.sum)
: Return the sum of the group values.

`DataFrameGroupBy.sum`
: Return the sum of the group values.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "b", "b"]
>>> ser = pd.Series([1, 2, 3, 4], index=lst)
>>> ser
a    1
a    2
b    3
b    4
dtype: int64
>>> ser.groupby(level=0).sum()
a    3
b    7
dtype: int64
```

For DataFrameGroupBy:

```pycon
>>> data = [[1, 8, 2], [1, 2, 5], [2, 5, 8], [2, 6, 9]]
>>> df = pd.DataFrame(
...     data,
...     columns=["a", "b", "c"],
...     index=["tiger", "leopard", "cheetah", "lion"],
... )
>>> df
          a  b  c
  tiger   1  8  2
leopard   1  2  5
cheetah   2  5  8
   lion   2  6  9
>>> df.groupby("a").sum()
     b   c
a
1   10   7
2   11  17
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.transform.html.md

# dask.dataframe.api.SeriesGroupBy.transform

#### SeriesGroupBy.transform(func, meta=<no_default>, shuffle_method=None, \*args, \*\*kwargs)

Parallel version of pandas GroupBy.transform

This mimics the pandas version except for the following:

1. If the grouper does not align with the index then this causes a full
   shuffle.  The order of rows within each group may not be preserved.
2. Dask’s GroupBy.transform is not appropriate for aggregations. For custom
   aggregations, use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

#### WARNING
Pandas’ groupby-transform can be used to apply arbitrary functions,
including aggregations that result in one row per group. Dask’s
groupby-transform will apply `func` once on each group, doing a shuffle
if needed, such that each group is contained in one partition.
When `func` is a reduction, e.g., you’ll end up with one row
per group. To apply a custom aggregation with Dask,
use [`dask.dataframe.groupby.Aggregation`](dask.dataframe.Aggregation.md#dask.dataframe.Aggregation).

* **Parameters:**
  **func: function**
  : Function to apply

  **args, kwargs**
  : Arguments and keywords to pass to the function.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes
    and column names of the output. This metadata is necessary for
    many algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`,
    a `dict` of `{name: dtype}` or iterable of `(name, dtype)`
    can be provided (note that the order of the names should match the
    order of the columns). Instead of a series, a tuple of `(name,
    dtype)` can be used. If not provided, dask will try to infer the
    metadata. This may lead to unexpected results, so providing
    `meta` is recommended. For more information, see
    `dask.dataframe.utils.make_meta`.
* **Returns:**
  **applied**

<!-- !! processed by numpydoc !! -->


# dask.dataframe.api.SeriesGroupBy.var.html.md

# dask.dataframe.api.SeriesGroupBy.var

#### SeriesGroupBy.var(ddof=1, split_every=None, split_out=None, numeric_only=False, shuffle_method=None)

Compute variance of groups, excluding missing values.

This docstring was copied from pandas.core.groupby.groupby.GroupBy.var.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex.

* **Parameters:**
  **ddof**
  : Degrees of freedom.

  **engine**
  : * `'cython'` : Runs the operation through C-extensions from cython.
    * `'numba'` : Runs the operation through JIT compiled code from numba.
    * `None` : Defaults to `'cython'` or globally setting
      `compute.use_numba`

  **engine_kwargs**
  : * For `'cython'` engine, there are no accepted `engine_kwargs`
    * For `'numba'` engine, the engine can accept `nopython`, `nogil`
      and `parallel` dictionary keys. The values must either be `True` or
      `False`. The default `engine_kwargs` for the `'numba'` engine is
      `{'nopython': True, 'nogil': False, 'parallel': False}`

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
    <br/>
    #### Versionadded
    Added in version 3.0.0.
* **Returns:**
  Series or DataFrame
  : Variance of values within each group.

#### SEE ALSO
`Series.var`
: Apply function var to a Series.

`DataFrame.var`
: Apply function var to each row or column of a DataFrame.

### Examples

For SeriesGroupBy:

```pycon
>>> lst = ["a", "a", "a", "b", "b", "b"]
>>> ser = pd.Series([7, 2, 8, 4, 3, 3], index=lst)
>>> ser
a     7
a     2
a     8
b     4
b     3
b     3
dtype: int64
>>> ser.groupby(level=0).var()
a    10.333333
b     0.333333
dtype: float64
```

For DataFrameGroupBy:

```pycon
>>> data = {"a": [1, 3, 5, 7, 7, 8, 3], "b": [1, 4, 8, 4, 4, 2, 1]}
>>> df = pd.DataFrame(
...     data, index=["dog", "dog", "dog", "mouse", "mouse", "mouse", "mouse"]
... )
>>> df
         a  b
  dog    1  1
  dog    3  4
  dog    5  8
mouse    7  4
mouse    7  4
mouse    8  2
mouse    3  1
>>> df.groupby(level=0).var()
              a          b
dog    4.000000  12.333333
mouse  4.916667   2.250000
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.compute.html.md

# dask.dataframe.compute

### dask.dataframe.compute(\*args, traverse=True, optimize_graph=True, scheduler=None, get=None, \*\*kwargs)

Compute several dask collections at once.

* **Parameters:**
  **args**
  : Any number of objects. If it is a dask object, it’s computed and the
    result is returned. By default, python builtin collections are also
    traversed to look for dask objects (for more information see the
    `traverse` keyword). Non-dask arguments are passed through unchanged.

  **traverse**
  : By default dask traverses builtin python collections looking for dask
    objects passed to `compute`. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

  **scheduler**
  : Which scheduler to use like “threads”, “synchronous” or “processes”.
    If not provided, the default is to check the global settings first,
    and then fall back to the collection defaults.

  **optimize_graph**
  : If True [default], the optimizations for each collection are applied
    before computation. Otherwise the graph is run as is. This can be
    useful for debugging.

  **get**
  : Should be left to `None` The get= keyword has been removed.

  **kwargs**
  : Extra keywords to forward to the scheduler function.

### Examples

```pycon
>>> import dask
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> dask.compute(a, b)
(np.int64(45), np.float64(4.5))
```

By default, dask objects inside python collections will also be computed:

```pycon
>>> dask.compute({'a': a, 'b': b, 'c': 1})
({'a': np.int64(45), 'b': np.float64(4.5), 'c': 1},)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.concat.html.md

# dask.dataframe.concat

### dask.dataframe.concat(dfs, axis=0, join='outer', ignore_unknown_divisions=False, ignore_order=False, interleave_partitions=False, \*\*kwargs)

Concatenate DataFrames along rows.

- When axis=0 (default), concatenate DataFrames row-wise:
  - If all divisions are known and ordered, concatenate DataFrames keeping
    divisions. When divisions are not ordered, specifying
    interleave_partition=True allows concatenate divisions each by each.
  - If any of division is unknown, concatenate DataFrames resetting its
    division to unknown (None)
- When axis=1, concatenate DataFrames column-wise:
  - Allowed if all divisions are known.
  - If any of division is unknown, it raises ValueError.

* **Parameters:**
  **dfs**
  : List of dask.DataFrames to be concatenated

  **axis**
  : The axis to concatenate along

  **join**
  : How to handle indexes on other axis

  **interleave_partitions**
  : Whether to concatenate DataFrames ignoring its order. If True, every
    divisions are concatenated each by each.

  **ignore_unknown_divisions**
  : By default a warning is raised if any input has unknown divisions.
    Set to True to disable this warning.

  **ignore_order**
  : Whether to ignore order when doing the union of categoricals.

### Notes

This differs in from `pd.concat` in the when concatenating Categoricals
with different categories. Pandas currently coerces those to objects
before concatenating. Coercing to objects is very expensive for large
arrays, so dask preserves the Categoricals by taking the union of
the categories.

### Examples

If all divisions are known and ordered, divisions are kept.

```pycon
>>> import dask.dataframe as dd
>>> a
dd.DataFrame<x, divisions=(1, 3, 5)>
>>> b
dd.DataFrame<y, divisions=(6, 8, 10)>
>>> dd.concat([a, b])
dd.DataFrame<concat-..., divisions=(1, 3, 6, 8, 10)>
```

Unable to concatenate if divisions are not ordered.

```pycon
>>> a
dd.DataFrame<x, divisions=(1, 3, 5)>
>>> b
dd.DataFrame<y, divisions=(2, 3, 6)>
>>> dd.concat([a, b])
ValueError: All inputs have known divisions which cannot be concatenated
in order. Specify interleave_partitions=True to ignore order
```

Specify interleave_partitions=True to ignore the division order.

```pycon
>>> dd.concat([a, b], interleave_partitions=True)
dd.DataFrame<concat-..., divisions=(1, 2, 3, 5, 6)>
```

If any of division is unknown, the result division will be unknown

```pycon
>>> a
dd.DataFrame<x, divisions=(None, None)>
>>> b
dd.DataFrame<y, divisions=(1, 4, 10)>
>>> dd.concat([a, b])
dd.DataFrame<concat-..., divisions=(None, None, None, None)>
```

By default concatenating with unknown divisions will raise a warning.
Set `ignore_unknown_divisions=True` to disable this:

```pycon
>>> dd.concat([a, b], ignore_unknown_divisions=True)
dd.DataFrame<concat-..., divisions=(None, None, None, None)>
```

Different categoricals are unioned

```pycon
>>> dd.concat([
...     dd.from_pandas(pd.Series(['a', 'b'], dtype='category'), 1),
...     dd.from_pandas(pd.Series(['a', 'c'], dtype='category'), 1),
... ], interleave_partitions=True).dtype
CategoricalDtype(categories=['a', 'b', 'c'], ordered=False, categories_dtype=str)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.from_array.html.md

# dask.dataframe.from_array

### dask.dataframe.from_array(arr, chunksize=50000, columns=None, meta=None)

Read any sliceable array into a Dask Dataframe

Uses getitem syntax to pull slices out of the array.  The array need not be
a NumPy array but must support slicing syntax

> x[50000:100000]

and have 2 dimensions:

> x.ndim == 2

or have a record dtype:

> x.dtype == [(‘name’, ‘O’), (‘balance’, ‘i8’)]
* **Parameters:**
  **x**

  **chunksize**
  : The number of rows per partition to use.

  **columns**
  : list of column names if DataFrame, single string if Series

  **meta**
  : An optional meta parameter can be passed for dask
    to specify the concrete dataframe type to use for partitions of
    the Dask dataframe. By default, pandas DataFrame is used.
* **Returns:**
  dask.DataFrame or dask.Series
  : A dask DataFrame/Series

<!-- !! processed by numpydoc !! -->


# dask.dataframe.from_dask_array.html.md

# dask.dataframe.from_dask_array

### dask.dataframe.from_dask_array(x, columns=None, index=None, meta=None) → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)

Create a Dask DataFrame from a Dask Array.

Converts a 2d array into a DataFrame and a 1d array into a Series.

* **Parameters:**
  **x**

  **columns**
  : list of column names if DataFrame, single string if Series

  **index**
  : An optional *dask* Index to use for the output Series or DataFrame.
    <br/>
    The default output index depends on whether x has any unknown
    chunks. If there are any unknown chunks, the output has `None`
    for all the divisions (one per chunk). If all the chunks are known,
    a default index with known divisions is created.
    <br/>
    Specifying index can be useful if you’re conforming a Dask Array
    to an existing dask Series or DataFrame, and you would like the
    indices to match.

  **meta**
  : An optional meta parameter can be passed for dask
    to specify the concrete dataframe type to be returned.
    By default, pandas DataFrame is used.

#### SEE ALSO
`dask.bag.to_dataframe`
: from dask.bag

[`dask.dataframe.DataFrame.values`](dask.dataframe.DataFrame.values.md#dask.dataframe.DataFrame.values)
: Reverse conversion

[`dask.dataframe.DataFrame.to_records`](dask.dataframe.DataFrame.to_records.md#dask.dataframe.DataFrame.to_records)
: Reverse conversion

### Examples

```pycon
>>> import dask.array as da
>>> import dask.dataframe as dd
>>> x = da.ones((4, 2), chunks=(2, 2))
>>> df = dd.io.from_dask_array(x, columns=['a', 'b'])
>>> df.compute()
     a    b
0  1.0  1.0
1  1.0  1.0
2  1.0  1.0
3  1.0  1.0
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.from_delayed.html.md

# dask.dataframe.from_delayed

### dask.dataframe.from_delayed(dfs: [Delayed](../delayed-api.md#dask.delayed.Delayed) | [distributed.Future](../futures.md#distributed.Future) | Collection[[Delayed](../delayed-api.md#dask.delayed.Delayed) | [distributed.Future](../futures.md#distributed.Future)], meta=None, divisions: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) | [None](https://docs.python.org/3/library/constants.html#None) = None, prefix: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, verify_meta: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Create Dask DataFrame from many Dask Delayed objects

#### WARNING
`from_delayed` should only be used if the objects that create
the data are complex and cannot be easily represented as a single
function in an embarrassingly parallel fashion.

`from_map` is recommended if the query can be expressed as a single
function like:

def read_xml(path):
: return pd.read_xml(path)

ddf = dd.from_map(read_xml, paths)

`from_delayed` might be deprecated in the future.

* **Parameters:**
  **dfs**
  : A `dask.delayed.Delayed`, a `distributed.Future`, or an iterable of either
    of these objects, e.g. returned by `client.submit`. These comprise the
    individual partitions of the resulting dataframe.
    If a single object is provided (not an iterable), then the resulting dataframe
    will have only one partition.

  **$META**

  **divisions**
  : Partition boundaries along the index.
    For tuple, see [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions)
    If None, then won’t use index information

  **prefix**
  : Prefix to prepend to the keys.

  **verify_meta**
  : If True check that the partitions have consistent metadata, defaults to True.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.from_map.html.md

# dask.dataframe.from_map

### dask.dataframe.from_map(func, \*iterables, args=None, meta=<no_default>, divisions=None, label=None, enforce_metadata=False, \*\*kwargs)

Create a DataFrame collection from a custom function map.

`from_map` is the preferred option when reading from data sources
that are not natively supported by Dask or if the data source
requires custom handling before handing things of to Dask DataFrames.
Examples are things like binary files or other unstructured data that
doesn’t have an IO connector.

`from_map` supports column projection by the optimizer. The optimizer
tries to push column selections into the from_map call if the function
supports a `columns` argument.

* **Parameters:**
  **func**
  : Function used to create each partition. Column projection will be
    enabled if the function has a `columns` keyword argument.

  **\*iterables**
  : Iterable objects to map to each output partition. All iterables must
    be the same length. This length determines the number of partitions
    in the output collection (only one element of each iterable will
    be passed to `func` for each partition).

  **args**
  : Positional arguments to broadcast to each output partition. Note
    that these arguments will always be passed to `func` after the
    `iterables` positional arguments.

  **$META**

  **divisions**
  : Partition boundaries along the index.
    For tuple, see [https://docs.dask.org/en/latest/dataframe-design.html#partitions](https://docs.dask.org/en/latest/dataframe-design.html#partitions)
    For string ‘sorted’ will compute the delayed values to find index
    values.  Assumes that the indexes are mutually sorted.
    If None, then won’t use index information

  **label**
  : String to use as the function-name label in the output
    collection-key names.

  **token**
  : String to use as the “token” in the output collection-key names.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition,
    and will raise an error if this doesn’t work,
    but it won’t raise if dtypes don’t match.

  **\*\*kwargs:**
  : Key-word arguments to broadcast to each output partition. These
    same arguments will be passed to `func` for every output partition.

### Examples

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> func = lambda x, size=0: pd.Series([x] * size)
>>> inputs = ["A", "B"]
>>> dd.from_map(func, inputs, size=2).compute()
0    A
1    A
0    B
1    B
dtype: string
```

The optimizer will identify a column selection that happens after from_map
and push the columns argument into the actual map call to drop unnecessary
columns as early as possible.

```pycon
>>> def map_function(x, columns=None):
...     df = pd.DataFrame({"a": [1, 2], "b": x})
...     if columns is not None:
...         df = df[columns]
...     return df
>>> dd.from_map(map_function, [1, 2])["b"].compute()
0    1
1    1
0    2
1    2
Name: b, dtype: int64
```

This API can also be used as an alternative to other file-based
IO functions, like `read_csv` (which are already just
`from_map` wrapper functions):

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> paths = ["0.csv", "1.csv", "2.csv"]
>>> dd.from_map(pd.read_csv, paths).head()
                    name
timestamp
2000-01-01 00:00:00   Laura
2000-01-01 00:00:01  Oliver
2000-01-01 00:00:02   Alice
2000-01-01 00:00:03  Victor
2000-01-01 00:00:04     Bob
```

Since `from_map` allows you to map an arbitrary function
to any number of iterable objects, it can be a very convenient
means of implementing functionality that may be missing
from other DataFrame-creation methods. For example, if you
happen to have apriori knowledge about the number of rows
in each of the files in a dataset, you can generate a
DataFrame collection with a global RangeIndex:

```pycon
>>> import pandas as pd
>>> import numpy as np
>>> import dask.dataframe as dd
>>> paths = ["0.csv", "1.csv", "2.csv"]
>>> file_sizes = [86400, 86400, 86400]
>>> def func(path, row_offset):
...     # Read parquet file and set RangeIndex offset
...     df = pd.read_csv(path)
...     return df.set_index(
...         pd.RangeIndex(row_offset, row_offset+len(df))
...     )
>>> def get_ddf(paths, file_sizes):
...     offsets = [0] + list(np.cumsum(file_sizes))
...     return dd.from_map(
...         func, paths, offsets[:-1], divisions=offsets
...     )
>>> ddf = get_ddf(paths, file_sizes)
>>> ddf.index
Dask Index Structure:
npartitions=3
0         int64
86400       ...
172800      ...
259200      ...
dtype: int64
Dask Name: myfunc, 6 tasks
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.from_pandas.html.md

# dask.dataframe.from_pandas

### dask.dataframe.from_pandas(data, npartitions=None, sort=True, chunksize=None)

Construct a Dask DataFrame from a Pandas DataFrame

This splits an in-memory Pandas dataframe into several parts and constructs
a dask.dataframe from those parts on which Dask.dataframe can operate in
parallel.  By default, the input dataframe will be sorted by the index to
produce cleanly-divided partitions (with known divisions).  To preserve the
input ordering, make sure the input index is monotonically-increasing. The
`sort=False` option will also avoid reordering, but will not result in
known divisions.

* **Parameters:**
  **data**
  : The DataFrame/Series with which to construct a Dask DataFrame/Series

  **npartitions**
  : The number of partitions of the index to create. Note that if there
    are duplicate values or insufficient elements in `data.index`, the
    output may have fewer partitions than requested.

  **chunksize**
  : The desired number of rows per index partition to use. Note that
    depending on the size and index of the dataframe, actual partition
    sizes may vary.

  **sort: bool, default True**
  : Sort the input by index first to obtain cleanly divided partitions
    (with known divisions).  If False, the input will not be sorted, and
    all divisions will be set to None. Default is True.
* **Returns:**
  dask.DataFrame or dask.Series
  : A dask DataFrame/Series partitioned along the index
* **Raises:**
  TypeError
  : If something other than a `pandas.DataFrame` or `pandas.Series` is
    passed in.

#### SEE ALSO
[`from_array`](dask.dataframe.from_array.md#dask.dataframe.from_array)
: Construct a dask.DataFrame from an array that has record dtype

[`read_csv`](dask.dataframe.read_csv.md#dask.dataframe.read_csv)
: Construct a dask.DataFrame from a CSV file

### Examples

```pycon
>>> from dask.dataframe import from_pandas
>>> df = pd.DataFrame(dict(a=list('aabbcc'), b=list(range(6))),
...                   index=pd.date_range(start='20100101', periods=6))
>>> ddf = from_pandas(df, npartitions=3)
>>> ddf.divisions
(Timestamp('2010-01-01 00:00:00'),
 Timestamp('2010-01-03 00:00:00'),
 Timestamp('2010-01-05 00:00:00'),
 Timestamp('2010-01-06 00:00:00'))
>>> ddf = from_pandas(df.a, npartitions=3)  # Works with Series too!
>>> ddf.divisions
(Timestamp('2010-01-01 00:00:00'),
 Timestamp('2010-01-03 00:00:00'),
 Timestamp('2010-01-05 00:00:00'),
 Timestamp('2010-01-06 00:00:00'))
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.get_dummies.html.md

# dask.dataframe.get_dummies

### dask.dataframe.get_dummies(data, prefix=None, prefix_sep='_', dummy_na=False, columns=None, sparse=False, drop_first=False, dtype=<class 'bool'>, \*\*kwargs)

Convert categorical variable into dummy/indicator variables.

Data must have category dtype to infer result’s `columns`.

* **Parameters:**
  **data**
  : For Series, the dtype must be categorical.
    For DataFrame, at least one column must be categorical.

  **prefix**
  : String to append DataFrame column names.
    Pass a list with length equal to the number of columns
    when calling get_dummies on a DataFrame. Alternatively, prefix
    can be a dictionary mapping column names to prefixes.

  **prefix_sep**
  : If appending prefix, separator/delimiter to use. Or pass a
    list or dictionary as with prefix.

  **dummy_na**
  : Add a column to indicate NaNs, if False NaNs are ignored.

  **columns**
  : Column names in the DataFrame to be encoded.
    If columns is None then all the columns with
    category dtype will be converted.

  **sparse**
  : Whether the dummy columns should be sparse or not.  Returns
    SparseDataFrame if data is a Series or if all columns are included.
    Otherwise returns a DataFrame with some SparseBlocks.
    <br/>
    #### Versionadded
    Added in version 0.18.2.

  **drop_first**
  : Whether to get k-1 dummies out of k categorical levels by removing the
    first level.

  **dtype**
  : Data type for new columns. Only a single dtype is allowed.
    <br/>
    #### Versionadded
    Added in version 0.18.2.
* **Returns:**
  **dummies**

#### SEE ALSO
[`pandas.get_dummies`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html#pandas.get_dummies)

### Examples

Dask’s version only works with Categorical data, as this is the only way to
know the output shape without computing all the data.

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> s = dd.from_pandas(pd.Series(list('abca')), npartitions=2)
>>> dd.get_dummies(s)
Traceback (most recent call last):
    ...
NotImplementedError: `get_dummies` with non-categorical dtypes is not supported...
```

With categorical data:

```pycon
>>> s = dd.from_pandas(pd.Series(list('abca'), dtype='category'), npartitions=2)
>>> dd.get_dummies(s)
Dask DataFrame Structure:
                  a     b     c
npartitions=2
0              bool  bool  bool
2               ...   ...   ...
3               ...   ...   ...
Dask Name: operation, 2 expressions
Expr=GetDummies(frame=df)
>>> dd.get_dummies(s).compute()
       a      b      c
0   True  False  False
1  False   True  False
2  False  False   True
3   True  False  False
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.map_overlap.html.md

# dask.dataframe.map_overlap

### dask.dataframe.map_overlap(func, df, before, after, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, \*\*kwargs)

Apply a function to each partition, sharing rows with adjacent partitions.

* **Parameters:**
  **func**
  : The function applied to each partition. If this function accepts
    the special `partition_info` keyword argument, it will receive
    information on the partition’s relative location within the
    dataframe.

  **df: dd.DataFrame, dd.Series**

  **args, kwargs**
  : Positional and keyword arguments to pass to the function.
    Positional arguments are computed on a per-partition basis, while
    keyword arguments are shared across all partitions. The partition
    itself will be the first positional argument, with all other
    arguments passed *after*. Arguments can be `Scalar`, `Delayed`,
    or regular Python objects. DataFrame-like args (both dask and
    pandas) will be repartitioned to align (if necessary) before
    applying the function; see `align_dataframes` to control this
    behavior.

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition,
    and will raise an error if this doesn’t work,
    but it won’t raise if dtypes don’t match.

  **before**
  : The rows to prepend to partition `i` from the end of
    partition `i - 1`.

  **after**
  : The rows to append to partition `i` from the beginning
    of partition `i + 1`.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **align_dataframes**
  : Whether to repartition DataFrame- or Series-like args
    (both dask and pandas) so their divisions align before applying
    the function. This requires all inputs to have known divisions.
    Single-partition inputs will be split into multiple partitions.
    <br/>
    If False, all inputs must have either the same number of partitions
    or a single partition. Single-partition inputs will be broadcast to
    every partition of multi-partition inputs.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes and
    column names of the output. This metadata is necessary for many
    algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`, a
    `dict` of `{name: dtype}` or iterable of `(name, dtype)` can be
    provided (note that the order of the names should match the order of
    the columns). Instead of a series, a tuple of `(name, dtype)` can be
    used. If not provided, dask will try to infer the metadata. This may
    lead to unexpected results, so providing `meta` is recommended. For
    more information, see `dask.dataframe.utils.make_meta`.

#### SEE ALSO
`dd.DataFrame.map_overlap`

<!-- !! processed by numpydoc !! -->


# dask.dataframe.map_partitions.html.md

# dask.dataframe.map_partitions

### dask.dataframe.map_partitions(func, \*args, meta=<no_default>, enforce_metadata=True, transform_divisions=True, clear_divisions=False, align_dataframes=False, parent_meta=None, required_columns=None, \*\*kwargs)

Apply Python function on each DataFrame partition.

* **Parameters:**
  **func**
  : Function applied to each partition.

  **args, kwargs**
  : Arguments and keywords to pass to the function.  At least one of the
    args should be a Dask.dataframe. Arguments and keywords may contain
    `Scalar`, `Delayed` or regular python objects. DataFrame-like args
    (both dask and pandas) will be repartitioned to align (if necessary)
    before applying the function (see `align_dataframes` to control).

  **enforce_metadata**
  : Whether to enforce at runtime that the structure of the DataFrame
    produced by `func` actually matches the structure of `meta`.
    This will rename and reorder columns for each partition,
    and will raise an error if this doesn’t work,
    but it won’t raise if dtypes don’t match.

  **transform_divisions**
  : Whether to apply the function onto the divisions and apply those
    transformed divisions to the output.

  **align_dataframes**
  : Whether to repartition DataFrame- or Series-like args
    (both dask and pandas) so their divisions align before applying
    the function. This requires all inputs to have known divisions.
    Single-partition inputs will be split into multiple partitions.
    <br/>
    If False, all inputs must have either the same number of partitions
    or a single partition. Single-partition inputs will be broadcast to
    every partition of multi-partition inputs.

  **required_columns**
  : List of columns that `func` requires for execution. These columns
    must belong to the first DataFrame argument (in `args`). If None
    is specified (the default), the query optimizer will assume that
    all input columns are required.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes and
    column names of the output. This metadata is necessary for many
    algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`, a
    `dict` of `{name: dtype}` or iterable of `(name, dtype)` can be
    provided (note that the order of the names should match the order of
    the columns). Instead of a series, a tuple of `(name, dtype)` can be
    used. If not provided, dask will try to infer the metadata. This may
    lead to unexpected results, so providing `meta` is recommended. For
    more information, see `dask.dataframe.utils.make_meta`.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.melt.html.md

# dask.dataframe.melt

### dask.dataframe.melt(frame, id_vars=None, value_vars=None, var_name=None, value_name='value', col_level=None)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.merge.html.md

# dask.dataframe.merge

### dask.dataframe.merge(left: DataFrame | Series, right: DataFrame | Series, how: MergeHow = 'inner', on: IndexLabel | AnyArrayLike | None = None, left_on: IndexLabel | AnyArrayLike | None = None, right_on: IndexLabel | AnyArrayLike | None = None, left_index: bool = False, right_index: bool = False, sort: bool = False, suffixes: Suffixes = ('_x', '_y'), copy: bool | lib.NoDefault = <no_default>, indicator: str | bool = False, validate: str | None = None) → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)

Merge DataFrame or named Series objects with a database-style join.

A named Series object is treated as a DataFrame with a single named column.

The join is done on columns or indexes. If joining columns on
columns, the DataFrame indexes *will be ignored*. Otherwise if joining indexes
on indexes or indexes on a column or columns, the index will be passed on.
When performing a cross merge, no column specifications to merge on are
allowed.

#### WARNING
If both key columns contain rows where the key is a null value, those
rows will be matched against each other. This is different from usual SQL
join behaviour and can lead to unexpected results.

* **Parameters:**
  **left**
  : First pandas object to merge.

  **right**
  : Second pandas object to merge.

  **how**
  : default ‘inner’
    Type of merge to be performed.
    * left: use only keys from left frame, similar to a SQL left outer join;
      preserve key order.
    * right: use only keys from right frame, similar to a SQL right outer join;
      preserve key order.
    * outer: use union of keys from both frames, similar to a SQL full outer
      join; sort keys lexicographically.
    * inner: use intersection of keys from both frames, similar to a SQL inner
      join; preserve the order of the left keys.
    * cross: creates the cartesian product from both frames, preserves the order
      of the left keys.
    * left_anti: use only keys from left frame that are not in right frame, similar
      to SQL left anti join; preserve key order.
    * right_anti: use only keys from right frame that are not in left frame, similar
      to SQL right anti join; preserve key order.

  **on**
  : Column or index level names to join on. These must be found in both
    DataFrames. If on is None and not merging on indexes then this defaults
    to the intersection of the columns in both DataFrames.

  **left_on**
  : Column or index level names to join on in the left DataFrame. Can also
    be an array or list of arrays of the length of the left DataFrame.
    These arrays are treated as if they are columns.

  **right_on**
  : Column or index level names to join on in the right DataFrame. Can also
    be an array or list of arrays of the length of the right DataFrame.
    These arrays are treated as if they are columns.

  **left_index**
  : Use the index from the left DataFrame as the join key(s). If it is a
    MultiIndex, the number of keys in the other DataFrame (either the index
    or a number of columns) must match the number of levels.

  **right_index**
  : Use the index from the right DataFrame as the join key. Same caveats as
    left_index.

  **sort**
  : Sort the join keys lexicographically in the result DataFrame. If False,
    the order of the join keys depends on the join type (how keyword).

  **suffixes**
  : A length-2 sequence where each element is optionally a string
    indicating the suffix to add to overlapping column names in
    left and right respectively. Pass a value of None instead
    of a string to indicate that the column name from left or
    right should be left as-is, with no suffix. At least one of the
    values must not be None.

  **copy**
  : This keyword is now ignored; changing its value will have no
    impact on the method.
    <br/>
    #### Deprecated
    Deprecated since version 3.0.0: This keyword is ignored and will be removed in pandas 4.0. Since
    pandas 3.0, this method always returns a new object using a lazy
    copy mechanism that defers copies until necessary
    (Copy-on-Write). See the [user guide on Copy-on-Write](https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html)
    for more details.

  **indicator**
  : If True, adds a column to the output DataFrame called “_merge” with
    information on the source of each row. The column can be given a different
    name by providing a string argument. The column will have a Categorical
    type with the value of “left_only” for observations whose merge key only
    appears in the left DataFrame, “right_only” for observations
    whose merge key only appears in the right DataFrame, and “both”
    if the observation’s merge key is found in both DataFrames.

  **validate**
  : If specified, checks if merge is of specified type.
    * “one_to_one” or “1:1”: check if merge keys are unique in both
      left and right datasets.
    * “one_to_many” or “1:m”: check if merge keys are unique in left
      dataset.
    * “many_to_one” or “m:1”: check if merge keys are unique in right
      dataset.
    * “many_to_many” or “m:m”: allowed, but does not result in checks.
* **Returns:**
  DataFrame
  : A DataFrame of the two merged objects.

#### SEE ALSO
`merge_ordered`
: Merge with optional filling/interpolation.

[`merge_asof`](dask.dataframe.merge_asof.md#dask.dataframe.merge_asof)
: Merge on nearest keys.

[`DataFrame.join`](dask.dataframe.DataFrame.join.md#dask.dataframe.DataFrame.join)
: Similar method using indices.

### Examples

```pycon
>>> df1 = pd.DataFrame(
...     {"lkey": ["foo", "bar", "baz", "foo"], "value": [1, 2, 3, 5]}
... )
>>> df2 = pd.DataFrame(
...     {"rkey": ["foo", "bar", "baz", "foo"], "value": [5, 6, 7, 8]}
... )
>>> df1
    lkey value
0   foo      1
1   bar      2
2   baz      3
3   foo      5
>>> df2
    rkey value
0   foo      5
1   bar      6
2   baz      7
3   foo      8
```

Merge df1 and df2 on the lkey and rkey columns. The value columns have
the default suffixes, \_x and \_y, appended.

```pycon
>>> df1.merge(df2, left_on="lkey", right_on="rkey")
  lkey  value_x rkey  value_y
0  foo        1  foo        5
1  foo        1  foo        8
2  bar        2  bar        6
3  baz        3  baz        7
4  foo        5  foo        5
5  foo        5  foo        8
```

Merge DataFrames df1 and df2 with specified left and right suffixes
appended to any overlapping columns.

```pycon
>>> df1.merge(df2, left_on="lkey", right_on="rkey", suffixes=("_left", "_right"))
  lkey  value_left rkey  value_right
0  foo           1  foo            5
1  foo           1  foo            8
2  bar           2  bar            6
3  baz           3  baz            7
4  foo           5  foo            5
5  foo           5  foo            8
```

Merge DataFrames df1 and df2, but raise an exception if the DataFrames have
any overlapping columns.

```pycon
>>> df1.merge(df2, left_on="lkey", right_on="rkey", suffixes=(False, False))
Traceback (most recent call last):
...
ValueError: columns overlap but no suffix specified:
    Index(['value'], dtype='str')
```

```pycon
>>> df1 = pd.DataFrame({"a": ["foo", "bar"], "b": [1, 2]})
>>> df2 = pd.DataFrame({"a": ["foo", "baz"], "c": [3, 4]})
>>> df1
      a  b
0   foo  1
1   bar  2
>>> df2
      a  c
0   foo  3
1   baz  4
```

```pycon
>>> df1.merge(df2, how="inner", on="a")
      a  b  c
0   foo  1  3
```

```pycon
>>> df1.merge(df2, how="left", on="a")
      a  b  c
0   foo  1  3.0
1   bar  2  NaN
```

```pycon
>>> df1 = pd.DataFrame({"left": ["foo", "bar"]})
>>> df2 = pd.DataFrame({"right": [7, 8]})
>>> df1
    left
0   foo
1   bar
>>> df2
    right
0   7
1   8
```

```pycon
>>> df1.merge(df2, how="cross")
   left  right
0   foo      7
1   foo      8
2   bar      7
3   bar      8
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.merge_asof.html.md

# dask.dataframe.merge_asof

### dask.dataframe.merge_asof(left: [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame) | [Series](dask.dataframe.Series.md#dask.dataframe.Series), right: [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame) | [Series](dask.dataframe.Series.md#dask.dataframe.Series), on: IndexLabel | [None](https://docs.python.org/3/library/constants.html#None) = None, left_on: IndexLabel | [None](https://docs.python.org/3/library/constants.html#None) = None, right_on: IndexLabel | [None](https://docs.python.org/3/library/constants.html#None) = None, left_index: [bool](https://docs.python.org/3/library/functions.html#bool) = False, right_index: [bool](https://docs.python.org/3/library/functions.html#bool) = False, by=None, left_by=None, right_by=None, suffixes: Suffixes = ('_x', '_y'), tolerance: [int](https://docs.python.org/3/library/functions.html#int) | [datetime.timedelta](https://docs.python.org/3/library/datetime.html#datetime.timedelta) | [None](https://docs.python.org/3/library/constants.html#None) = None, allow_exact_matches: [bool](https://docs.python.org/3/library/functions.html#bool) = True, direction: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'backward') → [DataFrame](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)

Perform a merge by key distance.

This is similar to a left-join except that we match on nearest
key rather than equal keys. Both DataFrames must be first sorted by
the merge key in ascending order before calling this function.
Sorting by any additional ‘by’ grouping columns is not required.

For each row in the left DataFrame:

> - A “backward” search selects the last row in the right DataFrame whose
>   ‘on’ key is less than or equal to the left’s key.
> - A “forward” search selects the first row in the right DataFrame whose
>   ‘on’ key is greater than or equal to the left’s key.
> - A “nearest” search selects the row in the right DataFrame whose ‘on’
>   key is closest in absolute distance to the left’s key.

Optionally match on equivalent keys with ‘by’ before searching with ‘on’.

* **Parameters:**
  **left**
  : First pandas object to merge.

  **right**
  : Second pandas object to merge.

  **on**
  : Field name to join on. Must be found in both DataFrames.
    The data MUST be in ascending order. Furthermore this must be
    a numeric column, such as datetimelike, integer, or float. `on`
    or `left_on` / `right_on` must be given.

  **left_on**
  : Field name to join on in left DataFrame. If specified, sort the left
    DataFrame by this column in ascending order before merging.

  **right_on**
  : Field name to join on in right DataFrame. If specified, sort the right
    DataFrame by this column in ascending order before merging.

  **left_index**
  : Use the index of the left DataFrame as the join key.

  **right_index**
  : Use the index of the right DataFrame as the join key.

  **by**
  : Match on these columns before performing merge operation. It is not required
    to sort by these columns.

  **left_by**
  : Field names to match on in the left DataFrame.

  **right_by**
  : Field names to match on in the right DataFrame.

  **suffixes**
  : Suffix to apply to overlapping column names in the left and right
    side, respectively.

  **tolerance**
  : Select asof tolerance within this range; must be compatible
    with the merge index.

  **allow_exact_matches**
  : - If True, allow matching with the same ‘on’ value
      (i.e. less-than-or-equal-to / greater-than-or-equal-to)
    - If False, don’t match the same ‘on’ value
      (i.e., strictly less-than / strictly greater-than).

  **direction**
  : Whether to search for prior, subsequent, or closest matches.
* **Returns:**
  DataFrame
  : A DataFrame of the two merged objects, containing all rows from the
    left DataFrame and the nearest matches from the right DataFrame.

#### SEE ALSO
[`merge`](dask.dataframe.merge.md#dask.dataframe.merge)
: Merge with a database-style join.

`merge_ordered`
: Merge with optional filling/interpolation.

### Examples

```pycon
>>> left = pd.DataFrame({"a": [1, 5, 10], "left_val": ["a", "b", "c"]})
>>> left
    a left_val
0   1        a
1   5        b
2  10        c
```

```pycon
>>> right = pd.DataFrame({"a": [1, 2, 3, 6, 7], "right_val": [1, 2, 3, 6, 7]})
>>> right
   a  right_val
0  1          1
1  2          2
2  3          3
3  6          6
4  7          7
```

```pycon
>>> pd.merge_asof(left, right, on="a")
    a left_val  right_val
0   1        a          1
1   5        b          3
2  10        c          7
```

```pycon
>>> pd.merge_asof(left, right, on="a", allow_exact_matches=False)
    a left_val  right_val
0   1        a        NaN
1   5        b        3.0
2  10        c        7.0
```

```pycon
>>> pd.merge_asof(left, right, on="a", direction="forward")
    a left_val  right_val
0   1        a        1.0
1   5        b        6.0
2  10        c        NaN
```

```pycon
>>> pd.merge_asof(left, right, on="a", direction="nearest")
    a left_val  right_val
0   1        a          1
1   5        b          6
2  10        c          7
```

We can use indexed DataFrames as well.

```pycon
>>> left = pd.DataFrame({"left_val": ["a", "b", "c"]}, index=[1, 5, 10])
>>> left
   left_val
1         a
5         b
10        c
```

```pycon
>>> right = pd.DataFrame({"right_val": [1, 2, 3, 6, 7]}, index=[1, 2, 3, 6, 7])
>>> right
   right_val
1          1
2          2
3          3
6          6
7          7
```

```pycon
>>> pd.merge_asof(left, right, left_index=True, right_index=True)
   left_val  right_val
1         a          1
5         b          3
10        c          7
```

Here is a real-world times-series example

```pycon
>>> quotes = pd.DataFrame(
...     {
...         "time": [
...             pd.Timestamp("2016-05-25 13:30:00.023"),
...             pd.Timestamp("2016-05-25 13:30:00.023"),
...             pd.Timestamp("2016-05-25 13:30:00.030"),
...             pd.Timestamp("2016-05-25 13:30:00.041"),
...             pd.Timestamp("2016-05-25 13:30:00.048"),
...             pd.Timestamp("2016-05-25 13:30:00.049"),
...             pd.Timestamp("2016-05-25 13:30:00.072"),
...             pd.Timestamp("2016-05-25 13:30:00.075"),
...         ],
...         "ticker": [
...             "GOOG",
...             "MSFT",
...             "MSFT",
...             "MSFT",
...             "GOOG",
...             "AAPL",
...             "GOOG",
...             "MSFT",
...         ],
...         "bid": [720.50, 51.95, 51.97, 51.99, 720.50, 97.99, 720.50, 52.01],
...         "ask": [720.93, 51.96, 51.98, 52.00, 720.93, 98.01, 720.88, 52.03],
...     }
... )
>>> quotes
                     time ticker     bid     ask
0 2016-05-25 13:30:00.023   GOOG  720.50  720.93
1 2016-05-25 13:30:00.023   MSFT   51.95   51.96
2 2016-05-25 13:30:00.030   MSFT   51.97   51.98
3 2016-05-25 13:30:00.041   MSFT   51.99   52.00
4 2016-05-25 13:30:00.048   GOOG  720.50  720.93
5 2016-05-25 13:30:00.049   AAPL   97.99   98.01
6 2016-05-25 13:30:00.072   GOOG  720.50  720.88
7 2016-05-25 13:30:00.075   MSFT   52.01   52.03
```

```pycon
>>> trades = pd.DataFrame(
...     {
...         "time": [
...             pd.Timestamp("2016-05-25 13:30:00.023"),
...             pd.Timestamp("2016-05-25 13:30:00.038"),
...             pd.Timestamp("2016-05-25 13:30:00.048"),
...             pd.Timestamp("2016-05-25 13:30:00.048"),
...             pd.Timestamp("2016-05-25 13:30:00.048"),
...         ],
...         "ticker": ["MSFT", "MSFT", "GOOG", "GOOG", "AAPL"],
...         "price": [51.95, 51.95, 720.77, 720.92, 98.0],
...         "quantity": [75, 155, 100, 100, 100],
...     }
... )
>>> trades
                     time ticker   price  quantity
0 2016-05-25 13:30:00.023   MSFT   51.95        75
1 2016-05-25 13:30:00.038   MSFT   51.95       155
2 2016-05-25 13:30:00.048   GOOG  720.77       100
3 2016-05-25 13:30:00.048   GOOG  720.92       100
4 2016-05-25 13:30:00.048   AAPL   98.00       100
```

By default we are taking the asof of the quotes

```pycon
>>> pd.merge_asof(trades, quotes, on="time", by="ticker")
                     time ticker   price  quantity     bid     ask
0 2016-05-25 13:30:00.023   MSFT   51.95        75   51.95   51.96
1 2016-05-25 13:30:00.038   MSFT   51.95       155   51.97   51.98
2 2016-05-25 13:30:00.048   GOOG  720.77       100  720.50  720.93
3 2016-05-25 13:30:00.048   GOOG  720.92       100  720.50  720.93
4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN
```

We only asof within 2ms between the quote time and the trade time

```pycon
>>> pd.merge_asof(
...     trades, quotes, on="time", by="ticker", tolerance=pd.Timedelta("2ms")
... )
                     time ticker   price  quantity     bid     ask
0 2016-05-25 13:30:00.023   MSFT   51.95        75   51.95   51.96
1 2016-05-25 13:30:00.038   MSFT   51.95       155     NaN     NaN
2 2016-05-25 13:30:00.048   GOOG  720.77       100  720.50  720.93
3 2016-05-25 13:30:00.048   GOOG  720.92       100  720.50  720.93
4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN
```

We only asof within 10ms between the quote time and the trade time
and we exclude exact matches on time. However *prior* data will
propagate forward

```pycon
>>> pd.merge_asof(
...     trades,
...     quotes,
...     on="time",
...     by="ticker",
...     tolerance=pd.Timedelta("10ms"),
...     allow_exact_matches=False,
... )
                     time ticker   price  quantity     bid     ask
0 2016-05-25 13:30:00.023   MSFT   51.95        75     NaN     NaN
1 2016-05-25 13:30:00.038   MSFT   51.95       155   51.97   51.98
2 2016-05-25 13:30:00.048   GOOG  720.77       100     NaN     NaN
3 2016-05-25 13:30:00.048   GOOG  720.92       100     NaN     NaN
4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.pivot_table.html.md

# dask.dataframe.pivot_table

### dask.dataframe.pivot_table(df, index, columns, values, aggfunc='mean')

Create a spreadsheet-style pivot table as a DataFrame. Target `columns`
must have category dtype to infer result’s `columns`.
`index`, `columns`, and `aggfunc` must be all scalar.
`values` can be scalar or list-like.

* **Parameters:**
  **df**

  **index**
  : column to be index

  **columns**
  : column to be columns

  **values**
  : column(s) to aggregate

  **aggfunc**
* **Returns:**
  **table**

#### SEE ALSO
[`pandas.DataFrame.pivot_table`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot_table.html#pandas.DataFrame.pivot_table)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_csv.html.md

# dask.dataframe.read_csv

### dask.dataframe.read_csv(urlpath, blocksize='default', lineterminator=None, compression='infer', sample=256000, sample_rows=10, enforce=False, assume_missing=False, storage_options=None, include_path_column=False, \*\*kwargs)

Read CSV files into a Dask.DataFrame

This parallelizes the [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv) function in the following ways:

- It supports loading many files at once using globstrings:
  ```pycon
  >>> df = dd.read_csv('myfiles.*.csv')
  ```
- In some cases it can break up large files:
  ```pycon
  >>> df = dd.read_csv('largefile.csv', blocksize=25e6)  # 25MB chunks
  ```
- It can read CSV files from external resources (e.g. S3, HDFS) by
  providing a URL:
  ```pycon
  >>> df = dd.read_csv('s3://bucket/myfiles.*.csv')
  >>> df = dd.read_csv('hdfs:///myfiles.*.csv')
  >>> df = dd.read_csv('hdfs://namenode.example.com/myfiles.*.csv')
  ```

Internally `dd.read_csv` uses [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv) and supports many of the
same keyword arguments with the same performance guarantees. See the docstring
for [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv) for more information on available keyword arguments.

* **Parameters:**
  **urlpath**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to read from alternative filesystems. To read from multiple files you
    can pass a globstring or a list of paths, with the caveat that they
    must all have the same protocol.

  **blocksize**
  : Number of bytes by which to cut up larger files. Default value is computed
    based on available physical memory and the number of cores, up to a maximum
    of 64MB. Can be a number like `64000000` or a string like `"64MB"`. If
    `None`, a single block is used for each file.

  **sample**
  : Number of bytes to use when determining dtypes

  **assume_missing**
  : If True, all integer columns that aren’t specified in `dtype` are assumed
    to contain missing values, and are converted to floats. Default is False.

  **storage_options**
  : Extra options that make sense for a particular storage connection, e.g.
    host, port, username, password, etc.

  **include_path_column**
  : Whether or not to include the path to each particular file. If True a new
    column is added to the dataframe called `path`. If str, sets new column
    name. Default is False.

  **\*\*kwargs**
  : Extra keyword arguments to forward to [`pandas.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html#pandas.read_csv).

### Notes

Dask dataframe tries to infer the `dtype` of each column by reading a sample
from the start of the file (or of the first file if it’s a glob). Usually this
works fine, but if the `dtype` is different later in the file (or in other
files) this can cause issues. For example, if all the rows in the sample had
integer dtypes, but later on there was a `NaN`, then this would error at
compute time. To fix this, you have a few options:

- Provide explicit dtypes for the offending columns using the `dtype`
  keyword. This is the recommended solution.
- Use the `assume_missing` keyword to assume that all columns inferred as
  integers contain missing values, and convert them to floats.
- Increase the size of the sample using the `sample` keyword.

It should also be noted that this function may fail if a CSV file
includes quoted strings that contain the line terminator. To get around this
you can specify `blocksize=None` to not split files into multiple partitions,
at the cost of reduced parallelism.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_fwf.html.md

# dask.dataframe.read_fwf

### dask.dataframe.read_fwf(urlpath, blocksize='default', lineterminator=None, compression='infer', sample=256000, sample_rows=10, enforce=False, assume_missing=False, storage_options=None, include_path_column=False, \*\*kwargs)

Read fixed-width files into a Dask.DataFrame

This parallelizes the [`pandas.read_fwf()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf) function in the following ways:

- It supports loading many files at once using globstrings:
  ```pycon
  >>> df = dd.read_fwf('myfiles.*.csv')
  ```
- In some cases it can break up large files:
  ```pycon
  >>> df = dd.read_fwf('largefile.csv', blocksize=25e6)  # 25MB chunks
  ```
- It can read CSV files from external resources (e.g. S3, HDFS) by
  providing a URL:
  ```pycon
  >>> df = dd.read_fwf('s3://bucket/myfiles.*.csv')
  >>> df = dd.read_fwf('hdfs:///myfiles.*.csv')
  >>> df = dd.read_fwf('hdfs://namenode.example.com/myfiles.*.csv')
  ```

Internally `dd.read_fwf` uses [`pandas.read_fwf()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf) and supports many of the
same keyword arguments with the same performance guarantees. See the docstring
for [`pandas.read_fwf()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf) for more information on available keyword arguments.

* **Parameters:**
  **urlpath**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to read from alternative filesystems. To read from multiple files you
    can pass a globstring or a list of paths, with the caveat that they
    must all have the same protocol.

  **blocksize**
  : Number of bytes by which to cut up larger files. Default value is computed
    based on available physical memory and the number of cores, up to a maximum
    of 64MB. Can be a number like `64000000` or a string like `"64MB"`. If
    `None`, a single block is used for each file.

  **sample**
  : Number of bytes to use when determining dtypes

  **assume_missing**
  : If True, all integer columns that aren’t specified in `dtype` are assumed
    to contain missing values, and are converted to floats. Default is False.

  **storage_options**
  : Extra options that make sense for a particular storage connection, e.g.
    host, port, username, password, etc.

  **include_path_column**
  : Whether or not to include the path to each particular file. If True a new
    column is added to the dataframe called `path`. If str, sets new column
    name. Default is False.

  **\*\*kwargs**
  : Extra keyword arguments to forward to [`pandas.read_fwf()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_fwf.html#pandas.read_fwf).

### Notes

Dask dataframe tries to infer the `dtype` of each column by reading a sample
from the start of the file (or of the first file if it’s a glob). Usually this
works fine, but if the `dtype` is different later in the file (or in other
files) this can cause issues. For example, if all the rows in the sample had
integer dtypes, but later on there was a `NaN`, then this would error at
compute time. To fix this, you have a few options:

- Provide explicit dtypes for the offending columns using the `dtype`
  keyword. This is the recommended solution.
- Use the `assume_missing` keyword to assume that all columns inferred as
  integers contain missing values, and convert them to floats.
- Increase the size of the sample using the `sample` keyword.

It should also be noted that this function may fail if a fixed-width file
includes quoted strings that contain the line terminator. To get around this
you can specify `blocksize=None` to not split files into multiple partitions,
at the cost of reduced parallelism.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_hdf.html.md

# dask.dataframe.read_hdf

### dask.dataframe.read_hdf(pattern, key, start=0, stop=None, columns=None, chunksize=1000000, sorted_index=False, lock=True, mode='r')

Read HDF files into a Dask DataFrame

Read hdf files into a dask dataframe. This function is like
`pandas.read_hdf`, except it can read from a single large file, or from
multiple files, or from multiple keys from the same file.

* **Parameters:**
  **pattern**
  : File pattern (string), pathlib.Path, buffer to read from, or list of
    file paths. Can contain wildcards.

  **key**

  **start**

  **stop**
  : stop at

  **columns**
  : A list of columns that if not None, will limit the return
    columns (default is None)

  **chunksize**
  : Maximal number of rows per partition (default is 1000000).

  **sorted_index**
  : Option to specify whether or not the input hdf files have a sorted
    index (default is False).

  **lock**
  : Option to use a lock to prevent concurrency issues (default is True).

  **mode**
  : ‘r’
    : Read-only; no data can be modified.
    <br/>
    ‘a’
    : Append; an existing file is opened for reading and writing,
      and if the file does not exist it is created.
    <br/>
    ‘r+’
    : It is similar to ‘a’, but the file must already exist.
* **Returns:**
  dask.DataFrame

### Examples

Load single file

```pycon
>>> dd.read_hdf('myfile.1.hdf5', '/x')
```

Load multiple files

```pycon
>>> dd.read_hdf('myfile.*.hdf5', '/x')
```

```pycon
>>> dd.read_hdf(['myfile.1.hdf5', 'myfile.2.hdf5'], '/x')
```

Load multiple datasets

```pycon
>>> dd.read_hdf('myfile.1.hdf5', '/*')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_json.html.md

# dask.dataframe.read_json

### dask.dataframe.read_json(url_path, orient='records', lines=None, storage_options=None, blocksize=None, sample=1048576, encoding='utf-8', errors='strict', compression='infer', meta=None, engine=<function read_json>, include_path_column=False, path_converter=None, \*\*kwargs)

Create a dataframe from a set of JSON files

This utilises `pandas.read_json()`, and most parameters are
passed through - see its docstring.

Differences: orient is ‘records’ by default, with lines=True; this
is appropriate for line-delimited “JSON-lines” data, the kind of JSON output
that is most common in big-data scenarios, and which can be chunked when
reading (see `read_json()`). All other options require blocksize=None,
i.e., one partition per input file.

* **Parameters:**
  **url_path: str, list of str**
  : Location to read from. If a string, can include a glob character to
    find a set of file names.
    Supports protocol specifications such as `"s3://"`.

  **encoding, errors:**
  : The text encoding to implement, e.g., “utf-8” and how to respond
    to errors in the conversion (see `str.encode()`).

  **orient, lines, kwargs**
  : passed to pandas; if not specified, lines=True when orient=’records’,
    False otherwise.

  **storage_options: dict**
  : Passed to backend file-system implementation

  **blocksize: None or int**
  : If None, files are not blocked, and you get one partition per input
    file. If int, which can only be used for line-delimited JSON files,
    each partition will be approximately this size in bytes, to the nearest
    newline character.

  **sample: int**
  : Number of bytes to pre-load, to provide an empty dataframe structure
    to any blocks without data. Only relevant when using blocksize.

  **encoding, errors:**
  : Text conversion, `see bytes.decode()`

  **compression**
  : String like ‘gzip’ or ‘xz’.

  **engine**
  : The underlying function that dask will use to read JSON files. By
    default, this will be the pandas JSON reader (`pd.read_json`).
    If a string is specified, this value will be passed under the `engine`
    key-word argument to `pd.read_json` (only supported for pandas>=2.0).

  **include_path_column**
  : Include a column with the file path where each row in the dataframe
    originated. If `True`, a new column is added to the dataframe called
    `path`. If `str`, sets new column name. Default is `False`.

  **path_converter**
  : A function that takes one argument and returns a string. Used to convert
    paths in the `path` column, for instance, to strip a common prefix from
    all the paths.

  **meta**
  : An empty `pd.DataFrame` or `pd.Series` that matches the dtypes and
    column names of the output. This metadata is necessary for many
    algorithms in dask dataframe to work.  For ease of use, some
    alternative inputs are also available. Instead of a `DataFrame`, a
    `dict` of `{name: dtype}` or iterable of `(name, dtype)` can be
    provided (note that the order of the names should match the order of
    the columns). Instead of a series, a tuple of `(name, dtype)` can be
    used. If not provided, dask will try to infer the metadata. This may
    lead to unexpected results, so providing `meta` is recommended. For
    more information, see `dask.dataframe.utils.make_meta`.
* **Returns:**
  dask.DataFrame

### Examples

Load single file

```pycon
>>> dd.read_json('myfile.1.json')
```

Load multiple files

```pycon
>>> dd.read_json('myfile.*.json')
```

```pycon
>>> dd.read_json(['myfile.1.json', 'myfile.2.json'])
```

Load large line-delimited JSON files using partitions of approx
256MB size

>> dd.read_json(‘data/file\*.csv’, blocksize=2\*\*28)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_orc.html.md

# dask.dataframe.read_orc

### dask.dataframe.read_orc(path, engine='pyarrow', columns=None, index=None, split_stripes=1, aggregate_files=None, storage_options=None)

Read dataframe from ORC file(s)

* **Parameters:**
  **path: str or list(str)**
  : Location of file(s), which can be a full URL with protocol
    specifier, and may include glob character if a single string.

  **engine: ‘pyarrow’ or ORCEngine**
  : Backend ORC engine to use for I/O. Default is “pyarrow”.

  **columns: None or list(str)**
  : Columns to load. If None, loads all.

  **index: str**
  : Column name to set as index.

  **split_stripes: int or False**
  : Maximum number of ORC stripes to include in each output-DataFrame
    partition. Use False to specify a 1-to-1 mapping between files
    and partitions. Default is 1.

  **aggregate_files**
  : Whether distinct file paths may be aggregated into the same output
    partition. A setting of True means that any two file paths may be
    aggregated into the same output partition, while False means that
    inter-file aggregation is prohibited.

  **storage_options: None or dict**
  : Further parameters to pass to the bytes backend.
* **Returns:**
  Dask.DataFrame (even if there is only one column)

### Examples

```pycon
>>> df = dd.read_orc('https://github.com/apache/orc/raw/'
...                  'master/examples/demo-11-zlib.orc')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_parquet.html.md

# dask.dataframe.read_parquet

### dask.dataframe.read_parquet(path=None, columns=None, filters=None, categories=None, index=None, storage_options=None, dtype_backend=None, calculate_divisions=False, ignore_metadata_file=False, metadata_task_size=None, split_row_groups='infer', blocksize='default', aggregate_files=None, parquet_file_extension=('.parq', '.parquet', '.pq'), filesystem='fsspec', engine=None, arrow_to_pandas=None, \*\*kwargs)

Read a Parquet file into a Dask DataFrame

This reads a directory of Parquet data into a Dask.dataframe, one file per
partition.  It selects the index among the sorted columns if any exist.

#### NOTE
Dask automatically resizes partitions to ensure that each partition is of
adequate size. The optimizer uses the ratio of selected columns to total
columns to squash multiple files into one partition.

Additionally, the Optimizer uses a minimum size per partition (default 75MB)
to avoid too many small partitions. This configuration can be set with

```pycon
>>> dask.config.set({"dataframe.parquet.minimum-partition-size": "100MB"})
```

#### NOTE
Specifying `filesystem="arrow"` leverages a complete reimplementation of
the Parquet reader that is solely based on PyArrow. It is significantly faster
than the legacy implementation, but doesn’t yet support all features.

* **Parameters:**
  **path**
  : Source directory for data, or path(s) to individual parquet files.
    Prefix with a protocol like `s3://` to read from alternative
    filesystems. To read from multiple files you can pass a globstring or a
    list of paths, with the caveat that they must all have the same
    protocol.

  **columns**
  : Field name(s) to read in as columns in the output. By default all
    non-index fields will be read (as determined by the pandas parquet
    metadata, if present). Provide a single field name instead of a list to
    read in the data as a Series.

  **filters**
  : List of filters to apply, like `[[('col1', '==', 0), ...], ...]`.
    Using this argument will result in row-wise filtering of the final partitions.
    <br/>
    Predicates can be expressed in disjunctive normal form (DNF). This means that
    the inner-most tuple describes a single column predicate. These inner predicates
    are combined with an AND conjunction into a larger predicate. The outer-most
    list then combines all of the combined filters with an OR disjunction.
    <br/>
    Predicates can also be expressed as a `List[Tuple]`. These are evaluated
    as an AND conjunction. To express OR in predicates, one must use the
    (preferred for “pyarrow”) `List[List[Tuple]]` notation.

  **index**
  : Field name(s) to use as the output frame index. By default will be
    inferred from the pandas parquet file metadata, if present. Use `False`
    to read all fields as columns.

  **categories**
  : For any fields listed here, if the parquet encoding is Dictionary,
    the column will be created with dtype category. Use only if it is
    guaranteed that the column is encoded as dictionary in all row-groups.
    If a list, assumes up to 2\*\*16-1 labels; if a dict, specify the number
    of labels expected; if None, will load categories automatically for
    data written by dask, not otherwise.

  **storage_options**
  : Key/value pairs to be passed on to the file-system backend, if any.
    Note that the default file-system backend can be configured with the
    `filesystem` argument, described below.

  **open_file_options**
  : Key/value arguments to be passed along to `AbstractFileSystem.open`
    when each parquet data file is open for reading. Experimental
    (optimized) “precaching” for remote file systems (e.g. S3, GCS) can
    be enabled by adding `{"method": "parquet"}` under the
    `"precache_options"` key. Also, a custom file-open function can be
    used (instead of `AbstractFileSystem.open`), by specifying the
    desired function under the `"open_file_func"` key.

  **dtype_backend**
  : Which dtype_backend to use, e.g. whether a DataFrame should have NumPy arrays,
    nullable dtypes are used for all dtypes that have a nullable implementation
    when ‘numpy_nullable’ is set, pyarrow is used for all dtypes if ‘pyarrow’
    is set.
    `dtype_backend="pyarrow"` requires `pandas` 1.5+.

  **calculate_divisions**
  : Whether to use min/max statistics from the footer metadata (or global
    `_metadata` file) to calculate divisions for the output DataFrame
    collection. Divisions will not be calculated if statistics are missing.
    This option will be ignored if `index` is not specified and there is
    no physical index column specified in the custom “pandas” Parquet
    metadata. Note that `calculate_divisions=True` may be extremely slow
    when no global `_metadata` file is present, especially when reading
    from remote storage. Set this to `True` only when known divisions
    are needed for your workload (see [Partitions](../dataframe-design.md#dataframe-design-partitions)).

  **ignore_metadata_file**
  : Whether to ignore the global `_metadata` file (when one is present).
    If `True`, or if the global `_metadata` file is missing, the parquet
    metadata may be gathered and processed in parallel. Parallel metadata
    processing is currently supported for `ArrowDatasetEngine` only.

  **metadata_task_size**
  : If parquet metadata is processed in parallel (see `ignore_metadata_file`
    description above), this argument can be used to specify the number of
    dataset files to be processed by each task in the Dask graph.  If this
    argument is set to `0`, parallel metadata processing will be disabled.
    The default values for local and remote filesystems can be specified
    with the “metadata-task-size-local” and “metadata-task-size-remote”
    config fields, respectively (see “dataframe.parquet”).

  **split_row_groups**
  : If True, then each output dataframe partition will correspond to a single
    parquet-file row-group. If False, each partition will correspond to a
    complete file.  If a positive integer value is given, each dataframe
    partition will correspond to that number of parquet row-groups (or fewer).
    If ‘adaptive’, the metadata of each file will be used to ensure that every
    partition satisfies `blocksize`. If ‘infer’ (the default), the
    uncompressed storage-size metadata in the first file will be used to
    automatically set `split_row_groups` to either ‘adaptive’ or `False`.

  **blocksize**
  : The desired size of each output `DataFrame` partition in terms of total
    (uncompressed) parquet storage space. This argument is currently used to
    set the default value of `split_row_groups` (using row-group metadata
    from a single file), and will be ignored if `split_row_groups` is not
    set to ‘infer’ or ‘adaptive’. Default is 256 MiB.

  **aggregate_files**
  : WARNING: Passing a string argument to `aggregate_files` will result
    in experimental behavior. This behavior may change in the future.
    <br/>
    Whether distinct file paths may be aggregated into the same output
    partition. This parameter is only used when split_row_groups is set to
    ‘infer’, ‘adaptive’ or to an integer >1. A setting of True means that any
    two file paths may be aggregated into the same output partition, while
    False means that inter-file aggregation is prohibited.
    <br/>
    For “hive-partitioned” datasets, a “partition”-column name can also be
    specified. In this case, we allow the aggregation of any two files
    sharing a file path up to, and including, the corresponding directory name.
    For example, if `aggregate_files` is set to `"section"` for the
    directory structure below, `03.parquet` and `04.parquet` may be
    aggregated together, but `01.parquet` and `02.parquet` cannot be.
    If, however, `aggregate_files` is set to `"region"`, `01.parquet`
    may be aggregated with `02.parquet`, and `03.parquet` may be aggregated
    with `04.parquet`:
    ```default
    dataset-path/
    ├── region=1/
    │   ├── section=a/
    │   │   └── 01.parquet
    │   ├── section=b/
    │   └── └── 02.parquet
    └── region=2/
        ├── section=a/
        │   ├── 03.parquet
        └── └── 04.parquet
    ```
    <br/>
    Note that the default behavior of `aggregate_files` is `False`.

  **parquet_file_extension: str, tuple[str], or None, default (“.parq”, “.parquet”, “.pq”)**
  : A file extension or an iterable of extensions to use when discovering
    parquet files in a directory. Files that don’t match these extensions
    will be ignored. This argument only applies when `paths` corresponds
    to a directory and no `_metadata` file is present (or
    `ignore_metadata_file=True`). Passing in `parquet_file_extension=None`
    will treat all files in the directory as parquet files.
    <br/>
    The purpose of this argument is to ensure that the engine will ignore
    unsupported metadata files (like Spark’s ‘_SUCCESS’ and ‘crc’ files).
    It may be necessary to change this argument if the data files in your
    parquet dataset do not end in “.parq”, “.parquet”, or “.pq”.

  **filesystem: “fsspec”, “arrow”, or fsspec.AbstractFileSystem backend to use.**

  **dataset: dict, default None**
  : Dictionary of options to use when creating a `pyarrow.dataset.Dataset` object.
    These options may include a “filesystem” key to configure the desired
    file-system backend. However, the top-level `filesystem` argument will always
    take precedence.
    <br/>
    **Note**: The `dataset` options may include a “partitioning” key.
    However, since `pyarrow.dataset.Partitioning`
    objects cannot be serialized, the value can be a dict of key-word
    arguments for the `pyarrow.dataset.partitioning` API
    (e.g. `dataset={"partitioning": {"flavor": "hive", "schema": ...}}`).
    Note that partitioned columns will not be converted to categorical
    dtypes when a custom partitioning schema is specified in this way.

  **read: dict, default None**
  : Dictionary of options to pass through to `engine.read_partitions`
    using the `read` key-word argument.

  **arrow_to_pandas: dict, default None**
  : Dictionary of options to use when converting from `pyarrow.Table` to
    a pandas `DataFrame` object. Only used by the “arrow” engine.

  **\*\*kwargs: dict (of dicts)**
  : Options to pass through to `engine.read_partitions` as stand-alone
    key-word arguments. Note that these options will be ignored by the
    engines defined in `dask.dataframe`, but may be used by other custom
    implementations.

#### SEE ALSO
[`to_parquet`](dask.dataframe.to_parquet.md#dask.dataframe.to_parquet)

[`pyarrow.parquet.ParquetDataset`](https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetDataset.html#pyarrow.parquet.ParquetDataset)

### Examples

```pycon
>>> df = dd.read_parquet('s3://bucket/my-parquet-data')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_sql.html.md

# dask.dataframe.read_sql

### dask.dataframe.read_sql(sql, con, index_col, \*\*kwargs)

Read SQL query or database table into a DataFrame.

This function is a convenience wrapper around `read_sql_table` and
`read_sql_query`. It will delegate to the specific function depending
on the provided input. A SQL query will be routed to `read_sql_query`,
while a database table name will be routed to `read_sql_table`.
Note that the delegated function might have more specific notes about
their functionality not listed here.

* **Parameters:**
  **sql**
  : Name of SQL table in database or SQL query to be executed. TextClause is not supported

  **con**
  : Full sqlalchemy URI for the database connection

  **index_col**
  : Column which becomes the index, and defines the partitioning. Should
    be a indexed column in the SQL server, and any orderable type. If the
    type is number or time, then partition boundaries can be inferred from
    `npartitions` or `bytes_per_chunk`; otherwise must supply explicit
    `divisions`.
* **Returns:**
  dask.dataframe

#### SEE ALSO
[`read_sql_table`](dask.dataframe.read_sql_table.md#dask.dataframe.read_sql_table)
: Read SQL database table into a DataFrame.

[`read_sql_query`](dask.dataframe.read_sql_query.md#dask.dataframe.read_sql_query)
: Read SQL query into a DataFrame.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_sql_query.html.md

# dask.dataframe.read_sql_query

### dask.dataframe.read_sql_query(sql, con, index_col, divisions=None, npartitions=None, limits=None, bytes_per_chunk='256 MiB', head_rows=5, meta=None, engine_kwargs=None, \*\*kwargs)

Read SQL query into a DataFrame.

If neither `divisions` or `npartitions` is given, the memory footprint of the
first few rows will be determined, and partitions of size ~256MB will
be used.

* **Parameters:**
  **sql**
  : SQL query to be executed. TextClause is not supported

  **con**
  : Full sqlalchemy URI for the database connection

  **index_col**
  : Column which becomes the index, and defines the partitioning. Should
    be a indexed column in the SQL server, and any orderable type. If the
    type is number or time, then partition boundaries can be inferred from
    `npartitions` or `bytes_per_chunk`; otherwise must supply explicit
    `divisions`.

  **divisions: sequence**
  : Values of the index column to split the table by. If given, this will
    override `npartitions` and `bytes_per_chunk`. The divisions are the value
    boundaries of the index column used to define the partitions. For
    example, `divisions=list('acegikmoqsuwz')` could be used to partition
    a string column lexicographically into 12 partitions, with the implicit
    assumption that each partition contains similar numbers of records.

  **npartitions**
  : Number of partitions, if `divisions` is not given. Will split the values
    of the index column linearly between `limits`, if given, or the column
    max/min. The index column must be numeric or time for this to work

  **limits: 2-tuple or None**
  : Manually give upper and lower range of values for use with `npartitions`;
    if None, first fetches max/min from the DB. Upper limit, if
    given, is inclusive.

  **bytes_per_chunk**
  : If both `divisions` and `npartitions` is None, this is the target size of
    each partition, in bytes

  **head_rows**
  : How many rows to load for inferring the data-types, and memory per row

  **meta**
  : If provided, do not attempt to infer dtypes, but use these, coercing
    all chunks on load

  **engine_kwargs**
  : Specific db engine parameters for sqlalchemy

  **kwargs**
  : Additional parameters to pass to pd.read_sql()
* **Returns:**
  dask.dataframe

#### SEE ALSO
[`read_sql_table`](dask.dataframe.read_sql_table.md#dask.dataframe.read_sql_table)
: Read SQL database table into a DataFrame.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_sql_table.html.md

# dask.dataframe.read_sql_table

### dask.dataframe.read_sql_table(table_name, con, index_col, divisions=None, npartitions=None, limits=None, columns=None, bytes_per_chunk='256 MiB', head_rows=5, schema=None, meta=None, engine_kwargs=None, \*\*kwargs)

Read SQL database table into a DataFrame.

If neither `divisions` or `npartitions` is given, the memory footprint of the
first few rows will be determined, and partitions of size ~256MB will
be used.

* **Parameters:**
  **table_name**
  : Name of SQL table in database.

  **con**
  : Full sqlalchemy URI for the database connection

  **index_col**
  : Column which becomes the index, and defines the partitioning. Should
    be a indexed column in the SQL server, and any orderable type. If the
    type is number or time, then partition boundaries can be inferred from
    `npartitions` or `bytes_per_chunk`; otherwise must supply explicit
    `divisions`.

  **columns**
  : Which columns to select; if None, gets all. Note can be a mix of str and SqlAlchemy columns

  **schema**
  : Pass this to sqlalchemy to select which DB schema to use within the
    URI connection

  **divisions: sequence**
  : Values of the index column to split the table by. If given, this will
    override `npartitions` and `bytes_per_chunk`. The divisions are the value
    boundaries of the index column used to define the partitions. For
    example, `divisions=list('acegikmoqsuwz')` could be used to partition
    a string column lexicographically into 12 partitions, with the implicit
    assumption that each partition contains similar numbers of records.

  **npartitions**
  : Number of partitions, if `divisions` is not given. Will split the values
    of the index column linearly between `limits`, if given, or the column
    max/min. The index column must be numeric or time for this to work

  **limits: 2-tuple or None**
  : Manually give upper and lower range of values for use with `npartitions`;
    if None, first fetches max/min from the DB. Upper limit, if
    given, is inclusive.

  **bytes_per_chunk**
  : If both `divisions` and `npartitions` is None, this is the target size of
    each partition, in bytes

  **head_rows**
  : How many rows to load for inferring the data-types, and memory per row

  **meta**
  : If provided, do not attempt to infer dtypes, but use these, coercing
    all chunks on load

  **engine_kwargs**
  : Specific db engine parameters for sqlalchemy

  **kwargs**
  : Additional parameters to pass to pd.read_sql()
* **Returns:**
  dask.dataframe

#### SEE ALSO
[`read_sql_query`](dask.dataframe.read_sql_query.md#dask.dataframe.read_sql_query)
: Read SQL query into a DataFrame.

### Examples

```pycon
>>> df = dd.read_sql_table('accounts', 'sqlite:///path/to/bank.db',
...                  npartitions=10, index_col='id')
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.read_table.html.md

# dask.dataframe.read_table

### dask.dataframe.read_table(urlpath, blocksize='default', lineterminator=None, compression='infer', sample=256000, sample_rows=10, enforce=False, assume_missing=False, storage_options=None, include_path_column=False, \*\*kwargs)

Read delimited files into a Dask.DataFrame

This parallelizes the [`pandas.read_table()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table) function in the following ways:

- It supports loading many files at once using globstrings:
  ```pycon
  >>> df = dd.read_table('myfiles.*.csv')
  ```
- In some cases it can break up large files:
  ```pycon
  >>> df = dd.read_table('largefile.csv', blocksize=25e6)  # 25MB chunks
  ```
- It can read CSV files from external resources (e.g. S3, HDFS) by
  providing a URL:
  ```pycon
  >>> df = dd.read_table('s3://bucket/myfiles.*.csv')
  >>> df = dd.read_table('hdfs:///myfiles.*.csv')
  >>> df = dd.read_table('hdfs://namenode.example.com/myfiles.*.csv')
  ```

Internally `dd.read_table` uses [`pandas.read_table()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table) and supports many of the
same keyword arguments with the same performance guarantees. See the docstring
for [`pandas.read_table()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table) for more information on available keyword arguments.

* **Parameters:**
  **urlpath**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to read from alternative filesystems. To read from multiple files you
    can pass a globstring or a list of paths, with the caveat that they
    must all have the same protocol.

  **blocksize**
  : Number of bytes by which to cut up larger files. Default value is computed
    based on available physical memory and the number of cores, up to a maximum
    of 64MB. Can be a number like `64000000` or a string like `"64MB"`. If
    `None`, a single block is used for each file.

  **sample**
  : Number of bytes to use when determining dtypes

  **assume_missing**
  : If True, all integer columns that aren’t specified in `dtype` are assumed
    to contain missing values, and are converted to floats. Default is False.

  **storage_options**
  : Extra options that make sense for a particular storage connection, e.g.
    host, port, username, password, etc.

  **include_path_column**
  : Whether or not to include the path to each particular file. If True a new
    column is added to the dataframe called `path`. If str, sets new column
    name. Default is False.

  **\*\*kwargs**
  : Extra keyword arguments to forward to [`pandas.read_table()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_table.html#pandas.read_table).

### Notes

Dask dataframe tries to infer the `dtype` of each column by reading a sample
from the start of the file (or of the first file if it’s a glob). Usually this
works fine, but if the `dtype` is different later in the file (or in other
files) this can cause issues. For example, if all the rows in the sample had
integer dtypes, but later on there was a `NaN`, then this would error at
compute time. To fix this, you have a few options:

- Provide explicit dtypes for the offending columns using the `dtype`
  keyword. This is the recommended solution.
- Use the `assume_missing` keyword to assume that all columns inferred as
  integers contain missing values, and convert them to floats.
- Increase the size of the sample using the `sample` keyword.

It should also be noted that this function may fail if a delimited file
includes quoted strings that contain the line terminator. To get around this
you can specify `blocksize=None` to not split files into multiple partitions,
at the cost of reduced parallelism.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_csv.html.md

# dask.dataframe.to_csv

### dask.dataframe.to_csv(df, filename, single_file=False, encoding='utf-8', mode='wt', name_function=None, compression=None, compute=True, scheduler=None, storage_options=None, header_first_partition_only=None, compute_kwargs=None, \*\*kwargs)

Store Dask DataFrame to CSV files

One filename per partition will be created. You can specify the
filenames in a variety of ways.

Use a globstring:

```default
>>> df.to_csv('/path/to/data/export-*.csv')
```

The \* will be replaced by the increasing sequence 0, 1, 2, …

```default
/path/to/data/export-0.csv
/path/to/data/export-1.csv
```

Use a globstring and a `name_function=` keyword argument.  The
name_function function should expect an integer and produce a string.
Strings produced by name_function must preserve the order of their
respective partition indices.

```pycon
>>> from datetime import date, timedelta
>>> def name(i):
...     return str(date(2015, 1, 1) + i * timedelta(days=1))
```

```pycon
>>> name(0)
'2015-01-01'
>>> name(15)
'2015-01-16'
```

```pycon
>>> df.to_csv('/path/to/data/export-*.csv', name_function=name)
```

```default
/path/to/data/export-2015-01-01.csv
/path/to/data/export-2015-01-02.csv
...
```

You can also provide an explicit list of paths:

```default
>>> paths = ['/path/to/data/alice.csv', '/path/to/data/bob.csv', ...]
>>> df.to_csv(paths)
```

You can also provide a directory name:

```pycon
>>> df.to_csv('/path/to/data')
```

The files will be numbered 0, 1, 2, (and so on) suffixed with ‘.part’:

```default
/path/to/data/0.part
/path/to/data/1.part
```

* **Parameters:**
  **df**
  : Data to save

  **filename**
  : Absolute or relative filepath(s). Prefix with a protocol like `s3://`
    to save to remote filesystems.

  **single_file**
  : Whether to save everything into a single CSV file. Under the
    single file mode, each partition is appended at the end of the
    specified CSV file.

  **encoding**
  : A string representing the encoding to use in the output file.

  **mode**
  : Python file mode. The default is ‘w’ (or ‘wt’), for writing
    a new file or overwriting an existing file in text mode. ‘a’
    (or ‘at’) will append to an existing file in text mode or
    create a new file if it does not already exist. See [`open()`](https://docs.python.org/3/library/functions.html#open).

  **name_function**
  : Function accepting an integer (partition index) and producing a
    string to replace the asterisk in the given filename globstring.
    Should preserve the lexicographic order of partitions. Not
    supported when `single_file` is True.

  **compression**
  : A string representing the compression to use in the output file,
    allowed values are ‘gzip’, ‘bz2’, ‘xz’,
    only used when the first argument is a filename.

  **compute**
  : If True, immediately executes. If False, returns a set of delayed
    objects, which can be computed at a later time.

  **storage_options**
  : Parameters passed on to the backend filesystem class.

  **header_first_partition_only**
  : If set to True, only write the header row in the first output
    file. By default, headers are written to all partitions under
    the multiple file mode (`single_file` is False) and written
    only once under the single file mode (`single_file` is True).
    It must be True under the single file mode.

  **compute_kwargs**
  : Options to be passed in to the compute method

  **kwargs**
  : Additional parameters to pass to [`pandas.DataFrame.to_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html#pandas.DataFrame.to_csv).
* **Returns:**
  The names of the file written if they were computed right away.

  If not, the delayed tasks associated with writing the files.
* **Raises:**
  ValueError
  : If `header_first_partition_only` is set to False or
    `name_function` is specified when `single_file` is True.

#### SEE ALSO
[`fsspec.open_files`](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open_files)

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_datetime.html.md

# dask.dataframe.to_datetime

### dask.dataframe.to_datetime(arg: DatetimeScalarOrArrayConvertible | DictConvertible, errors: DateTimeErrorChoices = 'raise', dayfirst: bool = False, yearfirst: bool = False, utc: bool = False, format: str | None = None, exact: bool | lib.NoDefault = <no_default>, unit: str | None = None, origin: str = 'unix', cache: bool = True) → DatetimeIndex | [Series](dask.dataframe.Series.md#dask.dataframe.Series) | DatetimeScalar | NaTType

Convert argument to datetime.

This function converts a scalar, array-like, [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) or
[`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)/dict-like to a pandas datetime object.

* **Parameters:**
  **arg**
  : The object to convert to a datetime. If a [`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame) is provided, the
    method expects minimally the following columns: `"year"`,
    `"month"`, `"day"`. The column “year”
    must be specified in 4-digit format.

  **errors**
  : - If `'raise'`, then invalid parsing will raise an exception.
    - If `'coerce'`, then invalid parsing will be set as `NaT`.

  **dayfirst**
  : Specify a date parse order if arg is str or is list-like.
    If `True`, parses dates with the day first, e.g. `"10/11/12"`
    is parsed as `2012-11-10`.
    <br/>
    #### WARNING
    `dayfirst=True` is not strict, but will prefer to parse
    with day first.

  **yearfirst**
  : Specify a date parse order if arg is str or is list-like.
    - If `True` parses dates with the year first, e.g.
      `"10/11/12"` is parsed as `2010-11-12`.
    - If both dayfirst and yearfirst are `True`, yearfirst is
      preceded (same as `dateutil`).
    <br/>
    #### WARNING
    `yearfirst=True` is not strict, but will prefer to parse
    with year first.

  **utc**
  : Control timezone-related parsing, localization and conversion.
    - If `True`, the function *always* returns a timezone-aware
      UTC-localized `Timestamp`, [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) or
      `DatetimeIndex`. To do this, timezone-naive inputs are
      *localized* as UTC, while timezone-aware inputs are *converted* to UTC.
    - If `False` (default), inputs will not be coerced to UTC.
      Timezone-naive inputs will remain naive, while timezone-aware ones
      will keep their time offsets. Limitations exist for mixed
      offsets (typically, daylight savings), see [Examples](#to-datetime-tz-examples) section for details.
    <br/>
    See also: pandas general documentation about [timezone conversion and
    localization](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#time-zone-handling).

  **format**
  : The strftime to parse time, e.g. `"%d/%m/%Y"`. See
    [strftime documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior) for more information on choices, though
    note that `"%f"` will parse all the way up to nanoseconds.
    You can also pass:
    - “ISO8601”, to parse any [ISO8601](https://en.wikipedia.org/wiki/ISO_8601)
      time string (not necessarily in exactly the same format);
    - “mixed”, to infer the format for each element individually. This is risky,
      and you should probably use it along with dayfirst.
    <br/>
    #### NOTE
    If a [`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame) is passed, then format has no effect.

  **exact**
  : Control how format is used:
    - If `True`, require an exact format match.
    - If `False`, allow the format to match anywhere in the target
      string.
    <br/>
    Cannot be used alongside `format='ISO8601'` or `format='mixed'`.

  **unit**
  : The unit of the arg (D,s,ms,us,ns) denote the unit, which is an
    integer or float number. This will be based off the origin.
    Example, with `unit='ms'` and `origin='unix'`, this would calculate
    the number of milliseconds to the unix epoch start.

  **origin**
  : Define the reference date. The numeric values would be parsed as number
    of units (defined by unit) since this reference date.
    - If `'unix'` (or POSIX) time; origin is set to 1970-01-01.
    - If `'julian'`, unit must be `'D'`, and origin is set to
      beginning of Julian Calendar. Julian day number `0` is assigned
      to the day starting at noon on January 1, 4713 BC.
    - If Timestamp convertible (Timestamp, dt.datetime, np.datetimt64 or date
      string), origin is set to Timestamp identified by origin.
    - If a float or integer, origin is the difference
      (in units determined by the `unit` argument) relative to 1970-01-01.

  **cache**
  : If `True`, use a cache of unique, converted dates to apply the
    datetime conversion. May produce significant speed-up when parsing
    duplicate date strings, especially ones with timezone offsets. The cache
    is only used when there are at least 50 values. The presence of
    out-of-bounds values will render the cache unusable and may slow down
    parsing.
* **Returns:**
  datetime
  : If parsing succeeded.
    Return type depends on input (types in parenthesis correspond to
    fallback in case of unsuccessful timezone or out-of-range timestamp
    parsing):
    - scalar: `Timestamp` (or [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime))
    - array-like: `DatetimeIndex` (or [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with
      [`object`](https://docs.python.org/3/library/functions.html#object) dtype containing [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime))
    - Series: [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) of `datetime64` dtype (or
      [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) of [`object`](https://docs.python.org/3/library/functions.html#object) dtype containing
      [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime))
    - DataFrame: [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) of `datetime64` dtype (or
      [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) of [`object`](https://docs.python.org/3/library/functions.html#object) dtype containing
      [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime))
* **Raises:**
  ParserError
  : When parsing a date from string fails.

  ValueError
  : When another datetime conversion error happens. For example when one
    of ‘year’, ‘month’, day’ columns is missing in a [`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame), or
    when a Timezone-aware [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime) is found in an array-like
    of mixed time offsets, and `utc=False`, or when parsing datetimes
    with mixed time zones unless `utc=True`. If parsing datetimes with mixed
    time zones, please specify `utc=True`.

#### SEE ALSO
[`DataFrame.astype`](dask.dataframe.DataFrame.astype.md#dask.dataframe.DataFrame.astype)
: Cast argument to a specified dtype.

[`to_timedelta`](dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)
: Convert argument to timedelta.

`convert_dtypes`
: Convert dtypes.

### Notes

Many input types are supported, and lead to different output types:

- **scalars** can be int, float, str, datetime object (from stdlib [`datetime`](https://docs.python.org/3/library/datetime.html#module-datetime)
  module or [`numpy`](https://numpy.org/doc/stable/reference/index.html#module-numpy)). They are converted to `Timestamp` when
  possible, otherwise they are converted to [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime).
  None/NaN/null scalars are converted to `NaT`.
- **array-like** can contain int, float, str, datetime objects. They are
  converted to `DatetimeIndex` when possible, otherwise they are
  converted to [`Index`](dask.dataframe.Index.md#dask.dataframe.Index) with [`object`](https://docs.python.org/3/library/functions.html#object) dtype, containing
  [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime). None/NaN/null entries are converted to
  `NaT` in both cases.
- **Series** are converted to [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with `datetime64`
  dtype when possible, otherwise they are converted to [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with
  [`object`](https://docs.python.org/3/library/functions.html#object) dtype, containing [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime). None/NaN/null
  entries are converted to `NaT` in both cases.
- **DataFrame/dict-like** are converted to [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with
  `datetime64` dtype. For each row a datetime is created from assembling
  the various dataframe columns. Column keys can be common abbreviations
  like [‘year’, ‘month’, ‘day’, ‘minute’, ‘second’, ‘ms’, ‘us’, ‘ns’]) or
  plurals of the same.

The following causes are responsible for [`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime) objects
being returned (possibly inside an [`Index`](dask.dataframe.Index.md#dask.dataframe.Index) or a [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with
[`object`](https://docs.python.org/3/library/functions.html#object) dtype) instead of a proper pandas designated type
(`Timestamp`, `DatetimeIndex` or [`Series`](dask.dataframe.Series.md#dask.dataframe.Series)
with `datetime64` dtype):

- when any input element is before `Timestamp.min` or after
  `Timestamp.max`, see [timestamp limitations](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-timestamp-limits).
- when `utc=False` (default) and the input is an array-like or
  [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) containing mixed naive/aware datetime, or aware with mixed
  time offsets. Note that this happens in the (quite frequent) situation when
  the timezone has a daylight savings policy. In that case you may wish to
  use `utc=True`.

### Examples

**Handling various input formats**

Assembling a datetime from multiple columns of a [`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame). The keys
can be common abbreviations like [‘year’, ‘month’, ‘day’, ‘minute’, ‘second’,
‘ms’, ‘us’, ‘ns’]) or plurals of the same

```pycon
>>> df = pd.DataFrame({"year": [2015, 2016], "month": [2, 3], "day": [4, 5]})
>>> pd.to_datetime(df)
0   2015-02-04
1   2016-03-05
dtype: datetime64[us]
```

Using a unix epoch time

```pycon
>>> pd.to_datetime(1490195805, unit="s")
Timestamp('2017-03-22 15:16:45')
>>> pd.to_datetime(1490195805433502912, unit="ns")
Timestamp('2017-03-22 15:16:45.433502912')
```

#### WARNING
For float arg, precision rounding might happen. To prevent
unexpected behavior use a fixed-width exact type.

Using a non-unix epoch origin

```pycon
>>> pd.to_datetime([1, 2, 3], unit="D", origin=pd.Timestamp("1960-01-01"))
DatetimeIndex(['1960-01-02', '1960-01-03', '1960-01-04'],
              dtype='datetime64[s]', freq=None)
```

**Differences with strptime behavior**

`"%f"` will parse all the way up to nanoseconds.

```pycon
>>> pd.to_datetime("2018-10-26 12:00:00.0000000011", format="%Y-%m-%d %H:%M:%S.%f")
Timestamp('2018-10-26 12:00:00.000000001')
```

**Non-convertible date/times**

Passing `errors='coerce'` will force an out-of-bounds date to `NaT`,
in addition to forcing non-dates (or non-parseable dates) to `NaT`.

```pycon
>>> pd.to_datetime("invalid for Ymd", format="%Y%m%d", errors="coerce")
NaT
```

<a id="to-datetime-tz-examples"></a>

**Timezones and time offsets**

The default behaviour (`utc=False`) is as follows:

- Timezone-naive inputs are converted to timezone-naive `DatetimeIndex`:

```pycon
>>> pd.to_datetime(["2018-10-26 12:00:00", "2018-10-26 13:00:15"])
DatetimeIndex(['2018-10-26 12:00:00', '2018-10-26 13:00:15'],
              dtype='datetime64[us]', freq=None)
```

- Timezone-aware inputs *with constant time offset* are converted to
  timezone-aware `DatetimeIndex`:

```pycon
>>> pd.to_datetime(["2018-10-26 12:00 -0500", "2018-10-26 13:00 -0500"])
DatetimeIndex(['2018-10-26 12:00:00-05:00', '2018-10-26 13:00:00-05:00'],
              dtype='datetime64[us, UTC-05:00]', freq=None)
```

- However, timezone-aware inputs *with mixed time offsets* (for example
  issued from a timezone with daylight savings, such as Europe/Paris)
  are **not successfully converted** to a `DatetimeIndex`.
  Parsing datetimes with mixed time zones will raise a ValueError unless
  `utc=True`:

```pycon
>>> pd.to_datetime(
...     ["2020-10-25 02:00 +0200", "2020-10-25 04:00 +0100"]
... )
ValueError: Mixed timezones detected. Pass utc=True in to_datetime
or tz='UTC' in DatetimeIndex to convert to a common timezone.
```

- To create a [`Series`](dask.dataframe.Series.md#dask.dataframe.Series) with mixed offsets and `object` dtype, please use
  [`Series.apply()`](dask.dataframe.Series.apply.md#dask.dataframe.Series.apply) and `datetime.datetime.strptime()`:

```pycon
>>> import datetime as dt
>>> ser = pd.Series(["2020-10-25 02:00 +0200", "2020-10-25 04:00 +0100"])
>>> ser.apply(lambda x: dt.datetime.strptime(x, "%Y-%m-%d %H:%M %z"))
0    2020-10-25 02:00:00+02:00
1    2020-10-25 04:00:00+01:00
dtype: object
```

- A mix of timezone-aware and timezone-naive inputs will also raise a ValueError
  unless `utc=True`:

```pycon
>>> from datetime import datetime
>>> pd.to_datetime(
...     ["2020-01-01 01:00:00-01:00", datetime(2020, 1, 1, 3, 0)]
... )
ValueError: Mixed timezones detected. Pass utc=True in to_datetime
or tz='UTC' in DatetimeIndex to convert to a common timezone.
```

<br/>

Setting `utc=True` solves most of the above issues:

- Timezone-naive inputs are *localized* as UTC

```pycon
>>> pd.to_datetime(["2018-10-26 12:00", "2018-10-26 13:00"], utc=True)
DatetimeIndex(['2018-10-26 12:00:00+00:00', '2018-10-26 13:00:00+00:00'],
              dtype='datetime64[us, UTC]', freq=None)
```

- Timezone-aware inputs are *converted* to UTC (the output represents the
  exact same datetime, but viewed from the UTC time offset +00:00).

```pycon
>>> pd.to_datetime(["2018-10-26 12:00 -0530", "2018-10-26 12:00 -0500"], utc=True)
DatetimeIndex(['2018-10-26 17:30:00+00:00', '2018-10-26 17:00:00+00:00'],
              dtype='datetime64[us, UTC]', freq=None)
```

- Inputs can contain both string or datetime, the above
  rules still apply

```pycon
>>> pd.to_datetime(["2018-10-26 12:00", datetime(2020, 1, 1, 18)], utc=True)
DatetimeIndex(['2018-10-26 12:00:00+00:00', '2020-01-01 18:00:00+00:00'],
              dtype='datetime64[us, UTC]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_hdf.html.md

# dask.dataframe.to_hdf

### dask.dataframe.to_hdf(df, path, key, mode='a', append=False, scheduler=None, name_function=None, compute=True, lock=None, dask_kwargs=None, \*\*kwargs)

Store Dask Dataframe to Hierarchical Data Format (HDF) files

This is a parallel version of the Pandas function of the same name.  Please
see the Pandas docstring for more detailed information about shared keyword
arguments.

This function differs from the Pandas version by saving the many partitions
of a Dask DataFrame in parallel, either to many files, or to many datasets
within the same file.  You may specify this parallelism with an asterix
`*` within the filename or datapath, and an optional `name_function`.
The asterix will be replaced with an increasing sequence of integers
starting from `0` or with the result of calling `name_function` on each
of those integers.

This function only supports the Pandas `'table'` format, not the more
specialized `'fixed'` format.

* **Parameters:**
  **path**
  : Path to a target filename. Supports strings, `pathlib.Path`, or any
    object implementing the `__fspath__` protocol. May contain a `*` to
    denote many filenames.

  **key**
  : Datapath within the files.  May contain a `*` to denote many locations

  **name_function**
  : A function to convert the `*` in the above options to a string.
    Should take in a number from 0 to the number of partitions and return a
    string. (see examples below)

  **compute**
  : Whether or not to execute immediately.  If False then this returns a
    `dask.Delayed` value.

  **lock**
  : Lock to use to prevent concurrency issues.  By default a
    `threading.Lock`, `multiprocessing.Lock` or `SerializableLock`
    will be used depending on your scheduler if a lock is required. See
    dask.utils.get_scheduler_lock for more information about lock
    selection.

  **scheduler**
  : The scheduler to use, like “threads” or “processes”

  **\*\*other:**
  : See pandas.to_hdf for more information
* **Returns:**
  **filenames**
  : Returned if `compute` is True. List of file names that each partition
    is saved to.

  **delayed**
  : Returned if `compute` is False. Delayed object to execute `to_hdf`
    when computed.

#### SEE ALSO
[`read_hdf`](dask.dataframe.read_hdf.md#dask.dataframe.read_hdf)

[`to_parquet`](dask.dataframe.to_parquet.md#dask.dataframe.to_parquet)

### Examples

Save Data to a single file

```pycon
>>> df.to_hdf('output.hdf', '/data')
```

Save data to multiple datapaths within the same file:

```pycon
>>> df.to_hdf('output.hdf', '/data-*')
```

Save data to multiple files:

```pycon
>>> df.to_hdf('output-*.hdf', '/data')
```

Save data to multiple files, using the multiprocessing scheduler:

```pycon
>>> df.to_hdf('output-*.hdf', '/data', scheduler='processes')
```

Specify custom naming scheme.  This writes files as
‘2000-01-01.hdf’, ‘2000-01-02.hdf’, ‘2000-01-03.hdf’, etc..

```pycon
>>> from datetime import date, timedelta
>>> base = date(year=2000, month=1, day=1)
>>> def name_function(i):
...     ''' Convert integer 0 to n to a string '''
...     return base + timedelta(days=i)
```

```pycon
>>> df.to_hdf('*.hdf', '/data', name_function=name_function)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_json.html.md

# dask.dataframe.to_json

### dask.dataframe.to_json(df, url_path, orient='records', lines=None, storage_options=None, compute=True, encoding='utf-8', errors='strict', compression=None, compute_kwargs=None, name_function=None, \*\*kwargs)

Write dataframe into JSON text files

This utilises `pandas.DataFrame.to_json()`, and most parameters are
passed through - see its docstring.

Differences: orient is ‘records’ by default, with lines=True; this
produces the kind of JSON output that is most common in big-data
applications, and which can be chunked when reading (see `read_json()`).

* **Parameters:**
  **df: dask.DataFrame**
  : Data to save

  **url_path: str, list of str**
  : Location to write to. If a string, and there are more than one
    partitions in df, should include a glob character to expand into a
    set of file names, or provide a `name_function=` parameter.
    Supports protocol specifications such as `"s3://"`.

  **encoding, errors:**
  : The text encoding to implement, e.g., “utf-8” and how to respond
    to errors in the conversion (see `str.encode()`).

  **orient, lines, kwargs**
  : passed to pandas; if not specified, lines=True when orient=’records’,
    False otherwise.

  **storage_options: dict**
  : Passed to backend file-system implementation

  **compute: bool**
  : If true, immediately executes. If False, returns a set of delayed
    objects, which can be computed at a later time.

  **compute_kwargs**
  : Options to be passed in to the compute method

  **compression**
  : String like ‘gzip’ or ‘xz’.

  **name_function**
  : Function accepting an integer (partition index) and producing a
    string to replace the asterisk in the given filename globstring.
    Should preserve the lexicographic order of partitions.

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_numeric.html.md

# dask.dataframe.to_numeric

### dask.dataframe.to_numeric(arg, errors='raise', downcast=None, meta=None)

Convert argument to a numeric type.

This docstring was copied from pandas.to_numeric.

Some inconsistencies with the Dask version may exist.

Return type depends on input. Delayed if scalar, otherwise same as input.
For errors, only “raise” and “coerce” are allowed.

If the input is already of a numeric dtype, the dtype will be preserved.
For non-numeric inputs, the default return dtype is float64 or int64
depending on the data supplied. Use the downcast parameter
to obtain other dtypes.

Please note that precision loss may occur if really large numbers
are passed in. Due to the internal limitations of ndarray, if
numbers smaller than -9223372036854775808 (np.iinfo(np.int64).min)
or larger than 18446744073709551615 (np.iinfo(np.uint64).max) are
passed in, it is very likely they will be converted to float so that
they can be stored in an ndarray. These warnings apply similarly to
Series since it internally leverages ndarray.

* **Parameters:**
  **arg**
  : Argument to be converted.

  **errors**
  : - If ‘raise’, then invalid parsing will raise an exception.
    - If ‘coerce’, then invalid parsing will be set as NaN.

  **downcast**
  : Can be ‘integer’, ‘signed’, ‘unsigned’, or ‘float’.
    If not None, and if the data has been successfully cast to a
    numerical dtype (or if the data was numeric to begin with),
    downcast that resulting data to the smallest numerical dtype
    possible according to the following rules:
    - ‘integer’ or ‘signed’: smallest signed int dtype (min.: np.int8)
    - ‘unsigned’: smallest unsigned int dtype (min.: np.uint8)
    - ‘float’: smallest float dtype (min.: np.float32)
    <br/>
    As this behaviour is separate from the core conversion to
    numeric values, any errors raised during the downcasting
    will be surfaced regardless of the value of the ‘errors’ input.
    <br/>
    In addition, downcasting will only occur if the size
    of the resulting data’s dtype is strictly larger than
    the dtype it is to be cast to, so if none of the dtypes
    checked satisfy that specification, no downcasting will be
    performed on the data.

  **dtype_backend**
  : Back-end data type applied to the resultant [`DataFrame`](dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)
    (still experimental). If not specified, the default behavior
    is to not use nullable data types. If specified, the behavior
    is as follows:
    * `"numpy_nullable"`: returns nullable-dtype-backed object
    * `"pyarrow"`: returns with pyarrow-backed nullable object
    <br/>
    #### Versionadded
    Added in version 2.0.
* **Returns:**
  ret
  : Numeric if parsing succeeded.
    Return type depends on input.  Series if Series, otherwise ndarray.
* **Raises:**
  ValueError
  : If the input contains non-numeric values and errors=’raise’.

  TypeError
  : If the input is not list-like, 1D, or scalar convertible to numeric,
    such as nested lists or unsupported input types (e.g., dict).

#### SEE ALSO
[`DataFrame.astype`](dask.dataframe.DataFrame.astype.md#dask.dataframe.DataFrame.astype)
: Cast argument to a specified dtype.

[`to_datetime`](dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)
: Convert argument to datetime.

[`to_timedelta`](dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)
: Convert argument to timedelta.

[`numpy.ndarray.astype`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html#numpy.ndarray.astype)
: Cast a numpy array to a specified type.

`DataFrame.convert_dtypes`
: Convert dtypes.

### Examples

Take separate series and convert to numeric, coercing when told to

```pycon
>>> s = pd.Series(["1.0", "2", -3])
>>> pd.to_numeric(s)
0    1.0
1    2.0
2   -3.0
dtype: float64
>>> pd.to_numeric(s, downcast="float")
0    1.0
1    2.0
2   -3.0
dtype: float32
>>> pd.to_numeric(s, downcast="signed")
0    1
1    2
2   -3
dtype: int8
>>> s = pd.Series(["apple", "1.0", "2", -3])
>>> pd.to_numeric(s, errors="coerce")
0    NaN
1    1.0
2    2.0
3   -3.0
dtype: float64
```

Downcasting of nullable integer and floating dtypes is supported:

```pycon
>>> s = pd.Series([1, 2, 3], dtype="Int64")
>>> pd.to_numeric(s, downcast="integer")
0    1
1    2
2    3
dtype: Int8
>>> s = pd.Series([1.0, 2.1, 3.0], dtype="Float64")
>>> pd.to_numeric(s, downcast="float")
0    1.0
1    2.1
2    3.0
dtype: Float32
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_orc.html.md

# dask.dataframe.to_orc

### dask.dataframe.to_orc(df, path, engine='pyarrow', write_index=True, storage_options=None, compute=True, compute_kwargs=None)

Store Dask.dataframe to ORC files

* **Parameters:**
  **df**

  **path**
  : Destination directory for data.  Prepend with protocol like `s3://`
    or `hdfs://` for remote data.

  **engine: ‘pyarrow’ or ORCEngine**
  : Backend ORC engine to use for I/O. Default is “pyarrow”.

  **write_index**
  : Whether or not to write the index. Defaults to True.

  **storage_options**
  : Key/value pairs to be passed on to the file-system backend, if any.

  **compute**
  : If True (default) then the result is computed immediately. If False
    then a `dask.delayed` object is returned for future computation.

  **compute_kwargs**
  : Options to be passed in to the compute method

#### SEE ALSO
[`read_orc`](dask.dataframe.read_orc.md#dask.dataframe.read_orc)
: Read ORC data to dask.dataframe

### Notes

Each partition will be written to a separate file.

### Examples

```pycon
>>> df = dd.read_csv(...)
>>> df.to_orc('/path/to/output/', ...)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_parquet.html.md

# dask.dataframe.to_parquet

### dask.dataframe.to_parquet(df, path, compression='snappy', write_index=True, append=False, overwrite=False, ignore_divisions=False, partition_on=None, storage_options=None, custom_metadata=None, write_metadata_file=None, compute=True, compute_kwargs=None, schema='infer', name_function=None, filesystem=None, engine=None, \*\*kwargs)

Store Dask.dataframe to Parquet files

* **Parameters:**
  **df**

  **path**
  : Destination directory for data.  Prepend with protocol like `s3://`
    or `hdfs://` for remote data.

  **compression**
  : Either a string like `"snappy"` or a dictionary mapping column names
    to compressors like `{"name": "gzip", "values": "snappy"}`. Defaults
    to `"snappy"`.

  **write_index**
  : Whether or not to write the index. Defaults to True.

  **append**
  : If False (default), construct data-set from scratch. If True, add new
    row-group(s) to an existing data-set. In the latter case, the data-set
    must exist, and the schema must match the input data.

  **overwrite**
  : Whether or not to remove the contents of path before writing the dataset.
    The default is False.  If True, the specified path must correspond to
    a directory (but not the current working directory).  This option cannot
    be set to True if append=True.
    NOTE: overwrite=True will remove the original data even if the current
    write operation fails.  Use at your own risk.

  **ignore_divisions**
  : If False (default) raises error when previous divisions overlap with
    the new appended divisions. Ignored if append=False.

  **partition_on**
  : Construct directory-based partitioning by splitting on these fields’
    values. Each dask partition will result in one or more datafiles,
    there will be no global groupby.

  **storage_options**
  : Key/value pairs to be passed on to the file-system backend, if any.

  **custom_metadata**
  : Custom key/value metadata to include in all footer metadata (and
    in the global “_metadata” file, if applicable).  Note that the custom
    metadata may not contain the reserved b”pandas” key.

  **write_metadata_file**
  : Whether to write the special `_metadata` file. If `None` (the
    default), a `_metadata` file will only be written if `append=True`
    and the dataset already has a `_metadata` file.

  **compute**
  : If `True` (default) then the result is computed immediately. If
    `False` then a `dask.dataframe.Scalar` object is returned for
    future computation.

  **compute_kwargs**
  : Options to be passed in to the compute method

  **schema**
  : Global schema to use for the output dataset. Defaults to “infer”, which
    will infer the schema from the dask dataframe metadata. This is usually
    sufficient for common schemas, but notably will fail for `object`
    dtype columns that contain things other than strings. These columns
    will require an explicit schema be specified. The schema for a subset
    of columns can be overridden by passing in a dict of column names to
    pyarrow types (for example `schema={"field": pa.string()}`); columns
    not present in this dict will still be automatically inferred.
    Alternatively, a full `pyarrow.Schema` may be passed, in which case
    no schema inference will be done. Passing in `schema=None` will
    disable the use of a global file schema - each written file may use a
    different schema dependent on the dtypes of the corresponding
    partition.

  **name_function**
  : Function to generate the filename for each output partition.
    The function should accept an integer (partition index) as input and
    return a string which will be used as the filename for the corresponding
    partition. Should preserve the lexicographic order of partitions.
    If not specified, files will created using the convention
    `part.0.parquet`, `part.1.parquet`, `part.2.parquet`, …
    and so on for each partition in the DataFrame.

  **filesystem: “fsspec”, “arrow”, or fsspec.AbstractFileSystem backend to use.**

  **\*\*kwargs**
  : Extra options to be passed on to the specific backend.

#### SEE ALSO
[`read_parquet`](dask.dataframe.read_parquet.md#dask.dataframe.read_parquet)
: Read parquet data to dask.dataframe

### Notes

Each partition will be written to a separate file.

### Examples

```pycon
>>> df = dd.read_csv(...)
>>> df.to_parquet('/path/to/output/', ...)
```

By default, files will be created in the specified output directory using the
convention `part.0.parquet`, `part.1.parquet`, `part.2.parquet`, … and so on for
each partition in the DataFrame. To customize the names of each file, you can use the
`name_function=` keyword argument. The function passed to `name_function` will be
used to generate the filename for each partition and should expect a partition’s index
integer as input and return a string which will be used as the filename for the corresponding
partition. Strings produced by `name_function` must preserve the order of their respective
partition indices.

For example:

```pycon
>>> name_function = lambda x: f"data-{x}.parquet"
>>> df.to_parquet('/path/to/output/', name_function=name_function)
```

will result in the following files being created:

```default
/path/to/output/
    ├── data-0.parquet
    ├── data-1.parquet
    ├── data-2.parquet
    └── ...
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_records.html.md

# dask.dataframe.to_records

### dask.dataframe.to_records(df)

Create Dask Array from a Dask Dataframe

Warning: This creates a dask.array without precise shape information.
Operations that depend on shape information, like slicing or reshaping,
will not work.

#### SEE ALSO
[`dask.dataframe.DataFrame.values`](dask.dataframe.DataFrame.values.md#dask.dataframe.DataFrame.values)

[`dask.dataframe.from_dask_array`](dask.dataframe.from_dask_array.md#dask.dataframe.from_dask_array)

### Examples

```pycon
>>> df.to_records()
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_sql.html.md

# dask.dataframe.to_sql

### dask.dataframe.to_sql(df, name: [str](https://docs.python.org/3/library/stdtypes.html#str), uri: [str](https://docs.python.org/3/library/stdtypes.html#str), schema=None, if_exists: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'fail', index: [bool](https://docs.python.org/3/library/functions.html#bool) = True, index_label=None, chunksize=None, dtype=None, method=None, compute=True, parallel=False, engine_kwargs=None)

Store Dask Dataframe to a SQL table

An empty table is created based on the “meta” DataFrame (and conforming to the caller’s “if_exists” preference), and
then each block calls pd.DataFrame.to_sql (with if_exists=”append”).

Databases supported by SQLAlchemy [[1]](#r62277b28829f-1) are supported. Tables can be
newly created, appended to, or overwritten.

* **Parameters:**
  **name**
  : Name of SQL table.

  **uri**
  : Full sqlalchemy URI for the database connection

  **schema**
  : Specify the schema (if database flavor supports this). If None, use
    default schema.

  **if_exists**
  : How to behave if the table already exists.
    * fail: Raise a ValueError.
    * replace: Drop the table before inserting new values.
    * append: Insert new values to the existing table.

  **index**
  : Write DataFrame index as a column. Uses index_label as the column
    name in the table.

  **index_label**
  : Column label for index column(s). If None is given (default) and
    index is True, then the index names are used.
    A sequence should be given if the DataFrame uses MultiIndex.

  **chunksize**
  : Specify the number of rows in each batch to be written at a time.
    By default, all rows will be written at once.

  **dtype**
  : Specifying the datatype for columns. If a dictionary is used, the
    keys should be the column names and the values should be the
    SQLAlchemy types or strings for the sqlite3 legacy mode. If a
    scalar is provided, it will be applied to all columns.

  **method**
  : Controls the SQL insertion clause used:
    * None : Uses standard SQL `INSERT` clause (one per row).
    * ‘multi’: Pass multiple values in a single `INSERT` clause.
    * callable with signature `(pd_table, conn, keys, data_iter)`.
    <br/>
    Details and a sample callable implementation can be found in the
    section [insert method](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-sql-method).

  **compute**
  : When true, call dask.compute and perform the load into SQL; otherwise, return a Dask object (or array of
    per-block objects when parallel=True)

  **parallel**
  : When true, have each block append itself to the DB table concurrently. This can result in DB rows being in a
    different order than the source DataFrame’s corresponding rows. When false, load each block into the SQL DB in
    sequence.

  **engine_kwargs**
  : Specific db engine parameters for sqlalchemy
* **Raises:**
  ValueError
  : When the table already exists and if_exists is ‘fail’ (the
    default).

#### SEE ALSO
[`read_sql`](dask.dataframe.read_sql.md#dask.dataframe.read_sql)
: Read a DataFrame from a table.

### Notes

Timezone aware datetime columns will be written as
`Timestamp with timezone` type with SQLAlchemy if supported by the
database. Otherwise, the datetimes will be stored as timezone unaware
timestamps local to the original timezone.

#### Versionadded
Added in version 0.24.0.

### References

### Examples

Create a table from scratch with 4 rows.

```pycon
>>> import pandas as pd
>>> import dask.dataframe as dd
>>> df = pd.DataFrame([ {'i':i, 's':str(i)*2 } for i in range(4) ])
>>> ddf = dd.from_pandas(df, npartitions=2)
>>> ddf
Dask DataFrame Structure:
                   i       s
npartitions=2
0              int64  object
2                ...     ...
3                ...     ...
Dask Name: from_pandas, 2 tasks
```

```pycon
>>> from dask.utils import tmpfile
>>> from sqlalchemy import create_engine, text
>>> with tmpfile() as f:
...     db = 'sqlite:///%s' %f
...     ddf.to_sql('test', db)
...     engine = create_engine(db, echo=False)
...     with engine.connect() as conn:
...         result = conn.execute(text("SELECT * FROM test")).fetchall()
>>> result
[(0, 0, '00'), (1, 1, '11'), (2, 2, '22'), (3, 3, '33')]
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.to_timedelta.html.md

# dask.dataframe.to_timedelta

### dask.dataframe.to_timedelta(arg: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | timedelta | [list](https://docs.python.org/3/library/stdtypes.html#list) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) | [range](https://docs.python.org/3/library/stdtypes.html#range) | ArrayLike | [Index](dask.dataframe.Index.md#dask.dataframe.Index) | [Series](dask.dataframe.Series.md#dask.dataframe.Series), unit: UnitChoices | [None](https://docs.python.org/3/library/constants.html#None) = None, errors: DateTimeErrorChoices = 'raise') → Timedelta | TimedeltaIndex | [Series](dask.dataframe.Series.md#dask.dataframe.Series) | NaTType | Any

Convert argument to timedelta.

Timedeltas are absolute differences in times, expressed in difference
units (e.g. days, hours, minutes, seconds). This method converts
an argument from a recognized timedelta format / value into
a Timedelta type.

* **Parameters:**
  **arg**
  : The data to be converted to timedelta.
    <br/>
    #### Versionchanged
    Changed in version 2.0: Strings with units ‘M’, ‘Y’ and ‘y’ do not represent
    unambiguous timedelta values and will raise an exception.

  **unit**
  : Denotes the unit of the arg for numeric arg. Defaults to `"ns"`.
    <br/>
    Possible values:
    * ‘W’
    * ‘D’ / ‘days’ / ‘day’
    * ‘hours’ / ‘hour’ / ‘hr’ / ‘h’
    * ‘m’ / ‘minute’ / ‘min’ / ‘minutes’
    * ‘s’ / ‘seconds’ / ‘sec’ / ‘second’
    * ‘ms’ / ‘milliseconds’ / ‘millisecond’ / ‘milli’ / ‘millis’
    * ‘us’ / ‘microseconds’ / ‘microsecond’ / ‘micro’ / ‘micros’
    * ‘ns’ / ‘nanoseconds’ / ‘nano’ / ‘nanos’ / ‘nanosecond’
    <br/>
    Must not be specified when arg contains strings and `errors="raise"`.

  **errors**
  : - If ‘raise’, then invalid parsing will raise an exception.
    - If ‘coerce’, then invalid parsing will be set as NaT.
* **Returns:**
  timedelta
  : If parsing succeeded.
    Return type depends on input:
    - list-like: TimedeltaIndex of timedelta64 dtype
    - Series: Series of timedelta64 dtype
    - scalar: Timedelta

#### SEE ALSO
[`DataFrame.astype`](dask.dataframe.DataFrame.astype.md#dask.dataframe.DataFrame.astype)
: Cast argument to a specified dtype.

[`to_datetime`](dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)
: Convert argument to datetime.

`convert_dtypes`
: Convert dtypes.

### Notes

If the precision is higher than nanoseconds, the precision of the duration is
truncated to nanoseconds for string inputs.

### Examples

Parsing a single string to a Timedelta:

```pycon
>>> pd.to_timedelta("1 days 06:05:01.00003")
Timedelta('1 days 06:05:01.000030')
>>> pd.to_timedelta("15.5us")
Timedelta('0 days 00:00:00.000015500')
```

Parsing a list or array of strings:

```pycon
>>> pd.to_timedelta(["1 days 06:05:01.00003", "15.5us", "nan"])
TimedeltaIndex(['1 days 06:05:01.000030', '0 days 00:00:00.000015500', NaT],
               dtype='timedelta64[ns]', freq=None)
```

Converting numbers by specifying the unit keyword argument:

```pycon
>>> pd.to_timedelta(np.arange(5), unit="s")
TimedeltaIndex(['0 days 00:00:00', '0 days 00:00:01', '0 days 00:00:02',
                '0 days 00:00:03', '0 days 00:00:04'],
               dtype='timedelta64[s]', freq=None)
>>> pd.to_timedelta(np.arange(5), unit="D")
TimedeltaIndex(['0 days', '1 days', '2 days', '3 days', '4 days'],
               dtype='timedelta64[s]', freq=None)
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.agg.html.md

# dask.dataframe.tseries.resample.Resampler.agg

#### Resampler.agg(func, \*args, \*\*kwargs)

Aggregate using one or more operations over the specified axis.

This docstring was copied from pandas.api.typing.Resampler.agg.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **func**
  : Function to use for aggregating the data. If a function, must either
    work when passed a DataFrame or when passed to DataFrame.apply.
    <br/>
    Accepted combinations are:
    - function
    - string function name
    - list of functions and/or function names, e.g. `[np.sum, 'mean']`
    - dict of axis labels -> functions, function names or list of such.

  **\*args**
  : Positional arguments to pass to func.

  **\*\*kwargs**
  : Keyword arguments to pass to func.
* **Returns:**
  scalar, Series or DataFrame
  : The return can be:
    * scalar : when Series.agg is called with single function
    * Series : when DataFrame.agg is called with a single function
    * DataFrame : when DataFrame.agg is called with several functions

#### SEE ALSO
`DataFrame.groupby.aggregate`
: Aggregate using callable, string, dict, or list of string/callables.

`DataFrame.resample.transform`
: Transforms the Series on each group based on the given function.

`DataFrame.aggregate`
: Aggregate using one or more operations over the specified axis.

### Notes

The aggregation operations are always performed over an axis, either the
index (default) or the column axis. This behavior is different from
numpy aggregation functions (mean, median, prod, sum, std,
var), where the default is to compute the aggregation of the flattened
array, e.g., `numpy.mean(arr_2d)` as opposed to
`numpy.mean(arr_2d, axis=0)`.

agg is an alias for aggregate. Use the alias.

Functions that mutate the passed object can produce unexpected
behavior or errors and are not supported. See [Mutating with User Defined Function (UDF) methods](https://pandas.pydata.org/pandas-docs/stable/user_guide/gotchas.html#gotchas-udf-mutation)
for more details.

A passed user-defined-function will be passed a Series for evaluation.

If `func` defines an index relabeling, `axis` must be `0` or `index`.

### Examples

```pycon
>>> s = pd.Series(
...     [1, 2, 3, 4, 5], index=pd.date_range("20130101", periods=5, freq="s")
... )
>>> s
2013-01-01 00:00:00    1
2013-01-01 00:00:01    2
2013-01-01 00:00:02    3
2013-01-01 00:00:03    4
2013-01-01 00:00:04    5
Freq: s, dtype: int64
```

```pycon
>>> r = s.resample("2s")
```

```pycon
>>> r.agg("sum")
2013-01-01 00:00:00    3
2013-01-01 00:00:02    7
2013-01-01 00:00:04    5
Freq: 2s, dtype: int64
```

```pycon
>>> r.agg(["sum", "mean", "max"])
                    sum  mean  max
2013-01-01 00:00:00    3   1.5    2
2013-01-01 00:00:02    7   3.5    4
2013-01-01 00:00:04    5   5.0    5
```

```pycon
>>> r.agg({"result": lambda x: x.mean() / x.std(), "total": "sum"})
                    result  total
2013-01-01 00:00:00  2.121320      3
2013-01-01 00:00:02  4.949747      7
2013-01-01 00:00:04       NaN      5
```

```pycon
>>> r.agg(average="mean", total="sum")
                        average  total
2013-01-01 00:00:00      1.5      3
2013-01-01 00:00:02      3.5      7
2013-01-01 00:00:04      5.0      5
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.count.html.md

# dask.dataframe.tseries.resample.Resampler.count

#### Resampler.count()

Compute count of group, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.count.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series or DataFrame
  : Count of values within each group.

#### SEE ALSO
`Series.groupby`
: Apply a function groupby to a Series.

`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").count()
2023-01-01    2
2023-02-01    2
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.first.html.md

# dask.dataframe.tseries.resample.Resampler.first

#### Resampler.first()

Compute the first non-null entry of each column.

This docstring was copied from pandas.api.typing.Resampler.first.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
* **Returns:**
  Series or DataFrame
  : First values within each group.

#### SEE ALSO
`core.resample.Resampler.last`
: Compute the last non-null value in each group.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

### Examples

```pycon
>>> s = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> s
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> s.resample("MS").first()
2023-01-01    1
2023-02-01    3
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.html.md

# dask.dataframe.tseries.resample.Resampler

### *class* dask.dataframe.tseries.resample.Resampler(obj, rule, \*\*kwargs)

Aggregate using one or more operations

The purpose of this class is to expose an API similar
to Pandas’ Resampler for dask-expr

<!-- !! processed by numpydoc !! -->

#### \_\_init_\_(obj, rule, \*\*kwargs)

<!-- !! processed by numpydoc !! -->

### Methods

| [`__init__`](#dask.dataframe.tseries.resample.Resampler.__init__)(obj, rule, \*\*kwargs)                                          |                                                                                |
|-----------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| [`agg`](dask.dataframe.tseries.resample.Resampler.agg.md#dask.dataframe.tseries.resample.Resampler.agg)(func, \*args, \*\*kwargs) | Aggregate using one or more operations over the specified axis.                |
| [`count`](dask.dataframe.tseries.resample.Resampler.count.md#dask.dataframe.tseries.resample.Resampler.count)()                   | Compute count of group, excluding missing values.                              |
| [`first`](dask.dataframe.tseries.resample.Resampler.first.md#dask.dataframe.tseries.resample.Resampler.first)()                   | Compute the first non-null entry of each column.                               |
| [`last`](dask.dataframe.tseries.resample.Resampler.last.md#dask.dataframe.tseries.resample.Resampler.last)()                      | Compute the last non-null entry of each column.                                |
| [`max`](dask.dataframe.tseries.resample.Resampler.max.md#dask.dataframe.tseries.resample.Resampler.max)()                         | Compute max value of group.                                                    |
| [`mean`](dask.dataframe.tseries.resample.Resampler.mean.md#dask.dataframe.tseries.resample.Resampler.mean)()                      | Compute mean of groups, excluding missing values.                              |
| [`median`](dask.dataframe.tseries.resample.Resampler.median.md#dask.dataframe.tseries.resample.Resampler.median)()                | Compute median of groups, excluding missing values.                            |
| [`min`](dask.dataframe.tseries.resample.Resampler.min.md#dask.dataframe.tseries.resample.Resampler.min)()                         | Compute min value of group.                                                    |
| [`nunique`](dask.dataframe.tseries.resample.Resampler.nunique.md#dask.dataframe.tseries.resample.Resampler.nunique)()             | Return number of unique elements in the group.                                 |
| [`ohlc`](dask.dataframe.tseries.resample.Resampler.ohlc.md#dask.dataframe.tseries.resample.Resampler.ohlc)()                      | Compute open, high, low and close values of a group, excluding missing values. |
| [`prod`](dask.dataframe.tseries.resample.Resampler.prod.md#dask.dataframe.tseries.resample.Resampler.prod)()                      | Compute prod of group values.                                                  |
| [`quantile`](dask.dataframe.tseries.resample.Resampler.quantile.md#dask.dataframe.tseries.resample.Resampler.quantile)()          | Return value at the given quantile.                                            |
| [`sem`](dask.dataframe.tseries.resample.Resampler.sem.md#dask.dataframe.tseries.resample.Resampler.sem)()                         | Compute standard error of the mean of groups, excluding missing values.        |
| [`size`](dask.dataframe.tseries.resample.Resampler.size.md#dask.dataframe.tseries.resample.Resampler.size)()                      | Compute group sizes.                                                           |
| [`std`](dask.dataframe.tseries.resample.Resampler.std.md#dask.dataframe.tseries.resample.Resampler.std)()                         | Compute standard deviation of groups, excluding missing values.                |
| [`sum`](dask.dataframe.tseries.resample.Resampler.sum.md#dask.dataframe.tseries.resample.Resampler.sum)()                         | Compute sum of group values.                                                   |
| [`var`](dask.dataframe.tseries.resample.Resampler.var.md#dask.dataframe.tseries.resample.Resampler.var)()                         | Compute variance of groups, excluding missing values.                          |


# dask.dataframe.tseries.resample.Resampler.last.html.md

# dask.dataframe.tseries.resample.Resampler.last

#### Resampler.last()

Compute the last non-null entry of each column.

This docstring was copied from pandas.api.typing.Resampler.last.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.

  **skipna**
  : Exclude NA/null values. If an entire group is NA, the result will be NA.
* **Returns:**
  Series or DataFrame
  : Last of values within each group.

#### SEE ALSO
`core.resample.Resampler.first`
: Compute the first non-null value in each group.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

### Examples

```pycon
>>> s = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> s
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> s.resample("MS").last()
2023-01-01    2
2023-02-01    4
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.max.html.md

# dask.dataframe.tseries.resample.Resampler.max

#### Resampler.max()

Compute max value of group.

This docstring was copied from pandas.api.typing.Resampler.max.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.
* **Returns:**
  Series or DataFrame
  : Computes the maximum value in the given Series or Dataframe.

#### SEE ALSO
`core.resample.Resampler.min`
: Compute min value of group.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").max()
2023-01-01    2
2023-02-01    4
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.mean.html.md

# dask.dataframe.tseries.resample.Resampler.mean

#### Resampler.mean()

Compute mean of groups, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.mean.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.
* **Returns:**
  DataFrame or Series
  : Mean of values within each group.

#### SEE ALSO
`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

`core.resample.Resampler.sum`
: Compute sum of groups, excluding missing values.

`core.resample.Resampler.std`
: Compute standard deviation of groups, excluding missing values.

`core.resample.Resampler.var`
: Compute variance of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").mean()
2023-01-01    1.5
2023-02-01    3.5
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.median.html.md

# dask.dataframe.tseries.resample.Resampler.median

#### Resampler.median()

Compute median of groups, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.median.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None` and defaults to False.
* **Returns:**
  Series or DataFrame
  : Median of values within each group.

#### SEE ALSO
`Series.groupby`
: Apply a function groupby to a Series.

`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 3, 4, 5],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").median()
2023-01-01    2.0
2023-02-01    4.0
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.min.html.md

# dask.dataframe.tseries.resample.Resampler.min

#### Resampler.min()

Compute min value of group.

This docstring was copied from pandas.api.typing.Resampler.min.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.
* **Returns:**
  Series or DataFrame
  : Compute the minimum value in the given Series or DataFrame.

#### SEE ALSO
`core.resample.Resampler.max`
: Compute max value of group.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").min()
2023-01-01    1
2023-02-01    3
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.nunique.html.md

# dask.dataframe.tseries.resample.Resampler.nunique

#### Resampler.nunique()

Return number of unique elements in the group.

This docstring was copied from pandas.api.typing.Resampler.nunique.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series
  : Number of unique values within each group.

#### SEE ALSO
`core.groupby.SeriesGroupBy.nunique`
: Method nunique for SeriesGroupBy.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 3],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    3
dtype: int64
>>> ser.resample("MS").nunique()
2023-01-01    2
2023-02-01    1
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.ohlc.html.md

# dask.dataframe.tseries.resample.Resampler.ohlc

#### Resampler.ohlc()

Compute open, high, low and close values of a group, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.ohlc.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  DataFrame
  : Open, high, low and close values within each group.

#### SEE ALSO
`DataFrame.agg`
: Aggregate using one or more operations over the specified axis.

`DataFrame.resample`
: Resample time-series data.

`DataFrame.groupby`
: Group DataFrame using a mapper or by a Series of columns.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 3, 2, 4, 3, 5],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").ohlc()
            open  high  low  close
2023-01-01     1     3    1      2
2023-02-01     4     5    3      5
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.prod.html.md

# dask.dataframe.tseries.resample.Resampler.prod

#### Resampler.prod()

Compute prod of group values.

This docstring was copied from pandas.api.typing.Resampler.prod.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.
* **Returns:**
  Series or DataFrame
  : Computed prod of values within each group.

#### SEE ALSO
`core.resample.Resampler.sum`
: Compute sum of groups, excluding missing values.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").prod()
2023-01-01    2
2023-02-01   12
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.quantile.html.md

# dask.dataframe.tseries.resample.Resampler.quantile

#### Resampler.quantile()

Return value at the given quantile.

This docstring was copied from pandas.api.typing.Resampler.quantile.

Some inconsistencies with the Dask version may exist.

Computes the quantile of values within each resampled group.

* **Parameters:**
  **q**
  : Value between 0 <= q <= 1, the quantile(s) to compute.

  **\*\*kwargs**
  : Additional keyword arguments to be passed to the function.
* **Returns:**
  DataFrame or Series
  : Quantile of values within each group.

#### SEE ALSO
`Series.quantile`
: Return a series, where the index is q and the values are the quantiles.

`DataFrame.quantile`
: Return a DataFrame, where the columns are the columns of self, and the values are the quantiles.

`DataFrameGroupBy.quantile`
: Return a DataFrame, where the columns are groupby columns, and the values are its quantiles.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 3, 2, 4, 3, 8],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").quantile()
2023-01-01    2.0
2023-02-01    4.0
Freq: MS, dtype: float64
```

```pycon
>>> ser.resample("MS").quantile(0.25)
2023-01-01    1.5
2023-02-01    3.5
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.sem.html.md

# dask.dataframe.tseries.resample.Resampler.sem

#### Resampler.sem()

Compute standard error of the mean of groups, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.sem.

Some inconsistencies with the Dask version may exist.

For multiple groupings, the result index will be a MultiIndex.

* **Parameters:**
  **ddof**
  : Degrees of freedom.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.
* **Returns:**
  Series or DataFrame
  : Standard error of the mean of values within each group.

#### SEE ALSO
`DataFrame.sem`
: Return unbiased standard error of the mean over requested axis.

`Series.sem`
: Return unbiased standard error of the mean over requested axis.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 3, 2, 4, 3, 8],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").sem()
2023-01-01    0.577350
2023-02-01    1.527525
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.size.html.md

# dask.dataframe.tseries.resample.Resampler.size

#### Resampler.size()

Compute group sizes.

This docstring was copied from pandas.api.typing.Resampler.size.

Some inconsistencies with the Dask version may exist.

* **Returns:**
  Series
  : Number of rows in each group.

#### SEE ALSO
`Series.groupby`
: Apply a function groupby to a Series.

`DataFrame.groupby`
: Apply a function groupby to each row or column of a DataFrame.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3],
...     index=pd.DatetimeIndex(["2023-01-01", "2023-01-15", "2023-02-01"]),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
dtype: int64
>>> ser.resample("MS").size()
2023-01-01    2
2023-02-01    1
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.std.html.md

# dask.dataframe.tseries.resample.Resampler.std

#### Resampler.std()

Compute standard deviation of groups, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.std.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ddof**
  : Degrees of freedom.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.
* **Returns:**
  DataFrame or Series
  : Standard deviation of values within each group.

#### SEE ALSO
`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

`core.resample.Resampler.var`
: Compute variance of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 3, 2, 4, 3, 8],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").std()
2023-01-01    1.000000
2023-02-01    2.645751
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.sum.html.md

# dask.dataframe.tseries.resample.Resampler.sum

#### Resampler.sum()

Compute sum of group values.

This docstring was copied from pandas.api.typing.Resampler.sum.

Some inconsistencies with the Dask version may exist.

This method provides a simple way to compute the sum of values within each
resampled group, particularly useful for aggregating time-based data into
daily, monthly, or yearly sums.

* **Parameters:**
  **numeric_only**
  : Include only float, int, boolean columns.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only no longer accepts `None`.

  **min_count**
  : The required number of valid values to perform the operation. If fewer
    than `min_count` non-NA values are present the result will be NA.
* **Returns:**
  Series or DataFrame
  : Computed sum of values within each group.

#### SEE ALSO
`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.count`
: Compute count of group, excluding missing values.

`DataFrame.resample`
: Resample time-series data.

`Series.sum`
: Return the sum of the values over the requested axis.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 2, 3, 4],
...     index=pd.DatetimeIndex(
...         ["2023-01-01", "2023-01-15", "2023-02-01", "2023-02-15"]
...     ),
... )
>>> ser
2023-01-01    1
2023-01-15    2
2023-02-01    3
2023-02-15    4
dtype: int64
>>> ser.resample("MS").sum()
2023-01-01    3
2023-02-01    7
Freq: MS, dtype: int64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.tseries.resample.Resampler.var.html.md

# dask.dataframe.tseries.resample.Resampler.var

#### Resampler.var()

Compute variance of groups, excluding missing values.

This docstring was copied from pandas.api.typing.Resampler.var.

Some inconsistencies with the Dask version may exist.

* **Parameters:**
  **ddof**
  : Degrees of freedom.

  **numeric_only**
  : Include only float, int or boolean data.
    <br/>
    #### Versionchanged
    Changed in version 2.0.0: numeric_only now defaults to `False`.
* **Returns:**
  DataFrame or Series
  : Variance of values within each group.

#### SEE ALSO
`core.resample.Resampler.std`
: Compute standard deviation of groups, excluding missing values.

`core.resample.Resampler.mean`
: Compute mean of groups, excluding missing values.

`core.resample.Resampler.median`
: Compute median of groups, excluding missing values.

### Examples

```pycon
>>> ser = pd.Series(
...     [1, 3, 2, 4, 3, 8],
...     index=pd.DatetimeIndex(
...         [
...             "2023-01-01",
...             "2023-01-10",
...             "2023-01-15",
...             "2023-02-01",
...             "2023-02-10",
...             "2023-02-15",
...         ]
...     ),
... )
>>> ser.resample("MS").var()
2023-01-01    1.0
2023-02-01    7.0
Freq: MS, dtype: float64
```

```pycon
>>> ser.resample("MS").var(ddof=0)
2023-01-01    0.666667
2023-02-01    4.666667
Freq: MS, dtype: float64
```

<!-- !! processed by numpydoc !! -->


# dask.dataframe.utils.make_meta.html.md

# dask.dataframe.utils.make_meta

### dask.dataframe.utils.make_meta(x, index=None, parent_meta=None)

This method creates meta-data based on the type of `x`,
and `parent_meta` if supplied.

* **Parameters:**
  **x**
  : Object to construct meta-data from.

  **index**
  : Any index to use in the metadata. This is a pass-through
    parameter to dispatches registered.

  **parent_meta**
  : If `x` is of arbitrary types and thus Dask cannot determine
    which back-end to be used to generate the meta-data for this
    object type, in which case `parent_meta` will be used to
    determine which back-end to select and dispatch to. To
    utilize this parameter `make_meta_obj` has be dispatched.
    If `parent_meta` is `None`, a pandas DataFrame is used for
    `parent_meta` that chooses pandas as the backend.
* **Returns:**
  A valid meta-data

<!-- !! processed by numpydoc !! -->


# dask.tokenize.TokenizationError.html.md

# dask.tokenize.TokenizationError

### *exception* dask.tokenize.TokenizationError

<!-- !! processed by numpydoc !! -->


# dask.tokenize.tokenize.html.md

# dask.tokenize.tokenize

### dask.tokenize.tokenize(\*args: [object](https://docs.python.org/3/library/functions.html#object), ensure_deterministic: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs: [object](https://docs.python.org/3/library/functions.html#object)) → [str](https://docs.python.org/3/library/stdtypes.html#str)

Deterministic token

```pycon
>>> tokenize([1, 2, '3'])
'06961e8de572e73c2e74b51348177918'
```

```pycon
>>> tokenize('Hello') == tokenize('Hello')
True
```

* **Parameters:**
  **args, kwargs:**
  : objects to tokenize

  **ensure_deterministic: bool, optional**
  : If True, raise TokenizationError if the objects cannot be deterministically
    tokenized, e.g. two identical objects will return different tokens.
    Defaults to the tokenize.ensure-deterministic configuration parameter.

<!-- !! processed by numpydoc !! -->


# dataframe-api.html.md

# Dask DataFrame API with Logical Query Planning

## DataFrame

| [`DataFrame`](generated/dask.dataframe.DataFrame.md#dask.dataframe.DataFrame)(expr)                                                                                   | DataFrame-like Expr Collection.                                                                                  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| [`DataFrame.abs`](generated/dask.dataframe.DataFrame.abs.md#dask.dataframe.DataFrame.abs)()                                                                           | Return a Series/DataFrame with absolute numeric value of each element.                                           |
| [`DataFrame.add`](generated/dask.dataframe.DataFrame.add.md#dask.dataframe.DataFrame.add)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.align`](generated/dask.dataframe.DataFrame.align.md#dask.dataframe.DataFrame.align)(other[, join, axis, fill_value])                                      | Align two objects on their axes with the specified join method.                                                  |
| [`DataFrame.all`](generated/dask.dataframe.DataFrame.all.md#dask.dataframe.DataFrame.all)([axis, skipna, split_every])                                                | Return whether all elements are True, potentially over an axis.                                                  |
| [`DataFrame.any`](generated/dask.dataframe.DataFrame.any.md#dask.dataframe.DataFrame.any)([axis, skipna, split_every])                                                | Return whether any element is True, potentially over an axis.                                                    |
| [`DataFrame.apply`](generated/dask.dataframe.DataFrame.apply.md#dask.dataframe.DataFrame.apply)(function, \*args[, meta, axis])                                       | Parallel version of pandas.DataFrame.apply                                                                       |
| [`DataFrame.assign`](generated/dask.dataframe.DataFrame.assign.md#dask.dataframe.DataFrame.assign)(\*\*pairs)                                                         | Assign new columns to a DataFrame.                                                                               |
| [`DataFrame.astype`](generated/dask.dataframe.DataFrame.astype.md#dask.dataframe.DataFrame.astype)(dtypes)                                                            | Cast a pandas object to a specified dtype `dtype`.                                                               |
| [`DataFrame.bfill`](generated/dask.dataframe.DataFrame.bfill.md#dask.dataframe.DataFrame.bfill)([axis, limit])                                                        | Fill NA/NaN values by using the next valid observation to fill the gap.                                          |
| [`DataFrame.categorize`](generated/dask.dataframe.DataFrame.categorize.md#dask.dataframe.DataFrame.categorize)([columns, index, ...])                                 | Convert columns of the DataFrame to category dtype.                                                              |
| [`DataFrame.columns`](generated/dask.dataframe.DataFrame.columns.md#dask.dataframe.DataFrame.columns)                                                                 |                                                                                                                  |
| [`DataFrame.compute`](generated/dask.dataframe.DataFrame.compute.md#dask.dataframe.DataFrame.compute)(\*\*kwargs)                                                     | Compute this dask collection                                                                                     |
| [`DataFrame.copy`](generated/dask.dataframe.DataFrame.copy.md#dask.dataframe.DataFrame.copy)([deep])                                                                  | Make a copy of the dataframe                                                                                     |
| [`DataFrame.corr`](generated/dask.dataframe.DataFrame.corr.md#dask.dataframe.DataFrame.corr)([method, min_periods, ...])                                              | Compute pairwise correlation of columns, excluding NA/null values.                                               |
| [`DataFrame.count`](generated/dask.dataframe.DataFrame.count.md#dask.dataframe.DataFrame.count)([axis, numeric_only, ...])                                            | Count non-NA cells for each column or row.                                                                       |
| [`DataFrame.cov`](generated/dask.dataframe.DataFrame.cov.md#dask.dataframe.DataFrame.cov)([min_periods, numeric_only, ...])                                           | Compute pairwise covariance of columns, excluding NA/null values.                                                |
| [`DataFrame.cummax`](generated/dask.dataframe.DataFrame.cummax.md#dask.dataframe.DataFrame.cummax)([axis, skipna])                                                    | Return cumulative maximum over a DataFrame or Series axis.                                                       |
| [`DataFrame.cummin`](generated/dask.dataframe.DataFrame.cummin.md#dask.dataframe.DataFrame.cummin)([axis, skipna])                                                    | Return cumulative minimum over a DataFrame or Series axis.                                                       |
| [`DataFrame.cumprod`](generated/dask.dataframe.DataFrame.cumprod.md#dask.dataframe.DataFrame.cumprod)([axis, skipna])                                                 | Return cumulative product over a DataFrame or Series axis.                                                       |
| [`DataFrame.cumsum`](generated/dask.dataframe.DataFrame.cumsum.md#dask.dataframe.DataFrame.cumsum)([axis, skipna])                                                    | Return cumulative sum over a DataFrame or Series axis.                                                           |
| [`DataFrame.describe`](generated/dask.dataframe.DataFrame.describe.md#dask.dataframe.DataFrame.describe)([split_every, ...])                                          | Generate descriptive statistics.                                                                                 |
| [`DataFrame.diff`](generated/dask.dataframe.DataFrame.diff.md#dask.dataframe.DataFrame.diff)([periods, axis])                                                         | First discrete difference of element.                                                                            |
| [`DataFrame.div`](generated/dask.dataframe.DataFrame.div.md#dask.dataframe.DataFrame.div)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.divide`](generated/dask.dataframe.DataFrame.divide.md#dask.dataframe.DataFrame.divide)(other[, axis, level, ...])                                         |                                                                                                                  |
| [`DataFrame.divisions`](generated/dask.dataframe.DataFrame.divisions.md#dask.dataframe.DataFrame.divisions)                                                           | Tuple of `npartitions + 1` values, in ascending order, marking the lower/upper bounds of each partition's index. |
| [`DataFrame.drop`](generated/dask.dataframe.DataFrame.drop.md#dask.dataframe.DataFrame.drop)([labels, axis, columns, errors])                                         | Drop specified labels from rows or columns.                                                                      |
| [`DataFrame.drop_duplicates`](generated/dask.dataframe.DataFrame.drop_duplicates.md#dask.dataframe.DataFrame.drop_duplicates)([subset, ...])                          | Return DataFrame with duplicate rows removed.                                                                    |
| [`DataFrame.dropna`](generated/dask.dataframe.DataFrame.dropna.md#dask.dataframe.DataFrame.dropna)([how, subset, thresh])                                             | Remove missing values.                                                                                           |
| [`DataFrame.dtypes`](generated/dask.dataframe.DataFrame.dtypes.md#dask.dataframe.DataFrame.dtypes)                                                                    | Return data types                                                                                                |
| [`DataFrame.eq`](generated/dask.dataframe.DataFrame.eq.md#dask.dataframe.DataFrame.eq)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.eval`](generated/dask.dataframe.DataFrame.eval.md#dask.dataframe.DataFrame.eval)(expr, \*\*kwargs)                                                        | Evaluate a string describing operations on DataFrame columns.                                                    |
| [`DataFrame.explode`](generated/dask.dataframe.DataFrame.explode.md#dask.dataframe.DataFrame.explode)(column)                                                         | Transform each element of a list-like to a row, replicating index values.                                        |
| [`DataFrame.ffill`](generated/dask.dataframe.DataFrame.ffill.md#dask.dataframe.DataFrame.ffill)([axis, limit])                                                        | Fill NA/NaN values by propagating the last valid observation to next valid.                                      |
| [`DataFrame.fillna`](generated/dask.dataframe.DataFrame.fillna.md#dask.dataframe.DataFrame.fillna)([value, axis])                                                     | Fill NA/NaN values with value.                                                                                   |
| [`DataFrame.floordiv`](generated/dask.dataframe.DataFrame.floordiv.md#dask.dataframe.DataFrame.floordiv)(other[, axis, level, ...])                                   |                                                                                                                  |
| [`DataFrame.ge`](generated/dask.dataframe.DataFrame.ge.md#dask.dataframe.DataFrame.ge)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.get_partition`](generated/dask.dataframe.DataFrame.get_partition.md#dask.dataframe.DataFrame.get_partition)(n)                                            | Get a dask DataFrame/Series representing the nth partition.                                                      |
| [`DataFrame.groupby`](generated/dask.dataframe.DataFrame.groupby.md#dask.dataframe.DataFrame.groupby)(by[, group_keys, sort, ...])                                    | Group DataFrame using a mapper or by a Series of columns.                                                        |
| [`DataFrame.gt`](generated/dask.dataframe.DataFrame.gt.md#dask.dataframe.DataFrame.gt)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.head`](generated/dask.dataframe.DataFrame.head.md#dask.dataframe.DataFrame.head)([n, npartitions, compute])                                               | First n rows of the dataset                                                                                      |
| [`DataFrame.idxmax`](generated/dask.dataframe.DataFrame.idxmax.md#dask.dataframe.DataFrame.idxmax)([axis, skipna, ...])                                               | Return index of first occurrence of maximum over requested axis.                                                 |
| [`DataFrame.idxmin`](generated/dask.dataframe.DataFrame.idxmin.md#dask.dataframe.DataFrame.idxmin)([axis, skipna, ...])                                               | Return index of first occurrence of minimum over requested axis.                                                 |
| [`DataFrame.iloc`](generated/dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)                                                                          | Purely integer-location based indexing for selection by position.                                                |
| [`DataFrame.index`](generated/dask.dataframe.DataFrame.index.md#dask.dataframe.DataFrame.index)                                                                       | Return dask Index instance                                                                                       |
| [`DataFrame.info`](generated/dask.dataframe.DataFrame.info.md#dask.dataframe.DataFrame.info)([buf, verbose, memory_usage])                                            | Concise summary of a Dask DataFrame                                                                              |
| [`DataFrame.isin`](generated/dask.dataframe.DataFrame.isin.md#dask.dataframe.DataFrame.isin)(values)                                                                  | Whether each element in the DataFrame is contained in values.                                                    |
| [`DataFrame.isna`](generated/dask.dataframe.DataFrame.isna.md#dask.dataframe.DataFrame.isna)()                                                                        | Detect missing values.                                                                                           |
| [`DataFrame.isnull`](generated/dask.dataframe.DataFrame.isnull.md#dask.dataframe.DataFrame.isnull)()                                                                  | DataFrame.isnull is an alias for DataFrame.isna.                                                                 |
| [`DataFrame.items`](generated/dask.dataframe.DataFrame.items.md#dask.dataframe.DataFrame.items)()                                                                     | Iterate over (column name, Series) pairs.                                                                        |
| [`DataFrame.iterrows`](generated/dask.dataframe.DataFrame.iterrows.md#dask.dataframe.DataFrame.iterrows)()                                                            | Iterate over DataFrame rows as (index, Series) pairs.                                                            |
| [`DataFrame.itertuples`](generated/dask.dataframe.DataFrame.itertuples.md#dask.dataframe.DataFrame.itertuples)([index, name])                                         | Iterate over DataFrame rows as namedtuples.                                                                      |
| [`DataFrame.join`](generated/dask.dataframe.DataFrame.join.md#dask.dataframe.DataFrame.join)(other[, on, how, lsuffix, ...])                                          | Join columns of another DataFrame.                                                                               |
| [`DataFrame.known_divisions`](generated/dask.dataframe.DataFrame.known_divisions.md#dask.dataframe.DataFrame.known_divisions)                                         | Whether the divisions are known.                                                                                 |
| [`DataFrame.le`](generated/dask.dataframe.DataFrame.le.md#dask.dataframe.DataFrame.le)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.loc`](generated/dask.dataframe.DataFrame.loc.md#dask.dataframe.DataFrame.loc)                                                                             | Purely label-location based indexer for selection by label.                                                      |
| [`DataFrame.lt`](generated/dask.dataframe.DataFrame.lt.md#dask.dataframe.DataFrame.lt)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.map_partitions`](generated/dask.dataframe.DataFrame.map_partitions.md#dask.dataframe.DataFrame.map_partitions)(func, \*args[, ...])                       | Apply a Python function to each partition                                                                        |
| [`DataFrame.mask`](generated/dask.dataframe.DataFrame.mask.md#dask.dataframe.DataFrame.mask)(cond[, other])                                                           | Replace values where the condition is True.                                                                      |
| [`DataFrame.max`](generated/dask.dataframe.DataFrame.max.md#dask.dataframe.DataFrame.max)([axis, skipna, numeric_only, ...])                                          | Return the maximum of the values over the requested axis.                                                        |
| [`DataFrame.mean`](generated/dask.dataframe.DataFrame.mean.md#dask.dataframe.DataFrame.mean)([axis, skipna, numeric_only, ...])                                       | Return the mean of the values over the requested axis.                                                           |
| [`DataFrame.median`](generated/dask.dataframe.DataFrame.median.md#dask.dataframe.DataFrame.median)([axis, numeric_only])                                              | Return the median of the values over the requested axis.                                                         |
| [`DataFrame.median_approximate`](generated/dask.dataframe.DataFrame.median_approximate.md#dask.dataframe.DataFrame.median_approximate)([axis, method, ...])           | Return the approximate median of the values over the requested axis.                                             |
| [`DataFrame.melt`](generated/dask.dataframe.DataFrame.melt.md#dask.dataframe.DataFrame.melt)([id_vars, value_vars, ...])                                              | Unpivot DataFrame from wide to long format, optionally leaving identifiers set.                                  |
| [`DataFrame.memory_usage`](generated/dask.dataframe.DataFrame.memory_usage.md#dask.dataframe.DataFrame.memory_usage)([deep, index])                                   | Return the memory usage of each column in bytes.                                                                 |
| [`DataFrame.memory_usage_per_partition`](generated/dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition)([...]) | Return the memory usage of each partition                                                                        |
| [`DataFrame.merge`](generated/dask.dataframe.DataFrame.merge.md#dask.dataframe.DataFrame.merge)(right[, how, on, left_on, ...])                                       | Merge the DataFrame with another DataFrame                                                                       |
| [`DataFrame.min`](generated/dask.dataframe.DataFrame.min.md#dask.dataframe.DataFrame.min)([axis, skipna, numeric_only, ...])                                          | Return the minimum of the values over the requested axis.                                                        |
| [`DataFrame.mod`](generated/dask.dataframe.DataFrame.mod.md#dask.dataframe.DataFrame.mod)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.mode`](generated/dask.dataframe.DataFrame.mode.md#dask.dataframe.DataFrame.mode)([dropna, split_every, ...])                                              | Get the mode(s) of each element along the selected axis.                                                         |
| [`DataFrame.mul`](generated/dask.dataframe.DataFrame.mul.md#dask.dataframe.DataFrame.mul)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.ndim`](generated/dask.dataframe.DataFrame.ndim.md#dask.dataframe.DataFrame.ndim)                                                                          | Return dimensionality                                                                                            |
| [`DataFrame.ne`](generated/dask.dataframe.DataFrame.ne.md#dask.dataframe.DataFrame.ne)(other[, level, axis])                                                          |                                                                                                                  |
| [`DataFrame.nlargest`](generated/dask.dataframe.DataFrame.nlargest.md#dask.dataframe.DataFrame.nlargest)([n, columns, split_every])                                   | Return the first n rows ordered by columns in descending order.                                                  |
| [`DataFrame.npartitions`](generated/dask.dataframe.DataFrame.npartitions.md#dask.dataframe.DataFrame.npartitions)                                                     | Return number of partitions                                                                                      |
| [`DataFrame.nsmallest`](generated/dask.dataframe.DataFrame.nsmallest.md#dask.dataframe.DataFrame.nsmallest)([n, columns, split_every])                                | Return the first n rows ordered by columns in ascending order.                                                   |
| [`DataFrame.partitions`](generated/dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions)                                                        | Slice dataframe by partitions                                                                                    |
| [`DataFrame.persist`](generated/dask.dataframe.DataFrame.persist.md#dask.dataframe.DataFrame.persist)([fuse])                                                         | Persist this dask collection into memory                                                                         |
| [`DataFrame.pivot_table`](generated/dask.dataframe.DataFrame.pivot_table.md#dask.dataframe.DataFrame.pivot_table)(index, columns, values)                             | Create a spreadsheet-style pivot table as a DataFrame.                                                           |
| [`DataFrame.pop`](generated/dask.dataframe.DataFrame.pop.md#dask.dataframe.DataFrame.pop)(item)                                                                       | Return item and drop it from DataFrame.                                                                          |
| [`DataFrame.pow`](generated/dask.dataframe.DataFrame.pow.md#dask.dataframe.DataFrame.pow)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.prod`](generated/dask.dataframe.DataFrame.prod.md#dask.dataframe.DataFrame.prod)([axis, skipna, numeric_only, ...])                                       | Return the product of the values over the requested axis.                                                        |
| [`DataFrame.quantile`](generated/dask.dataframe.DataFrame.quantile.md#dask.dataframe.DataFrame.quantile)([q, axis, numeric_only, ...])                                | Approximate row-wise and precise column-wise quantiles of DataFrame                                              |
| [`DataFrame.query`](generated/dask.dataframe.DataFrame.query.md#dask.dataframe.DataFrame.query)(expr, \*\*kwargs)                                                     | Filter dataframe with complex expression                                                                         |
| [`DataFrame.radd`](generated/dask.dataframe.DataFrame.radd.md#dask.dataframe.DataFrame.radd)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.random_split`](generated/dask.dataframe.DataFrame.random_split.md#dask.dataframe.DataFrame.random_split)(frac[, random_state, ...])                       | Pseudorandomly split dataframe into different pieces row-wise                                                    |
| [`DataFrame.rdiv`](generated/dask.dataframe.DataFrame.rdiv.md#dask.dataframe.DataFrame.rdiv)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.rename`](generated/dask.dataframe.DataFrame.rename.md#dask.dataframe.DataFrame.rename)([index, columns])                                                  | Rename columns or index labels.                                                                                  |
| [`DataFrame.rename_axis`](generated/dask.dataframe.DataFrame.rename_axis.md#dask.dataframe.DataFrame.rename_axis)([mapper, index, ...])                               | Set the name of the axis for the index or columns.                                                               |
| [`DataFrame.repartition`](generated/dask.dataframe.DataFrame.repartition.md#dask.dataframe.DataFrame.repartition)([divisions, ...])                                   | Repartition a collection                                                                                         |
| [`DataFrame.replace`](generated/dask.dataframe.DataFrame.replace.md#dask.dataframe.DataFrame.replace)([to_replace, value, regex])                                     | Replace values given in to_replace with value.                                                                   |
| [`DataFrame.resample`](generated/dask.dataframe.DataFrame.resample.md#dask.dataframe.DataFrame.resample)(rule[, closed, label])                                       | Resample time-series data.                                                                                       |
| [`DataFrame.reset_index`](generated/dask.dataframe.DataFrame.reset_index.md#dask.dataframe.DataFrame.reset_index)([drop])                                             | Reset the index to the default index.                                                                            |
| [`DataFrame.rfloordiv`](generated/dask.dataframe.DataFrame.rfloordiv.md#dask.dataframe.DataFrame.rfloordiv)(other[, axis, level, ...])                                |                                                                                                                  |
| [`DataFrame.rmod`](generated/dask.dataframe.DataFrame.rmod.md#dask.dataframe.DataFrame.rmod)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.rmul`](generated/dask.dataframe.DataFrame.rmul.md#dask.dataframe.DataFrame.rmul)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.round`](generated/dask.dataframe.DataFrame.round.md#dask.dataframe.DataFrame.round)([decimals])                                                           | Round numeric columns in a DataFrame to a variable number of decimal places.                                     |
| [`DataFrame.rpow`](generated/dask.dataframe.DataFrame.rpow.md#dask.dataframe.DataFrame.rpow)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.rsub`](generated/dask.dataframe.DataFrame.rsub.md#dask.dataframe.DataFrame.rsub)(other[, axis, level, fill_value])                                        |                                                                                                                  |
| [`DataFrame.rtruediv`](generated/dask.dataframe.DataFrame.rtruediv.md#dask.dataframe.DataFrame.rtruediv)(other[, axis, level, ...])                                   |                                                                                                                  |
| [`DataFrame.sample`](generated/dask.dataframe.DataFrame.sample.md#dask.dataframe.DataFrame.sample)([n, frac, replace, ...])                                           | Random sample of items                                                                                           |
| [`DataFrame.select_dtypes`](generated/dask.dataframe.DataFrame.select_dtypes.md#dask.dataframe.DataFrame.select_dtypes)([include, exclude])                           | Return a subset of the DataFrame's columns based on the column dtypes.                                           |
| [`DataFrame.sem`](generated/dask.dataframe.DataFrame.sem.md#dask.dataframe.DataFrame.sem)([axis, skipna, ddof, ...])                                                  | Return unbiased standard error of the mean over requested axis.                                                  |
| [`DataFrame.set_index`](generated/dask.dataframe.DataFrame.set_index.md#dask.dataframe.DataFrame.set_index)(other[, drop, sorted, ...])                               | Set the DataFrame index (row labels) using an existing column.                                                   |
| [`DataFrame.shape`](generated/dask.dataframe.DataFrame.shape.md#dask.dataframe.DataFrame.shape)                                                                       |                                                                                                                  |
| [`DataFrame.shuffle`](generated/dask.dataframe.DataFrame.shuffle.md#dask.dataframe.DataFrame.shuffle)([on, ignore_index, ...])                                        | Rearrange DataFrame into new partitions                                                                          |
| [`DataFrame.size`](generated/dask.dataframe.DataFrame.size.md#dask.dataframe.DataFrame.size)                                                                          | Size of the Series or DataFrame as a Delayed object.                                                             |
| [`DataFrame.sort_values`](generated/dask.dataframe.DataFrame.sort_values.md#dask.dataframe.DataFrame.sort_values)(by[, npartitions, ...])                             | Sort the dataset by a single column.                                                                             |
| [`DataFrame.squeeze`](generated/dask.dataframe.DataFrame.squeeze.md#dask.dataframe.DataFrame.squeeze)([axis])                                                         | Squeeze 1 dimensional axis objects into scalars.                                                                 |
| [`DataFrame.std`](generated/dask.dataframe.DataFrame.std.md#dask.dataframe.DataFrame.std)([axis, skipna, ddof, ...])                                                  | Return sample standard deviation over requested axis.                                                            |
| [`DataFrame.sub`](generated/dask.dataframe.DataFrame.sub.md#dask.dataframe.DataFrame.sub)(other[, axis, level, fill_value])                                           |                                                                                                                  |
| [`DataFrame.sum`](generated/dask.dataframe.DataFrame.sum.md#dask.dataframe.DataFrame.sum)([axis, skipna, numeric_only, ...])                                          | Return the sum of the values over the requested axis.                                                            |
| [`DataFrame.tail`](generated/dask.dataframe.DataFrame.tail.md#dask.dataframe.DataFrame.tail)([n, compute])                                                            | Last n rows of the dataset                                                                                       |
| [`DataFrame.to_backend`](generated/dask.dataframe.DataFrame.to_backend.md#dask.dataframe.DataFrame.to_backend)([backend])                                             | Move to a new DataFrame backend                                                                                  |
| [`DataFrame.to_bag`](generated/dask.dataframe.DataFrame.to_bag.md#dask.dataframe.DataFrame.to_bag)([index, format])                                                   | Create a Dask Bag from a Series                                                                                  |
| [`DataFrame.to_csv`](generated/dask.dataframe.DataFrame.to_csv.md#dask.dataframe.DataFrame.to_csv)(filename, \*\*kwargs)                                              | See dd.to_csv docstring for more information                                                                     |
| [`DataFrame.to_dask_array`](generated/dask.dataframe.DataFrame.to_dask_array.md#dask.dataframe.DataFrame.to_dask_array)([lengths, meta, ...])                         | Convert a dask DataFrame to a dask array.                                                                        |
| [`DataFrame.to_delayed`](generated/dask.dataframe.DataFrame.to_delayed.md#dask.dataframe.DataFrame.to_delayed)([optimize_graph])                                      | Convert into a list of `dask.delayed` objects, one per partition.                                                |
| [`DataFrame.to_hdf`](generated/dask.dataframe.DataFrame.to_hdf.md#dask.dataframe.DataFrame.to_hdf)(path_or_buf, key[, mode, ...])                                     | See dd.to_hdf docstring for more information                                                                     |
| [`DataFrame.to_html`](generated/dask.dataframe.DataFrame.to_html.md#dask.dataframe.DataFrame.to_html)([max_rows])                                                     | Render a DataFrame as an HTML table.                                                                             |
| [`DataFrame.to_json`](generated/dask.dataframe.DataFrame.to_json.md#dask.dataframe.DataFrame.to_json)(filename, \*args, \*\*kwargs)                                   | See dd.to_json docstring for more information                                                                    |
| [`DataFrame.to_orc`](generated/dask.dataframe.DataFrame.to_orc.md#dask.dataframe.DataFrame.to_orc)(path, \*args, \*\*kwargs)                                          | See dd.to_orc docstring for more information                                                                     |
| [`DataFrame.to_parquet`](generated/dask.dataframe.DataFrame.to_parquet.md#dask.dataframe.DataFrame.to_parquet)(path, \*\*kwargs)                                      |                                                                                                                  |
| [`DataFrame.to_records`](generated/dask.dataframe.DataFrame.to_records.md#dask.dataframe.DataFrame.to_records)([index, lengths])                                      |                                                                                                                  |
| [`DataFrame.to_string`](generated/dask.dataframe.DataFrame.to_string.md#dask.dataframe.DataFrame.to_string)([max_rows])                                               | Render a DataFrame to a console-friendly tabular output.                                                         |
| [`DataFrame.to_sql`](generated/dask.dataframe.DataFrame.to_sql.md#dask.dataframe.DataFrame.to_sql)(name, uri[, schema, ...])                                          |                                                                                                                  |
| [`DataFrame.to_timestamp`](generated/dask.dataframe.DataFrame.to_timestamp.md#dask.dataframe.DataFrame.to_timestamp)([freq, how])                                     | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.                                       |
| [`DataFrame.truediv`](generated/dask.dataframe.DataFrame.truediv.md#dask.dataframe.DataFrame.truediv)(other[, axis, level, ...])                                      |                                                                                                                  |
| [`DataFrame.values`](generated/dask.dataframe.DataFrame.values.md#dask.dataframe.DataFrame.values)                                                                    | Return a dask.array of the values of this dataframe                                                              |
| [`DataFrame.var`](generated/dask.dataframe.DataFrame.var.md#dask.dataframe.DataFrame.var)([axis, skipna, ddof, ...])                                                  | Return unbiased variance over requested axis.                                                                    |
| [`DataFrame.visualize`](generated/dask.dataframe.DataFrame.visualize.md#dask.dataframe.DataFrame.visualize)([tasks])                                                  | Visualize the expression or task graph                                                                           |
| [`DataFrame.where`](generated/dask.dataframe.DataFrame.where.md#dask.dataframe.DataFrame.where)(cond[, other])                                                        | Replace values where the condition is False.                                                                     |

## Series

| [`Series`](generated/dask.dataframe.Series.md#dask.dataframe.Series)(expr)                                                                                           | Series-like Expr Collection.                                                 |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| [`Series.add`](generated/dask.dataframe.Series.add.md#dask.dataframe.Series.add)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.align`](generated/dask.dataframe.Series.align.md#dask.dataframe.Series.align)(other[, join, axis, fill_value])                                              | Align two objects on their axes with the specified join method.              |
| [`Series.all`](generated/dask.dataframe.Series.all.md#dask.dataframe.Series.all)([axis, skipna, split_every])                                                        | Return whether all elements are True, potentially over an axis.              |
| [`Series.any`](generated/dask.dataframe.Series.any.md#dask.dataframe.Series.any)([axis, skipna, split_every])                                                        | Return whether any element is True, potentially over an axis.                |
| [`Series.apply`](generated/dask.dataframe.Series.apply.md#dask.dataframe.Series.apply)(function, \*args[, meta, axis])                                               | Parallel version of pandas.Series.apply                                      |
| [`Series.astype`](generated/dask.dataframe.Series.astype.md#dask.dataframe.Series.astype)(dtypes)                                                                    | Cast a pandas object to a specified dtype `dtype`.                           |
| [`Series.autocorr`](generated/dask.dataframe.Series.autocorr.md#dask.dataframe.Series.autocorr)([lag, split_every])                                                  | Compute the lag-N autocorrelation.                                           |
| [`Series.between`](generated/dask.dataframe.Series.between.md#dask.dataframe.Series.between)(left, right[, inclusive])                                               | Return boolean Series equivalent to left <= series <= right.                 |
| [`Series.bfill`](generated/dask.dataframe.Series.bfill.md#dask.dataframe.Series.bfill)([axis, limit])                                                                | Fill NA/NaN values by using the next valid observation to fill the gap.      |
| [`Series.clear_divisions`](generated/dask.dataframe.Series.clear_divisions.md#dask.dataframe.Series.clear_divisions)()                                               | Forget division information.                                                 |
| [`Series.clip`](generated/dask.dataframe.Series.clip.md#dask.dataframe.Series.clip)([lower, upper, axis])                                                            | Trim values at input threshold(s).                                           |
| [`Series.compute`](generated/dask.dataframe.Series.compute.md#dask.dataframe.Series.compute)(\*\*kwargs)                                                             | Compute this dask collection                                                 |
| [`Series.copy`](generated/dask.dataframe.Series.copy.md#dask.dataframe.Series.copy)([deep])                                                                          | Make a copy of the dataframe                                                 |
| [`Series.corr`](generated/dask.dataframe.Series.corr.md#dask.dataframe.Series.corr)(other[, method, min_periods, ...])                                               | Compute correlation with other Series, excluding missing values.             |
| [`Series.count`](generated/dask.dataframe.Series.count.md#dask.dataframe.Series.count)([axis, numeric_only, split_every])                                            | Count non-NA cells for each column or row.                                   |
| [`Series.cov`](generated/dask.dataframe.Series.cov.md#dask.dataframe.Series.cov)(other[, min_periods, split_every])                                                  | Compute covariance with Series, excluding missing values.                    |
| [`Series.cummax`](generated/dask.dataframe.Series.cummax.md#dask.dataframe.Series.cummax)([axis, skipna])                                                            | Return cumulative maximum over a DataFrame or Series axis.                   |
| [`Series.cummin`](generated/dask.dataframe.Series.cummin.md#dask.dataframe.Series.cummin)([axis, skipna])                                                            | Return cumulative minimum over a DataFrame or Series axis.                   |
| [`Series.cumprod`](generated/dask.dataframe.Series.cumprod.md#dask.dataframe.Series.cumprod)([axis, skipna])                                                         | Return cumulative product over a DataFrame or Series axis.                   |
| [`Series.cumsum`](generated/dask.dataframe.Series.cumsum.md#dask.dataframe.Series.cumsum)([axis, skipna])                                                            | Return cumulative sum over a DataFrame or Series axis.                       |
| [`Series.describe`](generated/dask.dataframe.Series.describe.md#dask.dataframe.Series.describe)([split_every, percentiles, ...])                                     | Generate descriptive statistics.                                             |
| [`Series.diff`](generated/dask.dataframe.Series.diff.md#dask.dataframe.Series.diff)([periods, axis])                                                                 | First discrete difference of element.                                        |
| [`Series.div`](generated/dask.dataframe.Series.div.md#dask.dataframe.Series.div)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.drop_duplicates`](generated/dask.dataframe.Series.drop_duplicates.md#dask.dataframe.Series.drop_duplicates)([ignore_index, ...])                            |                                                                              |
| [`Series.dropna`](generated/dask.dataframe.Series.dropna.md#dask.dataframe.Series.dropna)()                                                                          | Return a new Series with missing values removed.                             |
| [`Series.dtype`](generated/dask.dataframe.Series.dtype.md#dask.dataframe.Series.dtype)                                                                               |                                                                              |
| [`Series.eq`](generated/dask.dataframe.Series.eq.md#dask.dataframe.Series.eq)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.explode`](generated/dask.dataframe.Series.explode.md#dask.dataframe.Series.explode)()                                                                       | Transform each element of a list-like to a row.                              |
| [`Series.ffill`](generated/dask.dataframe.Series.ffill.md#dask.dataframe.Series.ffill)([axis, limit])                                                                | Fill NA/NaN values by propagating the last valid observation to next valid.  |
| [`Series.fillna`](generated/dask.dataframe.Series.fillna.md#dask.dataframe.Series.fillna)([value, axis])                                                             | Fill NA/NaN values with value.                                               |
| [`Series.floordiv`](generated/dask.dataframe.Series.floordiv.md#dask.dataframe.Series.floordiv)(other[, level, fill_value, axis])                                    |                                                                              |
| [`Series.ge`](generated/dask.dataframe.Series.ge.md#dask.dataframe.Series.ge)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.get_partition`](generated/dask.dataframe.Series.get_partition.md#dask.dataframe.Series.get_partition)(n)                                                    | Get a dask DataFrame/Series representing the nth partition.                  |
| [`Series.groupby`](generated/dask.dataframe.Series.groupby.md#dask.dataframe.Series.groupby)(by, \*\*kwargs)                                                         | Group Series using a mapper or by a Series of columns.                       |
| [`Series.gt`](generated/dask.dataframe.Series.gt.md#dask.dataframe.Series.gt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.head`](generated/dask.dataframe.Series.head.md#dask.dataframe.Series.head)([n, npartitions, compute])                                                       | First n rows of the dataset                                                  |
| [`Series.idxmax`](generated/dask.dataframe.Series.idxmax.md#dask.dataframe.Series.idxmax)([axis, skipna, numeric_only, ...])                                         | Return index of first occurrence of maximum over requested axis.             |
| [`Series.idxmin`](generated/dask.dataframe.Series.idxmin.md#dask.dataframe.Series.idxmin)([axis, skipna, numeric_only, ...])                                         | Return index of first occurrence of minimum over requested axis.             |
| [`Series.isin`](generated/dask.dataframe.Series.isin.md#dask.dataframe.Series.isin)(values)                                                                          | Whether each element in the DataFrame is contained in values.                |
| [`Series.isna`](generated/dask.dataframe.Series.isna.md#dask.dataframe.Series.isna)()                                                                                | Detect missing values.                                                       |
| [`Series.isnull`](generated/dask.dataframe.Series.isnull.md#dask.dataframe.Series.isnull)()                                                                          | DataFrame.isnull is an alias for DataFrame.isna.                             |
| [`Series.known_divisions`](generated/dask.dataframe.Series.known_divisions.md#dask.dataframe.Series.known_divisions)                                                 | Whether the divisions are known.                                             |
| [`Series.le`](generated/dask.dataframe.Series.le.md#dask.dataframe.Series.le)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.loc`](generated/dask.dataframe.Series.loc.md#dask.dataframe.Series.loc)                                                                                     | Purely label-location based indexer for selection by label.                  |
| [`Series.lt`](generated/dask.dataframe.Series.lt.md#dask.dataframe.Series.lt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.map`](generated/dask.dataframe.Series.map.md#dask.dataframe.Series.map)(arg[, na_action, meta])                                                             | Map values of Series according to an input mapping or function.              |
| [`Series.map_overlap`](generated/dask.dataframe.Series.map_overlap.md#dask.dataframe.Series.map_overlap)(func, before, after, \*args)                                | Apply a function to each partition, sharing rows with adjacent partitions.   |
| [`Series.map_partitions`](generated/dask.dataframe.Series.map_partitions.md#dask.dataframe.Series.map_partitions)(func, \*args[, meta, ...])                         | Apply a Python function to each partition                                    |
| [`Series.mask`](generated/dask.dataframe.Series.mask.md#dask.dataframe.Series.mask)(cond[, other])                                                                   | Replace values where the condition is True.                                  |
| [`Series.max`](generated/dask.dataframe.Series.max.md#dask.dataframe.Series.max)([axis, skipna, numeric_only, ...])                                                  | Return the maximum of the values over the requested axis.                    |
| [`Series.mean`](generated/dask.dataframe.Series.mean.md#dask.dataframe.Series.mean)([axis, skipna, numeric_only, ...])                                               | Return the mean of the values over the requested axis.                       |
| [`Series.median`](generated/dask.dataframe.Series.median.md#dask.dataframe.Series.median)()                                                                          | Return the median of the values over the requested axis.                     |
| [`Series.median_approximate`](generated/dask.dataframe.Series.median_approximate.md#dask.dataframe.Series.median_approximate)([method])                              | Return the approximate median of the values over the requested axis.         |
| [`Series.memory_usage`](generated/dask.dataframe.Series.memory_usage.md#dask.dataframe.Series.memory_usage)([deep, index])                                           | Return the memory usage of the Series.                                       |
| [`Series.memory_usage_per_partition`](generated/dask.dataframe.Series.memory_usage_per_partition.md#dask.dataframe.Series.memory_usage_per_partition)([index, deep]) | Return the memory usage of each partition                                    |
| [`Series.min`](generated/dask.dataframe.Series.min.md#dask.dataframe.Series.min)([axis, skipna, numeric_only, ...])                                                  | Return the minimum of the values over the requested axis.                    |
| [`Series.mod`](generated/dask.dataframe.Series.mod.md#dask.dataframe.Series.mod)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.mul`](generated/dask.dataframe.Series.mul.md#dask.dataframe.Series.mul)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.nbytes`](generated/dask.dataframe.Series.nbytes.md#dask.dataframe.Series.nbytes)                                                                            | Number of bytes                                                              |
| [`Series.ndim`](generated/dask.dataframe.Series.ndim.md#dask.dataframe.Series.ndim)                                                                                  | Return dimensionality                                                        |
| [`Series.ne`](generated/dask.dataframe.Series.ne.md#dask.dataframe.Series.ne)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Series.nlargest`](generated/dask.dataframe.Series.nlargest.md#dask.dataframe.Series.nlargest)([n, split_every])                                                    | Return the largest n elements.                                               |
| [`Series.notnull`](generated/dask.dataframe.Series.notnull.md#dask.dataframe.Series.notnull)()                                                                       | DataFrame.notnull is an alias for DataFrame.notna.                           |
| [`Series.nsmallest`](generated/dask.dataframe.Series.nsmallest.md#dask.dataframe.Series.nsmallest)([n, split_every])                                                 | Return the smallest n elements.                                              |
| [`Series.nunique`](generated/dask.dataframe.Series.nunique.md#dask.dataframe.Series.nunique)([dropna, split_every, split_out])                                       | Return number of unique elements in the object.                              |
| [`Series.nunique_approx`](generated/dask.dataframe.Series.nunique_approx.md#dask.dataframe.Series.nunique_approx)([split_every])                                     | Approximate number of unique rows.                                           |
| [`Series.persist`](generated/dask.dataframe.Series.persist.md#dask.dataframe.Series.persist)([fuse])                                                                 | Persist this dask collection into memory                                     |
| [`Series.pipe`](generated/dask.dataframe.Series.pipe.md#dask.dataframe.Series.pipe)(func, \*args, \*\*kwargs)                                                        | Apply chainable functions that expect Series or DataFrames.                  |
| [`Series.pow`](generated/dask.dataframe.Series.pow.md#dask.dataframe.Series.pow)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.prod`](generated/dask.dataframe.Series.prod.md#dask.dataframe.Series.prod)([axis, skipna, numeric_only, ...])                                               | Return the product of the values over the requested axis.                    |
| [`Series.quantile`](generated/dask.dataframe.Series.quantile.md#dask.dataframe.Series.quantile)([q, method])                                                         | Approximate quantiles of Series                                              |
| [`Series.radd`](generated/dask.dataframe.Series.radd.md#dask.dataframe.Series.radd)(other[, level, fill_value, axis])                                                |                                                                              |
| [`Series.random_split`](generated/dask.dataframe.Series.random_split.md#dask.dataframe.Series.random_split)(frac[, random_state, ...])                               | Pseudorandomly split dataframe into different pieces row-wise                |
| [`Series.rdiv`](generated/dask.dataframe.Series.rdiv.md#dask.dataframe.Series.rdiv)(other[, level, fill_value, axis])                                                |                                                                              |
| [`Series.repartition`](generated/dask.dataframe.Series.repartition.md#dask.dataframe.Series.repartition)([divisions, npartitions, ...])                              | Repartition a collection                                                     |
| [`Series.replace`](generated/dask.dataframe.Series.replace.md#dask.dataframe.Series.replace)([to_replace, value, regex])                                             | Replace values given in to_replace with value.                               |
| [`Series.rename`](generated/dask.dataframe.Series.rename.md#dask.dataframe.Series.rename)(index[, sorted_index])                                                     | Alter Series index labels or name                                            |
| [`Series.resample`](generated/dask.dataframe.Series.resample.md#dask.dataframe.Series.resample)(rule[, closed, label])                                               | Resample time-series data.                                                   |
| [`Series.reset_index`](generated/dask.dataframe.Series.reset_index.md#dask.dataframe.Series.reset_index)([drop])                                                     | Reset the index to the default index.                                        |
| [`Series.rolling`](generated/dask.dataframe.Series.rolling.md#dask.dataframe.Series.rolling)(window, \*\*kwargs)                                                     | Provides rolling transformations.                                            |
| [`Series.round`](generated/dask.dataframe.Series.round.md#dask.dataframe.Series.round)([decimals])                                                                   | Round numeric columns in a DataFrame to a variable number of decimal places. |
| [`Series.sample`](generated/dask.dataframe.Series.sample.md#dask.dataframe.Series.sample)([n, frac, replace, random_state])                                          | Random sample of items                                                       |
| [`Series.sem`](generated/dask.dataframe.Series.sem.md#dask.dataframe.Series.sem)([axis, skipna, ddof, ...])                                                          | Return unbiased standard error of the mean over requested axis.              |
| [`Series.shape`](generated/dask.dataframe.Series.shape.md#dask.dataframe.Series.shape)                                                                               | Return a tuple representing the dimensionality of the DataFrame.             |
| [`Series.shift`](generated/dask.dataframe.Series.shift.md#dask.dataframe.Series.shift)([periods, freq, axis])                                                        | Shift index by desired number of periods with an optional time freq.         |
| [`Series.size`](generated/dask.dataframe.Series.size.md#dask.dataframe.Series.size)                                                                                  | Size of the Series or DataFrame as a Delayed object.                         |
| [`Series.std`](generated/dask.dataframe.Series.std.md#dask.dataframe.Series.std)([axis, skipna, ddof, ...])                                                          | Return sample standard deviation over requested axis.                        |
| [`Series.sub`](generated/dask.dataframe.Series.sub.md#dask.dataframe.Series.sub)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Series.sum`](generated/dask.dataframe.Series.sum.md#dask.dataframe.Series.sum)([axis, skipna, numeric_only, ...])                                                  | Return the sum of the values over the requested axis.                        |
| [`Series.to_backend`](generated/dask.dataframe.Series.to_backend.md#dask.dataframe.Series.to_backend)([backend])                                                     | Move to a new DataFrame backend                                              |
| [`Series.to_bag`](generated/dask.dataframe.Series.to_bag.md#dask.dataframe.Series.to_bag)([index, format])                                                           | Create a Dask Bag from a Series                                              |
| [`Series.to_csv`](generated/dask.dataframe.Series.to_csv.md#dask.dataframe.Series.to_csv)(filename, \*\*kwargs)                                                      | See dd.to_csv docstring for more information                                 |
| [`Series.to_dask_array`](generated/dask.dataframe.Series.to_dask_array.md#dask.dataframe.Series.to_dask_array)([lengths, meta, optimize])                            | Convert a dask DataFrame to a dask array.                                    |
| [`Series.to_delayed`](generated/dask.dataframe.Series.to_delayed.md#dask.dataframe.Series.to_delayed)([optimize_graph])                                              | Convert into a list of `dask.delayed` objects, one per partition.            |
| [`Series.to_frame`](generated/dask.dataframe.Series.to_frame.md#dask.dataframe.Series.to_frame)([name])                                                              | Convert Series to DataFrame.                                                 |
| [`Series.to_hdf`](generated/dask.dataframe.Series.to_hdf.md#dask.dataframe.Series.to_hdf)(path_or_buf, key[, mode, append])                                          | See dd.to_hdf docstring for more information                                 |
| [`Series.to_string`](generated/dask.dataframe.Series.to_string.md#dask.dataframe.Series.to_string)([max_rows])                                                       | Render a string representation of the Series.                                |
| [`Series.to_timestamp`](generated/dask.dataframe.Series.to_timestamp.md#dask.dataframe.Series.to_timestamp)([freq, how])                                             | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.   |
| [`Series.truediv`](generated/dask.dataframe.Series.truediv.md#dask.dataframe.Series.truediv)(other[, level, fill_value, axis])                                       |                                                                              |
| [`Series.unique`](generated/dask.dataframe.Series.unique.md#dask.dataframe.Series.unique)([split_every, split_out, ...])                                             | Return Series of unique values in the object.                                |
| [`Series.value_counts`](generated/dask.dataframe.Series.value_counts.md#dask.dataframe.Series.value_counts)([sort, ascending, ...])                                  | Return a Series containing counts of unique values.                          |
| [`Series.values`](generated/dask.dataframe.Series.values.md#dask.dataframe.Series.values)                                                                            | Return a dask.array of the values of this dataframe                          |
| [`Series.var`](generated/dask.dataframe.Series.var.md#dask.dataframe.Series.var)([axis, skipna, ddof, ...])                                                          | Return unbiased variance over requested axis.                                |
| [`Series.visualize`](generated/dask.dataframe.Series.visualize.md#dask.dataframe.Series.visualize)([tasks])                                                          | Visualize the expression or task graph                                       |
| [`Series.where`](generated/dask.dataframe.Series.where.md#dask.dataframe.Series.where)(cond[, other])                                                                | Replace values where the condition is False.                                 |

## Index

| [`Index`](generated/dask.dataframe.Index.md#dask.dataframe.Index)(expr)                                                                                           | Index-like Expr Collection.                                                  |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|
| [`Index.add`](generated/dask.dataframe.Index.add.md#dask.dataframe.Index.add)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.align`](generated/dask.dataframe.Index.align.md#dask.dataframe.Index.align)(other[, join, axis, fill_value])                                              | Align two objects on their axes with the specified join method.              |
| [`Index.all`](generated/dask.dataframe.Index.all.md#dask.dataframe.Index.all)([axis, skipna, split_every])                                                        | Return whether all elements are True, potentially over an axis.              |
| [`Index.any`](generated/dask.dataframe.Index.any.md#dask.dataframe.Index.any)([axis, skipna, split_every])                                                        | Return whether any element is True, potentially over an axis.                |
| [`Index.apply`](generated/dask.dataframe.Index.apply.md#dask.dataframe.Index.apply)(function, \*args[, meta, axis])                                               | Parallel version of pandas.Series.apply                                      |
| [`Index.astype`](generated/dask.dataframe.Index.astype.md#dask.dataframe.Index.astype)(dtypes)                                                                    | Cast a pandas object to a specified dtype `dtype`.                           |
| [`Index.autocorr`](generated/dask.dataframe.Index.autocorr.md#dask.dataframe.Index.autocorr)([lag, split_every])                                                  | Compute the lag-N autocorrelation.                                           |
| [`Index.between`](generated/dask.dataframe.Index.between.md#dask.dataframe.Index.between)(left, right[, inclusive])                                               | Return boolean Series equivalent to left <= series <= right.                 |
| [`Index.bfill`](generated/dask.dataframe.Index.bfill.md#dask.dataframe.Index.bfill)([axis, limit])                                                                | Fill NA/NaN values by using the next valid observation to fill the gap.      |
| [`Index.clear_divisions`](generated/dask.dataframe.Index.clear_divisions.md#dask.dataframe.Index.clear_divisions)()                                               | Forget division information.                                                 |
| [`Index.clip`](generated/dask.dataframe.Index.clip.md#dask.dataframe.Index.clip)([lower, upper, axis])                                                            | Trim values at input threshold(s).                                           |
| [`Index.compute`](generated/dask.dataframe.Index.compute.md#dask.dataframe.Index.compute)(\*\*kwargs)                                                             | Compute this dask collection                                                 |
| [`Index.copy`](generated/dask.dataframe.Index.copy.md#dask.dataframe.Index.copy)([deep])                                                                          | Make a copy of the dataframe                                                 |
| [`Index.corr`](generated/dask.dataframe.Index.corr.md#dask.dataframe.Index.corr)(other[, method, min_periods, ...])                                               | Compute correlation with other Series, excluding missing values.             |
| [`Index.count`](generated/dask.dataframe.Index.count.md#dask.dataframe.Index.count)([split_every])                                                                | Count non-NA cells for each column or row.                                   |
| [`Index.cov`](generated/dask.dataframe.Index.cov.md#dask.dataframe.Index.cov)(other[, min_periods, split_every])                                                  | Compute covariance with Series, excluding missing values.                    |
| [`Index.cummax`](generated/dask.dataframe.Index.cummax.md#dask.dataframe.Index.cummax)([axis, skipna])                                                            | Return cumulative maximum over a DataFrame or Series axis.                   |
| [`Index.cummin`](generated/dask.dataframe.Index.cummin.md#dask.dataframe.Index.cummin)([axis, skipna])                                                            | Return cumulative minimum over a DataFrame or Series axis.                   |
| [`Index.cumprod`](generated/dask.dataframe.Index.cumprod.md#dask.dataframe.Index.cumprod)([axis, skipna])                                                         | Return cumulative product over a DataFrame or Series axis.                   |
| [`Index.cumsum`](generated/dask.dataframe.Index.cumsum.md#dask.dataframe.Index.cumsum)([axis, skipna])                                                            | Return cumulative sum over a DataFrame or Series axis.                       |
| [`Index.describe`](generated/dask.dataframe.Index.describe.md#dask.dataframe.Index.describe)([split_every, percentiles, ...])                                     | Generate descriptive statistics.                                             |
| [`Index.diff`](generated/dask.dataframe.Index.diff.md#dask.dataframe.Index.diff)([periods, axis])                                                                 | First discrete difference of element.                                        |
| [`Index.div`](generated/dask.dataframe.Index.div.md#dask.dataframe.Index.div)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.drop_duplicates`](generated/dask.dataframe.Index.drop_duplicates.md#dask.dataframe.Index.drop_duplicates)([ignore_index, ...])                            |                                                                              |
| [`Index.dropna`](generated/dask.dataframe.Index.dropna.md#dask.dataframe.Index.dropna)()                                                                          | Return a new Series with missing values removed.                             |
| [`Index.dtype`](generated/dask.dataframe.Index.dtype.md#dask.dataframe.Index.dtype)                                                                               |                                                                              |
| [`Index.eq`](generated/dask.dataframe.Index.eq.md#dask.dataframe.Index.eq)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.explode`](generated/dask.dataframe.Index.explode.md#dask.dataframe.Index.explode)()                                                                       | Transform each element of a list-like to a row.                              |
| [`Index.ffill`](generated/dask.dataframe.Index.ffill.md#dask.dataframe.Index.ffill)([axis, limit])                                                                | Fill NA/NaN values by propagating the last valid observation to next valid.  |
| [`Index.fillna`](generated/dask.dataframe.Index.fillna.md#dask.dataframe.Index.fillna)([value, axis])                                                             | Fill NA/NaN values with value.                                               |
| [`Index.floordiv`](generated/dask.dataframe.Index.floordiv.md#dask.dataframe.Index.floordiv)(other[, level, fill_value, axis])                                    |                                                                              |
| [`Index.ge`](generated/dask.dataframe.Index.ge.md#dask.dataframe.Index.ge)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.get_partition`](generated/dask.dataframe.Index.get_partition.md#dask.dataframe.Index.get_partition)(n)                                                    | Get a dask DataFrame/Series representing the nth partition.                  |
| [`Index.groupby`](generated/dask.dataframe.Index.groupby.md#dask.dataframe.Index.groupby)(by, \*\*kwargs)                                                         | Group Series using a mapper or by a Series of columns.                       |
| [`Index.gt`](generated/dask.dataframe.Index.gt.md#dask.dataframe.Index.gt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.head`](generated/dask.dataframe.Index.head.md#dask.dataframe.Index.head)([n, npartitions, compute])                                                       | First n rows of the dataset                                                  |
| [`Index.is_monotonic_decreasing`](generated/dask.dataframe.Index.is_monotonic_decreasing.md#dask.dataframe.Index.is_monotonic_decreasing)                         | Return True if values in the object are monotonically decreasing.            |
| [`Index.is_monotonic_increasing`](generated/dask.dataframe.Index.is_monotonic_increasing.md#dask.dataframe.Index.is_monotonic_increasing)                         | Return True if values in the object are monotonically increasing.            |
| [`Index.isin`](generated/dask.dataframe.Index.isin.md#dask.dataframe.Index.isin)(values)                                                                          | Whether each element in the DataFrame is contained in values.                |
| [`Index.isna`](generated/dask.dataframe.Index.isna.md#dask.dataframe.Index.isna)()                                                                                | Detect missing values.                                                       |
| [`Index.isnull`](generated/dask.dataframe.Index.isnull.md#dask.dataframe.Index.isnull)()                                                                          | DataFrame.isnull is an alias for DataFrame.isna.                             |
| [`Index.known_divisions`](generated/dask.dataframe.Index.known_divisions.md#dask.dataframe.Index.known_divisions)                                                 | Whether the divisions are known.                                             |
| [`Index.le`](generated/dask.dataframe.Index.le.md#dask.dataframe.Index.le)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.loc`](generated/dask.dataframe.Index.loc.md#dask.dataframe.Index.loc)                                                                                     | Purely label-location based indexer for selection by label.                  |
| [`Index.lt`](generated/dask.dataframe.Index.lt.md#dask.dataframe.Index.lt)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.map`](generated/dask.dataframe.Index.map.md#dask.dataframe.Index.map)(arg[, na_action, meta, is_monotonic])                                               | Map values using an input mapping or function.                               |
| [`Index.map_overlap`](generated/dask.dataframe.Index.map_overlap.md#dask.dataframe.Index.map_overlap)(func, before, after, \*args)                                | Apply a function to each partition, sharing rows with adjacent partitions.   |
| [`Index.map_partitions`](generated/dask.dataframe.Index.map_partitions.md#dask.dataframe.Index.map_partitions)(func, \*args[, meta, ...])                         | Apply a Python function to each partition                                    |
| [`Index.mask`](generated/dask.dataframe.Index.mask.md#dask.dataframe.Index.mask)(cond[, other])                                                                   | Replace values where the condition is True.                                  |
| [`Index.max`](generated/dask.dataframe.Index.max.md#dask.dataframe.Index.max)([axis, skipna, numeric_only, ...])                                                  | Return the maximum of the values over the requested axis.                    |
| [`Index.median`](generated/dask.dataframe.Index.median.md#dask.dataframe.Index.median)()                                                                          | Return the median of the values over the requested axis.                     |
| [`Index.median_approximate`](generated/dask.dataframe.Index.median_approximate.md#dask.dataframe.Index.median_approximate)([method])                              | Return the approximate median of the values over the requested axis.         |
| [`Index.memory_usage`](generated/dask.dataframe.Index.memory_usage.md#dask.dataframe.Index.memory_usage)([deep])                                                  | Memory usage of the values.                                                  |
| [`Index.memory_usage_per_partition`](generated/dask.dataframe.Index.memory_usage_per_partition.md#dask.dataframe.Index.memory_usage_per_partition)([index, deep]) | Return the memory usage of each partition                                    |
| [`Index.min`](generated/dask.dataframe.Index.min.md#dask.dataframe.Index.min)([axis, skipna, numeric_only, ...])                                                  | Return the minimum of the values over the requested axis.                    |
| [`Index.mod`](generated/dask.dataframe.Index.mod.md#dask.dataframe.Index.mod)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.mul`](generated/dask.dataframe.Index.mul.md#dask.dataframe.Index.mul)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.nbytes`](generated/dask.dataframe.Index.nbytes.md#dask.dataframe.Index.nbytes)                                                                            | Number of bytes                                                              |
| [`Index.ndim`](generated/dask.dataframe.Index.ndim.md#dask.dataframe.Index.ndim)                                                                                  | Return dimensionality                                                        |
| [`Index.ne`](generated/dask.dataframe.Index.ne.md#dask.dataframe.Index.ne)(other[, level, fill_value, axis])                                                      |                                                                              |
| [`Index.nlargest`](generated/dask.dataframe.Index.nlargest.md#dask.dataframe.Index.nlargest)([n, split_every])                                                    | Return the largest n elements.                                               |
| [`Index.notnull`](generated/dask.dataframe.Index.notnull.md#dask.dataframe.Index.notnull)()                                                                       | DataFrame.notnull is an alias for DataFrame.notna.                           |
| [`Index.nsmallest`](generated/dask.dataframe.Index.nsmallest.md#dask.dataframe.Index.nsmallest)([n, split_every])                                                 | Return the smallest n elements.                                              |
| [`Index.nunique`](generated/dask.dataframe.Index.nunique.md#dask.dataframe.Index.nunique)([dropna, split_every, split_out])                                       | Return number of unique elements in the object.                              |
| [`Index.nunique_approx`](generated/dask.dataframe.Index.nunique_approx.md#dask.dataframe.Index.nunique_approx)([split_every])                                     | Approximate number of unique rows.                                           |
| [`Index.persist`](generated/dask.dataframe.Index.persist.md#dask.dataframe.Index.persist)([fuse])                                                                 | Persist this dask collection into memory                                     |
| [`Index.pipe`](generated/dask.dataframe.Index.pipe.md#dask.dataframe.Index.pipe)(func, \*args, \*\*kwargs)                                                        | Apply chainable functions that expect Series or DataFrames.                  |
| [`Index.pow`](generated/dask.dataframe.Index.pow.md#dask.dataframe.Index.pow)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.quantile`](generated/dask.dataframe.Index.quantile.md#dask.dataframe.Index.quantile)([q, method])                                                         | Approximate quantiles of Series                                              |
| [`Index.radd`](generated/dask.dataframe.Index.radd.md#dask.dataframe.Index.radd)(other[, level, fill_value, axis])                                                |                                                                              |
| [`Index.random_split`](generated/dask.dataframe.Index.random_split.md#dask.dataframe.Index.random_split)(frac[, random_state, shuffle])                           | Pseudorandomly split dataframe into different pieces row-wise                |
| [`Index.rdiv`](generated/dask.dataframe.Index.rdiv.md#dask.dataframe.Index.rdiv)(other[, level, fill_value, axis])                                                |                                                                              |
| [`Index.rename`](generated/dask.dataframe.Index.rename.md#dask.dataframe.Index.rename)(index[, sorted_index])                                                     | Alter Series index labels or name                                            |
| [`Index.repartition`](generated/dask.dataframe.Index.repartition.md#dask.dataframe.Index.repartition)([divisions, npartitions, ...])                              | Repartition a collection                                                     |
| [`Index.replace`](generated/dask.dataframe.Index.replace.md#dask.dataframe.Index.replace)([to_replace, value, regex])                                             | Replace values given in to_replace with value.                               |
| [`Index.resample`](generated/dask.dataframe.Index.resample.md#dask.dataframe.Index.resample)(rule[, closed, label])                                               | Resample time-series data.                                                   |
| [`Index.reset_index`](generated/dask.dataframe.Index.reset_index.md#dask.dataframe.Index.reset_index)([drop])                                                     | Reset the index to the default index.                                        |
| [`Index.rolling`](generated/dask.dataframe.Index.rolling.md#dask.dataframe.Index.rolling)(window, \*\*kwargs)                                                     | Provides rolling transformations.                                            |
| [`Index.round`](generated/dask.dataframe.Index.round.md#dask.dataframe.Index.round)([decimals])                                                                   | Round numeric columns in a DataFrame to a variable number of decimal places. |
| [`Index.sample`](generated/dask.dataframe.Index.sample.md#dask.dataframe.Index.sample)([n, frac, replace, random_state])                                          | Random sample of items                                                       |
| [`Index.sem`](generated/dask.dataframe.Index.sem.md#dask.dataframe.Index.sem)([axis, skipna, ddof, split_every, ...])                                             | Return unbiased standard error of the mean over requested axis.              |
| [`Index.shape`](generated/dask.dataframe.Index.shape.md#dask.dataframe.Index.shape)                                                                               | Return a tuple representing the dimensionality of the DataFrame.             |
| [`Index.shift`](generated/dask.dataframe.Index.shift.md#dask.dataframe.Index.shift)([periods, freq])                                                              | Shift index by desired number of periods with an optional time freq.         |
| [`Index.size`](generated/dask.dataframe.Index.size.md#dask.dataframe.Index.size)                                                                                  | Size of the Series or DataFrame as a Delayed object.                         |
| [`Index.sub`](generated/dask.dataframe.Index.sub.md#dask.dataframe.Index.sub)(other[, level, fill_value, axis])                                                   |                                                                              |
| [`Index.to_backend`](generated/dask.dataframe.Index.to_backend.md#dask.dataframe.Index.to_backend)([backend])                                                     | Move to a new DataFrame backend                                              |
| [`Index.to_bag`](generated/dask.dataframe.Index.to_bag.md#dask.dataframe.Index.to_bag)([index, format])                                                           | Create a Dask Bag from a Series                                              |
| [`Index.to_csv`](generated/dask.dataframe.Index.to_csv.md#dask.dataframe.Index.to_csv)(filename, \*\*kwargs)                                                      | See dd.to_csv docstring for more information                                 |
| [`Index.to_dask_array`](generated/dask.dataframe.Index.to_dask_array.md#dask.dataframe.Index.to_dask_array)([lengths, meta, optimize])                            | Convert a dask DataFrame to a dask array.                                    |
| [`Index.to_delayed`](generated/dask.dataframe.Index.to_delayed.md#dask.dataframe.Index.to_delayed)([optimize_graph])                                              | Convert into a list of `dask.delayed` objects, one per partition.            |
| [`Index.to_frame`](generated/dask.dataframe.Index.to_frame.md#dask.dataframe.Index.to_frame)([index, name])                                                       | Create a DataFrame with a column containing the Index.                       |
| [`Index.to_hdf`](generated/dask.dataframe.Index.to_hdf.md#dask.dataframe.Index.to_hdf)(path_or_buf, key[, mode, append])                                          | See dd.to_hdf docstring for more information                                 |
| [`Index.to_series`](generated/dask.dataframe.Index.to_series.md#dask.dataframe.Index.to_series)([index, name])                                                    | Create a Series with both index and values equal to the index keys.          |
| [`Index.to_string`](generated/dask.dataframe.Index.to_string.md#dask.dataframe.Index.to_string)([max_rows])                                                       | Render a string representation of the Series.                                |
| [`Index.to_timestamp`](generated/dask.dataframe.Index.to_timestamp.md#dask.dataframe.Index.to_timestamp)([freq, how])                                             | Cast PeriodIndex to DatetimeIndex of timestamps, at *beginning* of period.   |
| [`Index.truediv`](generated/dask.dataframe.Index.truediv.md#dask.dataframe.Index.truediv)(other[, level, fill_value, axis])                                       |                                                                              |
| [`Index.unique`](generated/dask.dataframe.Index.unique.md#dask.dataframe.Index.unique)([split_every, split_out, ...])                                             | Return Series of unique values in the object.                                |
| [`Index.value_counts`](generated/dask.dataframe.Index.value_counts.md#dask.dataframe.Index.value_counts)([sort, ascending, ...])                                  | Return a Series containing counts of unique values.                          |
| [`Index.values`](generated/dask.dataframe.Index.values.md#dask.dataframe.Index.values)                                                                            | Return a dask.array of the values of this dataframe                          |
| [`Index.visualize`](generated/dask.dataframe.Index.visualize.md#dask.dataframe.Index.visualize)([tasks])                                                          | Visualize the expression or task graph                                       |
| [`Index.where`](generated/dask.dataframe.Index.where.md#dask.dataframe.Index.where)(cond[, other])                                                                | Replace values where the condition is False.                                 |
| [`Index.to_frame`](generated/dask.dataframe.Index.to_frame.md#dask.dataframe.Index.to_frame)([index, name])                                                       | Create a DataFrame with a column containing the Index.                       |

## Accessors

Similar to pandas, Dask provides dtype-specific methods under various accessors.
These are separate namespaces within [`Series`](generated/dask.dataframe.Series.md#dask.dataframe.Series) that only apply to specific data types.

### Datetime Accessor

**Methods**

| [`Series.dt.ceil`](generated/dask.dataframe.Series.dt.ceil.md#dask.dataframe.Series.dt.ceil)(freq[, ambiguous, nonexistent])    | Perform ceil operation on the data to the specified freq.         |
|---------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`Series.dt.floor`](generated/dask.dataframe.Series.dt.floor.md#dask.dataframe.Series.dt.floor)(freq[, ambiguous, nonexistent]) | Perform floor operation on the data to the specified freq.        |
| [`Series.dt.isocalendar`](generated/dask.dataframe.Series.dt.isocalendar.md#dask.dataframe.Series.dt.isocalendar)()             | Calculate year, week, and day according to the ISO 8601 standard. |
| [`Series.dt.normalize`](generated/dask.dataframe.Series.dt.normalize.md#dask.dataframe.Series.dt.normalize)()                   | Convert times to midnight.                                        |
| [`Series.dt.round`](generated/dask.dataframe.Series.dt.round.md#dask.dataframe.Series.dt.round)(freq[, ambiguous, nonexistent]) | Perform round operation on the data to the specified freq.        |
| [`Series.dt.strftime`](generated/dask.dataframe.Series.dt.strftime.md#dask.dataframe.Series.dt.strftime)(date_format)           | Convert to Index using specified date_format.                     |

**Attributes**

| [`Series.dt.date`](generated/dask.dataframe.Series.dt.date.md#dask.dataframe.Series.dt.date)                                     | Returns numpy array of python [`datetime.date`](https://docs.python.org/3/library/datetime.html#datetime.date) objects.         |
|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
| [`Series.dt.day`](generated/dask.dataframe.Series.dt.day.md#dask.dataframe.Series.dt.day)                                        | The day of the datetime.                                                                                                        |
| [`Series.dt.day_of_week`](generated/dask.dataframe.Series.dt.day_of_week.md#dask.dataframe.Series.dt.day_of_week)                | The day of the week with Monday=0, Sunday=6.                                                                                    |
| [`Series.dt.day_of_year`](generated/dask.dataframe.Series.dt.day_of_year.md#dask.dataframe.Series.dt.day_of_year)                | The ordinal day of the year.                                                                                                    |
| [`Series.dt.dayofweek`](generated/dask.dataframe.Series.dt.dayofweek.md#dask.dataframe.Series.dt.dayofweek)                      | The day of the week with Monday=0, Sunday=6.                                                                                    |
| [`Series.dt.dayofyear`](generated/dask.dataframe.Series.dt.dayofyear.md#dask.dataframe.Series.dt.dayofyear)                      | The ordinal day of the year.                                                                                                    |
| [`Series.dt.days_in_month`](generated/dask.dataframe.Series.dt.days_in_month.md#dask.dataframe.Series.dt.days_in_month)          | The number of days in the month.                                                                                                |
| [`Series.dt.daysinmonth`](generated/dask.dataframe.Series.dt.daysinmonth.md#dask.dataframe.Series.dt.daysinmonth)                | The number of days in the month.                                                                                                |
| [`Series.dt.freq`](generated/dask.dataframe.Series.dt.freq.md#dask.dataframe.Series.dt.freq)                                     | Tries to return a string representing a frequency generated by infer_freq.                                                      |
| [`Series.dt.hour`](generated/dask.dataframe.Series.dt.hour.md#dask.dataframe.Series.dt.hour)                                     | The hours of the datetime.                                                                                                      |
| [`Series.dt.is_leap_year`](generated/dask.dataframe.Series.dt.is_leap_year.md#dask.dataframe.Series.dt.is_leap_year)             | Boolean indicator if the date belongs to a leap year.                                                                           |
| [`Series.dt.is_month_end`](generated/dask.dataframe.Series.dt.is_month_end.md#dask.dataframe.Series.dt.is_month_end)             | Indicates whether the date is the last day of the month.                                                                        |
| [`Series.dt.is_month_start`](generated/dask.dataframe.Series.dt.is_month_start.md#dask.dataframe.Series.dt.is_month_start)       | Indicates whether the date is the first day of the month.                                                                       |
| [`Series.dt.is_quarter_end`](generated/dask.dataframe.Series.dt.is_quarter_end.md#dask.dataframe.Series.dt.is_quarter_end)       | Indicator for whether the date is the last day of a quarter.                                                                    |
| [`Series.dt.is_quarter_start`](generated/dask.dataframe.Series.dt.is_quarter_start.md#dask.dataframe.Series.dt.is_quarter_start) | Indicator for whether the date is the first day of a quarter.                                                                   |
| [`Series.dt.is_year_end`](generated/dask.dataframe.Series.dt.is_year_end.md#dask.dataframe.Series.dt.is_year_end)                | Indicate whether the date is the last day of the year.                                                                          |
| [`Series.dt.is_year_start`](generated/dask.dataframe.Series.dt.is_year_start.md#dask.dataframe.Series.dt.is_year_start)          | Indicate whether the date is the first day of a year.                                                                           |
| [`Series.dt.microsecond`](generated/dask.dataframe.Series.dt.microsecond.md#dask.dataframe.Series.dt.microsecond)                | The microseconds of the datetime.                                                                                               |
| [`Series.dt.minute`](generated/dask.dataframe.Series.dt.minute.md#dask.dataframe.Series.dt.minute)                               | The minutes of the datetime.                                                                                                    |
| [`Series.dt.month`](generated/dask.dataframe.Series.dt.month.md#dask.dataframe.Series.dt.month)                                  | The month as January=1, December=12.                                                                                            |
| [`Series.dt.nanosecond`](generated/dask.dataframe.Series.dt.nanosecond.md#dask.dataframe.Series.dt.nanosecond)                   | The nanoseconds of the datetime.                                                                                                |
| [`Series.dt.quarter`](generated/dask.dataframe.Series.dt.quarter.md#dask.dataframe.Series.dt.quarter)                            | The quarter of the date.                                                                                                        |
| [`Series.dt.second`](generated/dask.dataframe.Series.dt.second.md#dask.dataframe.Series.dt.second)                               | The seconds of the datetime.                                                                                                    |
| [`Series.dt.time`](generated/dask.dataframe.Series.dt.time.md#dask.dataframe.Series.dt.time)                                     | Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects.                |
| [`Series.dt.timetz`](generated/dask.dataframe.Series.dt.timetz.md#dask.dataframe.Series.dt.timetz)                               | Returns numpy array of [`datetime.time`](https://docs.python.org/3/library/datetime.html#datetime.time) objects with timezones. |
| [`Series.dt.tz`](generated/dask.dataframe.Series.dt.tz.md#dask.dataframe.Series.dt.tz)                                           | Return the timezone.                                                                                                            |
| [`Series.dt.week`](generated/dask.dataframe.Series.dt.week.md#dask.dataframe.Series.dt.week)                                     | The week ordinal of the year.                                                                                                   |
| [`Series.dt.weekday`](generated/dask.dataframe.Series.dt.weekday.md#dask.dataframe.Series.dt.weekday)                            | The day of the week with Monday=0, Sunday=6.                                                                                    |
| [`Series.dt.weekofyear`](generated/dask.dataframe.Series.dt.weekofyear.md#dask.dataframe.Series.dt.weekofyear)                   | The week ordinal of the year.                                                                                                   |
| [`Series.dt.year`](generated/dask.dataframe.Series.dt.year.md#dask.dataframe.Series.dt.year)                                     | The year of the datetime.                                                                                                       |

### String Accessor

**Methods**

| [`Series.str.capitalize`](generated/dask.dataframe.Series.str.capitalize.md#dask.dataframe.Series.str.capitalize)()                      | Convert strings in the Series/Index to be capitalized.                                    |
|------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------|
| [`Series.str.casefold`](generated/dask.dataframe.Series.str.casefold.md#dask.dataframe.Series.str.casefold)()                            | Convert strings in the Series/Index to be casefolded.                                     |
| [`Series.str.cat`](generated/dask.dataframe.Series.str.cat.md#dask.dataframe.Series.str.cat)([others, sep, na_rep])                      |                                                                                           |
| [`Series.str.center`](generated/dask.dataframe.Series.str.center.md#dask.dataframe.Series.str.center)(width[, fillchar])                 | Pad left and right side of strings in the Series/Index.                                   |
| [`Series.str.contains`](generated/dask.dataframe.Series.str.contains.md#dask.dataframe.Series.str.contains)(pat[, case, flags, na, ...]) | Test if pattern or regex is contained within a string of a Series or Index.               |
| [`Series.str.count`](generated/dask.dataframe.Series.str.count.md#dask.dataframe.Series.str.count)(pat[, flags])                         | Count occurrences of pattern in each string of the Series/Index.                          |
| [`Series.str.decode`](generated/dask.dataframe.Series.str.decode.md#dask.dataframe.Series.str.decode)(encoding[, errors, dtype])         | Decode character string in the Series/Index using indicated encoding.                     |
| [`Series.str.encode`](generated/dask.dataframe.Series.str.encode.md#dask.dataframe.Series.str.encode)(encoding[, errors])                | Encode character string in the Series/Index using indicated encoding.                     |
| [`Series.str.endswith`](generated/dask.dataframe.Series.str.endswith.md#dask.dataframe.Series.str.endswith)(pat[, na])                   | Test if the end of each string element matches a pattern.                                 |
| [`Series.str.extract`](generated/dask.dataframe.Series.str.extract.md#dask.dataframe.Series.str.extract)(pat[, flags, expand])           | Extract capture groups in the regex pat as columns in a DataFrame.                        |
| [`Series.str.extractall`](generated/dask.dataframe.Series.str.extractall.md#dask.dataframe.Series.str.extractall)(pat[, flags])          | Extract capture groups in the regex pat as columns in DataFrame.                          |
| [`Series.str.find`](generated/dask.dataframe.Series.str.find.md#dask.dataframe.Series.str.find)(sub[, start, end])                       | Return lowest indexes in each strings in the Series/Index.                                |
| [`Series.str.findall`](generated/dask.dataframe.Series.str.findall.md#dask.dataframe.Series.str.findall)(pat[, flags])                   | Find all occurrences of pattern or regular expression in the Series/Index.                |
| [`Series.str.fullmatch`](generated/dask.dataframe.Series.str.fullmatch.md#dask.dataframe.Series.str.fullmatch)(pat[, case, flags, na])   | Determine if each string entirely matches a regular expression.                           |
| [`Series.str.get`](generated/dask.dataframe.Series.str.get.md#dask.dataframe.Series.str.get)(i)                                          | Extract element from each component at specified position or with specified key.          |
| [`Series.str.index`](generated/dask.dataframe.Series.str.index.md#dask.dataframe.Series.str.index)(sub[, start, end])                    | Return lowest indexes in each string in Series/Index.                                     |
| [`Series.str.isalnum`](generated/dask.dataframe.Series.str.isalnum.md#dask.dataframe.Series.str.isalnum)()                               | Check whether all characters in each string are alphanumeric.                             |
| [`Series.str.isalpha`](generated/dask.dataframe.Series.str.isalpha.md#dask.dataframe.Series.str.isalpha)()                               | Check whether all characters in each string are alphabetic.                               |
| [`Series.str.isdecimal`](generated/dask.dataframe.Series.str.isdecimal.md#dask.dataframe.Series.str.isdecimal)()                         | Check whether all characters in each string are decimal.                                  |
| [`Series.str.isdigit`](generated/dask.dataframe.Series.str.isdigit.md#dask.dataframe.Series.str.isdigit)()                               | Check whether all characters in each string are digits.                                   |
| [`Series.str.islower`](generated/dask.dataframe.Series.str.islower.md#dask.dataframe.Series.str.islower)()                               | Check whether all characters in each string are lowercase.                                |
| [`Series.str.isnumeric`](generated/dask.dataframe.Series.str.isnumeric.md#dask.dataframe.Series.str.isnumeric)()                         | Check whether all characters in each string are numeric.                                  |
| [`Series.str.isspace`](generated/dask.dataframe.Series.str.isspace.md#dask.dataframe.Series.str.isspace)()                               | Check whether all characters in each string are whitespace.                               |
| [`Series.str.istitle`](generated/dask.dataframe.Series.str.istitle.md#dask.dataframe.Series.str.istitle)()                               | Check whether all characters in each string are titlecase.                                |
| [`Series.str.isupper`](generated/dask.dataframe.Series.str.isupper.md#dask.dataframe.Series.str.isupper)()                               | Check whether all characters in each string are uppercase.                                |
| [`Series.str.join`](generated/dask.dataframe.Series.str.join.md#dask.dataframe.Series.str.join)(sep)                                     | Join lists contained as elements in the Series/Index with passed delimiter.               |
| [`Series.str.len`](generated/dask.dataframe.Series.str.len.md#dask.dataframe.Series.str.len)()                                           | Compute the length of each element in the Series/Index.                                   |
| [`Series.str.ljust`](generated/dask.dataframe.Series.str.ljust.md#dask.dataframe.Series.str.ljust)(width[, fillchar])                    | Pad right side of strings in the Series/Index.                                            |
| [`Series.str.lower`](generated/dask.dataframe.Series.str.lower.md#dask.dataframe.Series.str.lower)()                                     | Convert strings in the Series/Index to lowercase.                                         |
| [`Series.str.lstrip`](generated/dask.dataframe.Series.str.lstrip.md#dask.dataframe.Series.str.lstrip)([to_strip])                        | Remove leading characters.                                                                |
| [`Series.str.match`](generated/dask.dataframe.Series.str.match.md#dask.dataframe.Series.str.match)(pat[, case, flags, na])               | Determine if each string starts with a match of a regular expression.                     |
| [`Series.str.normalize`](generated/dask.dataframe.Series.str.normalize.md#dask.dataframe.Series.str.normalize)(form)                     | Return the Unicode normal form for the strings in the Series/Index.                       |
| [`Series.str.pad`](generated/dask.dataframe.Series.str.pad.md#dask.dataframe.Series.str.pad)(width[, side, fillchar])                    | Pad strings in the Series/Index up to width.                                              |
| [`Series.str.partition`](generated/dask.dataframe.Series.str.partition.md#dask.dataframe.Series.str.partition)([sep, expand])            | Split the string at the first occurrence of sep.                                          |
| [`Series.str.repeat`](generated/dask.dataframe.Series.str.repeat.md#dask.dataframe.Series.str.repeat)(repeats)                           | Duplicate each string in the Series or Index.                                             |
| [`Series.str.replace`](generated/dask.dataframe.Series.str.replace.md#dask.dataframe.Series.str.replace)(pat[, repl, n, case, ...])      | Replace each occurrence of pattern/regex in the Series/Index.                             |
| [`Series.str.rfind`](generated/dask.dataframe.Series.str.rfind.md#dask.dataframe.Series.str.rfind)(sub[, start, end])                    | Return highest indexes in each strings in the Series/Index.                               |
| [`Series.str.rindex`](generated/dask.dataframe.Series.str.rindex.md#dask.dataframe.Series.str.rindex)(sub[, start, end])                 | Return highest indexes in each string in Series/Index.                                    |
| [`Series.str.rjust`](generated/dask.dataframe.Series.str.rjust.md#dask.dataframe.Series.str.rjust)(width[, fillchar])                    | Pad left side of strings in the Series/Index.                                             |
| [`Series.str.rpartition`](generated/dask.dataframe.Series.str.rpartition.md#dask.dataframe.Series.str.rpartition)([sep, expand])         | Split the string at the last occurrence of sep.                                           |
| [`Series.str.rsplit`](generated/dask.dataframe.Series.str.rsplit.md#dask.dataframe.Series.str.rsplit)([pat, n, expand])                  |                                                                                           |
| [`Series.str.rstrip`](generated/dask.dataframe.Series.str.rstrip.md#dask.dataframe.Series.str.rstrip)([to_strip])                        | Remove trailing characters.                                                               |
| [`Series.str.slice`](generated/dask.dataframe.Series.str.slice.md#dask.dataframe.Series.str.slice)([start, stop, step])                  | Slice substrings from each element in the Series or Index.                                |
| [`Series.str.split`](generated/dask.dataframe.Series.str.split.md#dask.dataframe.Series.str.split)([pat, n, expand])                     | Known inconsistencies: `expand=True` with unknown `n` will raise a `NotImplementedError`. |
| [`Series.str.startswith`](generated/dask.dataframe.Series.str.startswith.md#dask.dataframe.Series.str.startswith)(pat[, na])             | Test if the start of each string element matches a pattern.                               |
| [`Series.str.strip`](generated/dask.dataframe.Series.str.strip.md#dask.dataframe.Series.str.strip)([to_strip])                           | Remove leading and trailing characters.                                                   |
| [`Series.str.swapcase`](generated/dask.dataframe.Series.str.swapcase.md#dask.dataframe.Series.str.swapcase)()                            | Convert strings in the Series/Index to be swapcased.                                      |
| [`Series.str.title`](generated/dask.dataframe.Series.str.title.md#dask.dataframe.Series.str.title)()                                     | Convert strings in the Series/Index to titlecase.                                         |
| [`Series.str.translate`](generated/dask.dataframe.Series.str.translate.md#dask.dataframe.Series.str.translate)(table)                    | Map all characters in the string through the given mapping table.                         |
| [`Series.str.upper`](generated/dask.dataframe.Series.str.upper.md#dask.dataframe.Series.str.upper)()                                     | Convert strings in the Series/Index to uppercase.                                         |
| [`Series.str.wrap`](generated/dask.dataframe.Series.str.wrap.md#dask.dataframe.Series.str.wrap)(width[, expand_tabs, ...])               | Wrap strings in Series/Index at specified line width.                                     |
| [`Series.str.zfill`](generated/dask.dataframe.Series.str.zfill.md#dask.dataframe.Series.str.zfill)(width)                                | Pad strings in the Series/Index by prepending '0' characters.                             |

### Categorical Accessor

**Methods**

| [`Series.cat.add_categories`](generated/dask.dataframe.Series.cat.add_categories.md#dask.dataframe.Series.cat.add_categories)(new_categories)                 | Add new categories.                                 |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|
| [`Series.cat.as_known`](generated/dask.dataframe.Series.cat.as_known.md#dask.dataframe.Series.cat.as_known)(\*\*kwargs)                                       | Ensure the categories in this series are known.     |
| [`Series.cat.as_ordered`](generated/dask.dataframe.Series.cat.as_ordered.md#dask.dataframe.Series.cat.as_ordered)()                                           | Set the Categorical to be ordered.                  |
| [`Series.cat.as_unknown`](generated/dask.dataframe.Series.cat.as_unknown.md#dask.dataframe.Series.cat.as_unknown)()                                           | Ensure the categories in this series are unknown    |
| [`Series.cat.as_unordered`](generated/dask.dataframe.Series.cat.as_unordered.md#dask.dataframe.Series.cat.as_unordered)()                                     | Set the Categorical to be unordered.                |
| [`Series.cat.remove_categories`](generated/dask.dataframe.Series.cat.remove_categories.md#dask.dataframe.Series.cat.remove_categories)(removals)              | Remove the specified categories.                    |
| [`Series.cat.remove_unused_categories`](generated/dask.dataframe.Series.cat.remove_unused_categories.md#dask.dataframe.Series.cat.remove_unused_categories)() | Removes categories which are not used               |
| [`Series.cat.rename_categories`](generated/dask.dataframe.Series.cat.rename_categories.md#dask.dataframe.Series.cat.rename_categories)(new_categories)        | Rename categories.                                  |
| [`Series.cat.reorder_categories`](generated/dask.dataframe.Series.cat.reorder_categories.md#dask.dataframe.Series.cat.reorder_categories)(new_categories)     | Reorder categories as specified in new_categories.  |
| [`Series.cat.set_categories`](generated/dask.dataframe.Series.cat.set_categories.md#dask.dataframe.Series.cat.set_categories)(new_categories[, ...])          | Set the categories to the specified new categories. |

**Attributes**

| [`Series.cat.categories`](generated/dask.dataframe.Series.cat.categories.md#dask.dataframe.Series.cat.categories)   | The categories of this categorical.                 |
|---------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|
| [`Series.cat.codes`](generated/dask.dataframe.Series.cat.codes.md#dask.dataframe.Series.cat.codes)                  | The codes of this categorical.                      |
| [`Series.cat.known`](generated/dask.dataframe.Series.cat.known.md#dask.dataframe.Series.cat.known)                  | Whether the categories are fully known              |
| [`Series.cat.ordered`](generated/dask.dataframe.Series.cat.ordered.md#dask.dataframe.Series.cat.ordered)            | Whether the categories have an ordered relationship |

## Groupby Operations

### DataFrame Groupby

| [`GroupBy.aggregate`](generated/dask.dataframe.api.GroupBy.aggregate.md#dask.dataframe.api.GroupBy.aggregate)([arg, split_every, ...])      | Aggregate using one or more specified operations                       |
|---------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------|
| [`GroupBy.apply`](generated/dask.dataframe.api.GroupBy.apply.md#dask.dataframe.api.GroupBy.apply)(func, \*args[, meta, ...])                | Parallel version of pandas GroupBy.apply                               |
| [`GroupBy.bfill`](generated/dask.dataframe.api.GroupBy.bfill.md#dask.dataframe.api.GroupBy.bfill)([limit, shuffle_method])                  | Backward fill the values.                                              |
| [`GroupBy.count`](generated/dask.dataframe.api.GroupBy.count.md#dask.dataframe.api.GroupBy.count)(\*\*kwargs)                               | Compute count of group, excluding missing values.                      |
| [`GroupBy.cumcount`](generated/dask.dataframe.api.GroupBy.cumcount.md#dask.dataframe.api.GroupBy.cumcount)()                                | Number each item in each group from 0 to the length of that group - 1. |
| [`GroupBy.cumprod`](generated/dask.dataframe.api.GroupBy.cumprod.md#dask.dataframe.api.GroupBy.cumprod)([numeric_only])                     | Cumulative product for each group.                                     |
| [`GroupBy.cumsum`](generated/dask.dataframe.api.GroupBy.cumsum.md#dask.dataframe.api.GroupBy.cumsum)([numeric_only])                        | Cumulative sum for each group.                                         |
| [`GroupBy.ffill`](generated/dask.dataframe.api.GroupBy.ffill.md#dask.dataframe.api.GroupBy.ffill)([limit, shuffle_method])                  | Forward fill the values.                                               |
| [`GroupBy.get_group`](generated/dask.dataframe.api.GroupBy.get_group.md#dask.dataframe.api.GroupBy.get_group)(key)                          | Construct DataFrame from group with provided name.                     |
| [`GroupBy.max`](generated/dask.dataframe.api.GroupBy.max.md#dask.dataframe.api.GroupBy.max)([numeric_only])                                 | Compute max of group values.                                           |
| [`GroupBy.mean`](generated/dask.dataframe.api.GroupBy.mean.md#dask.dataframe.api.GroupBy.mean)([numeric_only, split_out])                   | Compute mean of groups, excluding missing values.                      |
| [`GroupBy.min`](generated/dask.dataframe.api.GroupBy.min.md#dask.dataframe.api.GroupBy.min)([numeric_only])                                 | Compute min of group values.                                           |
| [`GroupBy.size`](generated/dask.dataframe.api.GroupBy.size.md#dask.dataframe.api.GroupBy.size)(\*\*kwargs)                                  | Compute group sizes.                                                   |
| [`GroupBy.std`](generated/dask.dataframe.api.GroupBy.std.md#dask.dataframe.api.GroupBy.std)([ddof, split_every, split_out, ...])            | Compute standard deviation of groups, excluding missing values.        |
| [`GroupBy.sum`](generated/dask.dataframe.api.GroupBy.sum.md#dask.dataframe.api.GroupBy.sum)([numeric_only, min_count])                      | Compute sum of group values.                                           |
| [`GroupBy.var`](generated/dask.dataframe.api.GroupBy.var.md#dask.dataframe.api.GroupBy.var)([ddof, split_every, split_out, ...])            | Compute variance of groups, excluding missing values.                  |
| [`GroupBy.cov`](generated/dask.dataframe.api.GroupBy.cov.md#dask.dataframe.api.GroupBy.cov)([ddof, split_every, split_out, ...])            | Compute pairwise covariance of columns, excluding NA/null values.      |
| [`GroupBy.corr`](generated/dask.dataframe.api.GroupBy.corr.md#dask.dataframe.api.GroupBy.corr)([split_every, split_out, ...])               | Compute pairwise correlation of columns, excluding NA/null values.     |
| [`GroupBy.first`](generated/dask.dataframe.api.GroupBy.first.md#dask.dataframe.api.GroupBy.first)([numeric_only, sort])                     | Compute the first entry of each column within each group.              |
| [`GroupBy.last`](generated/dask.dataframe.api.GroupBy.last.md#dask.dataframe.api.GroupBy.last)([numeric_only, sort])                        | Compute the last entry of each column within each group.               |
| [`GroupBy.idxmin`](generated/dask.dataframe.api.GroupBy.idxmin.md#dask.dataframe.api.GroupBy.idxmin)([split_every, split_out, ...])         | Return index of first occurrence of minimum over requested axis.       |
| [`GroupBy.idxmax`](generated/dask.dataframe.api.GroupBy.idxmax.md#dask.dataframe.api.GroupBy.idxmax)([split_every, split_out, ...])         | Return index of first occurrence of maximum over requested axis.       |
| [`GroupBy.rolling`](generated/dask.dataframe.api.GroupBy.rolling.md#dask.dataframe.api.GroupBy.rolling)(window[, min_periods, ...])         | Provides rolling transformations.                                      |
| [`GroupBy.transform`](generated/dask.dataframe.api.GroupBy.transform.md#dask.dataframe.api.GroupBy.transform)(func[, meta, shuffle_method]) | Parallel version of pandas GroupBy.transform                           |

### Series Groupby

| [`SeriesGroupBy.aggregate`](generated/dask.dataframe.api.SeriesGroupBy.aggregate.md#dask.dataframe.api.SeriesGroupBy.aggregate)([arg, split_every, ...])   | Aggregate using one or more specified operations                       |
|------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------|
| [`SeriesGroupBy.apply`](generated/dask.dataframe.api.SeriesGroupBy.apply.md#dask.dataframe.api.SeriesGroupBy.apply)(func, \*args[, meta, ...])             | Parallel version of pandas GroupBy.apply                               |
| [`SeriesGroupBy.bfill`](generated/dask.dataframe.api.SeriesGroupBy.bfill.md#dask.dataframe.api.SeriesGroupBy.bfill)([limit, shuffle_method])               | Backward fill the values.                                              |
| [`SeriesGroupBy.count`](generated/dask.dataframe.api.SeriesGroupBy.count.md#dask.dataframe.api.SeriesGroupBy.count)(\*\*kwargs)                            | Compute count of group, excluding missing values.                      |
| [`SeriesGroupBy.cumcount`](generated/dask.dataframe.api.SeriesGroupBy.cumcount.md#dask.dataframe.api.SeriesGroupBy.cumcount)()                             | Number each item in each group from 0 to the length of that group - 1. |
| [`SeriesGroupBy.cumprod`](generated/dask.dataframe.api.SeriesGroupBy.cumprod.md#dask.dataframe.api.SeriesGroupBy.cumprod)([numeric_only])                  | Cumulative product for each group.                                     |
| [`SeriesGroupBy.cumsum`](generated/dask.dataframe.api.SeriesGroupBy.cumsum.md#dask.dataframe.api.SeriesGroupBy.cumsum)([numeric_only])                     | Cumulative sum for each group.                                         |
| [`SeriesGroupBy.ffill`](generated/dask.dataframe.api.SeriesGroupBy.ffill.md#dask.dataframe.api.SeriesGroupBy.ffill)([limit, shuffle_method])               | Forward fill the values.                                               |
| [`SeriesGroupBy.get_group`](generated/dask.dataframe.api.SeriesGroupBy.get_group.md#dask.dataframe.api.SeriesGroupBy.get_group)(key)                       | Construct DataFrame from group with provided name.                     |
| [`SeriesGroupBy.max`](generated/dask.dataframe.api.SeriesGroupBy.max.md#dask.dataframe.api.SeriesGroupBy.max)([numeric_only])                              | Compute max of group values.                                           |
| [`SeriesGroupBy.mean`](generated/dask.dataframe.api.SeriesGroupBy.mean.md#dask.dataframe.api.SeriesGroupBy.mean)([numeric_only, split_out])                | Compute mean of groups, excluding missing values.                      |
| [`SeriesGroupBy.min`](generated/dask.dataframe.api.SeriesGroupBy.min.md#dask.dataframe.api.SeriesGroupBy.min)([numeric_only])                              | Compute min of group values.                                           |
| [`SeriesGroupBy.nunique`](generated/dask.dataframe.api.SeriesGroupBy.nunique.md#dask.dataframe.api.SeriesGroupBy.nunique)([split_every, ...])              | Return number of unique elements in the group.                         |
| [`SeriesGroupBy.size`](generated/dask.dataframe.api.SeriesGroupBy.size.md#dask.dataframe.api.SeriesGroupBy.size)(\*\*kwargs)                               | Compute group sizes.                                                   |
| [`SeriesGroupBy.std`](generated/dask.dataframe.api.SeriesGroupBy.std.md#dask.dataframe.api.SeriesGroupBy.std)([ddof, split_every, ...])                    | Compute standard deviation of groups, excluding missing values.        |
| [`SeriesGroupBy.sum`](generated/dask.dataframe.api.SeriesGroupBy.sum.md#dask.dataframe.api.SeriesGroupBy.sum)([numeric_only, min_count])                   | Compute sum of group values.                                           |
| [`SeriesGroupBy.var`](generated/dask.dataframe.api.SeriesGroupBy.var.md#dask.dataframe.api.SeriesGroupBy.var)([ddof, split_every, ...])                    | Compute variance of groups, excluding missing values.                  |
| [`SeriesGroupBy.first`](generated/dask.dataframe.api.SeriesGroupBy.first.md#dask.dataframe.api.SeriesGroupBy.first)([numeric_only, sort])                  | Compute the first entry of each column within each group.              |
| [`SeriesGroupBy.last`](generated/dask.dataframe.api.SeriesGroupBy.last.md#dask.dataframe.api.SeriesGroupBy.last)([numeric_only, sort])                     | Compute the last entry of each column within each group.               |
| [`SeriesGroupBy.idxmin`](generated/dask.dataframe.api.SeriesGroupBy.idxmin.md#dask.dataframe.api.SeriesGroupBy.idxmin)([split_every, ...])                 | Return index of first occurrence of minimum over requested axis.       |
| [`SeriesGroupBy.idxmax`](generated/dask.dataframe.api.SeriesGroupBy.idxmax.md#dask.dataframe.api.SeriesGroupBy.idxmax)([split_every, ...])                 | Return index of first occurrence of maximum over requested axis.       |
| [`SeriesGroupBy.rolling`](generated/dask.dataframe.api.SeriesGroupBy.rolling.md#dask.dataframe.api.SeriesGroupBy.rolling)(window[, min_periods, ...])      | Provides rolling transformations.                                      |
| [`SeriesGroupBy.transform`](generated/dask.dataframe.api.SeriesGroupBy.transform.md#dask.dataframe.api.SeriesGroupBy.transform)(func[, meta, ...])         | Parallel version of pandas GroupBy.transform                           |

### Custom Aggregation

| [`Aggregation`](generated/dask.dataframe.Aggregation.md#dask.dataframe.Aggregation)(name, chunk, agg[, finalize])   | User defined groupby-aggregation.   |
|---------------------------------------------------------------------------------------------------------------------|-------------------------------------|

## Rolling Operations

| [`Series.rolling`](generated/dask.dataframe.Series.rolling.md#dask.dataframe.Series.rolling)(window, \*\*kwargs)          | Provides rolling transformations.   |
|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------|
| [`DataFrame.rolling`](generated/dask.dataframe.DataFrame.rolling.md#dask.dataframe.DataFrame.rolling)(window, \*\*kwargs) | Provides rolling transformations.   |

| [`Rolling.apply`](generated/dask.dataframe.api.Rolling.apply.md#dask.dataframe.api.Rolling.apply)(func, \*args, \*\*kwargs)       | Calculate the rolling custom aggregation function.                  |
|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`Rolling.count`](generated/dask.dataframe.api.Rolling.count.md#dask.dataframe.api.Rolling.count)(\*args, \*\*kwargs)             | Calculate the rolling count of non NaN observations.                |
| [`Rolling.kurt`](generated/dask.dataframe.api.Rolling.kurt.md#dask.dataframe.api.Rolling.kurt)(\*args, \*\*kwargs)                | Calculate the rolling Fisher's definition of kurtosis without bias. |
| [`Rolling.max`](generated/dask.dataframe.api.Rolling.max.md#dask.dataframe.api.Rolling.max)(\*args, \*\*kwargs)                   | Calculate the rolling maximum.                                      |
| [`Rolling.mean`](generated/dask.dataframe.api.Rolling.mean.md#dask.dataframe.api.Rolling.mean)(\*args, \*\*kwargs)                | Calculate the rolling mean.                                         |
| [`Rolling.median`](generated/dask.dataframe.api.Rolling.median.md#dask.dataframe.api.Rolling.median)(\*args, \*\*kwargs)          | Calculate the rolling median.                                       |
| [`Rolling.min`](generated/dask.dataframe.api.Rolling.min.md#dask.dataframe.api.Rolling.min)(\*args, \*\*kwargs)                   | Calculate the rolling minimum.                                      |
| [`Rolling.quantile`](generated/dask.dataframe.api.Rolling.quantile.md#dask.dataframe.api.Rolling.quantile)(q, \*args, \*\*kwargs) | Calculate the rolling quantile.                                     |
| [`Rolling.skew`](generated/dask.dataframe.api.Rolling.skew.md#dask.dataframe.api.Rolling.skew)(\*args, \*\*kwargs)                | Calculate the rolling unbiased skewness.                            |
| [`Rolling.std`](generated/dask.dataframe.api.Rolling.std.md#dask.dataframe.api.Rolling.std)(\*args, \*\*kwargs)                   | Calculate the rolling standard deviation.                           |
| [`Rolling.sum`](generated/dask.dataframe.api.Rolling.sum.md#dask.dataframe.api.Rolling.sum)(\*args, \*\*kwargs)                   | Calculate the rolling sum.                                          |
| [`Rolling.var`](generated/dask.dataframe.api.Rolling.var.md#dask.dataframe.api.Rolling.var)(\*args, \*\*kwargs)                   | Calculate the rolling variance.                                     |

## Create DataFrames

| [`read_csv`](generated/dask.dataframe.read_csv.md#dask.dataframe.read_csv)(urlpath[, blocksize, ...])                                     | Read CSV files into a Dask.DataFrame                      |
|-------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------|
| [`read_table`](generated/dask.dataframe.read_table.md#dask.dataframe.read_table)(urlpath[, blocksize, ...])                               | Read delimited files into a Dask.DataFrame                |
| [`read_fwf`](generated/dask.dataframe.read_fwf.md#dask.dataframe.read_fwf)(urlpath[, blocksize, ...])                                     | Read fixed-width files into a Dask.DataFrame              |
| [`read_parquet`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet)([path, columns, filters, ...])                     | Read a Parquet file into a Dask DataFrame                 |
| [`read_hdf`](generated/dask.dataframe.read_hdf.md#dask.dataframe.read_hdf)(pattern, key[, start, stop, ...])                              | Read HDF files into a Dask DataFrame                      |
| [`read_json`](generated/dask.dataframe.read_json.md#dask.dataframe.read_json)(url_path[, orient, lines, ...])                             | Create a dataframe from a set of JSON files               |
| [`read_orc`](generated/dask.dataframe.read_orc.md#dask.dataframe.read_orc)(path[, engine, columns, index, ...])                           | Read dataframe from ORC file(s)                           |
| [`read_sql_table`](generated/dask.dataframe.read_sql_table.md#dask.dataframe.read_sql_table)(table_name, con, index_col[, ...])           | Read SQL database table into a DataFrame.                 |
| [`read_sql_query`](generated/dask.dataframe.read_sql_query.md#dask.dataframe.read_sql_query)(sql, con, index_col[, ...])                  | Read SQL query into a DataFrame.                          |
| [`read_sql`](generated/dask.dataframe.read_sql.md#dask.dataframe.read_sql)(sql, con, index_col, \*\*kwargs)                               | Read SQL query or database table into a DataFrame.        |
| [`from_array`](generated/dask.dataframe.from_array.md#dask.dataframe.from_array)(arr[, chunksize, columns, meta])                         | Read any sliceable array into a Dask Dataframe            |
| [`from_dask_array`](generated/dask.dataframe.from_dask_array.md#dask.dataframe.from_dask_array)(x[, columns, index, meta])                | Create a Dask DataFrame from a Dask Array.                |
| [`from_delayed`](generated/dask.dataframe.from_delayed.md#dask.dataframe.from_delayed)(dfs[, meta, divisions, prefix, ...])               | Create Dask DataFrame from many Dask Delayed objects      |
| [`from_map`](generated/dask.dataframe.from_map.md#dask.dataframe.from_map)(func, \*iterables[, args, meta, ...])                          | Create a DataFrame collection from a custom function map. |
| [`from_pandas`](generated/dask.dataframe.from_pandas.md#dask.dataframe.from_pandas)(data[, npartitions, sort, chunksize])                 | Construct a Dask DataFrame from a Pandas DataFrame        |
| [`DataFrame.from_dict`](generated/dask.dataframe.DataFrame.from_dict.md#dask.dataframe.DataFrame.from_dict)(data, \*[, npartitions, ...]) | Construct a Dask DataFrame from a Python Dictionary       |

## Store DataFrames

| [`to_csv`](generated/dask.dataframe.to_csv.md#dask.dataframe.to_csv)(df, filename[, single_file, ...])         | Store Dask DataFrame to CSV files                            |
|----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| [`to_parquet`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet)(df, path[, compression, ...]) | Store Dask.dataframe to Parquet files                        |
| [`to_hdf`](generated/dask.dataframe.to_hdf.md#dask.dataframe.to_hdf)(df, path, key[, mode, append, ...])       | Store Dask Dataframe to Hierarchical Data Format (HDF) files |
| [`to_records`](generated/dask.dataframe.to_records.md#dask.dataframe.to_records)(df)                           | Create Dask Array from a Dask Dataframe                      |
| [`to_sql`](generated/dask.dataframe.to_sql.md#dask.dataframe.to_sql)(df, name, uri[, schema, if_exists, ...])  | Store Dask Dataframe to a SQL table                          |
| [`to_json`](generated/dask.dataframe.to_json.md#dask.dataframe.to_json)(df, url_path[, orient, lines, ...])    | Write dataframe into JSON text files                         |
| [`to_orc`](generated/dask.dataframe.to_orc.md#dask.dataframe.to_orc)(df, path[, engine, write_index, ...])     | Store Dask.dataframe to ORC files                            |

## Convert DataFrames

| [`DataFrame.to_bag`](generated/dask.dataframe.DataFrame.to_bag.md#dask.dataframe.DataFrame.to_bag)([index, format])                           | Create a Dask Bag from a Series                                   |
|-----------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`DataFrame.to_dask_array`](generated/dask.dataframe.DataFrame.to_dask_array.md#dask.dataframe.DataFrame.to_dask_array)([lengths, meta, ...]) | Convert a dask DataFrame to a dask array.                         |
| [`DataFrame.to_delayed`](generated/dask.dataframe.DataFrame.to_delayed.md#dask.dataframe.DataFrame.to_delayed)([optimize_graph])              | Convert into a list of `dask.delayed` objects, one per partition. |

## Reshape DataFrames

| [`get_dummies`](generated/dask.dataframe.get_dummies.md#dask.dataframe.get_dummies)(data[, prefix, prefix_sep, ...])   | Convert categorical variable into dummy/indicator variables.   |
|------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------|
| [`pivot_table`](generated/dask.dataframe.pivot_table.md#dask.dataframe.pivot_table)(df, index, columns, values[, ...]) | Create a spreadsheet-style pivot table as a DataFrame.         |
| [`melt`](generated/dask.dataframe.melt.md#dask.dataframe.melt)(frame[, id_vars, value_vars, var_name, ...])            |                                                                |

## Concatenate DataFrames

| [`DataFrame.merge`](generated/dask.dataframe.DataFrame.merge.md#dask.dataframe.DataFrame.merge)(right[, how, on, left_on, ...])   | Merge the DataFrame with another DataFrame                          |
|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`concat`](generated/dask.dataframe.concat.md#dask.dataframe.concat)(dfs[, axis, join, ...])                                      | Concatenate DataFrames along rows.                                  |
| [`merge`](generated/dask.dataframe.merge.md#dask.dataframe.merge)(left, right[, how, on, left_on, ...])                           | Merge DataFrame or named Series objects with a database-style join. |
| [`merge_asof`](generated/dask.dataframe.merge_asof.md#dask.dataframe.merge_asof)(left, right[, on, left_on, ...])                 | Perform a merge by key distance.                                    |

## Resampling

| [`Resampler`](generated/dask.dataframe.tseries.resample.Resampler.md#dask.dataframe.tseries.resample.Resampler)(obj, rule, \*\*kwargs)                | Aggregate using one or more operations                                         |
|-------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| [`Resampler.agg`](generated/dask.dataframe.tseries.resample.Resampler.agg.md#dask.dataframe.tseries.resample.Resampler.agg)(func, \*args, \*\*kwargs) | Aggregate using one or more operations over the specified axis.                |
| [`Resampler.count`](generated/dask.dataframe.tseries.resample.Resampler.count.md#dask.dataframe.tseries.resample.Resampler.count)()                   | Compute count of group, excluding missing values.                              |
| [`Resampler.first`](generated/dask.dataframe.tseries.resample.Resampler.first.md#dask.dataframe.tseries.resample.Resampler.first)()                   | Compute the first non-null entry of each column.                               |
| [`Resampler.last`](generated/dask.dataframe.tseries.resample.Resampler.last.md#dask.dataframe.tseries.resample.Resampler.last)()                      | Compute the last non-null entry of each column.                                |
| [`Resampler.max`](generated/dask.dataframe.tseries.resample.Resampler.max.md#dask.dataframe.tseries.resample.Resampler.max)()                         | Compute max value of group.                                                    |
| [`Resampler.mean`](generated/dask.dataframe.tseries.resample.Resampler.mean.md#dask.dataframe.tseries.resample.Resampler.mean)()                      | Compute mean of groups, excluding missing values.                              |
| [`Resampler.median`](generated/dask.dataframe.tseries.resample.Resampler.median.md#dask.dataframe.tseries.resample.Resampler.median)()                | Compute median of groups, excluding missing values.                            |
| [`Resampler.min`](generated/dask.dataframe.tseries.resample.Resampler.min.md#dask.dataframe.tseries.resample.Resampler.min)()                         | Compute min value of group.                                                    |
| [`Resampler.nunique`](generated/dask.dataframe.tseries.resample.Resampler.nunique.md#dask.dataframe.tseries.resample.Resampler.nunique)()             | Return number of unique elements in the group.                                 |
| [`Resampler.ohlc`](generated/dask.dataframe.tseries.resample.Resampler.ohlc.md#dask.dataframe.tseries.resample.Resampler.ohlc)()                      | Compute open, high, low and close values of a group, excluding missing values. |
| [`Resampler.prod`](generated/dask.dataframe.tseries.resample.Resampler.prod.md#dask.dataframe.tseries.resample.Resampler.prod)()                      | Compute prod of group values.                                                  |
| [`Resampler.quantile`](generated/dask.dataframe.tseries.resample.Resampler.quantile.md#dask.dataframe.tseries.resample.Resampler.quantile)()          | Return value at the given quantile.                                            |
| [`Resampler.sem`](generated/dask.dataframe.tseries.resample.Resampler.sem.md#dask.dataframe.tseries.resample.Resampler.sem)()                         | Compute standard error of the mean of groups, excluding missing values.        |
| [`Resampler.size`](generated/dask.dataframe.tseries.resample.Resampler.size.md#dask.dataframe.tseries.resample.Resampler.size)()                      | Compute group sizes.                                                           |
| [`Resampler.std`](generated/dask.dataframe.tseries.resample.Resampler.std.md#dask.dataframe.tseries.resample.Resampler.std)()                         | Compute standard deviation of groups, excluding missing values.                |
| [`Resampler.sum`](generated/dask.dataframe.tseries.resample.Resampler.sum.md#dask.dataframe.tseries.resample.Resampler.sum)()                         | Compute sum of group values.                                                   |
| [`Resampler.var`](generated/dask.dataframe.tseries.resample.Resampler.var.md#dask.dataframe.tseries.resample.Resampler.var)()                         | Compute variance of groups, excluding missing values.                          |

## Dask Metadata

| [`make_meta`](generated/dask.dataframe.utils.make_meta.md#dask.dataframe.utils.make_meta)(x[, index, parent_meta])   | This method creates meta-data based on the type of `x`, and `parent_meta` if supplied.   |
|----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|

## Query Planning and Optimization

| [`DataFrame.explain`](generated/dask.dataframe.DataFrame.explain.md#dask.dataframe.DataFrame.explain)([stage, format])    | Create a graph representation of the Expression.       |
|---------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|
| [`DataFrame.visualize`](generated/dask.dataframe.DataFrame.visualize.md#dask.dataframe.DataFrame.visualize)([tasks])      | Visualize the expression or task graph                 |
| [`DataFrame.analyze`](generated/dask.dataframe.DataFrame.analyze.md#dask.dataframe.DataFrame.analyze)([filename, format]) | Outputs statistics about every node in the expression. |

## Other functions

| [`compute`](generated/dask.dataframe.compute.md#dask.dataframe.compute)(\*args[, traverse, optimize_graph, ...])        | Compute several dask collections at once.                                  |
|-------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|
| [`map_partitions`](generated/dask.dataframe.map_partitions.md#dask.dataframe.map_partitions)(func, \*args[, meta, ...]) | Apply Python function on each DataFrame partition.                         |
| [`map_overlap`](generated/dask.dataframe.map_overlap.md#dask.dataframe.map_overlap)(func, df, before, after, \*args)    | Apply a function to each partition, sharing rows with adjacent partitions. |
| [`to_datetime`](generated/dask.dataframe.to_datetime.md#dask.dataframe.to_datetime)(arg[, errors, dayfirst, ...])       | Convert argument to datetime.                                              |
| [`to_numeric`](generated/dask.dataframe.to_numeric.md#dask.dataframe.to_numeric)(arg[, errors, downcast, meta])         | Convert argument to a numeric type.                                        |
| [`to_timedelta`](generated/dask.dataframe.to_timedelta.md#dask.dataframe.to_timedelta)(arg[, unit, errors])             | Convert argument to timedelta.                                             |


# dataframe-best-practices.html.md

<a id="dataframe-performance"></a>

# Dask DataFrames Best Practices

It is easy to get started with Dask DataFrame, but using it *well* does require
some experience.  This page contains suggestions for Dask DataFrames best practices,
and includes solutions to common problems.

## Use Pandas

For data that fits into RAM, pandas can often be faster and easier to use than
Dask DataFrame.  While “Big Data” tools can be exciting, they are almost always
worse than normal data tools while those remain appropriate.

## Reduce, and then use pandas

Similar to above, even if you have a large dataset there may be a point in your
computation where you’ve reduced things to a more manageable level.  You may
want to switch to pandas at this point.

```python
df = dd.read_parquet('my-giant-file.parquet')
df = df[df.name == 'Alice']              # Select a subsection
result = df.groupby('id').value.mean()   # Reduce to a smaller size
result = result.compute()                # Convert to pandas dataframe
result...                                # Continue working with pandas
```

## Pandas Performance Tips Apply to Dask DataFrame

Usual pandas performance tips like avoiding apply, using vectorized
operations, using categoricals, etc., all apply equally to Dask DataFrame.  See
[Modern Pandas](https://tomaugspurger.github.io/modern-1-intro) by [Tom
Augspurger](https://github.com/TomAugspurger) for a good read on this topic.

## Use the Index

Dask DataFrame can be optionally sorted along a single index column.  Some
operations against this column can be very fast.  For example, if your dataset
is sorted by time, you can quickly select data for a particular day, perform
time series joins, etc.  You can check if your data is sorted by looking at the
`df.known_divisions` attribute.  You can set an index column using the
`.set_index(column_name)` method.  This operation is expensive though, so use
it sparingly (see below):

```python
df = df.set_index('timestamp')  # set the index to make some operations fast

df.loc['2001-01-05':'2001-01-12']  # this is very fast if you have an index
df.merge(df2, left_index=True, right_index=True)  # this is also very fast
```

For more information, see documentation on [dataframe partitions](dataframe-design.md#dataframe-design-partitions).

## Avoid Full-Data Shuffling

Setting an index is an important but expensive operation (see above).  You
should do it infrequently and you should persist afterwards (see below).

Some operations like `set_index` and `merge/join` are harder to do in a
parallel or distributed setting than if they are in-memory on a single machine.
In particular, *shuffling operations* that rearrange data become much more
communication intensive.  For example, if your data is arranged by customer ID
but now you want to arrange it by time, all of your partitions will have to talk
to each other to exchange shards of data.  This can be an intensive process,
particularly on a cluster.

So, definitely set the index but try doing so infrequently.

```python
df = df.set_index('column_name')  # do this infrequently
```

Additionally, `set_index` has a few options that can accelerate it in some
situations.  For example, if you know that your dataset is sorted or you already
know the values by which it is divided, you can provide these to accelerate the
`set_index` operation.  For more information, see the [set_index docstring](https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.DataFrame.set_index).

```python
df2 = df.set_index(d.timestamp, sorted=True)
```

## Persist Intelligently

#### NOTE
This section is only relevant to users on distributed systems.

#### WARNING
persist has a number of drawbacks with the query optimizer. It will block all
optimizations and prevent us from pushing column projections or filters into
the IO layer. Use persist sparingly only when absolutely necessary or you need
the full dataset afterwards.

Often DataFrame workloads look like the following:

1. Load data from files
2. Filter data to a particular subset
3. Shuffle data to set an intelligent index
4. Several complex queries on top of this indexed data

It is often ideal to load, filter, and shuffle data once and keep this result in
memory.  Afterwards, each of the several complex queries can be based off of
this in-memory data rather than have to repeat the full load-filter-shuffle
process each time.  To do this, use the [client.persist](https://distributed.dask.org/en/latest/api.html#distributed.Client.persist)
method:

```python
df = dd.read_csv('s3://bucket/path/to/*.csv')
df = df[df.balance < 0]
df = client.persist(df)

df = df.set_index('timestamp')
df = client.persist(df)

>>> df.customer_id.nunique().compute()
18452844

>>> df.groupby(df.city).size().compute()
...
```

Persist is important because Dask DataFrame is *lazy by default*.  It is a
way of telling the cluster that it should start executing the computations
that you have defined so far, and that it should try to keep those results in
memory.  You will get back a new DataFrame that is semantically equivalent to
your old DataFrame, but now points to running data.  Your old DataFrame still
points to lazy computations:

```python
# Don't do this
client.persist(df)  # persist doesn't change the input in-place

# Do this instead
df = client.persist(df)  # replace your old lazy DataFrame
```

## Repartition to Reduce Overhead

Your Dask DataFrame is split up into many pandas DataFrames.  We sometimes call
these “partitions”, and often the number of partitions is decided for you. For
example, it might be the number of CSV files from which you are reading. However,
over time, as you reduce or increase the size of your pandas DataFrames by
filtering or joining, it may be wise to reconsider how many partitions you need.
There is a cost to having too many or having too few.

![Individual partitions of a Dask DataFrame are pandas DataFrames. One tip from Dask DataFrames Best Practices is to repartition these partitions.](images/dask-dataframe.svg)

Partitions should fit comfortably in memory (smaller than a gigabyte) but also
not be too many.  Every operation on every partition takes the central
scheduler a few hundred microseconds to process.  If you have a few thousand
tasks this is barely noticeable, but it is nice to reduce the number if
possible.

A common situation is that you load lots of data into reasonably sized
partitions (Dask’s defaults make decent choices), but then you filter down your
dataset to only a small fraction of the original.  At this point, it is wise to
regroup your many small partitions into a few larger ones.  You can do this by
using the [`dask.dataframe.DataFrame.repartition`](generated/dask.dataframe.DataFrame.repartition.md#dask.dataframe.DataFrame.repartition) method:

```python
df = dd.read_csv('s3://bucket/path/to/*.csv')
df = df[df.name == 'Alice']  # only 1/100th of the data
df = df.repartition(npartitions=df.npartitions // 100)

df = df.persist()  # if on a distributed system
```

This helps to reduce overhead and increase the effectiveness of vectorized
Pandas operations.  You should aim for partitions that have around 100MB of
data each.

## Joins

Joining two DataFrames can be either very expensive or very cheap depending on
the situation.  It is cheap in the following cases:

1. Joining a Dask DataFrame with a pandas DataFrame
2. Joining a Dask DataFrame with another Dask DataFrame of a single partition
3. Joining Dask DataFrames along their indexes

And expensive in the following case:

1. Joining Dask DataFrames along columns that are not their index

The expensive case requires a shuffle.  This is fine, and Dask DataFrame will
complete the job well, but it will be more expensive than a typical linear-time
operation:

```python
dd.merge(a, pandas_df)  # fast
dd.merge(a, b, left_index=True, right_index=True)  # fast
dd.merge(a, b, left_index=True, right_on='id')  # half-fast, half-slow
dd.merge(a, b, left_on='id', right_on='id')  # slow
```

For more information see [Joins](dataframe-joins.md).

## Use Parquet

[Apache Parquet](https://parquet.apache.org/) is a columnar binary format.
It is the de-facto standard for the storage of large volumes of tabular data
and our recommended storage solution for basic tabular data.

```python
df.to_parquet('path/to/my-results/')
df = dd.read_parquet('path/to/my-results/')
```

When compared to formats like CSV, Parquet brings the following advantages:

1. It’s faster to read and write, often by 4-10x
2. It’s more compact to store, often by 2-5x
3. It has a schema, and so there’s no ambiguity about what types the columns are.  This avoids confusing errors.
4. It supports more advanced data types, like categoricals, proper datetimes, and more
5. It’s more portable, and can be used with other systems like databases or Apache Spark
6. Depending on how the data is partitioned Dask can identify sorted columns, and sometimes pick out subsets of data more efficiently

See [Dask Dataframe and Parquet](dataframe-parquet.md#dataframe-parquet) for more details.


# dataframe-categoricals.html.md

# Categoricals

Dask DataFrame divides [categorical data](https://pandas.pydata.org/pandas-docs/stable/categorical.html) into two types:

- Known categoricals have the `categories` known statically (on the `_meta`
  attribute).  Each partition **must** have the same categories as found on the
  `_meta` attribute
- Unknown categoricals don’t know the categories statically, and may have
  different categories in each partition.  Internally, unknown categoricals are
  indicated by the presence of `dd.utils.UNKNOWN_CATEGORIES` in the
  categories on the `_meta` attribute.  Since most DataFrame operations
  propagate the categories, the known/unknown status should propagate through
  operations (similar to how `NaN` propagates)

For metadata specified as a description (option 2 above), unknown categoricals
are created.

Certain operations are only available for known categoricals.  For example,
`df.col.cat.categories` would only work if `df.col` has known categories,
since the categorical mapping is only known statically on the metadata of known
categoricals.

The known/unknown status for a categorical column can be found using the
`known` property on the categorical accessor:

```python
>>> ddf.col.cat.known
False
```

Additionally, an unknown categorical can be converted to known using
`.cat.as_known()`.  If you have multiple categorical columns in a DataFrame,
you may instead want to use `df.categorize(columns=...)`, which will convert
all specified columns to known categoricals.  Since getting the categories
requires a full scan of the data, using `df.categorize()` is more efficient
than calling `.cat.as_known()` for each column (which would result in
multiple scans):

```python
>>> col_known = ddf.col.cat.as_known()  # use for single column
>>> col_known.cat.known
True
>>> ddf_known = ddf.categorize()        # use for multiple columns
>>> ddf_known.col.cat.known
True
```

To convert a known categorical to an unknown categorical, there is also the
`.cat.as_unknown()` method. This requires no computation as it’s just a
change in the metadata.

Non-categorical columns can be converted to categoricals in a few different
ways:

```python
# astype operates lazily, and results in unknown categoricals
ddf = ddf.astype({'mycol': 'category', ...})
# or
ddf['mycol'] = ddf.mycol.astype('category')

# categorize requires computation, and results in known categoricals
ddf = ddf.categorize(columns=['mycol', ...])
```

Additionally, with Pandas 0.19.2 and up, `dd.read_csv` and `dd.read_table`
can read data directly into unknown categorical columns by specifying a column
dtype as `'category'`:

```python
>>> ddf = dd.read_csv(..., dtype={col_name: 'category'})
```

Moreover, with Pandas 0.21.0 and up, `dd.read_csv` and `dd.read_table` can read
data directly into *known* categoricals by specifying instances of
`pd.api.types.CategoricalDtype`:

```python
>>> dtype = {'col': pd.api.types.CategoricalDtype(['a', 'b', 'c'])}
>>> ddf = dd.read_csv(..., dtype=dtype)
```

If you write and read to parquet, Dask will forget known categories.
This happens because, due to performance concerns, all the categories are
saved in every partition rather than in the parquet metadata.
It is possible to manually load the categories:

```python
>>> import dask.dataframe as dd
>>> import pandas as pd
>>> df = pd.DataFrame(data=list('abcaabbcc'), columns=['col'])
>>> df.col = df.col.astype('category')
>>> ddf = dd.from_pandas(df, npartitions=1)
>>> ddf.col.cat.known
True
>>> ddf.to_parquet('tmp')
>>> ddf2 = dd.read_parquet('tmp')
>>> ddf2.col.cat.known
False
>>> ddf2.col = ddf2.col.cat.set_categories(ddf2.col.head(1).cat.categories)
>>> ddf2.col.cat.known
True
```


# dataframe-create.html.md

# Load and Save Data with Dask DataFrames

You can create a Dask DataFrame from various data storage formats
like CSV, HDF, Apache Parquet, and others. For most formats, this data can live on various
storage systems including local disk, network file systems (NFS), the Hadoop
Distributed File System (HDFS), Google Cloud Storage, and Amazon S3
(excepting HDF, which is only available on POSIX like file systems).

See the [DataFrame overview page](dataframe.md) for more on
`dask.dataframe` scope, use, and limitations and
[DataFrame Best Practices](dataframe-best-practices.md) for more tips
and solutions to common problems.

## API

The following functions provide access to convert between Dask DataFrames,
file formats, and other Dask or Python collections.

File Formats:

| [`read_csv`](generated/dask.dataframe.read_csv.md#dask.dataframe.read_csv)(urlpath[, blocksize, ...])                           | Read CSV files into a Dask.DataFrame                         |
|---------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------|
| [`read_parquet`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet)([path, columns, filters, ...])           | Read a Parquet file into a Dask DataFrame                    |
| [`read_hdf`](generated/dask.dataframe.read_hdf.md#dask.dataframe.read_hdf)(pattern, key[, start, stop, ...])                    | Read HDF files into a Dask DataFrame                         |
| [`read_orc`](generated/dask.dataframe.read_orc.md#dask.dataframe.read_orc)(path[, engine, columns, index, ...])                 | Read dataframe from ORC file(s)                              |
| [`read_json`](generated/dask.dataframe.read_json.md#dask.dataframe.read_json)(url_path[, orient, lines, ...])                   | Create a dataframe from a set of JSON files                  |
| [`read_sql_table`](generated/dask.dataframe.read_sql_table.md#dask.dataframe.read_sql_table)(table_name, con, index_col[, ...]) | Read SQL database table into a DataFrame.                    |
| [`read_sql_query`](generated/dask.dataframe.read_sql_query.md#dask.dataframe.read_sql_query)(sql, con, index_col[, ...])        | Read SQL query into a DataFrame.                             |
| [`read_sql`](generated/dask.dataframe.read_sql.md#dask.dataframe.read_sql)(sql, con, index_col, \*\*kwargs)                     | Read SQL query or database table into a DataFrame.           |
| [`read_table`](generated/dask.dataframe.read_table.md#dask.dataframe.read_table)(urlpath[, blocksize, ...])                     | Read delimited files into a Dask.DataFrame                   |
| [`read_fwf`](generated/dask.dataframe.read_fwf.md#dask.dataframe.read_fwf)(urlpath[, blocksize, ...])                           | Read fixed-width files into a Dask.DataFrame                 |
| [`from_array`](generated/dask.dataframe.from_array.md#dask.dataframe.from_array)(arr[, chunksize, columns, meta])               | Read any sliceable array into a Dask Dataframe               |
| [`to_csv`](generated/dask.dataframe.to_csv.md#dask.dataframe.to_csv)(df, filename[, single_file, ...])                          | Store Dask DataFrame to CSV files                            |
| [`to_parquet`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet)(df, path[, compression, ...])                  | Store Dask.dataframe to Parquet files                        |
| [`to_hdf`](generated/dask.dataframe.to_hdf.md#dask.dataframe.to_hdf)(df, path, key[, mode, append, ...])                        | Store Dask Dataframe to Hierarchical Data Format (HDF) files |
| [`to_sql`](generated/dask.dataframe.to_sql.md#dask.dataframe.to_sql)(df, name, uri[, schema, if_exists, ...])                   | Store Dask Dataframe to a SQL table                          |

Dask Collections:

| [`from_delayed`](generated/dask.dataframe.from_delayed.md#dask.dataframe.from_delayed)(dfs[, meta, divisions, prefix, ...])      | Create Dask DataFrame from many Dask Delayed objects              |
|----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`from_dask_array`](generated/dask.dataframe.from_dask_array.md#dask.dataframe.from_dask_array)(x[, columns, index, meta])       | Create a Dask DataFrame from a Dask Array.                        |
| [`from_map`](generated/dask.dataframe.from_map.md#dask.dataframe.from_map)(func, \*iterables[, args, meta, ...])                 | Create a DataFrame collection from a custom function map.         |
| `dask.bag.core.Bag.to_dataframe`([meta, ...])                                                                                    | Create Dask Dataframe from a Dask Bag.                            |
| [`DataFrame.to_delayed`](generated/dask.dataframe.DataFrame.to_delayed.md#dask.dataframe.DataFrame.to_delayed)([optimize_graph]) | Convert into a list of `dask.delayed` objects, one per partition. |
| [`to_records`](generated/dask.dataframe.to_records.md#dask.dataframe.to_records)(df)                                             | Create Dask Array from a Dask Dataframe                           |
| `to_bag`(df[, index, format])                                                                                                    | Create Dask Bag from a Dask DataFrame                             |

Pandas:

| [`from_pandas`](generated/dask.dataframe.from_pandas.md#dask.dataframe.from_pandas)(data[, npartitions, sort, chunksize])                 | Construct a Dask DataFrame from a Pandas DataFrame   |
|-------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
| [`DataFrame.from_dict`](generated/dask.dataframe.DataFrame.from_dict.md#dask.dataframe.DataFrame.from_dict)(data, \*[, npartitions, ...]) | Construct a Dask DataFrame from a Python Dictionary  |

Other File Formats:

- [Snowflake](https://github.com/coiled/dask-snowflake)
- [Bigquery](https://github.com/coiled/dask-bigquery)
- [Delta Lake](https://github.com/dask-contrib/dask-deltatable)

## Creating

### Read from CSV

You can use [`read_csv()`](generated/dask.dataframe.read_csv.md#dask.dataframe.read_csv) to read one or more CSV files into a Dask DataFrame.
It supports loading multiple files at once using globstrings:

```python
>>> df = dd.read_csv('myfiles.*.csv')
```

You can break up a single large file with the `blocksize` parameter:

```python
>>> df = dd.read_csv('largefile.csv', blocksize=25e6)  # 25MB chunks
```

Changing the `blocksize` parameter will change the number of partitions (see the explanation on
[partitions](dataframe-design.md#dataframe-design-partitions)). A good rule of thumb when working with
Dask DataFrames is to keep your partitions under 100MB in size.

### Read from Parquet

Similarly, you can use [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) for reading one or more Parquet files.
You can read in a single Parquet file:

```python
>>> df = dd.read_parquet("path/to/mydata.parquet")
```

Or a directory of local Parquet files:

```python
>>> df = dd.read_parquet("path/to/my/parquet/")
```

For more details on working with Parquet files, including tips and best practices, see the documentation
on [Dask Dataframe and Parquet](dataframe-parquet.md).

### Read from cloud storage

Dask can read data from a variety of data stores including cloud object stores.
You can do this by prepending a protocol like `s3://` to paths
used in common data access functions like `dd.read_csv`:

```python
>>> df = dd.read_csv('s3://bucket/path/to/data-*.csv')
>>> df = dd.read_parquet('gcs://bucket/path/to/data-*.parq')
```

For remote systems like Amazon S3 or Google Cloud Storage, you may need to provide credentials.
These are usually stored in a configuration file, but in some cases you may want to pass
storage-specific options through to the storage backend.
You can do this with the `storage_options` parameter:

```python
>>> df = dd.read_csv('s3://bucket-name/my-data-*.csv',
...                  storage_options={'anon': True})
>>> df = dd.read_parquet('gs://dask-nyc-taxi/yellowtrip.parquet',
...                      storage_options={'token': 'anon'})
```

See the documentation on connecting to [Amazon S3](how-to/connect-to-remote-data.md#connect-to-remote-data-s3) or
[Google Cloud Storage](how-to/connect-to-remote-data.md#connect-to-remote-data-gc).

### Mapping from a function

For cases that are not covered by the functions above, but *can* be
captured by a simple `map` operation, [`from_map()`](generated/dask.dataframe.from_map.md#dask.dataframe.from_map) is likely to be
the most convenient means for DataFrame creation. For example, this
API can be used to convert an arbitrary PyArrow `Dataset` object into a
DataFrame collection by mapping fragments to DataFrame partitions:

```python
>>> import pyarrow.dataset as ds
>>> dataset = ds.dataset("hive_data_path", format="orc", partitioning="hive")
>>> fragments = dataset.get_fragments()
>>> func = lambda frag: frag.to_table().to_pandas()
>>> df = dd.from_map(func, fragments)
```

### Dask Delayed

[Dask delayed](delayed.md) is particularly useful when simple `map` operations aren’t sufficient to capture the complexity of your data layout. It lets you construct Dask DataFrames out of arbitrary Python function calls, which can be
helpful to handle custom data formats or bake in particular logic around loading data.
See the [documentation on using dask.delayed with collections](delayed-collections.md).

## Storing

### Writing files locally

You can save files locally, assuming each worker can access the same file system. The workers could be located on the same machine, or a network file system can be mounted and referenced at the same path location for every worker node.
See the documentation on [accessing data locally](how-to/connect-to-remote-data.md#connect-to-remote-data-local).

### Writing to remote locations

Dask can write to a variety of data stores including cloud object stores.
For example, you can write a `dask.dataframe` to an Azure storage blob as:

```python
>>> d = {'col1': [1, 2, 3, 4], 'col2': [5, 6, 7, 8]}
>>> df = dd.from_pandas(pd.DataFrame(data=d), npartitions=2)
>>> dd.to_parquet(df=df,
...               path='abfs://CONTAINER/FILE.parquet'
...               storage_options={'account_name': 'ACCOUNT_NAME',
...                                'account_key': 'ACCOUNT_KEY'}
```

See the [how-to guide on connecting to remote data](how-to/connect-to-remote-data.md).


# dataframe-design.html.md

<a id="dataframe-design"></a>

# Dask DataFrame Design

Dask DataFrames coordinate many Pandas DataFrames/Series arranged along an
index.  We define a Dask DataFrame object with the following components:

- A Dask graph with a special set of keys designating partitions, such as
  `('x', 0), ('x', 1), ...`
- A name to identify which keys in the Dask graph refer to this DataFrame, such
  as `'x'`
- An empty Pandas object containing appropriate metadata (e.g.  column names,
  dtypes, etc.)
- A sequence of partition boundaries along the index called `divisions`

## Metadata

Many DataFrame operations rely on knowing the name and dtype of columns.  To
keep track of this information, all Dask DataFrame objects have a `_meta`
attribute which contains an empty Pandas object with the same dtypes and names.
For example:

```python
>>> df = pd.DataFrame({'a': [1, 2, 3], 'b': ['x', 'y', 'z']})
>>> ddf = dd.from_pandas(df, npartitions=2)
>>> ddf._meta
Empty DataFrame
Columns: [a, b]
Index: []
>>> ddf._meta.dtypes
a     int64
b    object
dtype: object
```

Internally, Dask DataFrame does its best to propagate this information
through all operations, so most of the time a user shouldn’t have to worry
about this.  Usually this is done by evaluating the operation on a small sample
of fake data, which can be found on the `_meta_nonempty` attribute:

```python
>>> ddf._meta_nonempty
   a    b
0  1  foo
1  1  foo
```

Sometimes this operation may fail in user defined functions (e.g. when using
`DataFrame.apply`), or may be prohibitively expensive.  For these cases, many
functions support an optional `meta` keyword, which allows specifying the
metadata directly, avoiding the inference step.  For convenience, this supports
several options:

1. A Pandas object with appropriate dtypes and names.  If not empty, an empty
   slice will be taken:

```python
>>> ddf.map_partitions(foo, meta=pd.DataFrame({'a': [1], 'b': [2]}))
```

1. A description of the appropriate names and dtypes.  This can take several forms:

* A `dict` of `{name: dtype}` or an iterable of `(name, dtype)`
  specifies a DataFrame. Note that order is important: the order of the
  names in `meta` should match the order of the columns
* A tuple of `(name, dtype)` specifies a series

This keyword is available on all functions/methods that take user provided
callables (e.g. `DataFrame.map_partitions`, `DataFrame.apply`, etc…), as
well as many creation functions (e.g. `dd.from_delayed`).

<a id="dataframe-design-partitions"></a>

## Partitions

Internally, a Dask DataFrame is split into many partitions, where each partition
is one Pandas DataFrame.  These DataFrames are split vertically along the
index.  When our index is sorted and we know the values of the divisions of our
partitions, then we can be clever and efficient with expensive algorithms (e.g.
groupby’s, joins, etc…).

For example, if we have a time-series index, then our partitions might be
divided by month: all of January will live in one partition while all of
February will live in the next.  In these cases, operations like `loc`,
`groupby`, and `join/merge` along the index can be *much* more efficient
than would otherwise be possible in parallel.  You can view the number of
partitions and divisions of your DataFrame with the following fields:

```python
>>> df.npartitions
4
>>> df.divisions
['2015-01-01', '2015-02-01', '2015-03-01', '2015-04-01', '2015-04-31']
```

The number of partitions and the division values might change during optimization.
The optimizer will try to create partitions with a sensible size to avoid straining
the scheduler with many small partitions.

Divisions includes the minimum value of every partition’s index and the maximum
value of the last partition’s index.  In the example above, if the user searches
for a specific datetime range, then we know which partitions we need to inspect
and which we can drop:

```python
>>> df.loc['2015-01-20': '2015-02-10']  # Must inspect first two partitions
```

Often we do not have such information about our partitions.  When reading CSV
files, for example, we do not know, without extra user input, how the data is
divided.  In this case `.divisions` will be all `None`:

```python
>>> df.divisions
[None, None, None, None, None]
```

In these cases, any operation that requires a cleanly partitioned DataFrame with
known divisions will have to perform a sort.  This can generally be achieved by
calling `df.set_index(...)`.

<a id="dataframe-design-groupby"></a>

## Groupby

By default, groupby will choose the number of output partitions based on a few
different factors. It will look at the number of grouping keys to guess the
cardinality of your data. It will use this information to calculate a factor
based on the number of input partitions. You can override this behavior by
specifying the number of output partitions using the split_out argument.

```python
result = df.groupby('id').value.mean()
result.npartitions  # returns 1

result = df.groupby(['id', 'id2']).value.mean()
result.npartitions  # returns 5

result = df.groupby('id').value.mean(split_out=8)
result.npartitions  # returns 8
```

Some groupby aggregation functions have a different split_out default value.
split_out=True will keep the number of partitions constant, which is useful
for operations that either don’t reduce the number of rows very much.

```python
result = df.groupby('id').value.nunique()
result.npartitions  # returns same as df.npartitions
```


# dataframe-extend.html.md

# Extending DataFrames

## Subclass DataFrames

There are a few projects that subclass or replicate the functionality of Pandas
objects:

- GeoPandas: for Geospatial analytics
- cuDF: for data analysis on GPUs
- …

These projects may also want to produce parallel variants of themselves with
Dask, and may want to reuse some of the code in Dask DataFrame. Subclassing
Dask DataFrames is intended for maintainers of these libraries and not for
general users.

### Implement dask, name, meta, and divisions

You will need to implement `._meta`, `.dask`, `.divisions`, and
`._name` as defined in the [DataFrame design docs](dataframe-design.md).

### Extend Dispatched Methods

If you are going to pass around Pandas-like objects that are not normal Pandas
objects, then we ask you to extend a few dispatched methods: `make_meta`,
`get_collection_type`, and `concat`.

#### make_meta

This function returns an empty version of one of your non-Dask objects, given a
non-empty non-Dask object:

```python
from dask.dataframe.dispatch import make_meta_dispatch

@make_meta_dispatch.register(MyDataFrame)
def make_meta_dataframe(df, index=None):
    return df.head(0)


@make_meta_dispatch.register(MySeries)
def make_meta_series(s, index=None):
    return s.head(0)


@make_meta_dispatch.register(MyIndex)
def make_meta_index(ind, index=None):
    return ind[:0]
```

For dispatching any arbitrary `object` types to a respective back-end, we
recommend registering a dispatch for `make_meta_obj`:

```python
from dask.dataframe.dispatch import make_meta_obj

@make_meta_obj.register(MyDataFrame)
def make_meta_object(x, index=None):
    if isinstance(x, dict):
        return MyDataFrame()
    elif isinstance(x, int):
        return MySeries
    .
    .
    .
```

Additionally, you should create a similar function that returns a non-empty
version of your non-Dask DataFrame objects filled with a few rows of
representative or random data.  This is used to guess types when they are not
provided.  It should expect an empty version of your object with columns,
dtypes, index name, and it should return a non-empty version:

```python
from dask.dataframe.utils import meta_nonempty

@meta_nonempty.register(MyDataFrame)
def meta_nonempty_dataframe(df):
    ...
    return MyDataFrame(..., columns=df.columns,
                       index=MyIndex(..., name=df.index.name)


@meta_nonempty.register(MySeries)
def meta_nonempty_series(s):
    ...


@meta_nonempty.register(MyIndex)
def meta_nonempty_index(ind):
    ...
```

#### get_collection_type

Given a non-Dask DataFrame object, return the Dask equivalent:

```python
from dask.dataframe import get_collection_type

@get_collection_type.register(MyDataFrame)
def get_collection_type_dataframe(df):
    return MyDaskDataFrame


@get_collection_type.register(MySeries)
def get_collection_type_series(s):
    return MyDaskSeries


@get_collection_type.register(MyIndex)
def get_collection_type_index(ind):
    return MyDaskIndex
```

#### concat

Concatenate many of your non-Dask DataFrame objects together.  It should expect
a list of your objects (homogeneously typed):

```python
from dask.dataframe.methods import concat_dispatch

@concat_dispatch.register((MyDataFrame, MySeries, MyIndex))
def concat_pandas(dfs, axis=0, join='outer', uniform=False, filter_warning=True):
    ...
```

<a id="extensionarrays"></a>

## Extension Arrays

Rather than subclassing Pandas DataFrames, you may be interested in extending
Pandas with [Extension Arrays](https://pandas.pydata.org/pandas-docs/stable/extending.html).

All of the first-party extension arrays (those implemented in pandas itself)
are supported directly by dask.

Developers implementing third-party extension arrays (outside of pandas) will
need to do register their `ExtensionDtype` with Dask so that it works
correctly in `dask.dataframe`.

For example, we’ll register the *test-only* `DecimalDtype` from pandas
test suite.

```python
from decimal import Decimal
from dask.dataframe.extensions import make_array_nonempty, make_scalar
from pandas.tests.extension.decimal import DecimalArray, DecimalDtype

@make_array_nonempty.register(DecimalDtype)
def _(dtype):
    return DecimalArray._from_sequence([Decimal('0'), Decimal('NaN')],
                                       dtype=dtype)


@make_scalar.register(Decimal)
def _(x):
   return Decimal('1')
```

Internally, Dask will use this to create a small dummy Series for tracking
metadata through operations.

```python
>>> make_array_nonempty(DecimalDtype())
<DecimalArray>
[Decimal('0'), Decimal('NaN')]
Length: 2, dtype: decimal
```

So you (or your users) can now create and store a dask `DataFrame` or
`Series` with your extension array contained within.

```python
>>> from decimal import Decimal
>>> import dask.dataframe as dd
>>> import pandas as pd
>>> from pandas.tests.extension.decimal import DecimalArray
>>> s = pd.Series(DecimalArray([Decimal('0.0')] * 10))
>>> ds = dd.from_pandas(s, 3)
>>> ds
Dask Series Structure:
npartitions=3
0    decimal
4        ...
8        ...
9        ...
dtype: decimal
Dask Name: from_pandas, 3 tasks
```

Notice the `decimal` dtype.

<a id="dataframe-accessors"></a>

## Accessors

Many extension arrays expose their functionality on Series or DataFrame objects
using accessors. Dask provides decorators to register accessors similar to pandas. See
[the pandas documentation on accessors](http://pandas.pydata.org/pandas-docs/stable/development/extending.html#registering-custom-accessors)
for more.

### dask.dataframe.extensions.register_dataframe_accessor(name)

Register a custom accessor on [`dask.dataframe.DataFrame`](generated/dask.dataframe.DataFrame.md#dask.dataframe.DataFrame).

See [`pandas.api.extensions.register_dataframe_accessor()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.register_dataframe_accessor.html#pandas.api.extensions.register_dataframe_accessor) for more.

<!-- !! processed by numpydoc !! -->

### dask.dataframe.extensions.register_series_accessor(name)

Register a custom accessor on [`dask.dataframe.Series`](generated/dask.dataframe.Series.md#dask.dataframe.Series).

See [`pandas.api.extensions.register_series_accessor()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.register_series_accessor.html#pandas.api.extensions.register_series_accessor) for more.

<!-- !! processed by numpydoc !! -->

### dask.dataframe.extensions.register_index_accessor(name)

Register a custom accessor on [`dask.dataframe.Index`](generated/dask.dataframe.Index.md#dask.dataframe.Index).

See [`pandas.api.extensions.register_index_accessor()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.register_index_accessor.html#pandas.api.extensions.register_index_accessor) for more.

<!-- !! processed by numpydoc !! -->


# dataframe-extra.html.md

# Additional Information

* [Parquet](dataframe-parquet.md)
* [Indexing](dataframe-indexing.md)
* [SQL](dataframe-sql.md)
* [Join Performance](dataframe-joins.md)
* [Shuffling Performance](dataframe-groupby.md)
* [Aggregate](dataframe-groupby.md#aggregate)
* [Categoricals](dataframe-categoricals.md)
* [Extend](dataframe-extend.md)
* [Hive Partitioning](dataframe-hive.md)


# dataframe-groupby.html.md

# Shuffling Performance

Operations like `groupby`, `join`, and `set_index` have special
performance considerations that are different from normal Pandas due to the
parallel, larger-than-memory, and distributed nature of Dask DataFrame.

## Easy Case

To start off, common groupby operations like
`df.groupby(columns).reduction()` for known reductions like `mean, sum, std,
var, count, nunique` are all quite fast and efficient, even if partitions are
not cleanly divided with known divisions.  This is the common case.

Additionally, if divisions are known, then applying an arbitrary function to
groups is efficient when the grouping columns include the index.

Joins are also quite fast when joining a Dask DataFrame to a Pandas DataFrame
or when joining two Dask DataFrames along their index.  No special
considerations need to be made when operating in these common cases.

So, if you’re doing common groupby and join operations, then you can stop reading
this.  Everything will scale nicely.  Fortunately, this is true most of the
time:

```python
>>> ddf.groupby(columns).known_reduction()            # Fast and common case
>>> ddf.groupby(columns_with_index).apply(user_fn)    # Fast and common case
>>> ddf.join(pandas_df, on=column)                    # Fast and common case
>>> lhs.join(rhs)                                     # Fast and common case
>>> lhs.merge(rhs, on=columns_with_index)             # Fast and common case
```

## Difficult Cases

In some cases, such as when applying an arbitrary function to groups (when not
grouping on index with known divisions), when joining along non-index columns,
or when explicitly setting an unsorted column to be the index, we may need to
trigger a full dataset shuffle:

```python
>>> ddf.groupby(columns_no_index).apply(user_fn)   # Requires shuffle
>>> lhs.join(rhs, on=columns_no_index)             # Requires shuffle
>>> ddf.set_index(column)                          # Requires shuffle
```

A shuffle is necessary when we need to re-sort our data along a new index.  For
example, if we have banking records that are organized by time and we now want
to organize them by user ID, then we’ll need to move a lot of data around.  In
Pandas all of this data fits in memory, so this operation was easy.  Now that we
don’t assume that all data fits in memory, we must be a bit more careful.

Re-sorting the data can be avoided by restricting yourself to the easy cases
mentioned above.

<a id="shuffle-methods"></a>

## Shuffle Methods

There are currently two strategies to shuffle data depending on whether you are
on a single machine or on a distributed cluster: shuffle on disk and shuffle
over the network.

### Shuffle on Disk

When operating on larger-than-memory data on a single machine, we shuffle by
dumping intermediate results to disk.  This is done using the [partd](https://github.com/dask/partd) project
for on-disk shuffles.

### Shuffle over the Network

When operating on a distributed cluster, the Dask workers may not have access to
a shared hard drive.  In this case, we shuffle data by breaking input partitions
into many pieces based on where they will end up and moving these pieces
throughout the network.

### Selecting methods

Dask will use on-disk shuffling by default, but will switch to a
distributed shuffling algorithm if the default scheduler is set to use a
`dask.distributed.Client`, such as would be the case if the user sets the
Client as default:

```python
client = Client('scheduler:8786', set_as_default=True)
```

Alternatively, if you prefer to avoid defaults, you can configure the global
shuffling method with the `dataframe.shuffle.method` configuration option.
This can be done globally:

```python
dask.config.set({"dataframe.shuffle.method": "p2p"})

ddf.groupby(...).apply(...)
```

or as a context manager:

```python
with dask.config.set({"dataframe.shuffle.method": "p2p"}):
    ddf.groupby(...).apply(...)
```

In addition, `set_index` also accepts a `shuffle_method` keyword argument that
can be used to select either on-disk or task-based shuffling:

```python
ddf.set_index(column, shuffle_method='disk')
ddf.set_index(column, shuffle_method='tasks')
ddf.set_index(column, shuffle_method='p2p')
```

<a id="dataframe-groupby-aggregate"></a>

# Aggregate

Dask supports Pandas’ `aggregate` syntax to run multiple reductions on the
same groups.  Common reductions such as `max`, `sum`, `list` and `mean` are
directly supported:

```python
>>> ddf.groupby(columns).aggregate(['sum', 'mean', 'max', 'min', list])
```

Dask also supports user defined reductions.  To ensure proper performance, the
reduction has to be formulated in terms of three independent steps. The
`chunk` step is applied to each partition independently and reduces the data
within a partition. The `aggregate` combines the within partition results.
The optional `finalize` step combines the results returned from the
`aggregate` step and should return a single final column. For Dask to
recognize the reduction, it has to be passed as an instance of
`dask.dataframe.Aggregation`.

For example, `sum` could be implemented as:

```python
custom_sum = dd.Aggregation('custom_sum', lambda s: s.sum(), lambda s0: s0.sum())
ddf.groupby('g').agg(custom_sum)
```

The name argument should be different from existing reductions to avoid data
corruption.  The arguments to each function are pre-grouped series objects,
similar to `df.groupby('g')['value']`.

Many reductions can only be implemented with multiple temporaries. To implement
these reductions, the steps should return tuples and expect multiple arguments.
A mean function can be implemented as:

```python
custom_mean = dd.Aggregation(
    'custom_mean',
    lambda s: (s.count(), s.sum()),
    lambda count, sum: (count.sum(), sum.sum()),
    lambda count, sum: sum / count,
)
ddf.groupby('g').agg(custom_mean)
```

For example, let’s compute the group-wise extent (maximum - minimum)
for a DataFrame.

```python
>>> df = pd.DataFrame({
...   'a': ['a', 'b', 'a', 'a', 'b'],
...   'b': [0, 1, 0, 2, 5],
... })
>>> ddf = dd.from_pandas(df, 2)
```

We define the building blocks to find the maximum and minimum of each chunk, and then
the maximum and minimum over all the chunks. We finalize by taking the difference between
the Series with the maxima and minima

```python
>>> def chunk(grouped):
...     return grouped.max(), grouped.min()

>>> def agg(chunk_maxes, chunk_mins):
...     return chunk_maxes.max(), chunk_mins.min()

>>> def finalize(maxima, minima):
...     return maxima - minima
```

Finally, we create and use the aggregation

```python
>>> extent = dd.Aggregation('extent', chunk, agg, finalize=finalize)
>>> ddf.groupby('a').agg(extent).compute()
   b
a
a  2
b  4
```

To apply `dask.dataframe.groupby.SeriesGroupBy.nunique` to more than one
column you can use:

```python
>>> df['c'] = [1, 2, 1, 1, 2]
>>> ddf = dd.from_pandas(df, 2)
>>> nunique = dd.Aggregation(
...     name="nunique",
...     chunk=lambda s: s.apply(lambda x: list(set(x))),
...     agg=lambda s0: s0.obj.groupby(level=list(range(s0.obj.index.nlevels))).sum(),
...     finalize=lambda s1: s1.apply(lambda final: len(set(final))),
... )
>>> ddf.groupby('a').agg({'b':nunique, 'c':nunique})
```

To access NumPy functions use `apply` with a lambda function such as `.apply(lambda r: np.sum(r))`. Here’s an example of how a sum of squares aggregation would look like:

```python
>>> dd.Aggregation(name="sum_of_squares", chunk=lambda s: s.apply(lambda r: np.sum(np.power(r, 2))), agg=lambda s: s.sum())
```


# dataframe-hive.html.md

<a id="dataframe-hive"></a>

# Using Hive Partitioning with Dask

It is sometimes useful to write your dataset with a hive-like directory scheme.
For example, if your dataframe contains `'year'` and `'semester'` columns,
a hive-based directory structure might look something like the following:

```default
output-path/
├── year=2022/
│   ├── semester=fall/
│   │   └── part.0.parquet
│   └── semester=spring/
│       ├── part.0.parquet
│       └── part.1.parquet
└── year=2023/
    └── semester=fall/
        └── part.1.parquet
```

The use of this self-describing structure implies that all rows within
the `'output-path/year=2022/semester=fall/'` directory will contain
the value `2022` in the `'year'` column and the value `'fall'`
in the `'semester'` column.

The primary advantage of generating a hive-partitioned dataset
is that certain IO filters can be applied by [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet)
without the need to parse any file metadata. In other words,
the following command will typically be faster when the dataset
is already hive-partitioned on the `'year'` column.

```python
>>> dd.read_parquet("output-path", filters=[("year", ">", 2022)])
```

## Writing Parquet Data with Hive Partitioning

Dask’s [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) function will produce a hive-partitioned
directory scheme automatically when the `partition_on` option is used.

```python
>>> df.to_parquet("output-path", partition_on=["year", "semester"])

>>> os.listdir("output-path")
["year=2022", "year=2023"]

>>> os.listdir("output-path/year=2022")
["semester=fall", "semester=spring"]

>>> os.listdir("output-path/year=2022/semester=spring")
['part.0.parquet', 'part.1.parquet']
```

It is important to recognize that Dask will **not** aggregate the
data files written within each of the leaf directories. This is
because each of the DataFrame partitions is written independently
during the execution of the [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) task graph. In order
to write out data for partition i, the partition-i write task will
perform a groupby operation on columns `["year", "semester"]`,
and then each distinct group will be written to the corresponding
directory using the file name `'part.{i}.parquet'`. Therefore, it
is possible for a hive-partitioned write to produce a large number
of files in every leaf directory (one for each DataFrame partition).

If your application requires you to produce a single parquet file
for each hive partition, one possible solution is to sort or shuffle
on the partitioning columns before calling [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet).

```python
>>> partition_on = ["year", "semester"]

>>> df.shuffle(on=partition_on).to_parquet(partition_on=partition_on)
```

Using a global shuffle like this is extremely expensive, and should be
avoided whenever possible. However, it is also guaranteed to produce
the minimum number of files, which may be worth the sacrifice at times.

## Reading Parquet Data with Hive Partitioning

In most cases, [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) will process hive-partitioned
data automatically. By default, all hive-partitioned columns will
be interpreted as categorical columns.

```python
>>> ddf = dd.read_parquet("output-path", columns=["year", "semester"])

>>> ddf
Dask DataFrame Structure:
                        year         semester
npartitions=4
            category[known]  category[known]
                        ...              ...
                        ...              ...
                        ...              ...
                        ...              ...
Dask Name: read-parquet, 1 graph layer

>>> ddf.compute()
year semester
0  2022     fall
1  2022     fall
2  2022     fall
3  2022   spring
4  2022   spring
5  2022   spring
6  2023     fall
7  2023     fall
```

## Defining a Custom Partitioning Schema

It is possible to specify a custom schema for the hive-partitioned columns. The columns
will then be read using the specified types and not as category.

```python
>>> schema = pa.schema([("year", pa.int16()), ("semester", pa.string())])

>>> ddf2 = dd.read_parquet(
...     path,
...     columns=["year", "semester"],
...     dataset={"partitioning": {"flavor": "hive", "schema": schema}}
... )
Dask DataFrame Structure:
                year semester
npartitions=4
            int16   object
                ...      ...
                ...      ...
                ...      ...
                ...      ...
```

If any of your hive-partitioned columns contain null values, you
**must** specify the partitioning schema in this way.

Although it is not required, we also recommend that you specify
the partitioning schema if you need to partition on high-cardinality
columns. This is because the default `'category'` dtype will
track the known categories in a way that can significantly increase
the overall memory footprint of your Dask collection. In fact,
[`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) already clears the “known categories” of other
columns for this same reason (see [Categoricals](dataframe-categoricals.md)).

### Best Practices

Although hive partitioning can sometimes improve read performance
by simplifying filtering, it can also lead to degraded performance
and errors in other cases.

## Avoid High Cardinality

A good rule of thumb is to avoid partitioning on float columns,
or any column containing many unique values (i.e. high cardinality).

The most common cause of poor user experience with hive partitioning
is high-cardinality of the partitioning column(s). For example, if
you try to partition on a column with millions of unique values, then
:func:\`to_parquet\` will need to generate millions of directories. The
management of these directories is likely to put strain on the file
system, and the need for many small files within each directory
is sure to compound the issue.

## Use Simple Data Types for Partitioning

Since hive-partitioned data is “self describing,” we suggest that
you avoid partitioning on complex data types, and opt for integer
or string-based data types whenever possible. If your data type
cannot be easily inferred from the string value used to define the
directory name, then the IO engine may struggle to parse the values.

For example, directly partitioning on a column with a `datetime64`
dtype might produce a directory name like the following:

```default
output-path/
├── date=2022-01-01 00:00:00/
├── date=2022-02-01 00:00:00/
├── ...
└── date=2022-12-01 00:00:00/
```

These directory names will not be correctly interpreted as
`datetime64` values, and are even considered illegal on Windows
systems. For more-reliable behavior, we recommend that such a column
be decomposed into one or more “simple” columns. For example, one
could easily use `'date'` to construct `'year'`, `'month'`,
and `'day'` columns (as needed).

## Aggregate Files at Read Time

#### WARNING
The `aggregate_files` argument is currently listed as
experimental. However, there are currently no plans to remove
the argument or change its behavior in a future release.

Since hive-partitioning will typically produce a large number of
small files, [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) performance will usually benefit
from proper usage of the `aggregate_files` argument. Take the
following dataset for example:

```default
dataset-path/
├── region=1/
│   ├── section=a/
│   │   └── 01.parquet
│   │   └── 02.parquet
│   │   └── 03.parquet
│   ├── section=b/
│   └── └── 04.parquet
│   └── └── 05.parquet
└── region=2/
    ├── section=a/
    │   ├── 06.parquet
    │   ├── 07.parquet
    │   ├── 08.parquet
```

If we set `aggregate_files=True` for this case, we are telling Dask
that any of the parquet data files may be aggregated into the same output
DataFrame partition. If, instead, we specify the name of a partitioning
column (e.g. `'region'` or `'section'`), we allow the aggregation of
any two files sharing a file path up to, and including, the corresponding
directory name. For example, if `aggregate_files` is set to `'section'`,
`04.parquet` and `05.parquet` may be aggregated together, but
`03.parquet` and `04.parquet` cannot be. If, however, `aggregate_files`
is set to `'region'`, `04.parquet` may be aggregated with `05.parquet`,
**and** `03.parquet` may be aggregated with `04.parquet`.

Using `aggregate_files` will typically improve performance by making it
more likely for DataFrame partitions to approach the size specified by the
`blocksize` argument. In contrast, default behavior may produce a large
number of partitions that are much smaller than `blocksize`.


# dataframe-indexing.html.md

<a id="dataframe-indexing"></a>

# Indexing into Dask DataFrames

Dask DataFrame supports some of Pandas’ indexing behavior.

| [`DataFrame.iloc`](generated/dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)   | Purely integer-location based indexing for selection by position.   |
|------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`DataFrame.loc`](generated/dask.dataframe.DataFrame.loc.md#dask.dataframe.DataFrame.loc)      | Purely label-location based indexer for selection by label.         |

## Label-based Indexing

Just like Pandas, Dask DataFrame supports label-based indexing with the `.loc`
accessor for selecting rows or columns, and `__getitem__` (square brackets)
for selecting just columns.

#### NOTE
To select rows, the DataFrame’s divisions must be known (see
[Dask DataFrame Design](dataframe-design.md#dataframe-design) and [Dask DataFrames Best Practices](dataframe-best-practices.md#dataframe-performance) for more information.)

```python
>>> import dask.dataframe as dd
>>> import pandas as pd
>>> df = pd.DataFrame({"A": [1, 2, 3], "B": [3, 4, 5]},
...                   index=['a', 'b', 'c'])
>>> ddf = dd.from_pandas(df, npartitions=2)
>>> ddf
Dask DataFrame Structure:
                   A      B
npartitions=1
a              int64  int64
c                ...    ...
Dask Name: from_pandas, 1 tasks
```

Selecting columns:

```python
>>> ddf[['B', 'A']]
Dask DataFrame Structure:
                   B      A
npartitions=1
a              int64  int64
c                ...    ...
Dask Name: getitem, 2 tasks
```

Selecting a single column reduces to a Dask Series:

```python
>>> ddf['A']
Dask Series Structure:
npartitions=1
a    int64
c      ...
Name: A, dtype: int64
Dask Name: getitem, 2 tasks
```

Slicing rows and (optionally) columns with `.loc`:

```python
>>> ddf.loc[['b', 'c'], ['A']]
Dask DataFrame Structure:
                   A
npartitions=1
b              int64
c                ...
Dask Name: loc, 2 tasks

>>> ddf.loc[df["A"] > 1, ["B"]]
Dask DataFrame Structure:
                   B
npartitions=1
a              int64
c                ...
Dask Name: try_loc, 2 tasks

>>> ddf.loc[lambda df: df["A"] > 1, ["B"]]
Dask DataFrame Structure:
                   B
npartitions=1
a              int64
c                ...
Dask Name: try_loc, 2 tasks
```

Dask DataFrame supports Pandas’ [partial-string indexing](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#partial-string-indexing):

```python
>>> ts = dd.demo.make_timeseries()
>>> ts
Dask DataFrame Structure:
                   id    name        x        y
npartitions=11
2000-01-31      int64  object  float64  float64
2000-02-29        ...     ...      ...      ...
...               ...     ...      ...      ...
2000-11-30        ...     ...      ...      ...
2000-12-31        ...     ...      ...      ...
Dask Name: make-timeseries, 11 tasks

>>> ts.loc['2000-02-12']
Dask DataFrame Structure:
                                  id    name        x        y
npartitions=1
2000-02-12 00:00:00.000000000  int64  object  float64  float64
2000-02-12 23:59:59.999999999    ...     ...      ...      ...
Dask Name: loc, 12 tasks
```

## Positional Indexing

Dask DataFrame does not track the length of partitions, making positional
indexing with `.iloc` inefficient for selecting rows. [`DataFrame.iloc()`](generated/dask.dataframe.DataFrame.iloc.md#dask.dataframe.DataFrame.iloc)
only supports indexers where the row indexer is `slice(None)` (which `:` is
a shorthand for.)

```python
>>> ddf.iloc[:, [1, 0]]
Dask DataFrame Structure:
                   B      A
npartitions=1
a              int64  int64
c                ...    ...
Dask Name: iloc, 2 tasks
```

Trying to select specific rows with `iloc` will raise an exception:

```python
>>> ddf.iloc[[0, 2], [1]]
Traceback (most recent call last)
  File "<stdin>", line 1, in <module>
ValueError: 'DataFrame.iloc' does not support slicing rows. The indexer must be a 2-tuple whose first item is 'slice(None)'.
```

## Partition Indexing

In addition to pandas-style indexing, Dask DataFrame also supports indexing at a
partition level with [`DataFrame.get_partition()`](generated/dask.dataframe.DataFrame.get_partition.md#dask.dataframe.DataFrame.get_partition) and
[`DataFrame.partitions`](generated/dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions). These can be used to select subsets of the data by
partition, rather than by position in the entire DataFrame or index label.

Use [`DataFrame.get_partition()`](generated/dask.dataframe.DataFrame.get_partition.md#dask.dataframe.DataFrame.get_partition) to select a single partition by position.

```python
>>> import dask
>>> ddf = dask.datasets.timeseries(start="2021-01-01", end="2021-01-07", freq="1h")
>>> ddf.get_partition(0)
Dask DataFrame Structure:
                 name     id        x        y
npartitions=1
2021-01-01     object  int64  float64  float64
2021-01-02        ...    ...      ...      ...
Dask Name: get-partition, 2 graph layers
```

Note that the result is also a Dask DataFrame.

Index into [`DataFrame.partitions`](generated/dask.dataframe.DataFrame.partitions.md#dask.dataframe.DataFrame.partitions) to select one or more partitions. For
example, you can select every other partition with a slice:

```python
>>> ddf.partitions[::2]
Dask DataFrame Structure:
                 name     id        x        y
npartitions=3
2021-01-01     object  int64  float64  float64
2021-01-03        ...    ...      ...      ...
2021-01-05        ...    ...      ...      ...
2021-01-06        ...    ...      ...      ...
Dask Name: blocks, 2 graph layers
```

Or even more complicated selections based on the data in the partitions
themselves (at the cost of computing the DataFrame up until that point). For
example, we can create a boolean mask with the partitions that have more than
some number of unique IDs using [`DataFrame.map_partitions()`](generated/dask.dataframe.DataFrame.map_partitions.md#dask.dataframe.DataFrame.map_partitions):

```python
>>> mask = ddf.id.map_partitions(lambda p: len(p.unique()) > 20).compute()
>>> ddf.partitions[mask]
Dask DataFrame Structure:
                 name     id        x        y
npartitions=5
2021-01-01     object  int64  float64  float64
2021-01-02        ...    ...      ...      ...
...               ...    ...      ...      ...
2021-01-06        ...    ...      ...      ...
2021-01-07        ...    ...      ...      ...
Dask Name: blocks, 2 graph layers
```


# dataframe-joins.html.md

# Joins

DataFrame joins are a common and expensive computation that benefit from a
variety of optimizations in different situations.  Understanding how your data
is laid out and what you’re trying to accomplish can have a large impact on
performance.  This documentation page goes through the various different
options and their performance impacts.

## Large to Large Unsorted Joins

In the worst case scenario you have two large tables with many partitions each
and you want to join them both along a column that may not be sorted.

This can be slow.  In this case Dask DataFrame will need to move all of your
data around so that rows with matching values in the joining columns are in the
same partition.  This large-scale movement can create communication costs, and
can require a large amount of memory.  If enough memory cannot be found then
Dask will have to read and write data to disk, which may cause other
performance costs.

These problems are solvable, but will be significantly slower than many other
operations.  They are best avoided if possible.

## Large to Small Joins

Many join or merge computations combine a large table with one small one.  If
the small table is either a single partition Dask DataFrame or even just a
normal Pandas DataFrame then the computation can proceed in an embarrassingly
parallel way, where each partition of the large DataFrame is joined against the
single small table.  This incurs almost no overhead relative to Pandas joins.

If your smaller table can easily fit in memory, then you might want to ensure
that it is a single partition with the repartition method.

```python
import dask
large = dask.datasets.timeseries(freq="10s", npartitions=10)
small = dask.datasets.timeseries(freq="1D", dtypes={"z": int})

small = small.repartition(npartitions=1)
result = large.merge(small, how="left", on=["timestamp"])
```

## Sorted Joins

The Pandas merge API supports the `left_index=` and `right_index=` options
to perform joins on the index.  For Dask DataFrames these keyword options hold
special significance if the index has known divisions
(see [Partitions](dataframe-design.md#dataframe-design-partitions)).
In this case the DataFrame partitions are aligned along these divisions (which
is generally fast) and then an embarrassingly parallel Pandas join happens
across partition pairs.  This is generally relatively fast.

Sorted or indexed joins are a good solution to the large-large join problem.
If you plan to join against a dataset repeatedly then it may be worthwhile to
set the index ahead of time, and possibly store the data in a format that
maintains that index, like Parquet.

```python
import dask
import dask.dataframe as dd

left = dask.datasets.timeseries(dtypes={"foo": int})

# timeseries returns a dataframe indexed by
# timestamp, we don't need to set_index.

# left.set_index("timestamp")

left.to_parquet("left", overwrite=True)
left = dd.read_parquet("left")

right_one = dask.datasets.timeseries(dtypes={"bar": int})
right_two = dask.datasets.timeseries(dtypes={"baz": int})

result = left.merge(
    right_one, how="left", left_index=True, right_index=True)
result = result.merge(
    right_two, how="left", left_index=True, right_index=True)
```


# dataframe-optimizer.html.md

# Optimizer

#### NOTE
Dask DataFrame supports Query Planning since version 2024.03.0

## Optimization steps

Dask DataFrame will run several optimizations before executing the computation. These
computations are aimed towards improving the efficiency of the query.

The optimizations entail the following steps (this list is not complete):

- **Projection Pushdown:**

  Select only the required columns in every step. This reduces the amount of data
  that needs to be read from storage but also the amount of data that is processed
  along the way. Columns are dropped at the earliest stage in the query.
- **Filter Pushdown:**

  Push filters down as far as possible, potentially into the IO step. Filters are
  executed in the earliest stage in the query.
- **Partition Pruning:**

  Partition selections are pushed down as far as possible, potentially into the IO
  step.
- **Avoiding Shuffles:**

  Dask DataFrame will try to avoid shuffling data between workers as much as possible.
  This can be achieved if the column layout is already known, i.e. if the DataFrame
  was shuffled on the same column before. For example, executing a
  `df.groupby(...).apply(...)` after a merge operation will not shuffle the data
  again if the groupby happens on the merge columns.
  ![image](images/optimizer/avoiding-shuffles.svg)
- **Automatically resizing partitions:**

  The IO layers automatically adjust the partition count based on the column subset
  that is selected from the dataset. Very small partitions impact the scheduler and
  expensive operations like shuffling negatively. This is addressed by adjusting
  the partition count automatically.
  ![image](images/optimizer/automatic-repartitioning.svg)

## Exploring the optimized query

Dask will call `df.optimize()` before executing the computation. This method applies
the steps mentioned above and returns a new Dask DataFrame that represents the optimized
query.

A rudimentary representation of the optimized query can be obtained by calling
`df.pprint()`. This will print the query plan in a human-readable format to the
command line/console. The advantage of this method is that it doesn’t require any
additional dependencies.

```default
pdf = pd.DataFrame({"a": [1, 2, 3] * 5, "b": [1, 2, 3] * 5})
df = dd.from_pandas(pdf, npartitions=2)
df = df.replace(1, 5)[["a"]]

df.optimize().pprint()

Replace: to_replace=1 value=5
    FromPandas: frame='<pandas>' npartitions=2 columns=['a'] pyarrow_strings_enabled=True
```

A more advanced and easier to read reprepresentation can be obtained by calling
`df.explain()`. This method requires the `graphviz` package to be installed.
The method will return a graph that represents the query plan and create an image
from it.

```default
df.explain()
```

![Optimized Query](images/dataframe-optimize-explain.png)

We can see in both representations that `FromPandas` consumed the column
projection, only selecting the column `a`.

The `explain()` method is significantly easier to understand for more complex
queries.


# dataframe-parquet.html.md

<a id="dataframe-parquet"></a>

# Dask Dataframe and Parquet

[Parquet](https://parquet.apache.org/) is a popular, columnar file format designed for efficient data storage
and retrieval. Dask dataframe includes [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) and
[`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) functions/methods for reading and writing parquet files
respectively. Here we document these methods, and provide some tips and best
practices.

Parquet I/O requires `pyarrow` to be installed.

## Reading Parquet Files

| [`read_parquet`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet)([path, columns, filters, ...])   | Read a Parquet file into a Dask DataFrame   |
|-------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|

Dask dataframe provides a [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) function for reading one or more
parquet files. Its first argument is one of:

- A path to a single parquet file
- A path to a directory of parquet files (files with `.parquet` or `.parq` extension)
- A [glob string](https://docs.python.org/3/library/glob.html) expanding to one or more parquet file paths
- A list of parquet file paths

These paths can be local, or point to some remote filesystem (for example [S3](https://aws.amazon.com/s3/)
or [GCS](https://cloud.google.com/storage)) by prepending the path with a protocol.

```python
>>> import dask.dataframe as dd

# Load a single local parquet file
>>> df = dd.read_parquet("path/to/mydata.parquet")

# Load a directory of local parquet files
>>> df = dd.read_parquet("path/to/my/parquet/")

# Load a directory of parquet files from S3
>>> df = dd.read_parquet("s3://bucket-name/my/parquet/")
```

Note that for remote filesystems you may need to configure credentials. When
possible we recommend handling these external to Dask through
filesystem-specific configuration files/environment variables. For example, you
may wish to store S3 credentials using the [AWS credentials file](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#shared-credentials-file>`_.).
Alternatively, you can pass configuration on to the [fsspec](https://filesystem-spec.readthedocs.io) backend through the
`storage_options` keyword argument:

```python
>>> df = dd.read_parquet(
...      "s3://bucket-name/my/parquet/",
...      storage_options={"anon": True}  # passed to `s3fs.S3FileSystem`
... )
```

For more information on connecting to remote data, see
[Connect to remote data](how-to/connect-to-remote-data.md).

[`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) has many configuration options affecting both behavior and
performance. Here we highlight a few common options.

### Metadata

When [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) is used to read *multiple files*, it first loads
metadata about the files in the dataset. This metadata may include:

- The dataset schema
- How the dataset is partitioned into files, and those files into row-groups

Some parquet datasets include a `_metadata` file which aggregates per-file
metadata into a single location. For small-to-medium sized datasets this *may*
be useful because it makes accessing the row-group metadata possible without
reading parts of *every* file in the dataset. Row-group metadata allows Dask to
split large files into smaller in-memory partitions, and merge many small files
into larger partitions, possibly leading to higher performance.

However, for large datasets the `_metadata` file can be problematic because
it may be too large for a single endpoint to parse! If this is true, you can
disable loading the `_metadata` file by specifying
`ignore_metadata_file=True`.

```python
>>> df = dd.read_parquet(
...      "s3://bucket-name/my/parquet/",
...      ignore_metadata_file=True  # don't read the _metadata file
... )
```

### Partition Size

By default, Dask will use metadata from the first parquet file in the dataset
to infer whether or not it is safe load each file individually as a partition
in the Dask dataframe. If the uncompressed byte size of the parquet data
exceeds `blocksize` (which is 256 MiB by default), then each partition will
correspond to a range of parquet row-groups instead of the entire file.

For best performance, use files that can be individually mapped to good
dataframe partition sizes, and set `blocksize` accordingly. If individual
files need to be divided into multiple row-group ranges, and the dataset
does not contain a `_metadata` file, Dask will need to load all footer
metadata up-front.

We recommend aiming for 100-300 MiB in-memory size per file once loaded into
pandas. Oversized partitions can lead to excessive memory usage on a single
worker, while undersized partitions can lead to poor performance as the
overhead of Dask dominates.

If you know your parquet dataset comprises oversized files, you can pass
`split_row_groups='adaptive'` to ensure that Dask will attempt to keep
each partition under the `blocksize` limit. Note that partitions may
still exceed `blocksize` if one or more row-groups are too large.

### Column Selection

When loading parquet data, sometimes you don’t need all the columns available
in the dataset. In this case, you likely want to specify the subset of columns
you need via the `columns` keyword argument. This is beneficial for a few
reasons:

- It lets Dask read less data from the backing filesystem, reducing IO costs
- It lets Dask load less data into memory, reducing memory usage

```python
>>> dd.read_parquet(
...     "s3://path/to/myparquet/",
...     columns=["a", "b", "c"]  # Only read columns 'a', 'b', and 'c'
... )
```

### Calculating Divisions

By default, [`read_parquet()`](generated/dask.dataframe.read_parquet.md#dask.dataframe.read_parquet) will **not** produce a collection with
known divisions. However, you can pass `calculate_divisions=True` to
tell Dask that you want to use row-group statistics from the footer
metadata (or global `_metadata` file) to calculate the divisions at
graph-creation time. Using this option will not produce known
divisions if any of the necessary row-group statistics are missing,
or if no index column is detected. Using the `index` argument is the
best way to ensure that the desired field will be treated as the index.

```python
>>> dd.read_parquet(
...     "s3://path/to/myparquet/",
...     index="timestamp",  # Specify a specific index column
...     calculate_divisions=True,  # Calculate divisions from metadata
... )
```

Although using `calculate_divisions=True` does not require any *real*
data to be read from the parquet file(s), it does require Dask to load
and process metadata for every row-group in the dataset. For this reason,
calculating divisions should be avoided for large datasets without a
global `_metadata` file. This is especially true for remote storage.

For more information about divisions, see [Dask DataFrame Design](dataframe-design.md#dataframe-design).

## Writing

| [`to_parquet`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet)(df, path[, compression, ...])                   | Store Dask.dataframe to Parquet files   |
|----------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| [`DataFrame.to_parquet`](generated/dask.dataframe.DataFrame.to_parquet.md#dask.dataframe.DataFrame.to_parquet)(path, \*\*kwargs) |                                         |

Dask dataframe provides a [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) function and method for writing
parquet files.

In its simplest usage, this takes a path to the directory in which to write the
dataset. This path may be local, or point to some remote filesystem (for
example [S3](https://aws.amazon.com/s3/) or [GCS](https://cloud.google.com/storage)) by prepending the path with a protocol.

```python
# Write to a local directory
>>> df.to_parquet("path/to/my/parquet/")

# Write to S3
>>> df.to_parquet("s3://bucket-name/my/parquet/")
```

Note that for remote filesystems you may need to configure credentials. When
possible we recommend handling these external to Dask through
filesystem-specific configuration files/environment variables. For example, you
may wish to store S3 credentials using the [AWS credentials file](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#shared-credentials-file>`_.).
Alternatively, you can pass configuration on to the [fsspec](https://filesystem-spec.readthedocs.io) backend through the
`storage_options` keyword argument:

```python
>>> df.to_parquet(
...     "s3://bucket-name/my/parquet/",
...     storage_options={"anon": True}  # passed to `s3fs.S3FileSystem`
... )
```

For more information on connecting to remote data, see
[Connect to remote data](how-to/connect-to-remote-data.md).

Dask will write one file per Dask dataframe partition to this directory. To
optimize access for downstream consumers, we recommend aiming for an in-memory
size of 100-300 MiB per partition. This helps balance worker memory usage
against Dask overhead. You may find the
[`DataFrame.memory_usage_per_partition()`](generated/dask.dataframe.DataFrame.memory_usage_per_partition.md#dask.dataframe.DataFrame.memory_usage_per_partition) method useful for determining if
your data is partitioned optimally.

[`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) has many configuration options affecting both behavior and
performance. Here we highlight a few common options.

### Metadata

In order to improve *read* performance, Dask can optionally write out
a global `_metadata` file at write time by aggregating the row-group
metadata from every file in the dataset. While potentially useful at
read time, the generation of this file may result in excessive memory
usage at scale (and potentially killed Dask workers). As such,
enabling the writing of this file is only recommended for small to
moderate dataset sizes.

```python
>>> df.to_parquet(
...     "s3://bucket-name/my/parquet/",
...     write_metadata_file=True  # enable writing the _metadata file
... )
```

### File Names

Unless the partition_on option is used (see [Using Hive Partitioning with Dask](dataframe-hive.md)),
[`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) will write one file per Dask dataframe partition to the
output directory. By default these files will have names like
`part.0.parquet`, `part.1.parquet`, etc. If you wish to alter this naming
scheme, you can use the `name_function` keyword argument. This takes a
function with the signature `name_function(partition: int) -> str`, taking
the partition index for each Dask dataframe partition and returning a string to
use as the filename. Note that names returned must sort in the same order as
their partition indices.

```python
>>> df.npartitions  # 3 partitions (0, 1, and 2)
3

>>> df.to_parquet("/path/to/output", name_function=lambda i: f"data-{i}.parquet")

>>> os.listdir("/path/to/parquet")
["data-0.parquet", "data-1.parquet", "data-2.parquet"]
```

### Hive Partitioning

It is sometimes useful to write a parquet dataset with a hive-like directory scheme
(e.g. `'/year=2022/month=12/day=25'`). [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) will automatically
produce a dataset with this kind of directory structure when the `partition_on`
option is used. In most cases, [`to_parquet()`](generated/dask.dataframe.to_parquet.md#dask.dataframe.to_parquet) will handle hive partitioning
automatically. See [Using Hive Partitioning with Dask](dataframe-hive.md) for more information.


# dataframe-sql.html.md

# Dask Dataframe and SQL

SQL is a method for executing tabular computation on database servers.
Similar operations can be done on Dask Dataframes. Users commonly wish
to link the two together.

This document describes the connection between Dask and SQL-databases
and serves to clarify several of the questions that we commonly
receive from users.

> * [Does Dask implement SQL?](#does-dask-implement-sql)
> * [Database or Dask?](#database-or-dask)
> * [Loading from SQL with read_sql_table or read_sql_query](#loading-from-sql-with-read-sql-table-or-read-sql-query)
> * [Load from SQL, manual approaches](#load-from-sql-manual-approaches)
> * [Query pushdown?](#query-pushdown)

## Does Dask implement SQL?

The short answer is “no”. Dask has no parser or query planner for SQL
queries. However, the Pandas API, which is largely identical for
Dask Dataframes, has many analogues to SQL operations. A good
description for mapping SQL onto Pandas syntax can be found in the
[pandas docs](https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_sql.html).

The following packages may be of interest:

- [dask-sql](https://dask-sql.readthedocs.io/en/latest/) adds a SQL query engine on top of Dask.
  In addition to working on CPU, it offers experimental support for CUDA-enabled GPUs through RAPIDS libraries such as [cuDF](https://docs.rapids.ai/api/cudf/stable/).
- [FugueSQL](https://fugue-tutorials.readthedocs.io/en/latest/tutorials/fugue_sql/index.html) provides a unified interface to run SQL code on a variety of different computing frameworks.
  Specifying `DaskExecutionEngine` or `DaskSQLExecutionEngine` as the execution engine for queries allows them to be computed using Dask or dask-sql, respectively.

## Database or Dask?

A database server is able to process tabular data and produce results just like
Dask Dataframe. Why would you choose to use one over the other?

These days a database server can be a sharded/distributed system, capable of
handling tables with millions of rows. Most database implementations are
geared towards row-wise retrieval and (atomic) updates of small subset of a
table. Configuring a database to be fast for a particular
sort of query can be challenging, but assuming all your data is already in the
database, it may well be the best solution - particularly if you understand
something about SQL query plan optimisation. A SQL implementation can
very efficiently analyse a query to only extract a small part of a table
for consideration, when the rest is excluded by conditionals.

Dask is much more flexible than a database, and designed explicitly
to work with larger-than-memory datasets, in parallel, and potentially distributed
across a cluster. If your workflow is not well suited to SQL, use dask. If
your database server struggles with volume, dask may do better. It
would be best to profile your queries
(and keep in mind other users of the resources!). If you need
to combine data from different sources, dask may be your best option.

You may find the dask API easier to use than writing SQL (if you
are already used to Pandas), and the diagnostic feedback more useful.
These points can debatably be in Dask’s favour.

## Loading from SQL with read_sql_table or read_sql_query

Dask allows you to build dataframes from SQL tables and queries using the
function [`dask.dataframe.read_sql_table()`](generated/dask.dataframe.read_sql_table.md#dask.dataframe.read_sql_table) and [`dask.dataframe.read_sql_query()`](generated/dask.dataframe.read_sql_query.md#dask.dataframe.read_sql_query),
based on the [Pandas version](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql_table.html), sharing most arguments, and using SQLAlchemy
for the actual handling of the queries. You may need to install additional
driver packages for your chosen database server.

Since Dask is designed to work with larger-than-memory datasets, or be distributed
on a cluster, the following are the main differences versus Pandas to watch out for

- Dask does not support arbitrary text queries, only whole tables and SQLAlchemy
  [sql expressions](https://docs.sqlalchemy.org/en/13/core/tutorial.html)
- the con argument must be a [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls), not an SQLAlchemy engine/connection
- partitioning information is *required*, which can be as simple as providing
  an index column argument, or can be more explicit (see below)
- the chunksize argument is not used, since the partitioning must be via an
  index column

If you need something more flexible than this, or the
method fails for you (e.g., on type inference), then skip to the next section.

### Why the differences

Dask is intended to make processing large volumes of data possible, including
potentially distributing that processing across a cluster. For the retrieval of
data from SQL servers, this means that the query must be partitionable: that
each partition can be fetched independently of others and not dependent on
some global state, and that the definitions of the tasks must be serialisable,
i.e., can be represented as a stream of bytes communicated to workers.

The constraints mean that we cannot directly accept SQLAlchemy engines
or connection objects, since they have internal state (buffers, etc.)
that cannot be serialised. A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls)  must be used, which can be
recreated into a fresh engine on the workers.
Similarly, we cannot accommodate chunked queries
which rely on the internal state of a database cursor; nor LIMIT/OFFSET
queries, which are not guaranteed to be repeatable, and involve scanning
the whole query on the server (which is very inefficient).

**If** your data is small enough not to require Dask’s out-of-core and/or
distributed capabilities, then you are probably better to use Pandas or SQLAlchemy
directly.

### Index Column

We need a way to turn a single main query into sub-queries for each
partition. For most reasonable database tables, there should be an obvious
column which can be used for partitioning - it is probably numeric,
and should certainly be indexed in the database. The latter condition
is important, since many simultaneous queries will hit your server once
Dask starts to compute.

By providing just a column name for the index argument, you imply that the
column is numeric, and Dask guesses a reasonable partitioning by evenly
splitting the space between minimum and maximum values into `npartitions`
intervals. You can also provide the max/min that you would like to
consider so that Dask doesn’t need to query for these. Alternatively,
you can have Dask fetch the first few row (5 by default) and use
them to guess the typical bytes/row, and base the partitioning size on
this. Needless to say, the results will vary a lot for tables that are
not uncommonly homogeneous.

### Specific partitioning

In some cases, you may have a very good idea of how to partition the data,
for example based on a column that has a finite number of unique values
or categories. This enables using string columns, or anything with a
natural ordering, for the index column, not only numerical types.

In this case, you would provide a specific set of `divisions`,
the start/end values of the index column for each partition. For example,
if a column happened to contain a random ID in hex string format, then you
could specify 16 partitions with

```python
df = read_sql_table("mytable", divisions=list("0123456789abcdefh"),
                    index_col="hexID")
```

so the first partition would have IDs with values `"0" <= hexID < "1"`, i.e.,
leading character “0”.

### SQLAlchemy expressions

Since we only send the database connection URI and not the engine object,
we cannot rely on SQLAlchemy’s table class inference and ORM to conduct queries. However, we can
use the “select” [sql expressions](https://docs.sqlalchemy.org/en/13/core/tutorial.html), which only get formatted into a text query at
the point of execution.

```python
from sqlalchemy import sql
number = sql.column("number")
name = sql.column("name")
s1 = sql.select([
        number, name, sql.func.length(name).label("lenname")
    ]
    ).select_from(sql.table("test"))
data = read_sql_query(
    s1, db, npartitions=2, index_col=number
)
```

Here we have also demonstrated the use of the function `length` to
perform an operation server-side. Note that it is necessary to *label* such
operations, but you can use them for the index column,
so long as it is also
in the set of selected columns. If using for the index/partitioning, the
column should still be indexed in the database, for performance.
One of the most important functions to consider is `cast` to specify the
output data type or conversion in the database, if pandas is having
trouble inferring the data type.

You should be warned, that SQLAlchemy expressions take some time to get
used to, and you can practice with Pandas first, reading only the first small
chunk of a query, until things look right. You can find a more complete
object-oriented example in [this gist](https://gist.github.com/quasiben/08a7f291039db2b04c2e28e1a6c21e3b)

## Load from SQL, manual approaches

If `read_sql_table` is not sufficient for your needs, you can try one of
the following methods.

### From Map functions

Often you know more about your data and server than the generic approach above
allows. Indeed, some database-like servers may simply not be supported by
SQLAlchemy, or provide an alternate API which is better optimised.

If you already have a way to fetch data from the database in partitions,
then you can use [`dask.dataframe.from_map()`](generated/dask.dataframe.from_map.md#dask.dataframe.from_map) and construct a
dataframe this way. It might look something like.

```python
import dask.dataframe as dd

def fetch_partition(part):
    conn = establish_connection()
    df = fetch_query(base_query.format(part))
    return df.astype(known_types)

ddf = dd.from_map(fetch_partition,
                  parts,
                  meta=known_types,
                  divisions=div_from_parts(parts))
```

Where you must provide your own functions for setting up a connection to the server,
your own query, and a way to format that query to be specific to each partition.
For example, you might have ranges or specific unique values with a WHERE
clause. The `known_types` here is used to transform the dataframe partition and provide
a `meta`, to help for consistency and avoid Dask having to analyse one partition
up front to guess the columns/types; you may also want to explicitly set the index.

### Stream via client

In some cases, the workers may not have access to data, but the client does;
or the initial loading time of the data is not important, so long as the
dataset is then held in cluster memory and available for dask-dataframe
queries. It is possible to construct the dataframe by uploading chunks of
data from the client:

See a complete example of how to do this [here](https://stackoverflow.com/questions/62818473/why-dasks-read-sql-table-requires-a-index-col-parameter/62821858#62821858)

### Access data files directly

Some database systems such as Apache Hive store their data in a location
and format that may be directly accessible to Dask, such as parquet files
on S3 or HDFS. In cases where your SQL query would read whole datasets and pass
them to Dask, the streaming of data from the database is very likely the
bottleneck, and it’s probably faster to read the source data files directly.

## Query pushdown?

If you define a query based on a database table, then only use some columns
of the output, you may expect that Dask is able to tell the database server
to only send some of the table’s data. Dask is not currently able to
do this “pushdown” optimisation, and you would need to change your query using
the SQL expression syntax.
We may be able to resolve this in the future ([dask#6388](https://github.com/dask/dask/issues/6388)).

If the divisions on your dataframe are well defined, then selections on the
index may successfully avoid reading irrelevant partitions.


# dataframe.html.md

# Dask DataFrame

Dask DataFrame helps you process large tabular data by parallelizing pandas,
either on your laptop for larger-than-memory computing, or on a distributed
cluster of computers.

- **Just pandas:** Dask DataFrames are a collection of many pandas DataFrames.

  The API is the same.  The execution is the same.
- **Large scale:** Works on 100 GiB on a laptop, or 100 TiB on a cluster.
- **Easy to use:** Pure Python, easy to set up and debug.

![Column of four squares collectively labeled as a Dask DataFrame with a single constituent square labeled as a pandas DataFrame.](images/dask-dataframe.svg)

Dask DataFrames coordinate many pandas DataFrames/Series arranged along the
index.  A Dask DataFrame is partitioned *row-wise*, grouping rows by index value
for efficiency.  These pandas objects may live on disk or on other machines.

## From pandas to Dask

Dask DataFrame copies pandas, and so should be familiar to most users

### Load Data

Pandas and Dask have the same API, and so switching from one to the other
is straightforward.

```python
>>> import pandas as pd

>>> df = pd.read_parquet('s3://mybucket/myfile.parquet')
>>> df.head()
0  1  a
1  2  b
2  3  c
```

```python
>>> import dask.dataframe as dd

>>> df = dd.read_parquet('s3://mybucket/myfile.*.parquet')
>>> df.head()
0  1  a
1  2  b
2  3  c
```

### Data Processing

Dask does pandas in parallel.
Dask is lazy; when you want an in-memory result add `.compute()`.

```python
>>> import pandas as pd

>>> df = df[df.value >= 0]
>>> joined = df.merge(other, on="account")
>>> result = joined.groupby("account").value.mean()

>>> result
alice 123
bob   456
```

```python
>>> import dask.dataframe as dd

>>> df = df[df.value >= 0]
>>> joined = df.merge(other, on="account")
>>> result = joined.groupby("account").value.mean()

>>> result.compute()
alice 123
bob   456
```

### Machine Learning

Machine learning libraries often have Dask submodules that
expect Dask DataFrames and operate in parallel.

```python
>>> import pandas as pd
>>> import xgboost
>>> from sklearn.cross_validation import train_test_split

>>> X_train, X_test, y_train, y_test = train_test_split(
...    X, y, test_size=0.2,
)
>>> dtrain = xgboost.DMatrix(X_train, label=y_train)

>>> xgboost.train(params, dtrain, 100)
<xgboost.Booster ...>
```

```python
>>> import dask.dataframe as dd
>>> import xgboost.dask
>>> from dask_ml.model_selection import train_test_split

>>> X_train, X_test, y_train, y_test = train_test_split(
...    X, y, test_size=0.2,
)
>>> dtrain = xgboost.dask.DaskDMatrix(client, X, y)

>>> xgboost.dask.train(params, dtrain, 100)
<xgboost.Booster ...>
```

As with all Dask collections, you trigger computation by calling the
`.compute()` method or persist data in distributed memory with the
`.persist()` method.

## When not to use Dask DataFrames

Dask DataFrames are often used either when …

1. Your data is too big
2. Your computation is too slow and other techniques don’t work

You should probably stick to just using pandas if …

1. Your data is small
2. Your computation is fast (subsecond)
3. There are simpler ways to accelerate your computation, like avoiding
   `.apply` or Python for loops and using a built-in pandas method instead.

## Examples

Dask DataFrame is used across a wide variety of applications — anywhere where working
with large tabular dataset. Here are a few large-scale examples:

- [Parquet ETL with Dask DataFrame](https://docs.coiled.io/user_guide/uber-lyft.html?utm_source=dask-docs&utm_medium=dataframe)
- [XGBoost model training with Dask DataFrame](https://docs.coiled.io/user_guide/xgboost.html?utm_source=dask-docs&utm_medium=dataframe)
- [Visualize 1,000,000,000 points](https://docs.coiled.io/user_guide/datashader.html?utm_source=dask-docs&utm_medium=dataframe)

These examples all process larger-than-memory datasets on Dask clusters deployed with
[Coiled](https://coiled.io/?utm_source=dask-docs&utm_medium=dataframe),
but there are many options for managing and deploying Dask.
See our [Deploy Dask Clusters](deploying.md) documentation for more information on deployment options.

You can also visit [https://examples.dask.org/dataframe.html](https://examples.dask.org/dataframe.html) for a collection of additional examples.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/AT2XtFehFSQ"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>


# debug.html.md

# Debug

Debugging parallel programs is hard.  Normal debugging tools like logging and
using `pdb` to interact with tracebacks stop working normally when exceptions
occur in far-away machines, different processes, or threads.

Dask has a variety of mechanisms to make this process easier.  Depending on
your situation, some of these approaches may be more appropriate than others.

These approaches are ordered from lightweight or easy solutions to more
involved solutions.

## Printing

One of the most basic methods of debugging is to simply print values and inspect them.
However, when using Python’s built-in [`print()`](https://docs.python.org/3/library/functions.html#print) function with Dask, those prints
often happen on remote machines instead of in the user’s Python session, which typically
isn’t the experience developers want when debugging.

Because of this, Dask offers a [`dask.distributed.print`](../futures.md#distributed.print) function which acts
just like Python’s built-in [`print()`](https://docs.python.org/3/library/functions.html#print) but also forwards the printed output to the
client side Python session. This makes distributed debugging feel more like debugging
locally.

## Exceptions

When a task in your computation fails, the standard way of understanding what
went wrong is to look at the exception and traceback.  Often people do this
with the `pdb` module, IPython `%debug` or `%pdb` magics, or by just
looking at the traceback and investigating where in their code the exception
occurred.

Normally when a computation executes in a separate thread or a different
machine, these approaches break down.  To address this, Dask provides a few
mechanisms to recreate the normal Python debugging experience.

### Inspect Exceptions and Tracebacks

By default, Dask already copies the exception and traceback wherever they
occur and reraises that exception locally.  If your task failed with a
`ZeroDivisionError` remotely, then you’ll get a `ZeroDivisionError` in your
interactive session.  Similarly you’ll see a full traceback of where this error
occurred, which, just like in normal Python, can help you to identify the
troublesome spot in your code.

However, you cannot use the `pdb` module or `%debug` IPython magics with
these tracebacks to look at the value of variables during failure.  You can
only inspect things visually.  Additionally, the top of the traceback may be
filled with functions that are Dask-specific and not relevant to your
problem, so you can safely ignore these.

Both the single-machine and distributed schedulers do this.

### Use the Single-Threaded Scheduler

Dask ships with a simple single-threaded scheduler.  This doesn’t offer any
parallel performance improvements but does run your Dask computation
faithfully in your local thread, allowing you to use normal tools like `pdb`,
`%debug` IPython magics, the profiling tools like the `cProfile` module, and
[snakeviz](https://jiffyclub.github.io/snakeviz/).  This allows you to use
all of your normal Python debugging tricks in Dask computations, as long as you
don’t need parallelism.

The single-threaded scheduler can be used, for example, by setting
`scheduler='single-threaded'` in a compute call:

```python
>>> x.compute(scheduler='single-threaded')
```

For more ways to configure schedulers, see the [scheduler configuration
documentation](../scheduling.md#scheduling-configuration).

This only works for single-machine schedulers.  It does not work with
`dask.distributed` unless you are comfortable using the Tornado API (look at the
[testing infrastructure](https://distributed.dask.org/en/latest/develop.html#writing-tests)
docs, which accomplish this).  Also, because this operates on a single machine,
it assumes that your computation can run on a single machine without exceeding
memory limits.  It may be wise to use this approach on smaller versions of your
problem if possible.

### Rerun Failed Task Locally

If a remote task fails, we can collect the function and all inputs, bring them
to the local thread, and then rerun the function in hopes of triggering the
same exception locally where normal debugging tools can be used.

With the single-machine schedulers, use the `rerun_exceptions_locally=True`
keyword:

```python
>>> x.compute(rerun_exceptions_locally=True)
```

On the distributed scheduler use the `recreate_error_locally` method on
anything that contains `Futures`:

```python
>>> x.compute()
ZeroDivisionError(...)

>>> %pdb
>>> future = client.compute(x)
>>> client.recreate_error_locally(future)
```

### Remove Failed Futures Manually

Sometimes only parts of your computations fail, for example, if some rows of a
CSV dataset are faulty in some way.  When running with the distributed
scheduler, you can remove chunks of your data that have produced bad results if
you switch to dealing with Futures:

```python
>>> import dask.dataframe as dd
>>> df = ...           # create dataframe
>>> df = df.persist()  # start computing on the cluster

>>> from distributed.client import futures_of
>>> futures = futures_of(df)  # get futures behind dataframe
>>> futures
[<Future: status: finished, type: pd.DataFrame, key: load-1>
 <Future: status: finished, type: pd.DataFrame, key: load-2>
 <Future: status: error, key: load-3>
 <Future: status: pending, key: load-4>
 <Future: status: error, key: load-5>]

>>> # wait until computation is done
>>> while any(f.status == 'pending' for f in futures):
...     sleep(0.1)

>>> # pick out only the successful futures and reconstruct the dataframe
>>> good_futures = [f for f in futures if f.status == 'finished']
>>> df = dd.from_delayed(good_futures, meta=df._meta)
```

This is a bit of a hack, but often practical when first exploring messy data.
If you are using the concurrent.futures API (map, submit, gather), then this
approach is more natural.

## Inspect Scheduling State

Not all errors present themselves as exceptions.  For example, in a distributed
system workers may die unexpectedly, your computation may be unreasonably
slow due to inter-worker communication or scheduler overhead, or one of several
other issues.  Getting feedback about what’s going on can help to identify
both failures and general performance bottlenecks.

For the single-machine scheduler, see [local diagnostic](../diagnostics-local.md) documentation.  The rest of the section will
assume that you are using the [distributed scheduler](https://distributed.dask.org/en/latest/) where these issues arise more
commonly.

### Web Diagnostics

First, the distributed scheduler has a number of [diagnostic tools](https://distributed.dask.org/en/latest/diagnosing-performance.html) showing dozens of
recorded metrics like CPU, memory, network, and disk use, a history of previous
tasks, allocation of tasks to workers, worker memory pressure, work stealing,
open file handle limits, etc.  *Many* problems can be correctly diagnosed by
inspecting these pages.  By default, these are available at
`http://scheduler:8787/` where `scheduler` should be replaced by the address of the
scheduler. See [diagnosing performance docs](https://distributed.dask.org/en/latest/diagnosing-performance.html) for more information.

### Logs

The scheduler, workers, and client all emits logs using [Python’s standard
logging module](https://docs.python.org/3/library/logging.html).  By default,
these emit to standard error.  When Dask is launched by a cluster job scheduler
(SGE/SLURM/YARN/Mesos/Marathon/Kubernetes/whatever), that system will track
these logs and will have an interface to help you access them.  If you are
launching Dask on your own, they will probably dump to the screen unless you
[redirect stderr to a file](https://en.wikipedia.org/wiki/Redirection_(computing)#Redirecting_to_and_from_the_standard_file_handles)
.

You can control the logging verbosity in the [Configuration](../configuration.md), for example,
the `~/.config/dask/*.yaml` files.
Defaults currently look like the following:

```yaml
logging:
  distributed: info
  distributed.client: warning
  bokeh: error
```

Logging for specific components like `distributed.client`,  `distributed.scheduler`,
`distributed.nanny`,  `distributed.worker`, etc. can each be independently configured.
So, for example, you could add a line like `distributed.worker: debug` to get
*very* verbose output from the workers.

Furthermore, you can explicitly assign handlers to loggers. The following example
assigns both file (“output.log”) and console output to the scheduler and workers.
See the [python logging](https://docs.python.org/3/library/logging.html) documentation for information on the meaning of
specific terms here.

```yaml
logging:
  version: 1
  handlers:
    file:
      class: logging.handlers.RotatingFileHandler
      filename: output.log
      level: INFO
    console:
      class: logging.StreamHandler
      level: INFO
  loggers:
    distributed.worker:
      level: INFO
      handlers:
        - file
        - console
    distributed.scheduler:
      level: INFO
      handlers:
        - file
        - console
```

## LocalCluster

If you are using the distributed scheduler from a single machine, you may be
setting up workers manually using the command line interface or you may be
using [LocalCluster](https://distributed.dask.org/en/latest/api.html#cluster)
which is what runs when you just call `Client()`:

```python
>>> from dask.distributed import Client, LocalCluster
>>> client = Client()  # This is actually the following two commands

>>> cluster = LocalCluster()
>>> client = Client(cluster.scheduler.address)
```

LocalCluster is useful because the scheduler and workers are in the same
process with you, so you can easily inspect their [state](https://distributed.dask.org/en/latest/scheduling-state.html) while
they run (they are running in a separate thread):

```python
>>> cluster.scheduler.processing
{'worker-one:59858': {'inc-123', 'add-443'},
 'worker-two:48248': {'inc-456'}}
```

You can also do this for the workers *if* you run them without nanny processes:

```python
>>> cluster = LocalCluster(nanny=False)
>>> client = Client(cluster)
```

This can be very helpful if you want to use the Dask distributed API and still
want to investigate what is going on directly within the workers.  Information
is not distilled for you like it is in the web diagnostics, but you have full
low-level access.


# debugging-performance.html.md

# Debugging and Performance

This section contains resources to help you debug and understand performance.

* [Debug](how-to/debug.md)
* [Visualize task graphs](graphviz.md)
* [Dashboard](dashboard.md)
* [Diagnostics (local)](diagnostics-local.md)
* [Diagnostics (distributed)](diagnostics-distributed.md)
* [Phases of computation](phases-of-computation.md)


# delayed-api.html.md

# API

The `dask.delayed` interface consists of one function, `delayed`:

- `delayed` wraps functions
  > Wraps functions. Can be used as a decorator, or around function calls
  > directly (i.e. `delayed(foo)(a, b, c)`). Outputs from functions wrapped in
  > `delayed` are proxy objects of type `Delayed` that contain a graph of
  > all operations done to get to this result.
- `delayed` wraps objects
  > Wraps objects. Used to create `Delayed` proxies directly.

`Delayed` objects can be thought of as representing a key in the dask task
graph. A `Delayed` supports *most* python operations, each of which creates
another `Delayed` representing the result:

- Most operators (`*`, `-`, and so on)
- Item access and slicing (`a[0]`)
- Attribute access (`a.size`)
- Method calls (`a.index(0)`)

Operations that aren’t supported include:

- Mutating operators (`a += 1`)
- Mutating magic methods such as `__setitem__`/`__setattr__` (`a[0] = 1`, `a.foo = 1`)
- Iteration. (`for i in a: ...`)
- Use as a predicate (`if a: ...`)

The last two points in particular mean that `Delayed` objects cannot be used for
control flow, meaning that no `Delayed` can appear in a loop or if statement.
In other words you can’t iterate over a `Delayed` object, or use it as part of
a condition in an if statement, but `Delayed` object can be used in a body of a loop
or if statement (i.e. the example above is fine, but if `data` was a `Delayed`
object it wouldn’t be).
Even with this limitation, many workflows can easily be parallelized.

| [`delayed`](#dask.delayed.delayed)([obj, name, pure, nout, traverse])   | Wraps a function or object to produce a `Delayed`.   |
|-------------------------------------------------------------------------|------------------------------------------------------|
| [`Delayed`](#dask.delayed.Delayed)(key, dsk[, length, layer])           | Represents a value to be computed by dask.           |

### dask.delayed.delayed(obj='_\_no_\_default_\_', name=None, pure=None, nout=None, traverse=True)

Wraps a function or object to produce a `Delayed`.

`Delayed` objects act as proxies for the object they wrap, but all
operations on them are done lazily by building up a dask graph internally.

* **Parameters:**
  **obj**
  : The function or object to wrap

  **name**
  : The key to use in the underlying graph for the wrapped object. Defaults
    to hashing content. Note that this only affects the name of the object
    wrapped by this call to delayed, and *not* the output of delayed
    function calls - for that use `dask_key_name=` as described below.
    <br/>
    #### NOTE
    Because this `name` is used as the key in task graphs, you should
    ensure that it uniquely identifies `obj`. If you’d like to provide
    a descriptive name that is still unique, combine the descriptive name
    with `dask.base.tokenize()` of the `array_like`. See
    [Task Graphs](graphs.md#graphs) for more.

  **pure**
  : Indicates whether calling the resulting `Delayed` object is a pure
    operation. If True, arguments to the call are hashed to produce
    deterministic keys. If not provided, the default is to check the global
    `delayed_pure` setting, and fallback to `False` if unset.

  **nout**
  : The number of outputs returned from calling the resulting `Delayed`
    object. If provided, the `Delayed` output of the call can be iterated
    into `nout` objects, allowing for unpacking of results. By default
    iteration over `Delayed` objects will error. Note, that `nout=1`
    expects `obj` to return a tuple of length 1, and consequently for
    `nout=0`, `obj` should return an empty tuple.

  **traverse**
  : By default dask traverses builtin python collections looking for dask
    objects passed to `delayed`. For large collections this can be
    expensive. If `obj` doesn’t contain any dask objects, set
    `traverse=False` to avoid doing this traversal.

### Examples

Apply to functions to delay execution:

```pycon
>>> from dask import delayed
>>> def inc(x):
...     return x + 1
```

```pycon
>>> inc(10)
11
```

```pycon
>>> x = delayed(inc, pure=True)(10)
>>> type(x) == Delayed
True
>>> x.compute()
11
```

Can be used as a decorator:

```pycon
>>> @delayed(pure=True)
... def add(a, b):
...     return a + b
>>> add(1, 2).compute()
3
```

`delayed` also accepts an optional keyword `pure`. If False, then
subsequent calls will always produce a different `Delayed`. This is
useful for non-pure functions (such as `time` or `random`).

```pycon
>>> from random import random
>>> out1 = delayed(random, pure=False)()
>>> out2 = delayed(random, pure=False)()
>>> out1.key == out2.key
False
```

If you know a function is pure (output only depends on the input, with no
global state), then you can set `pure=True`. This will attempt to apply a
consistent name to the output, but will fallback on the same behavior of
`pure=False` if this fails.

```pycon
>>> @delayed(pure=True)
... def add(a, b):
...     return a + b
>>> out1 = add(1, 2)
>>> out2 = add(1, 2)
>>> out1.key == out2.key
True
```

Instead of setting `pure` as a property of the callable, you can also set
it contextually using the `delayed_pure` setting. Note that this
influences the *call* and not the *creation* of the callable:

```pycon
>>> @delayed
... def mul(a, b):
...     return a * b
>>> import dask
>>> with dask.config.set(delayed_pure=True):
...     print(mul(1, 2).key == mul(1, 2).key)
True
>>> with dask.config.set(delayed_pure=False):
...     print(mul(1, 2).key == mul(1, 2).key)
False
```

The key name of the result of calling a delayed object is determined by
hashing the arguments by default. To explicitly set the name, you can use
the `dask_key_name` keyword when calling the function:

```pycon
>>> add(1, 2)
Delayed('add-3dce7c56edd1ac2614add714086e950f')
>>> add(1, 2, dask_key_name='three')
Delayed('three')
```

Note that objects with the same key name are assumed to have the same
result. If you set the names explicitly you should make sure your key names
are different for different results.

```pycon
>>> add(1, 2, dask_key_name='three')
Delayed('three')
>>> add(2, 1, dask_key_name='three')
Delayed('three')
>>> add(2, 2, dask_key_name='four')
Delayed('four')
```

`delayed` can also be applied to objects to make operations on them lazy:

```pycon
>>> a = delayed([1, 2, 3])
>>> isinstance(a, Delayed)
True
>>> a.compute()
[1, 2, 3]
```

The key name of a delayed object is hashed by default if `pure=True` or
is generated randomly if `pure=False` (default).  To explicitly set the
name, you can use the `name` keyword. To ensure that the key is unique
you should include the tokenized value as well, or otherwise ensure that
it’s unique:

```pycon
>>> from dask.base import tokenize
>>> data = [1, 2, 3]
>>> a = delayed(data, name='mylist-' + tokenize(data))
>>> a
Delayed('mylist-55af65871cb378a4fa6de1660c3e8fb7')
```

Delayed results act as a proxy to the underlying object. Many operators
are supported:

```pycon
>>> (a + [1, 2]).compute()
[1, 2, 3, 1, 2]
>>> a[1].compute()
2
```

Method and attribute access also works:

```pycon
>>> a.count(2).compute()
1
```

Note that if a method doesn’t exist, no error will be thrown until runtime:

```pycon
>>> res = a.not_a_real_method()
>>> res.compute()
AttributeError("'list' object has no attribute 'not_a_real_method'")
```

“Magic” methods (e.g. operators and attribute access) are assumed to be
pure, meaning that subsequent calls must return the same results. This
behavior is not overridable through the `delayed` call, but can be
modified using other ways as described below.

To invoke an impure attribute or operator, you’d need to use it in a
delayed function with `pure=False`:

```pycon
>>> class Incrementer:
...     def __init__(self):
...         self._n = 0
...     @property
...     def n(self):
...         self._n += 1
...         return self._n
...
>>> x = delayed(Incrementer())
>>> x.n.key == x.n.key
True
>>> get_n = delayed(lambda x: x.n, pure=False)
>>> get_n(x).key == get_n(x).key
False
```

In contrast, methods are assumed to be impure by default, meaning that
subsequent calls may return different results. To assume purity, set
`pure=True`. This allows sharing of any intermediate values.

```pycon
>>> a.count(2, pure=True).key == a.count(2, pure=True).key
True
```

As with function calls, method calls also respect the global
`delayed_pure` setting and support the `dask_key_name` keyword:

```pycon
>>> a.count(2, dask_key_name="count_2")
Delayed('count_2')
>>> import dask
>>> with dask.config.set(delayed_pure=True):
...     print(a.count(2).key == a.count(2).key)
True
```

<!-- !! processed by numpydoc !! -->

### *class* dask.delayed.Delayed(key, dsk, length=None, layer=None)

Represents a value to be computed by dask.

Equivalent to the output from a single key in a dask graph.

<!-- !! processed by numpydoc !! -->


# delayed-best-practices.html.md

# Best Practices

It is easy to get started with Dask delayed, but using it *well* does require
some experience.  This page contains suggestions for best practices, and
includes solutions to common problems.

## Call delayed on the function, not the result

Dask delayed operates on functions like `dask.delayed(f)(x, y)`, not on their results like `dask.delayed(f(x, y))`.  When you do the latter, Python first calculates `f(x, y)` before Dask has a chance to step in.

| **Don’t**                                                                        | **Do**                                                                                              |
|----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| ```python<br/># This executes immediately<br/><br/>dask.delayed(f(x, y))<br/>``` | ```python<br/># This makes a delayed function, acting lazily<br/><br/>dask.delayed(f)(x, y)<br/>``` |

## Compute on lots of computation at once

To improve parallelism, you want to include lots of computation in each compute call.
Ideally, you want to make many `dask.delayed` calls to define your computation and
then call `dask.compute` only at the end.  It is ok to call `dask.compute`
in the middle of your computation as well, but everything will stop there as
Dask computes those results before moving forward with your code.

| **Don’t**                                                                                                                                                                          | **Do**                                                                                                                                                                                              |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Avoid calling compute repeatedly<br/><br/>results = []<br/>for x in L:<br/>    y = dask.delayed(f)(x)<br/>    results.append(y.compute())<br/><br/>results<br/>``` | ```python<br/># Collect many calls for one compute<br/><br/>results = []<br/>for x in L:<br/>    y = dask.delayed(f)(x)<br/>    results.append(y)<br/><br/>results = dask.compute(*results)<br/>``` |

Calling y.compute() within the loop would await the result of the computation every time, and
so inhibit parallelism.

## Don’t mutate inputs

Your functions should not change the inputs directly.

| **Don’t**                                                                                                               | **Do**                                                                                                                      |
|-------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Mutate inputs in functions<br/><br/>@dask.delayed<br/>def f(x):<br/>    x += 1<br/>    return x<br/>``` | ```python<br/># Return new values or copies<br/><br/>@dask.delayed<br/>def f(x):<br/>    x = x + 1<br/>    return x<br/>``` |

If you need to use a mutable operation, then make a copy within your function first:

```python
@dask.delayed
def f(x):
    x = copy(x)
    x += 1
    return x
```

## Avoid global state

Ideally, your operations shouldn’t rely on global state.  Using global state
*might* work if you only use threads, but when you move to multiprocessing or
distributed computing then you will likely encounter confusing errors.

| **Don’t**                                                                                                                          |
|------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/>L = []<br/><br/># This references global variable L<br/><br/>@dask.delayed<br/>def f(x):<br/>    L.append(x)<br/>``` |

## Don’t rely on side effects

Delayed functions only do something if they are computed.  You will always need
to pass the output to something that eventually calls compute.

| **Don’t**                                                                                     | **Do**                                                                                                                           |
|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Forget to call compute<br/><br/>dask.delayed(f)(1, 2, 3)<br/><br/>...<br/>``` | ```python<br/># Ensure delayed tasks are computed<br/><br/>x = dask.delayed(f)(1, 2, 3)<br/>...<br/>dask.compute(x, ...)<br/>``` |

In the first case here, nothing happens, because `compute()` is never called.

## Break up computations into many pieces

Every `dask.delayed` function call is a single operation from Dask’s perspective.
You achieve parallelism by having many delayed calls, not by using only a
single one: Dask will not look inside a function decorated with `@dask.delayed`
and parallelize that code internally.  To accomplish that, it needs your help to
find good places to break up a computation.

| **Don’t**                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | **Do**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># One giant task<br/><br/><br/>def load(filename):<br/>    ...<br/><br/><br/>def process(data):<br/>    ...<br/><br/><br/>def save(data):<br/>    ...<br/><br/>@dask.delayed<br/>def f(filenames):<br/>    results = []<br/>    for filename in filenames:<br/>        data = load(filename)<br/>        data = process(data)<br/>        result = save(data)<br/>        results.append(result)<br/><br/>    return results<br/><br/>dask.compute(f(filenames))<br/>``` | ```python<br/># Break up into many tasks<br/><br/>@dask.delayed<br/>def load(filename):<br/>    ...<br/><br/>@dask.delayed<br/>def process(data):<br/>    ...<br/><br/>@dask.delayed<br/>def save(data):<br/>    ...<br/><br/><br/>def f(filenames):<br/>    results = []<br/>    for filename in filenames:<br/>        data = load(filename)<br/>        data = process(data)<br/>        result = save(data)<br/>        results.append(result)<br/><br/>    return results<br/><br/>dask.compute(f(filenames))<br/>``` |

The first version only has one delayed task, and so cannot parallelize.

## Avoid too many tasks

Every delayed task has an overhead of a few hundred microseconds.  Usually this
is ok, but it can become a problem if you apply `dask.delayed` too finely.  In
this case, it’s often best to break up your many tasks into batches or use one
of the Dask collections to help you.

| **Don’t**                                                                                                                                           | **Do**                                                                                                                                                      |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Too many tasks<br/><br/>results = []<br/>for x in range(10000000):<br/>    y = dask.delayed(f)(x)<br/>    results.append(y)<br/>``` | ```python<br/># Use collections<br/><br/>import dask.bag as db<br/>b = db.from_sequence(range(10000000), npartitions=1000)<br/>b = b.map(f)<br/>...<br/>``` |

Here we use `dask.bag` to automatically batch applying our function. We could also have constructed
our own batching as follows

```python
def batch(seq):
    sub_results = []
    for x in seq:
        sub_results.append(f(x))
    return sub_results

 batches = []
 for i in range(0, 10000000, 10000):
     result_batch = dask.delayed(batch)(range(i, i + 10000))
     batches.append(result_batch)
```

Here we construct batches where each delayed function call computes for many data points from
the original input.

## Avoid calling delayed within delayed functions

Often, if you are new to using Dask delayed, you place `dask.delayed` calls
everywhere and hope for the best.  While this may actually work, it’s usually
slow and results in hard-to-understand solutions.

Usually you never call `dask.delayed` within `dask.delayed` functions.

| **Don’t**                                                                                                                                                                                                                           | **Do**                                                                                                                                                                                                                |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Delayed function calls delayed<br/><br/>@dask.delayed<br/>def process_all(L):<br/>    result = []<br/>    for x in L:<br/>        y = dask.delayed(f)(x)<br/>        result.append(y)<br/>    return result<br/>``` | ```python<br/># Normal function calls delayed<br/><br/><br/>def process_all(L):<br/>    result = []<br/>    for x in L:<br/>        y = dask.delayed(f)(x)<br/>        result.append(y)<br/>    return result<br/>``` |

Because the normal function only does delayed work it is very fast and so
there is no reason to delay it.

## Don’t call dask.delayed on other Dask collections

When you place a Dask array or Dask DataFrame into a delayed call, that function
will receive the NumPy or Pandas equivalent.  Beware that if your array is
large, then this might crash your workers.

Instead, it’s more common to use methods like `da.map_blocks`

| **Don’t**                                                                                                                                                                       | **Do**                                                                                                                                                                  |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/># Call delayed functions on Dask collections<br/><br/>import dask.dataframe as dd<br/>df = dd.read_csv('/path/to/*.csv')<br/><br/>dask.delayed(train)(df)<br/>``` | ```python<br/># Use mapping methods if applicable<br/><br/>import dask.dataframe as dd<br/>df = dd.read_csv('/path/to/*.csv')<br/><br/>df.map_partitions(train)<br/>``` |

Alternatively, if the procedure doesn’t fit into a mapping, you can always
turn your arrays or dataframes into *many* delayed
objects, for example

```python
partitions = df.to_delayed()
delayed_values = [dask.delayed(train)(part)
                  for part in partitions]
```

However, if you don’t mind turning your Dask array/DataFrame into a single
chunk, then this is ok.

```python
dask.delayed(train)(..., y=df.sum())
```

## Avoid repeatedly putting large inputs into delayed calls

Every time you pass a concrete result (anything that isn’t delayed) Dask will
hash it by default to give it a name.  This is fairly fast (around 500 MB/s)
but can be slow if you do it over and over again.  Instead, it is better to
delay your data as well.

This is especially important when using a distributed cluster to avoid sending
your data separately for each function call.

| **Don’t**                                                                                                                                      | **Do**                                                                                                                                                                                     |
|------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ```python<br/>x = np.array(...)  # some large array<br/><br/>results = [dask.delayed(train)(x, i)<br/>           for i in range(1000)]<br/>``` | ```python<br/>x = np.array(...)    # some large array<br/>x = dask.delayed(x)  # delay the data once<br/>results = [dask.delayed(train)(x, i)<br/>           for i in range(1000)]<br/>``` |

Every call to `dask.delayed(train)(x, ...)` has to hash the NumPy array `x`, which slows things down.


# delayed-collections.html.md

# Working with Collections

Often we want to do a bit of custom work with `dask.delayed` (for example,
for complex data ingest), then leverage the algorithms in `dask.array` or
`dask.dataframe`, and then switch back to custom work.  To this end, all
collections support `from_delayed` functions and `to_delayed`
methods.

As an example, consider the case where we store tabular data in a custom format
not known by Dask DataFrame.  This format is naturally broken apart into
pieces and we have a function that reads one piece into a Pandas DataFrame.
We use `dask.delayed` to lazily read these files into Pandas DataFrames,
use `dd.from_delayed` to wrap these pieces up into a single
Dask DataFrame, use the complex algorithms within the DataFrame
(groupby, join, etc.), and then switch back to `dask.delayed` to save our results
back to the custom format:

```python
import dask.dataframe as dd
from dask.delayed import delayed

from my_custom_library import load, save

filenames = ...
dfs = [delayed(load)(fn) for fn in filenames]

df = dd.from_delayed(dfs)
df = ... # do work with dask.dataframe

dfs = df.to_delayed()
writes = [delayed(save)(df, fn) for df, fn in zip(dfs, filenames)]

dd.compute(*writes)
```

Data science is often complex, and `dask.delayed` provides a release valve for
users to manage this complexity on their own, and solve the last mile problem
for custom formats and complex situations.


# delayed.html.md

# Dask Delayed

Sometimes problems don’t fit into one of the collections like `dask.array` or
`dask.dataframe`. In these cases, users can parallelize custom algorithms
using the simpler `dask.delayed` interface. This allows you to create graphs
directly with a light annotation of normal python code:

```python
>>> x = dask.delayed(inc)(1)
>>> y = dask.delayed(inc)(2)
>>> z = dask.delayed(add)(x, y)
>>> z.compute()
5
>>> z.visualize()
```

![A Dask Delayed task graph with two "inc" functions combined using an "add" function resulting in an output node.](images/inc-add.svg)<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/SHqFmynRxVU"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

## Example

Visit [https://examples.dask.org/delayed.html](https://examples.dask.org/delayed.html) to see and run examples using Dask
Delayed.

Sometimes we face problems that are parallelizable, but don’t fit into high-level
abstractions like Dask Array or Dask DataFrame.  Consider the following example:

```python
def inc(x):
    return x + 1

def double(x):
    return x * 2

def add(x, y):
    return x + y

data = [1, 2, 3, 4, 5]

output = []
for x in data:
    a = inc(x)
    b = double(x)
    c = add(a, b)
    output.append(c)

total = sum(output)
```

There is clearly parallelism in this problem (many of the `inc`,
`double`, and `add` functions can be evaluated independently), but it’s not
clear how to convert this to an array or DataFrame computation. As written,
this code runs sequentially in a single thread.  However, we see that a
lot of this could be executed in parallel.

The Dask `delayed` function decorates your functions so that they operate
*lazily*.  Rather than executing your function immediately, it will defer
execution, placing the function and its arguments into a task graph.

| [`delayed`](delayed-api.md#dask.delayed.delayed)([obj, name, pure, nout, traverse])   | Wraps a function or object to produce a `Delayed`.   |
|---------------------------------------------------------------------------------------|------------------------------------------------------|

We slightly modify our code by wrapping functions in `delayed`.
This delays the execution of the function and generates a Dask graph instead:

```python
import dask

output = []
for x in data:
    a = dask.delayed(inc)(x)
    b = dask.delayed(double)(x)
    c = dask.delayed(add)(a, b)
    output.append(c)

total = dask.delayed(sum)(output)
```

We used the `dask.delayed` function to wrap the function calls that we want
to turn into tasks.  None of the `inc`, `double`, `add`, or `sum` calls
have happened yet. Instead, the object `total` is a `Delayed` result that
contains a task graph of the entire computation.  Looking at the graph we see
clear opportunities for parallel execution.  The [Dask schedulers](scheduling.md)
will exploit this parallelism, generally improving performance (although not in this
example, because these functions are already very small and fast.)

```python
total.visualize()  # see image to the right
```

![A task graph with many nodes for "inc" and "double" that combine with "add" nodes. The output of the "add" nodes finally aggregate with a "sum" node.](images/delayed-inc-double-add.svg)

We can now compute this lazy result to execute the graph in parallel:

```python
>>> total.compute()
45
```

## Decorator

It is also common to see the delayed function used as a decorator.  Here is a
reproduction of our original problem as a parallel code:

```python
import dask

@dask.delayed
def inc(x):
    return x + 1

@dask.delayed
def double(x):
    return x * 2

@dask.delayed
def add(x, y):
    return x + y

data = [1, 2, 3, 4, 5]

output = []
for x in data:
    a = inc(x)
    b = double(x)
    c = add(a, b)
    output.append(c)

total = dask.delayed(sum)(output)
```

## Real time

Sometimes you want to create and destroy work during execution, launch tasks
from other tasks, etc.  For this, see the [Futures](futures.md) interface.

## Best Practices

For a list of common problems and recommendations see
[Delayed Best Practices](delayed-best-practices.md).

## Indirect Dependencies

Sometimes you might find yourself wanting to add a dependency to a task that does
not take the result of that dependency as an input. For example when a task depends
on the side-effect of another task. In these cases you can use
`dask.graph_manipulation.bind`.

```python
import dask
from dask.graph_manipulation import bind

DATA = []

@dask.delayed
def inc(x):
    return x + 1

@dask.delayed
def add_data(x):
    DATA.append(x)

@dask.delayed
def sum_data(x):
    return sum(DATA) + x

a = inc(1)
b = add_data(a)
c = inc(3)
d = add_data(c)
e = inc(5)
f = bind(sum_data, [b, d])(e)
f.compute()
```

`sum_data` will operate on DATA only after both the expected items have
been appended to it. `bind` can also be used along with direct dependencies
passed through the function arguments.

## Execution

By default, Dask Delayed uses the threaded scheduler in order to avoid data
transfer costs.
You should consider using [multi-processing](https://docs.dask.org/en/stable/scheduling.html#local-processes) scheduler or [dask.distributed](https://distributed.dask.org/en/latest/) scheduler
on a local machine or on a cluster if your code does not [release the GIL well](https://docs.dask.org/en/stable/graphs.html#avoid-holding-the-gil)
(computations that are dominated by pure Python code, or computations wrapping external
code and holding onto it).


# deploying-cli.html.md

<a id="deploying-cli"></a>

# Command Line

This is the most fundamental way to deploy Dask on multiple machines.  In
production environments this process is often automated by some other resource
manager. Hence, it is rare that people need to follow these instructions
explicitly.  Instead, these instructions are useful to help understand what
*cluster managers* and other automated tooling is doing under the hood and to
help users deploy onto platforms that have no automated tools today.

A `dask.distributed` network consists of one `dask scheduler` process and
several `dask worker` processes that connect to that scheduler.  These are
normal Python processes that can be executed from the command line.  We launch
the `dask scheduler` executable in one process and the `dask worker`
executable in several processes, possibly on different machines.

To accomplish this, launch `dask scheduler` on one node:

```default
$ dask scheduler
Scheduler at:   tcp://192.0.0.100:8786
```

Then, launch `dask worker` on the rest of the nodes, providing the address to
the node that hosts `dask scheduler`:

```default
$ dask worker tcp://192.0.0.100:8786
Start worker at:  tcp://192.0.0.1:12345
Registered to:    tcp://192.0.0.100:8786

$ dask worker tcp://192.0.0.100:8786
Start worker at:  tcp://192.0.0.2:40483
Registered to:    tcp://192.0.0.100:8786

$ dask worker tcp://192.0.0.100:8786
Start worker at:  tcp://192.0.0.3:27372
Registered to:    tcp://192.0.0.100:8786
```

The workers connect to the scheduler, which then sets up a long-running network
connection back to the worker.  The workers will learn the location of other
workers from the scheduler.

## Handling Ports

The scheduler and workers both need to accept TCP connections on an open port.
By default, the scheduler binds to port `8786` and the worker binds to a
random open port.  If you are behind a firewall then you may have to open
particular ports or tell Dask to listen on particular ports with the `--port`
and `--worker-port` keywords.:

```default
dask scheduler --port 8000
dask worker --dashboard-address 8000 --nanny-port 8001
```

## Nanny Processes

Dask workers are run within a nanny process that monitors the worker process
and restarts it if necessary.

## Diagnostic Web Servers

Additionally, Dask schedulers and workers host interactive diagnostic web
servers using [Bokeh](https://docs.bokeh.org).  These are optional, but
generally useful to users.  The diagnostic server on the scheduler is
particularly valuable, and is served on port `8787` by default (configurable
with the `--dashboard-address` keyword).

For more information about relevant ports, please take a look at the available
[command line options](#worker-scheduler-cli-options).

## Automated Tools

There are various mechanisms to deploy these executables on a cluster, ranging
from manually SSH-ing into all of the machines to more automated systems like
SGE/SLURM/Torque or Yarn/Mesos.  Additionally, cluster SSH tools exist to send
the same commands to many machines.  We recommend searching online for “cluster
ssh” or “cssh”.

<a id="worker-scheduler-cli-options"></a>

## CLI Options

#### NOTE
The command line documentation here may differ depending on your installed
version. We recommend referring to the output of `dask scheduler --help`
and `dask worker --help`.

### dask scheduler

Launch a Dask scheduler.

### Usage

```shell
dask scheduler [OPTIONS] [PRELOAD_ARGV]...
```

### Options

### --host <host>

URI, IP or hostname of this server

### --port <port>

Serving port

### --interface <interface>

Preferred network interface like ‘eth0’ or ‘ib0’

### --protocol <protocol>

Protocol like tcp, tls, or ucx

### --tls-ca-file <tls_ca_file>

CA cert(s) file for TLS (in PEM format)

### --tls-cert <tls_cert>

certificate file for TLS (in PEM format)

### --tls-key <tls_key>

private key file for TLS (in PEM format)

### --dashboard-address <dashboard_address>

Address on which to listen for diagnostics dashboard

* **Default:**
  `':8787'`

### --dashboard, --no-dashboard

Launch the Dashboard [default: –dashboard]

### --jupyter, --no-jupyter

Start a Jupyter Server in the same process.  Warning: This will makeit possible for anyone with access to your dashboard address to runPython code

### --show, --no-show

Show web UI [default: –show] (DEPRECATED)

### --dashboard-prefix <dashboard_prefix>

Prefix for the dashboard app

### --use-xheaders <use_xheaders>

Use xheaders in dashboard app for ssl termination in header (DEPRECATED)

* **Default:**
  `False`

### --pid-file <pid_file>

File to write the process PID

### --scheduler-file <scheduler_file>

File to write connection information. This may be a good way to share connection information if your cluster is on a shared network file system.

### --preload <preload>

Module that should be loaded by the scheduler process  like “foo.bar” or “/path/to/foo.py”.

### --idle-timeout <idle_timeout>

Time of inactivity after which to kill the scheduler

### --version

Show the version and exit.

### Arguments

### PRELOAD_ARGV

Optional argument(s)

### dask worker

Launch a Dask worker attached to an existing scheduler

### Usage

```shell
dask worker [OPTIONS] [SCHEDULER] [PRELOAD_ARGV]...
```

### Options

### --tls-ca-file <tls_ca_file>

CA cert(s) file for TLS (in PEM format)

### --tls-cert <tls_cert>

certificate file for TLS (in PEM format)

### --tls-key <tls_key>

private key file for TLS (in PEM format)

### --worker-port <worker_port>

Serving computation port, defaults to random. When creating multiple workers with –nworkers, a sequential range of worker ports may be used by specifying the first and last available ports like <first-port>:<last-port>. For example, –worker-port=3000:3026 will use ports 3000, 3001, …, 3025, 3026.

### --nanny-port <nanny_port>

Serving nanny port, defaults to random. When creating multiple nannies with –nworkers, a sequential range of nanny ports may be used by specifying the first and last available ports like <first-port>:<last-port>. For example, –nanny-port=3000:3026 will use ports 3000, 3001, …, 3025, 3026.

### --dashboard-address <dashboard_address>

Address on which to listen for diagnostics dashboard

### --dashboard, --no-dashboard

Launch the Dashboard [default: –dashboard]

### --listen-address <listen_address>

The address to which the worker binds. Example: [tcp://0.0.0.0:9000](tcp://0.0.0.0:9000) or [tcp://:9000](tcp://:9000) for IPv4+IPv6

### --contact-address <contact_address>

The address the worker advertises to the scheduler for communication with it and other workers. Example: [tcp://127.0.0.1:9000](tcp://127.0.0.1:9000)

### --host <host>

Serving host. Should be an ip address that is visible to the scheduler and other workers. See –listen-address and –contact-address if you need different listen and contact addresses. See –interface.

### --interface <interface>

Network interface like ‘eth0’ or ‘ib0’

### --protocol <protocol>

Protocol like tcp, tls, or ucx

### --nthreads <nthreads>

Number of threads per process.

### --nworkers <n_workers>

Number of worker processes to launch. If negative, then (CPU_COUNT + 1 + nworkers) is used. Set to ‘auto’ to set nworkers and nthreads dynamically based on CPU_COUNT

### --name <name>

A unique name for this worker like ‘worker-1’. If used with –nworkers then the process number will be appended like name-0, name-1, name-2, …

### --memory-limit <memory_limit>

Bytes of memory per process that the worker can use.
<br/>
This can be:
<br/>
- an integer (bytes), note 0 is a special case for no memory management.
<br/>
- a float (fraction of total system memory).
<br/>
- a string (like 5GB or 5000M).
<br/>
- ‘auto’ for automatically computing the memory limit.
<br/>
* **Default:**
  `'auto'`

### --nanny, --no-nanny

Start workers in nanny process for management [default: –nanny]

### --pid-file <pid_file>

File to write the process PID

### --local-directory <local_directory>

Directory to place worker files

### --resources <resources>

Resources for task constraints like “GPU=2 MEM=10e9”. Resources are applied separately to each worker process (only relevant when starting multiple worker processes with ‘–nworkers’).

### --scheduler-file <scheduler_file>

Filename to JSON encoded scheduler information. Use with dask scheduler –scheduler-file

### --death-timeout <death_timeout>

Seconds to wait for a scheduler before closing

### --dashboard-prefix <dashboard_prefix>

Prefix for the dashboard (DEPRECATED)

### --lifetime <lifetime>

If provided, shut down the worker after this duration.

### --lifetime-stagger <lifetime_stagger>

Random amount by which to stagger lifetime values

### --worker-class <worker_class>

Worker class used to instantiate workers from.

* **Default:**
  `'dask.distributed.Worker'`

### --lifetime-restart, --no-lifetime-restart

Whether or not to restart the worker after the lifetime lapses. This assumes that you are using the –lifetime and –nanny keywords

### --preload <preload>

Module that should be loaded by each worker process like “foo.bar” or “/path/to/foo.py”

### --preload-nanny <preload_nanny>

Module that should be loaded by each nanny like “foo.bar” or “/path/to/foo.py”

### --scheduler-sni <scheduler_sni>

Scheduler SNI (if different from scheduler hostname)

### --version

Show the version and exit.

### Arguments

### SCHEDULER

Optional argument

### PRELOAD_ARGV

Optional argument(s)


# deploying-cloud.html.md

# Cloud

There are a variety of ways to deploy Dask on the cloud.
Cloud providers offer managed services,
like VMs, Kubernetes, Yarn, or custom APIs with which Dask can connect easily.

Some common deployment options you may want to consider are:

- A commercial Dask deployment option like [Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=deploying-cloud)
  to handle the creation and management of Dask clusters on AWS, GCP, and Azure.
- A managed Kubernetes service and Dask’s
  [Kubernetes integration](deploying-kubernetes.md).
- Directly launching cloud resources such as VMs or containers via a cluster manager with
  [Dask Cloud Provider](https://cloudprovider.dask.org/en/latest/).
- A managed Yarn service,
  like [Amazon EMR](https://aws.amazon.com/emr/)
  or [Google Cloud DataProc](https://cloud.google.com/dataproc/)
  and [Dask-Yarn](https://yarn.dask.org)
  (specific documentation for the popular Amazon EMR service can be found
  [here](https://yarn.dask.org/en/latest/aws-emr.html).)

![image](images/cloud-provider-logos.svg)

## Cloud Deployment Examples

### Coiled

[Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=deploying-cloud)
deploys managed Dask clusters on AWS, GCP, and Azure. It’s free for most users and
has several features that address common
[deployment pain points](deployment-considerations.md) like:

- Easy to use API
- Automatic software synchronization
- Easy access to any cloud hardware (like GPUs) in any region
- Robust logging, cost controls, and metrics collection

```python
>>> import coiled
>>> cluster = coiled.Cluster(
...     n_workers=100,             # Size of cluster
...     region="us-west-2",        # Same region as data
...     vm_type="m6i.xlarge",      # Hardware of your choosing
... )
>>> client = cluster.get_client()
```

Coiled is recommended for deploying Dask on the cloud.
Though there are non-commercial, open source options like
Dask Cloud Provider, Dask-Gateway, and Dask-Yarn that are also available
(see [cloud deployment options](deploying.md#cloud-deployment-options)
for additional options.)

### Dask Cloud Provider

Using [Dask Cloud Provider](https://cloudprovider.dask.org/en/latest/) to launch a cluster of
VMs on a platform like [DigitalOcean](https://www.digitalocean.com/) can be as convenient as
launching a local cluster.

```python
>>> import dask.config
>>> dask.config.set({"cloudprovider.digitalocean.token": "yourAPItoken"})
>>> from dask_cloudprovider.digitalocean import DropletCluster
>>> cluster = DropletCluster(n_workers=1)
Creating scheduler instance
Created droplet dask-38b817c1-scheduler
Waiting for scheduler to run
Scheduler is running
Creating worker instance
Created droplet dask-38b817c1-worker-dc95260d
```

Many of the cluster managers in Dask Cloud Provider work by launching VMs with a startup script
that pulls down the [Dask Docker image](deploying-docker.md) and runs Dask components within that container.
As with all cluster managers the VM resources, Docker image, etc are all configurable.

You can then connect a client and work with the cluster as if it were on your local machine.

```python
>>> client = cluster.get_client()
```

## Data Access

In addition to deploying Dask clusters on the cloud, most cloud users will also want
to access cloud-hosted data on their respective cloud provider.

We recommend installing additional libraries (listed below) for easy data access on your cloud provider.
See [Connect to remote data](how-to/connect-to-remote-data.md) for more information.

### AWS

Use [s3fs](https://s3fs.readthedocs.io/) for accessing data on Amazon’s S3.

```bash
python -m pip install s3fs
```

### GCP

Use [gcsfs](https://gcsfs.readthedocs.io/) for accessing data on Google’s GCS.

```bash
python -m pip install gcsfs
```

### Azure

Use [adlfs](https://github.com/dask/adlfs/) for accessing data on Microsoft’s Data Lake or Blob Storage.

```bash
python -m pip install adlfs
```


# deploying-docker.html.md

# Docker Images

Example docker images are maintained at [https://github.com/dask/dask-docker](https://github.com/dask/dask-docker) .

Each image installs the full Dask conda environment (including the distributed
scheduler), Numpy, and Pandas on top of a Miniconda installation on top of
a Debian image.

These images are large, around 1GB.

- `ghcr.io/dask/dask`: This is a normal debian + miniconda image with the full Dask
  conda package (including the distributed scheduler), Numpy, and Pandas.
  This image is about 1GB in size.
- `ghcr.io/dask/dask-notebook`: This is based on the
  [Jupyter base-notebook image](https://hub.docker.com/r/jupyter/base-notebook/)
  and so it is suitable for use both normally as a Jupyter server, and also as
  part of a JupyterHub deployment.  It also includes a matching Dask software
  environment described above.  This image is about 2GB in size.

## Example

Here is a simple example on a dedicated virtual network

```bash
docker network create dask

docker run --network dask -p 8787:8787 --name scheduler ghcr.io/dask/dask dask-scheduler  # start scheduler

docker run --network dask ghcr.io/dask/dask dask-worker scheduler:8786 # start worker
docker run --network dask ghcr.io/dask/dask dask-worker scheduler:8786 # start worker
docker run --network dask ghcr.io/dask/dask dask-worker scheduler:8786 # start worker

docker run --network dask -p 8888:8888 ghcr.io/dask/dask-notebook  # start Jupyter server
```

Then from within the notebook environment you can connect to the Dask cluster like this:

```python
from dask.distributed import Client
client = Client("scheduler:8786")
client
```

## Extensibility

Users can mildly customize the software environment by populating the
environment variables `EXTRA_APT_PACKAGES`, `EXTRA_CONDA_PACKAGES`, and
`EXTRA_PIP_PACKAGES`.  If these environment variables are set in the container,
they will trigger calls to the following respectively:

```default
apt-get install $EXTRA_APT_PACKAGES
conda install $EXTRA_CONDA_PACKAGES
python -m pip install $EXTRA_PIP_PACKAGES
```

For example, the following `conda` installs the `joblib` package into
the Dask worker software environment:

```bash
docker run --network dask -e EXTRA_CONDA_PACKAGES="joblib" ghcr.io/dask/dask dask-worker scheduler:8786
```

Note that using these can significantly delay the container from starting,
especially when using `apt`, or `conda` (`pip` is relatively fast).

Remember that it is important for software versions to match between Dask
workers and Dask clients.  As a result, it is often useful to include the same
extra packages in both Jupyter and Worker images.

## Source

Docker files are maintained at [https://github.com/dask/dask-docker](https://github.com/dask/dask-docker).
This repository also includes a docker-compose configuration.


# deploying-extra.html.md

# Additional Information

* [Adaptive deployments](adaptive.md)
* [Docker Images](deploying-docker.md)
* [Python API (advanced)](deploying-python-advanced.md)
* [Manage Environments](software-environments.md)
* [Prometheus](prometheus.md)
* [Customize Initialization](customize-initialization.md)
* [Deployment Considerations](deployment-considerations.md)


# deploying-hpc.html.md

# High Performance Computers

## Relevant Machines

This page includes instructions and guidelines when deploying Dask on high
performance supercomputers commonly found in scientific and industry research
labs.  These systems commonly have the following attributes:

1. Some mechanism to launch MPI applications or use job schedulers like
   SLURM, SGE, TORQUE, LSF, DRMAA, PBS, or others
2. A shared network file system visible to all machines in the cluster
3. A high performance network interconnect, such as Infiniband
4. Little or no node-local storage

## Where to start

Most of this page documents various ways and best practices to use Dask on an
HPC cluster.  This is technical and aimed both at users with some experience
deploying Dask and also system administrators.

The preferred and simplest way to run Dask on HPC systems today both for new,
experienced users or administrator is to use
[dask-jobqueue](https://jobqueue.dask.org).

However, dask-jobqueue is slightly oriented toward interactive analysis usage,
and it might be better to use tools like dask-mpi in some routine batch
production workloads.

## Dask-jobqueue and Dask-drmaa

[dask-jobqueue](https://jobqueue.dask.org) provides cluster managers for PBS,
SLURM, LSF, SGE and other resource managers. You can launch a Dask cluster on
these systems like this.

```python
from dask_jobqueue import PBSCluster

cluster = PBSCluster(cores=36,
                     memory="100GB",
                     project='P48500028',
                     queue='premium',
                     interface='ib0',
                     walltime='02:00:00')

cluster.scale(100)  # Start 100 workers in 100 jobs that match the description above

from dask.distributed import Client
client = Client(cluster)    # Connect to that cluster
```

Dask-jobqueue provides a lot of possibilities like adaptive dynamic scaling
of workers, we recommend reading the [dask-jobqueue documentation](https://jobqueue.dask.org) first to get a basic system running and then
returning to this documentation for fine-tuning if necessary.

## Using MPI

You can launch a Dask cluster using `mpirun` or `mpiexec` and the
[dask-mpi](http://mpi.dask.org/en/latest/) command line tool.

```bash
mpirun --np 4 dask-mpi --scheduler-file /home/$USER/scheduler.json
```

```python
from dask.distributed import Client
client = Client(scheduler_file='/path/to/scheduler.json')
```

This depends on the [mpi4py](https://mpi4py.readthedocs.io/) library.  It only
uses MPI to start the Dask cluster and not for inter-node communication. MPI
implementations differ: the use of `mpirun --np 4` is specific to the
`mpich` or `open-mpi` MPI implementation installed through conda and linked
to mpi4py.

```bash
conda install mpi4py
```

It is not necessary to use exactly this implementation, but you may want to
verify that your `mpi4py` Python library is linked against the proper
`mpirun/mpiexec` executable and that the flags used (like `--np 4`) are
correct for your system.  The system administrator of your cluster should be
very familiar with these concerns and able to help.

In some setups, MPI processes are not allowed to fork other processes. In this
case, we recommend using `--no-nanny` option in order to prevent dask from
using an additional nanny process to manage workers.

Run `dask-mpi --help` to see more options for the `dask-mpi` command.

## Using a Shared Network File System and a Job Scheduler

#### NOTE
This section is not necessary if you use a tool like dask-jobqueue.

Some clusters benefit from a shared File System (NFS, GPFS, Lustre or alike),
and can use this to communicate the scheduler location to the workers:

```default
dask-scheduler --scheduler-file /path/to/scheduler.json  # writes address to file

dask-worker --scheduler-file /path/to/scheduler.json  # reads file for address
dask-worker --scheduler-file /path/to/scheduler.json  # reads file for address
```

```python
>>> client = Client(scheduler_file='/path/to/scheduler.json')
```

This can be particularly useful when deploying `dask-scheduler` and
`dask-worker` processes using a job scheduler like
SGE/SLURM/Torque/etc.  Here is an example using SGE’s `qsub` command:

```default
# Start a dask-scheduler somewhere and write the connection information to a file
qsub -b y /path/to/dask-scheduler --scheduler-file /home/$USER/scheduler.json

# Start 100 dask-worker processes in an array job pointing to the same file
qsub -b y -t 1-100 /path/to/dask-worker --scheduler-file /home/$USER/scheduler.json
```

Note, the `--scheduler-file` option is *only* valuable if your scheduler and
workers share a network file system.

## High Performance Network

Many HPC systems have both standard Ethernet networks as well as
high-performance networks capable of increased bandwidth.  You can instruct
Dask to use the high-performance network interface by using the `--interface`
keyword with the `dask-worker`, `dask-scheduler`, or `dask-mpi` commands or
the `interface=` keyword with the dask-jobqueue `Cluster` objects:

```bash
mpirun --np 4 dask-mpi --scheduler-file /home/$USER/scheduler.json --interface ib0
```

In the code example above, we have assumed that your cluster has an Infiniband
network interface called `ib0`. You can check this by asking your system
administrator or by inspecting the output of `ifconfig`

```bash
$ ifconfig
lo          Link encap:Local Loopback                       # Localhost
                        inet addr:127.0.0.1  Mask:255.0.0.0
                        inet6 addr: ::1/128 Scope:Host
eth0        Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX   # Ethernet
                        inet addr:192.168.0.101
                        ...
ib0         Link encap:Infiniband                           # Fast InfiniBand
                        inet addr:172.42.0.101
```

[https://stackoverflow.com/questions/43881157/how-do-i-use-an-infiniband-network-with-dask](https://stackoverflow.com/questions/43881157/how-do-i-use-an-infiniband-network-with-dask)

## Local Storage

Users often exceed memory limits available to a specific Dask deployment.  In
normal operation, Dask spills excess data to disk, often to the default
temporary directory.

However, in HPC systems this default temporary directory may point to an
network file system (NFS) mount which can cause problems as Dask tries to read
and write many small files.  *Beware, reading and writing many tiny files from
many distributed processes is a good way to shut down a national
supercomputer*.

If available, it’s good practice to point Dask workers to local storage, or
hard drives that are physically on each node.  Your IT administrators will be
able to point you to these locations.  You can do this with the
`--local-directory` or `local_directory=` keyword in the `dask-worker`
command:

```default
dask-mpi ... --local-directory /path/to/local/storage
```

or any of the other Dask Setup utilities, or by specifying the
following [configuration value](configuration.md):

```yaml
temporary-directory: /path/to/local/storage
```

However, not all HPC systems have local storage.  If this is the case then you
may want to turn off Dask’s ability to spill to disk altogether.
See this page for more information on Dask’s memory policies.
Consider changing the following values in your `~/.config/dask/distributed.yaml` file
to disable spilling data to disk:

```yaml
distributed:
  worker:
    memory:
      target: false  # don't spill to disk
      spill: false  # don't spill to disk
      pause: 0.80  # pause execution at 80% memory use
      terminate: 0.95  # restart the worker at 95% use
```

This stops Dask workers from spilling to disk, and instead relies entirely on
mechanisms to stop them from processing when they reach memory limits.

As a reminder, you can set the memory limit for a worker using the
`--memory-limit` keyword:

```default
dask-mpi ... --memory-limit 10GB
```

## Launch Many Small Jobs

#### NOTE
This section is not necessary if you use a tool like dask-jobqueue.

HPC job schedulers are optimized for large monolithic jobs with many nodes that
all need to run as a group at the same time.  Dask jobs can be quite a bit more
flexible: workers can come and go without strongly affecting the job.  If we
split our job into many smaller jobs, we can often get through the job
scheduling queue much more quickly than a typical job.  This is particularly
valuable when we want to get started right away and interact with a Jupyter
notebook session rather than waiting for hours for a suitable allocation block
to become free.

So, to get a large cluster quickly, we recommend allocating a dask-scheduler
process on one node with a modest wall time (the intended time of your session)
and then allocating many small single-node dask-worker jobs with shorter wall
times (perhaps 30 minutes) that can easily squeeze into extra space in the job
scheduler.  As you need more computation, you can add more of these single-node
jobs or let them expire.

## Use Dask to co-launch a Jupyter server

Dask can help you by launching other services alongside it.  For example, you
can run a Jupyter notebook server on the machine running the `dask-scheduler`
process with the following commands

```python
from dask.distributed import Client
client = Client(scheduler_file='scheduler.json')

import socket
host = client.run_on_scheduler(socket.gethostname)

def start_jlab(dask_scheduler):
    import subprocess
    proc = subprocess.Popen(['/path/to/jupyter', 'lab', '--ip', host, '--no-browser'])
    dask_scheduler.jlab_proc = proc

client.run_on_scheduler(start_jlab)
```


# deploying-kubernetes.html.md

# Kubernetes

[Kubernetes](https://kubernetes.io/) is a popular system for deploying distributed applications on clusters,
particularly in the cloud.  You can use Kubernetes to launch Dask clusters in the
following ways:

## Dask Kubernetes Operator

The Dask Kubernetes Operator is a set of Custom Resource Definitions (CRDs) and a controller that
allows you to create and manage your Dask clusters as native Kubernetes resources.

Creating clusters can either be done via the Kubernetes API with `kubectl` or the
Python API with `KubeCluster`.

```bash
helm install --repo https://helm.dask.org --create-namespace -n dask-operator --generate-name dask-kubernetes-operator
```

```bash
# Create a cluster with kubectl
kubectl apply -f - <<EOF
apiVersion: kubernetes.dask.org/v1
kind: DaskCluster
metadata:
  name: my-dask-cluster
spec:
  ...
EOF
```

```python
# Create a cluster in Python
from dask_kubernetes.operator import KubeCluster
cluster = KubeCluster(name="my-dask-cluster", image='ghcr.io/dask/dask:latest')
cluster.scale(10)
```

This is a good choice if you want to do the following:

1. Have a Kubernetes native experience.
2. Manage Dask clusters via the Kubernetes API and tools like `kubectl`.
3. Integrate Dask with other tools and workloads running on Kubernetes.
4. Compose Dask clusters as part of a larger Kubernetes application.

Learn more at [kubernetes.dask.org](https://kubernetes.dask.org/en/latest/operator.html).

## Dask Gateway

Dask Gateway provides a secure, multi-tenant server for managing Dask clusters.
It allows users to launch and use Dask clusters in a shared, centrally managed cluster environment,
without requiring users to have direct access to the underlying cluster backend
(e.g. Kubernetes, Hadoop/YARN, HPC Job queues, etc…).

```bash
helm install --repo https://helm.dask.org --create-namespace -n dask-gateway --generate-name dask-gateway
```

```python
from dask_gateway import Gateway
gateway = Gateway("<gateway service address>")
cluster = gateway.new_cluster()
```

This is a good choice if you want to do the following:

1. Abstract users away from Kubernetes.
2. Provide a consistent Dask user experience across Kubernetes/Hadoop/HPC.

Learn more at [gateway.dask.org](https://gateway.dask.org/install-kube.html).

### DaskHub

You can also deploy Dask Gateway alongside [JupyterHub](https://jupyter.org/hub) using the DaskHub helm chart.

```bash
helm install --repo https://helm.dask.org --create-namespace -n daskhub --generate-name daskhub
```

Learn more at the [artifacthub.io DaskHub page](https://artifacthub.io/packages/helm/dask/daskhub).

## Single Cluster Helm Chart

You can deploy a single Dask cluster and (optionally) Jupyter on Kubernetes
easily using [Helm](https://helm.sh/)

```bash
helm install --repo https://helm.dask.org my-dask dask
```

This is a good choice if you want to do the following:

1. Try out Dask for the first time on a cloud-based system
   like Amazon, Google, or Microsoft Azure where you already have
   a Kubernetes cluster. If you don’t already have Kubernetes deployed,
   see our [Cloud documentation](deploying-cloud.md).

You can also use the `HelmCluster` cluster manager from dask-kubernetes to manage your
Helm Dask cluster from within your Python session.

```python
from dask_kubernetes import HelmCluster

cluster = HelmCluster(release_name="myrelease")
cluster.scale(10)
```

Learn more at the [artifacthub.io Dask page](https://artifacthub.io/packages/helm/dask/dask).

## Further Reading

You may also want to see the documentation on using
[Dask with Docker containers](deploying-docker.md)
to help you manage your software environments on Kubernetes.


# deploying-python-advanced.html.md

# Python API (advanced)

In some rare cases, experts may want to create `Scheduler`, `Worker`, and
`Nanny`  objects explicitly in Python.  This is often necessary when making
tools to automatically deploy Dask in custom settings.

It is more common to create a [Local cluster with Client() on a single
machine](deploying-python.md) or use the [Command Line Interface (CLI)](deploying-cli.md).
New readers are recommended to start there.

If you do want to start Scheduler and Worker objects yourself you should be a
little familiar with `async`/`await` style Python syntax.  These objects
are awaitable and are commonly used within `async with` context managers.
Here are a few examples to show a few ways to start and finish things.

## Full Example

| [`Scheduler`](#distributed.Scheduler)(\*[, services, service_kwargs, ...])   | Dynamic distributed task scheduler                  |
|------------------------------------------------------------------------------|-----------------------------------------------------|
| [`Worker`](#distributed.Worker)(scheduler_ip, scheduler_port, \*, ...)       | Worker node in a Dask distributed cluster           |
| [`Client`](futures.md#distributed.Client)([address, loop, timeout, ...])     | Connect to and submit computation to a Dask cluster |

We first start with a comprehensive example of setting up a Scheduler, two Workers,
and one Client in the same event loop, running a simple computation, and then
cleaning everything up.

```python
import asyncio
from dask.distributed import Scheduler, Worker, Client

async def f():
    async with Scheduler() as s:
        async with Worker(s.address) as w1, Worker(s.address) as w2:
            async with Client(s.address, asynchronous=True) as client:
                future = client.submit(lambda x: x + 1, 10)
                result = await future
                print(result)

asyncio.get_event_loop().run_until_complete(f())
```

Now we look at simpler examples that build up to this case.

## Scheduler

| [`Scheduler`](#distributed.Scheduler)(\*[, services, service_kwargs, ...])   | Dynamic distributed task scheduler   |
|------------------------------------------------------------------------------|--------------------------------------|

We create scheduler by creating a `Scheduler()` object, and then `await`
that object to wait for it to start up.  We can then wait on the `.finished`
method to wait until it closes.  In the meantime the scheduler will be active
managing the cluster..

```python
import asyncio
from dask.distributed import Scheduler, Worker

async def f():
    s = Scheduler()        # scheduler created, but not yet running
    s = await s            # the scheduler is running
    await s.finished()     # wait until the scheduler closes

asyncio.get_event_loop().run_until_complete(f())
```

This program will run forever, or until some external process connects to the
scheduler and tells it to stop.  If you want to close things yourself you can
close any `Scheduler`, `Worker`, `Nanny`, or `Client` class by awaiting
the `.close` method:

```python
await s.close()
```

## Worker

| [`Worker`](#distributed.Worker)(scheduler_ip, scheduler_port, \*, ...)   | Worker node in a Dask distributed cluster   |
|--------------------------------------------------------------------------|---------------------------------------------|

The worker follows the same API.
The only difference is that the worker needs to know the address of the
scheduler.

```python
import asyncio
from dask.distributed import Scheduler, Worker

async def f(scheduler_address):
    w = await Worker(scheduler_address)
    await w.finished()

asyncio.get_event_loop().run_until_complete(f("tcp://127.0.0.1:8786"))
```

## Start many in one event loop

| [`Scheduler`](#distributed.Scheduler)(\*[, services, service_kwargs, ...])   | Dynamic distributed task scheduler        |
|------------------------------------------------------------------------------|-------------------------------------------|
| [`Worker`](#distributed.Worker)(scheduler_ip, scheduler_port, \*, ...)       | Worker node in a Dask distributed cluster |

We can run as many of these objects as we like in the same event loop.

```python
import asyncio
from dask.distributed import Scheduler, Worker

async def f():
    s = await Scheduler()
    w = await Worker(s.address)
    await w.finished()
    await s.finished()

asyncio.get_event_loop().run_until_complete(f())
```

## Use Context Managers

We can also use `async with` context managers to make sure that we clean up
properly.  Here is the same example as from above:

```python
import asyncio
from dask.distributed import Scheduler, Worker

async def f():
    async with Scheduler() as s:
        async with Worker(s.address) as w:
            await w.finished()
            await s.finished()

asyncio.get_event_loop().run_until_complete(f())
```

Alternatively, in the example below we also include a `Client`, run a small
computation, and then allow things to clean up after that computation..

```python
import asyncio
from dask.distributed import Scheduler, Worker, Client

async def f():
    async with Scheduler() as s:
        async with Worker(s.address) as w1, Worker(s.address) as w2:
            async with Client(s.address, asynchronous=True) as client:
                future = client.submit(lambda x: x + 1, 10)
                result = await future
                print(result)

asyncio.get_event_loop().run_until_complete(f())
```

This is equivalent to creating and `awaiting` each server, and then calling
`.close` on each as we leave the context.
In this example we don’t wait on `s.finished()`, so this will terminate
relatively quickly.  You could have called `await s.finished()` though if you
wanted this to run forever.

<a id="nanny"></a>

## Nanny

| [`Nanny`](#distributed.Nanny)([scheduler_ip, scheduler_port, ...])   | A process to manage worker processes   |
|----------------------------------------------------------------------|----------------------------------------|

Alternatively, we can replace `Worker` with `Nanny` if we want your workers
to be managed in a separate process.  The `Nanny` constructor follows the
same API. This allows workers to restart themselves in case of failure. Also,
it provides some additional monitoring, and is useful when coordinating many
workers that should live in different processes in order to avoid the [GIL](https://docs.python.org/3/glossary.html#term-gil).

```python
# w = await Worker(s.address)
w = await Nanny(s.address)
```

## API

These classes have a variety of keyword arguments that you can use to control
their behavior.  See the API documentation below for more information.

### Scheduler

### *class* distributed.Scheduler(, services: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, service_kwargs: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, allowed_failures: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, extensions: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, validate: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, scheduler_file: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, security: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [Security](https://distributed.dask.org/en/latest/tls.html#distributed.security.Security) | [None](https://docs.python.org/3/library/constants.html#None) = None, worker_ttl: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, idle_timeout: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, interface: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, host: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, port: [int](https://docs.python.org/3/library/functions.html#int) = 0, protocol: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, dashboard_address: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, dashboard: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, http_prefix: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = '/', preload: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, preload_argv: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str)]] = (), plugins: [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[SchedulerPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.SchedulerPlugin)] = (), contact_address: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, transition_counter_max: [bool](https://docs.python.org/3/library/functions.html#bool) | [int](https://docs.python.org/3/library/functions.html#int) = False, jupyter: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any))

Dynamic distributed task scheduler

The scheduler tracks the current state of workers, data, and computations.
The scheduler listens for events and responds by controlling workers
appropriately.  It continuously tries to use the workers to execute an ever
growing dask graph.

All events are handled quickly, in linear time with respect to their input
(which is often of constant size) and generally within a millisecond.  To
accomplish this the scheduler tracks a lot of state.  Every operation
maintains the consistency of this state.

The scheduler communicates with the outside world through Comm objects.
It maintains a consistent and valid view of the world even when listening
to several clients at once.

A Scheduler is typically started either with the `dask scheduler`
executable:

```default
$ dask scheduler
Scheduler started at 127.0.0.1:8786
```

Or within a LocalCluster a Client starts up without connection
information:

```default
>>> c = Client()
>>> c.cluster.scheduler
Scheduler(...)
```

Users typically do not interact with the scheduler directly but rather with
the client object `Client`.

The `contact_address` parameter allows to advertise a specific address to
the workers for communication with the scheduler, which is different than
the address the scheduler binds to. This is useful when the scheduler
listens on a private address, which therefore cannot be used by the workers
to contact it.

**State**

The scheduler contains the following state variables.  Each variable is
listed along with what it stores and a brief description.

* **tasks:** `{task key: TaskState}`
  : Tasks currently known to the scheduler
* **unrunnable:** `{TaskState}`
  : Tasks in the “no-worker” state
* **workers:** `{worker key: WorkerState}`
  : Workers currently connected to the scheduler
* **idle:** `{WorkerState}`:
  : Set of workers that are not fully utilized
* **saturated:** `{WorkerState}`:
  : Set of workers that are not over-utilized
* **host_info:** `{hostname: dict}`:
  : Information about each worker host
* **clients:** `{client key: ClientState}`
  : Clients currently connected to the scheduler
* **services:** `{str: port}`:
  : Other services running on this scheduler, like Bokeh
* **loop:** `IOLoop`:
  : The running Tornado IOLoop
* **client_comms:** `{client key: Comm}`
  : For each client, a Comm object used to receive task requests and
    report task status updates.
* **stream_comms:** `{worker key: Comm}`
  : For each worker, a Comm object from which we both accept stimuli and
    report results
* **task_duration:** `{key-prefix: time}`
  : Time we expect certain functions to take, e.g. `{'sum': 0.25}`

<!-- !! processed by numpydoc !! -->

#### MEMORY_REBALANCE_HALF_GAP *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.worker.memory.rebalance.sender-recipient-gap / 2

<!-- !! processed by numpydoc !! -->

#### MEMORY_REBALANCE_MEASURE *: [str](https://docs.python.org/3/library/stdtypes.html#str)*

distributed.worker.memory.rebalance.measure

<!-- !! processed by numpydoc !! -->

#### MEMORY_REBALANCE_RECIPIENT_MAX *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.worker.memory.rebalance.recipient-max

<!-- !! processed by numpydoc !! -->

#### MEMORY_REBALANCE_SENDER_MIN *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.worker.memory.rebalance.sender-min

<!-- !! processed by numpydoc !! -->

#### MEMORY_RECENT_TO_OLD_TIME *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.worker.memory.recent-to-old-time

<!-- !! processed by numpydoc !! -->

#### UNKNOWN_TASK_DURATION *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.scheduler.unknown-task-duration

<!-- !! processed by numpydoc !! -->

#### WORKER_SATURATION *: [float](https://docs.python.org/3/library/functions.html#float)*

distributed.scheduler.worker-saturation

<!-- !! processed by numpydoc !! -->

#### adaptive_target(target_duration: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [int](https://docs.python.org/3/library/functions.html#int)

Desired number of workers based on the current workload

This looks at the current running tasks and memory use, and returns a
number of desired workers.  This is often used by adaptive scheduling.

* **Parameters:**
  **target_duration**
  : A desired duration of time for computations to take.  This affects
    how rapidly the scheduler will ask to scale.

#### SEE ALSO
[`distributed.deploy.Adaptive`](adaptive.md#distributed.deploy.Adaptive)

<!-- !! processed by numpydoc !! -->

#### *async* add_client(comm: [Comm](https://distributed.dask.org/en/latest/communications.html#distributed.comm.Comm), client: [str](https://docs.python.org/3/library/stdtypes.html#str), versions: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]) → [None](https://docs.python.org/3/library/constants.html#None)

Add client to network

We listen to all future messages from this Comm.

<!-- !! processed by numpydoc !! -->

#### add_keys(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), keys: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]] = (), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['OK', 'not found']

Learn that a worker has certain keys

This should not be used in practice and is mostly here for legacy
reasons.  However, it is sent by workers from time to time.

<!-- !! processed by numpydoc !! -->

#### add_plugin(plugin: [SchedulerPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.SchedulerPlugin), , idempotent: [bool](https://docs.python.org/3/library/functions.html#bool) = False, name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [None](https://docs.python.org/3/library/constants.html#None)

Add external plugin to scheduler.

See [https://distributed.readthedocs.io/en/latest/plugins.html](https://distributed.readthedocs.io/en/latest/plugins.html)

* **Parameters:**
  **plugin**
  : SchedulerPlugin instance to add

  **idempotent**
  : If true, the plugin is assumed to already exist and no
    action is taken.

  **name**
  : A name for the plugin, if None, the name attribute is
    checked on the Plugin instance and generated if not
    discovered.

<!-- !! processed by numpydoc !! -->

#### *async* add_worker(comm: [Comm](https://distributed.dask.org/en/latest/communications.html#distributed.comm.Comm), , address: [str](https://docs.python.org/3/library/stdtypes.html#str), status: [str](https://docs.python.org/3/library/stdtypes.html#str), server_id: [str](https://docs.python.org/3/library/stdtypes.html#str), nthreads: [int](https://docs.python.org/3/library/functions.html#int), name: [str](https://docs.python.org/3/library/stdtypes.html#str), resolve_address: [bool](https://docs.python.org/3/library/functions.html#bool) = True, now: [float](https://docs.python.org/3/library/functions.html#float), resources: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [float](https://docs.python.org/3/library/functions.html#float)], host_info: [None](https://docs.python.org/3/library/constants.html#None) = None, memory_limit: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None), metrics: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)], pid: [int](https://docs.python.org/3/library/functions.html#int) = 0, services: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [int](https://docs.python.org/3/library/functions.html#int)], local_directory: [str](https://docs.python.org/3/library/stdtypes.html#str), versions: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)], nanny: [str](https://docs.python.org/3/library/stdtypes.html#str), extra: [dict](https://docs.python.org/3/library/stdtypes.html#dict), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Add a new worker to the cluster

<!-- !! processed by numpydoc !! -->

#### aliases *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[Hashable, [str](https://docs.python.org/3/library/stdtypes.html#str)]*

Worker {name: address}

<!-- !! processed by numpydoc !! -->

#### *async* benchmark_hardware() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [float](https://docs.python.org/3/library/functions.html#float)]]

Run a benchmark on the workers for memory, disk, and network bandwidths

* **Returns:**
  result: dict
  : A dictionary mapping the names “disk”, “memory”, and “network” to
    dictionaries mapping sizes to bandwidths.  These bandwidths are
    averaged over many workers running computations across the cluster.

<!-- !! processed by numpydoc !! -->

#### *async* broadcast(, msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict), workers: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, hosts: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, nanny: [bool](https://docs.python.org/3/library/functions.html#bool) = False, serializers: [Any](https://docs.python.org/3/library/typing.html#typing.Any) = None, on_error: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['raise', 'return', 'return_pickle', 'ignore'] = 'raise') → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Broadcast message to workers, return all results

<!-- !! processed by numpydoc !! -->

#### client_heartbeat(client: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Handle heartbeats from Client

<!-- !! processed by numpydoc !! -->

#### client_releases_keys(keys: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], client: [str](https://docs.python.org/3/library/stdtypes.html#str), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Remove keys from client desired list

<!-- !! processed by numpydoc !! -->

#### client_send(client: [str](https://docs.python.org/3/library/stdtypes.html#str), msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict)) → [None](https://docs.python.org/3/library/constants.html#None)

Send message to client

<!-- !! processed by numpydoc !! -->

#### clients *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), ClientState]*

Clients currently connected to the scheduler

<!-- !! processed by numpydoc !! -->

#### *async* close(timeout: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'unknown') → [None](https://docs.python.org/3/library/constants.html#None)

Send cleanup signal to all coroutines then wait until finished.

<!-- !! processed by numpydoc !! -->

#### close_worker(worker: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Ask a worker to shut itself down. Do not wait for it to take effect.
Note that there is no guarantee that the worker will actually accept the
command.

Note that [`remove_worker()`](#distributed.Scheduler.remove_worker) sends the same command internally if close=True.

#### SEE ALSO
[`retire_workers`](#distributed.Scheduler.retire_workers)

[`remove_worker`](#distributed.Scheduler.remove_worker)

<!-- !! processed by numpydoc !! -->

#### coerce_address(addr: [str](https://docs.python.org/3/library/stdtypes.html#str) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple), resolve: [bool](https://docs.python.org/3/library/functions.html#bool) = True) → [str](https://docs.python.org/3/library/stdtypes.html#str)

Coerce possible input addresses to canonical form.
*resolve* can be disabled for testing with fake hostnames.

Handles strings, tuples, or aliases.

<!-- !! processed by numpydoc !! -->

#### computations *: deque[Computation]*

History of computations.
The length can be tweaked through
distributed.diagnostics.computations.max-history

<!-- !! processed by numpydoc !! -->

#### *async* delete_worker_data(worker_address: [str](https://docs.python.org/3/library/stdtypes.html#str), keys: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Delete data from a worker and update the corresponding worker/task states

* **Parameters:**
  **worker_address: str**
  : Worker address to delete keys from

  **keys: list[Key]**
  : List of keys to delete on the specified worker

<!-- !! processed by numpydoc !! -->

#### *async* dump_cluster_state_to_url(url: [str](https://docs.python.org/3/library/stdtypes.html#str), exclude: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)], format: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['msgpack', 'yaml'], \*\*storage_options: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]) → [None](https://docs.python.org/3/library/constants.html#None)

Write a cluster state dump to an fsspec-compatible URL.

<!-- !! processed by numpydoc !! -->

#### erred_tasks *: deque[ErredTask]*

History of erred tasks.
The length can be tweaked through
distributed.diagnostics.erred-tasks.max-history

<!-- !! processed by numpydoc !! -->

#### *async* feed(comm: [Comm](https://distributed.dask.org/en/latest/communications.html#distributed.comm.Comm), function: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [None](https://docs.python.org/3/library/constants.html#None) = None, setup: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [None](https://docs.python.org/3/library/constants.html#None) = None, teardown: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [None](https://docs.python.org/3/library/constants.html#None) = None, interval: [str](https://docs.python.org/3/library/stdtypes.html#str) | [float](https://docs.python.org/3/library/functions.html#float) = '1s', \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [None](https://docs.python.org/3/library/constants.html#None)

Provides a data Comm to external requester

Caution: this runs arbitrary Python code on the scheduler.  This should
eventually be phased out.  It is mostly used by diagnostics.

<!-- !! processed by numpydoc !! -->

#### *async* gather(keys: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], serializers: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [object](https://docs.python.org/3/library/functions.html#object)]

Collect data from workers to the scheduler

<!-- !! processed by numpydoc !! -->

#### *async* gather_on_worker(worker_address: [str](https://docs.python.org/3/library/stdtypes.html#str), who_has: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]]) → [set](https://docs.python.org/3/library/stdtypes.html#set)

Peer-to-peer copy of keys from multiple workers to a single worker

* **Parameters:**
  **worker_address: str**
  : Recipient worker address to copy keys to

  **who_has: dict[Key, list[str]]**
  : {key: [sender address, sender address, …], key: …}
* **Returns:**
  returns:
  : set of keys that failed to be copied

<!-- !! processed by numpydoc !! -->

#### *async* get_cluster_state(exclude: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)]) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Produce the state dict used in a cluster state dump

<!-- !! processed by numpydoc !! -->

#### *async* get_story(keys_or_stimuli: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]) → [list](https://docs.python.org/3/library/stdtypes.html#list)[Transition]

RPC hook for `SchedulerState.story()`.

Note that the msgpack serialization/deserialization round-trip will transform
the `Transition` namedtuples into regular tuples.

<!-- !! processed by numpydoc !! -->

#### get_task_stream_index() → [int](https://docs.python.org/3/library/functions.html#int)

Return the number of tasks recorded by the task stream so far.

Used as an opaque cursor by the `get_task_stream` context manager so
that it can collect exactly the tasks that ran during the block without
relying on (latency- and clock-sensitive) wall-clock boundaries.

<!-- !! processed by numpydoc !! -->

#### get_worker_service_addr(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), service_name: [str](https://docs.python.org/3/library/stdtypes.html#str), protocol: [bool](https://docs.python.org/3/library/functions.html#bool) = False) → [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str), [int](https://docs.python.org/3/library/functions.html#int)] | [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)

Get the (host, port) address of the named service on the *worker*.
Returns None if the service doesn’t exist.

* **Parameters:**
  **worker**

  **service_name**
  : Common services include ‘bokeh’ and ‘nanny’

  **protocol**
  : Whether or not to include a full address with protocol (True)
    or just a (host, port) pair

<!-- !! processed by numpydoc !! -->

#### handle_long_running(key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], worker: [str](https://docs.python.org/3/library/stdtypes.html#str), run_id: [int](https://docs.python.org/3/library/functions.html#int), compute_duration: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

A task has seceded from the thread pool

We stop the task from being stolen in the future, and change task
duration accounting as if the task has stopped.

<!-- !! processed by numpydoc !! -->

#### handle_request_refresh_who_has(keys: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], worker: [str](https://docs.python.org/3/library/stdtypes.html#str), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Request from a Worker to refresh the
who_has for some keys. Not to be confused with scheduler.who_has, which
is a dedicated comm RPC request from a Client.

<!-- !! processed by numpydoc !! -->

#### *async* handle_worker(comm: [Comm](https://distributed.dask.org/en/latest/communications.html#distributed.comm.Comm), worker: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Listen to responses from a single worker

This is the main loop for scheduler-worker interaction

#### SEE ALSO
`Scheduler.handle_client`
: Equivalent coroutine for clients

<!-- !! processed by numpydoc !! -->

#### identity(n_workers: [int](https://docs.python.org/3/library/functions.html#int) = -1) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Basic information about ourselves and our cluster

<!-- !! processed by numpydoc !! -->

#### idle *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), WorkerState]*

Workers that are currently in running state and not fully utilized
Definition based on occupancy
(actually a SortedDict, but the sortedcontainers package isn’t annotated).
Not to be confused with `is_idle()`.

<!-- !! processed by numpydoc !! -->

#### idle_task_count *: [set](https://docs.python.org/3/library/stdtypes.html#set)[WorkerState]*

Similar to idle
Definition based on assigned tasks

<!-- !! processed by numpydoc !! -->

#### log_event(topic: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)], msg: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [None](https://docs.python.org/3/library/constants.html#None)

Log an event under a given topic

* **Parameters:**
  **topic**
  : Name of the topic under which to log an event. To log the same
    event under multiple topics, pass a list of topic names.

  **msg**
  : Event message to log. Note this must be msgpack serializable.

#### SEE ALSO
[`Client.log_event`](futures.md#distributed.Client.log_event)

<!-- !! processed by numpydoc !! -->

#### n_tasks *: [int](https://docs.python.org/3/library/functions.html#int)*

Total number of tasks ever processed

<!-- !! processed by numpydoc !! -->

#### *async* proxy(msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict), worker: [str](https://docs.python.org/3/library/stdtypes.html#str), serializers: [Any](https://docs.python.org/3/library/typing.html#typing.Any) = None) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Proxy a communication through the scheduler to some other worker

<!-- !! processed by numpydoc !! -->

#### queued *: HeapSet[TaskState]*

Tasks in the “queued” state, ordered by priority

<!-- !! processed by numpydoc !! -->

#### *async* rebalance(keys: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]] | [None](https://docs.python.org/3/library/constants.html#None) = None, workers: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Rebalance keys so that each worker ends up with roughly the same process
memory (managed+unmanaged).

#### WARNING
This operation is generally not well tested against normal operation of the
scheduler. It is not recommended to use it while waiting on computations.

**Algorithm**

1. Find the mean occupancy of the cluster, defined as data managed by dask +
   unmanaged process memory that has been there for at least 30 seconds
   (`distributed.worker.memory.recent-to-old-time`).
   This lets us ignore temporary spikes caused by task heap usage.

   Alternatively, you may change how memory is measured both for the individual
   workers as well as to calculate the mean through
   `distributed.worker.memory.rebalance.measure`. Namely, this can be useful
   to disregard inaccurate OS memory measurements.
2. Discard workers whose occupancy is within 5% of the mean cluster occupancy
   (`distributed.worker.memory.rebalance.sender-recipient-gap` / 2).
   This helps avoid data from bouncing around the cluster repeatedly.
3. Workers above the mean are senders; those below are recipients.
4. Discard senders whose absolute occupancy is below 30%
   (`distributed.worker.memory.rebalance.sender-min`). In other words, no data
   is moved regardless of imbalancing as long as all workers are below 30%.
5. Discard recipients whose absolute occupancy is above 60%
   (`distributed.worker.memory.rebalance.recipient-max`).
   Note that this threshold by default is the same as
   `distributed.worker.memory.target` to prevent workers from accepting data
   and immediately spilling it out to disk.
6. Iteratively pick the sender and recipient that are farthest from the mean and
   move the *least recently inserted* key between the two, until either all
   senders or all recipients fall within 5% of the mean.

   A recipient will be skipped if it already has a copy of the data. In other
   words, this method does not degrade replication.
   A key will be skipped if there are no recipients available with enough memory
   to accept the key and that don’t already hold a copy.

The least recently insertd (LRI) policy is a greedy choice with the advantage of
being O(1), trivial to implement (it relies on python dict insertion-sorting)
and hopefully good enough in most cases. Discarded alternative policies were:

- Largest first. O(n\*log(n)) save for non-trivial additional data structures and
  risks causing the largest chunks of data to repeatedly move around the
  cluster like pinballs.
- Least recently used (LRU). This information is currently available on the
  workers only and not trivial to replicate on the scheduler; transmitting it
  over the network would be very expensive. Also, note that dask will go out of
  its way to minimise the amount of time intermediate keys are held in memory,
  so in such a case LRI is a close approximation of LRU.

* **Parameters:**
  **keys: optional**
  : allowlist of dask keys that should be considered for moving. All other keys
    will be ignored. Note that this offers no guarantee that a key will actually
    be moved (e.g. because it is unnecessary or because there are no viable
    recipient workers for it).

  **workers: optional**
  : allowlist of workers addresses to be considered as senders or recipients.
    All other workers will be ignored. The mean cluster occupancy will be
    calculated only using the allowed workers.

<!-- !! processed by numpydoc !! -->

#### *async* register_nanny_plugin(comm: [None](https://docs.python.org/3/library/constants.html#None), plugin: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes), name: [str](https://docs.python.org/3/library/stdtypes.html#str), idempotent: [bool](https://docs.python.org/3/library/functions.html#bool)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), OKMessage]

Registers a nanny plugin on all running and future nannies

<!-- !! processed by numpydoc !! -->

#### *async* register_scheduler_plugin(plugin: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [SchedulerPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.SchedulerPlugin), name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, , idempotent: [bool](https://docs.python.org/3/library/functions.html#bool)) → [None](https://docs.python.org/3/library/constants.html#None)

Register a plugin on the scheduler.

<!-- !! processed by numpydoc !! -->

#### *async* register_worker_plugin(comm: [None](https://docs.python.org/3/library/constants.html#None), plugin: [bytes](https://docs.python.org/3/library/stdtypes.html#bytes), name: [str](https://docs.python.org/3/library/stdtypes.html#str), , idempotent: [bool](https://docs.python.org/3/library/functions.html#bool)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), OKMessage]

Registers a worker plugin on all running and future workers

<!-- !! processed by numpydoc !! -->

#### remove_client(client: [str](https://docs.python.org/3/library/stdtypes.html#str), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Remove client from network

<!-- !! processed by numpydoc !! -->

#### remove_plugin(name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Remove external plugin from scheduler

* **Parameters:**
  **name**
  : Name of the plugin to remove

<!-- !! processed by numpydoc !! -->

#### *async* remove_worker(address: [str](https://docs.python.org/3/library/stdtypes.html#str), , stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str), expected: [bool](https://docs.python.org/3/library/functions.html#bool) = False, close: [bool](https://docs.python.org/3/library/functions.html#bool) = True) → [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['OK', 'already-removed']

Remove worker from cluster.

We do this when a worker reports that it plans to leave or when it appears to be
unresponsive. This may send its tasks back to a released state.

#### SEE ALSO
[`retire_workers`](#distributed.Scheduler.retire_workers)

[`close_worker`](#distributed.Scheduler.close_worker)

<!-- !! processed by numpydoc !! -->

#### *async* replicate(keys: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], n: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, workers: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable) | [None](https://docs.python.org/3/library/constants.html#None) = None, branching_factor: [int](https://docs.python.org/3/library/functions.html#int) = 2, delete: [bool](https://docs.python.org/3/library/functions.html#bool) = True, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None)

Replicate data throughout cluster

This performs a tree copy of the data throughout the network
individually on each piece of data.

* **Parameters:**
  **keys: Iterable**
  : list of keys to replicate

  **n: int**
  : Number of replications we expect to see within the cluster

  **branching_factor: int, optional**
  : The number of workers that can copy data in each generation.
    The larger the branching factor, the more data we copy in
    a single step, but the more a given worker risks being
    swamped by data requests.

#### SEE ALSO
[`Scheduler.rebalance`](#distributed.Scheduler.rebalance)

<!-- !! processed by numpydoc !! -->

#### replicated_tasks *: [set](https://docs.python.org/3/library/stdtypes.html#set)[TaskState]*

Subset of tasks that exist in memory on more than one worker

<!-- !! processed by numpydoc !! -->

#### report(msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict), ts: [TaskState](https://distributed.dask.org/en/latest/scheduling-state.html#distributed.scheduler.TaskState) | [None](https://docs.python.org/3/library/constants.html#None) = None, client: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Publish updates to all listening Queues and Comms

If the message contains a key then we only send the message to those
comms that care about the key.

<!-- !! processed by numpydoc !! -->

#### request_acquire_replicas(addr: [str](https://docs.python.org/3/library/stdtypes.html#str), keys: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], , stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Asynchronously ask a worker to acquire a replica of the listed keys from
other workers. This is a fire-and-forget operation which offers no feedback for
success or failure, and is intended for housekeeping and not for computation.

<!-- !! processed by numpydoc !! -->

#### request_remove_replicas(addr: [str](https://docs.python.org/3/library/stdtypes.html#str), keys: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], , stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Asynchronously ask a worker to discard its replica of the listed keys.
This must never be used to destroy the last replica of a key. This is a
fire-and-forget operation, intended for housekeeping and not for computation.

The replica disappears immediately from TaskState.who_has on the Scheduler side;
if the worker refuses to delete, e.g. because the task is a dependency of
another task running on it, it will (also asynchronously) inform the scheduler
to re-add itself to who_has. If the worker agrees to discard the task, there is
no feedback.

<!-- !! processed by numpydoc !! -->

#### resources *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [float](https://docs.python.org/3/library/functions.html#float)]]*

Cluster-wide resources. {resource name: {worker address: amount}}

<!-- !! processed by numpydoc !! -->

#### *async* restart(, client: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, timeout: [float](https://docs.python.org/3/library/functions.html#float) = 30, wait_for_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = True, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Forget all tasks and call restart_workers on all workers.

* **Parameters:**
  **timeout:**
  : See restart_workers

  **wait_for_workers:**
  : See restart_workers

#### SEE ALSO
[`Client.restart`](futures.md#distributed.Client.restart)

[`Client.restart_workers`](futures.md#distributed.Client.restart_workers)

[`Scheduler.restart_workers`](#distributed.Scheduler.restart_workers)

<!-- !! processed by numpydoc !! -->

#### *async* restart_workers(workers: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, , client: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, timeout: [float](https://docs.python.org/3/library/functions.html#float) = 30, wait_for_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = True, on_error: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['raise', 'return'] = 'raise', stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['OK', 'removed', 'timed out']]

Restart selected workers. Optionally wait for workers to return.

Workers without nannies are shut down, hoping an external deployment system
will restart them. Therefore, if not using nannies and your deployment system
does not automatically restart workers, `restart` will just shut down all
workers, then time out!

After `restart`, all connected workers are new, regardless of whether
`TimeoutError` was raised. Any workers that failed to shut down in time are
removed, and may or may not shut down on their own in the future.

* **Parameters:**
  **workers:**
  : List of worker addresses to restart. If omitted, restart all workers.

  **timeout:**
  : How long to wait for workers to shut down and come back, if `wait_for_workers`
    is True, otherwise just how long to wait for workers to shut down.
    Raises `asyncio.TimeoutError` if this is exceeded.

  **wait_for_workers:**
  : Whether to wait for all workers to reconnect, or just for them to shut down
    (default True). Use `restart(wait_for_workers=False)` combined with
    [`Client.wait_for_workers()`](futures.md#distributed.Client.wait_for_workers) for granular control over how many workers to
    wait for.

  **on_error:**
  : If ‘raise’ (the default), raise if any nanny times out while restarting the
    worker. If ‘return’, return error messages.
* **Returns:**
  {worker address: “OK”, “no nanny”, or “timed out” or error message}

#### SEE ALSO
[`Client.restart`](futures.md#distributed.Client.restart)

[`Client.restart_workers`](futures.md#distributed.Client.restart_workers)

[`Scheduler.restart`](#distributed.Scheduler.restart)

<!-- !! processed by numpydoc !! -->

#### *async* retire_workers(workers: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)], , close_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = False, remove: [bool](https://docs.python.org/3/library/functions.html#bool) = True, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), Any]

#### *async* retire_workers(, names: [list](https://docs.python.org/3/library/stdtypes.html#list), close_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = False, remove: [bool](https://docs.python.org/3/library/functions.html#bool) = True, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), Any]

#### *async* retire_workers(, close_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = False, remove: [bool](https://docs.python.org/3/library/functions.html#bool) = True, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None, memory_ratio: [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, n: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, key: Callable[[WorkerState], Hashable] | [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [None](https://docs.python.org/3/library/constants.html#None) = None, minimum: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, target: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, attribute: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'address') → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), Any]

Gracefully retire workers from cluster. Any key that is in memory exclusively
on the retired workers is replicated somewhere else.

* **Parameters:**
  **workers: list[str] (optional)**
  : List of worker addresses to retire.

  **names: list (optional)**
  : List of worker names to retire.
    Mutually exclusive with `workers`.
    If neither `workers` nor `names` are provided, we call
    `workers_to_close` which finds a good set.

  **close_workers: bool (defaults to False)**
  : Whether to actually close the worker explicitly from here.
    Otherwise, we expect some external job scheduler to finish off the worker.

  **remove: bool (defaults to True)**
  : Whether to remove the worker metadata immediately or else wait for the
    worker to contact us.
    <br/>
    If close_workers=False and remove=False, this method just flushes the tasks
    in memory out of the workers and then returns.
    If close_workers=True and remove=False, this method will return while the
    workers are still in the cluster, although they won’t accept new tasks.
    If close_workers=False or for whatever reason a worker doesn’t accept the
    close command, it will be left permanently unable to accept new tasks and
    it is expected to be closed in some other way.

  **\*\*kwargs: dict**
  : Extra options to pass to workers_to_close to determine which
    workers we should drop. Only accepted if `workers` and `names` are
    omitted.
* **Returns:**
  Dictionary mapping worker ID/address to dictionary of information about

  that worker for each retired worker.

  If there are keys that exist in memory only on the workers being retired and it

  was impossible to replicate them somewhere else (e.g. because there aren’t

  any other running workers), the workers holding such keys won’t be retired and

  won’t appear in the returned dict.

#### SEE ALSO
[`Scheduler.workers_to_close`](#distributed.Scheduler.workers_to_close)

<!-- !! processed by numpydoc !! -->

#### run_function(comm: [Comm](https://distributed.dask.org/en/latest/communications.html#distributed.comm.Comm), function: [Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable), args: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple) = (), kwargs: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, wait: [bool](https://docs.python.org/3/library/functions.html#bool) = True) → [Any](https://docs.python.org/3/library/typing.html#typing.Any)

Run a function within this process

#### SEE ALSO
[`Client.run_on_scheduler`](futures.md#distributed.Client.run_on_scheduler)

<!-- !! processed by numpydoc !! -->

#### running *: [set](https://docs.python.org/3/library/stdtypes.html#set)[WorkerState]*

Workers that are currently in running state

<!-- !! processed by numpydoc !! -->

#### saturated *: [set](https://docs.python.org/3/library/stdtypes.html#set)[WorkerState]*

Workers that are fully utilized. May include non-running workers.

<!-- !! processed by numpydoc !! -->

#### *async* scatter(data: [dict](https://docs.python.org/3/library/stdtypes.html#dict), workers: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable) | [None](https://docs.python.org/3/library/constants.html#None), client: [str](https://docs.python.org/3/library/stdtypes.html#str), broadcast: [bool](https://docs.python.org/3/library/functions.html#bool) = False, timeout: [float](https://docs.python.org/3/library/functions.html#float) = 2) → [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]

Send data out to workers

#### SEE ALSO
[`Scheduler.broadcast`](#distributed.Scheduler.broadcast)

<!-- !! processed by numpydoc !! -->

#### send_all(client_msgs: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]], worker_msgs: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]]) → [None](https://docs.python.org/3/library/constants.html#None)

Send messages to client and workers

<!-- !! processed by numpydoc !! -->

#### send_task_to_worker(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), ts: [TaskState](https://distributed.dask.org/en/latest/scheduling-state.html#distributed.scheduler.TaskState)) → [None](https://docs.python.org/3/library/constants.html#None)

Send a single computational task to a worker

<!-- !! processed by numpydoc !! -->

#### *async* start_unsafe() → Self

Clear out old state and restart all running coroutines

<!-- !! processed by numpydoc !! -->

#### stimulus_cancel(keys: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], client: [str](https://docs.python.org/3/library/stdtypes.html#str), force: [bool](https://docs.python.org/3/library/functions.html#bool), reason: [str](https://docs.python.org/3/library/stdtypes.html#str), msg: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Stop execution on a list of keys

<!-- !! processed by numpydoc !! -->

#### stimulus_queue_slots_maybe_opened(, stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Respond to an event which may have opened spots on worker threadpools

Selects the appropriate number of tasks from the front of the queue according to
the total number of task slots available on workers (potentially 0), and
transitions them to `processing`.

### Notes

Other transitions related to this stimulus should be fully processed beforehand,
so any tasks that became runnable are already in `processing`. Otherwise,
overproduction can occur if queued tasks get scheduled before downstream tasks.

Must be called after check_idle_saturated; i.e. idle_task_count must be up to date.

<!-- !! processed by numpydoc !! -->

#### stimulus_task_erred(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str), key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], run_id: [int](https://docs.python.org/3/library/functions.html#int), \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['released', 'waiting', 'no-worker', 'queued', 'processing', 'memory', 'erred', 'forgotten']], [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]], [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]]]

Mark that a task has erred on a particular worker

<!-- !! processed by numpydoc !! -->

#### stimulus_task_finished(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str), key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], run_id: [int](https://docs.python.org/3/library/functions.html#int), metadata: [dict](https://docs.python.org/3/library/stdtypes.html#dict), \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['released', 'waiting', 'no-worker', 'queued', 'processing', 'memory', 'erred', 'forgotten']], [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]], [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]]]]

Mark that a task has finished execution on a particular worker

<!-- !! processed by numpydoc !! -->

#### tasks *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[Key, TaskState]*

All tasks currently known to the scheduler

<!-- !! processed by numpydoc !! -->

#### total_memory *: [int](https://docs.python.org/3/library/functions.html#int)*

Current total memory across all workers (sum over memory_limit)

<!-- !! processed by numpydoc !! -->

#### total_nthreads *: [int](https://docs.python.org/3/library/functions.html#int)*

Current number of threads across all workers

<!-- !! processed by numpydoc !! -->

#### total_nthreads_history *: [list](https://docs.python.org/3/library/stdtypes.html#list)[[tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[float](https://docs.python.org/3/library/functions.html#float), [int](https://docs.python.org/3/library/functions.html#int)]]*

History of number of threads
(timestamp, new number of threads)

<!-- !! processed by numpydoc !! -->

#### transition(key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], finish: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['released', 'waiting', 'no-worker', 'queued', 'processing', 'memory', 'erred', 'forgotten'], stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str), \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['released', 'waiting', 'no-worker', 'queued', 'processing', 'memory', 'erred', 'forgotten']]

Transition a key from its current state to the finish state

* **Returns:**
  Dictionary of recommendations for future transitions

#### SEE ALSO
[`Scheduler.transitions`](#distributed.Scheduler.transitions)
: transitive version of this function

### Examples

```pycon
>>> self.transition('x', 'waiting')
{'x': 'processing'}
```

<!-- !! processed by numpydoc !! -->

#### transition_counter *: [int](https://docs.python.org/3/library/functions.html#int)*

Total number of transitions since the cluster was started

<!-- !! processed by numpydoc !! -->

#### transition_counter_max *: [int](https://docs.python.org/3/library/functions.html#int) | Literal[False]*

Raise an error if the [`transition_counter`](#distributed.Scheduler.transition_counter) ever reaches this value.
This is meant for debugging only, to catch infinite recursion loops.
In production, it should always be set to False.

<!-- !! processed by numpydoc !! -->

#### transition_log *: deque[Transition]*

History of task state transitions.
The length can be tweaked through
distributed.admin.low-level-log-length

<!-- !! processed by numpydoc !! -->

#### transitions(recommendations: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['released', 'waiting', 'no-worker', 'queued', 'processing', 'memory', 'erred', 'forgotten']], stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Process transitions until none are left

This includes feedback from previous transitions and continues until we
reach a steady state

<!-- !! processed by numpydoc !! -->

#### *async* unregister_nanny_plugin(comm: [None](https://docs.python.org/3/library/constants.html#None), name: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), ErrorMessage | OKMessage]

Unregisters a worker plugin

<!-- !! processed by numpydoc !! -->

#### *async* unregister_scheduler_plugin(name: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [None](https://docs.python.org/3/library/constants.html#None)

Unregister a plugin on the scheduler.

<!-- !! processed by numpydoc !! -->

#### *async* unregister_worker_plugin(comm: [None](https://docs.python.org/3/library/constants.html#None), name: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), ErrorMessage | OKMessage]

Unregisters a worker plugin

<!-- !! processed by numpydoc !! -->

#### unrunnable *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[TaskState, [float](https://docs.python.org/3/library/functions.html#float)]*

Tasks in the “no-worker” state with the (monotonic) time when they became unrunnable

<!-- !! processed by numpydoc !! -->

#### update_data(, who_has: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]], nbytes: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [int](https://docs.python.org/3/library/functions.html#int)], client: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Learn that new data has entered the network from an external source

<!-- !! processed by numpydoc !! -->

#### validate *: [bool](https://docs.python.org/3/library/functions.html#bool)*

If True, enable expensive internal consistency check.
Typically disabled in production.

<!-- !! processed by numpydoc !! -->

#### worker_send(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]) → [None](https://docs.python.org/3/library/constants.html#None)

Send message to worker

This also handles connection failures by adding a callback to remove
the worker on the next cycle.

<!-- !! processed by numpydoc !! -->

#### workers *: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), WorkerState]*

Workers currently connected to the scheduler
(actually a SortedDict, but the sortedcontainers package isn’t annotated)

<!-- !! processed by numpydoc !! -->

#### workers_list(workers: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None)) → [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]

List of qualifying workers

Takes a list of worker addresses or hostnames.
Returns a list of all worker addresses that match

<!-- !! processed by numpydoc !! -->

#### workers_to_close(memory_ratio: [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None, n: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, key: [Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[WorkerState](https://distributed.dask.org/en/latest/scheduling-state.html#distributed.scheduler.WorkerState)], [Hashable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)] | [bytes](https://docs.python.org/3/library/stdtypes.html#bytes) | [None](https://docs.python.org/3/library/constants.html#None) = None, minimum: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, target: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, attribute: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'address') → [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]

Find workers that we can close with low cost

This returns a list of workers that are good candidates to retire.
These workers are not running anything and are storing
relatively little data relative to their peers.  If all workers are
idle then we still maintain enough workers to have enough RAM to store
our data, with a comfortable buffer.

This is for use with systems like `distributed.deploy.adaptive`.

* **Parameters:**
  **memory_ratio**
  : Amount of extra space we want to have for our stored data.
    Defaults to 2, or that we want to have twice as much memory as we
    currently have data.

  **n**
  : Number of workers to close

  **minimum**
  : Minimum number of workers to keep around

  **key**
  : An optional callable mapping a WorkerState object to a group
    affiliation. Groups will be closed together. This is useful when
    closing workers must be done collectively, such as by hostname.

  **target**
  : Target number of workers to have after we close

  **attribute**
  : The attribute of the WorkerState object to return, like “address”
    or “name”.  Defaults to “address”.
* **Returns:**
  to_close: list of worker addresses that are OK to close

#### SEE ALSO
[`Scheduler.retire_workers`](#distributed.Scheduler.retire_workers)

### Examples

```pycon
>>> scheduler.workers_to_close()
['tcp://192.168.0.1:1234', 'tcp://192.168.0.2:1234']
```

Group workers by hostname prior to closing

```pycon
>>> scheduler.workers_to_close(key=lambda ws: ws.host)
['tcp://192.168.0.1:1234', 'tcp://192.168.0.1:4567']
```

Remove two workers

```pycon
>>> scheduler.workers_to_close(n=2)
```

Keep enough workers to have twice as much memory as we we need.

```pycon
>>> scheduler.workers_to_close(memory_ratio=2)
```

<!-- !! processed by numpydoc !! -->

### Worker

### *class* distributed.Worker(scheduler_ip: str | None = None, scheduler_port: int | None = None, \*, scheduler_file: str | None = None, nthreads: int | None = None, local_directory: str | None = None, services: dict | None = None, name: Any | None = None, executor: Executor | dict[str, Executor] | Literal['offload'] | None = None, resources: dict[str, float] | None = None, silence_logs: int | None = None, death_timeout: Any | None = None, preload: list[str] | None = None, preload_argv: list[str] | list[list[str]] | None = None, security: Security | dict[str, Any] | None = None, contact_address: str | None = None, heartbeat_interval: Any = '1s', extensions: dict[str, type] | None = None, metrics: Mapping[str, Callable[[Worker], Any]] = {}, startup_information: Mapping[str, Callable[[Worker], Any]] = {}, interface: str | None = None, host: str | None = None, port: int | str | Collection[int] | None = None, protocol: str | None = None, dashboard_address: str | None = None, dashboard: bool = False, http_prefix: str = '/', nanny: Nanny | None = None, plugins: tuple[WorkerPlugin, ...] = (), low_level_profiler: bool | None = None, validate: bool | None = None, profile_cycle_interval=None, lifetime: Any | None = None, lifetime_stagger: Any | None = None, lifetime_restart: bool | None = None, transition_counter_max: int | Literal[False] = False, memory_limit: str | float = 'auto', data: WorkerDataParameter = None, scheduler_sni: str | None = None, WorkerStateClass: type = <class 'distributed.worker_state_machine.WorkerState'>, \*\*kwargs)

Worker node in a Dask distributed cluster

Workers perform two functions:

1. **Serve data** from a local dictionary
2. **Perform computation** on that data and on data from peers

Workers keep the scheduler informed of their data and use that scheduler to
gather data from other workers when necessary to perform a computation.

You can start a worker with the `dask worker` command line application:

```default
$ dask worker scheduler-ip:port
```

Use the `--help` flag to see more options:

```default
$ dask worker --help
```

The rest of this docstring is about the internal state that the worker uses
to manage and track internal computations.

**State**

**Informational State**

These attributes don’t change significantly during execution.

* **nthreads:** `int`:
  : Number of nthreads used by this worker process
* **executors:** `dict[str, concurrent.futures.Executor]`:
  : Executors used to perform computation. Always contains the default
    executor.
* **local_directory:** `path`:
  : Path on local machine to store temporary files
* **scheduler:** `PooledRPCCall`:
  : Location of scheduler.  See `.ip/.port` attributes.
* **name:** `string`:
  : Alias
* **services:** `{str: Server}`:
  : Auxiliary web servers running on this worker
* **service_ports:** `{str: port}`:
* **transfer_outgoing_count_limit**: `int`
  : The maximum number of concurrent outgoing data transfers.
    See also
    [`distributed.worker_state_machine.WorkerState.transfer_incoming_count_limit`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.WorkerState.transfer_incoming_count_limit).
* **batched_stream**: `BatchedSend`
  : A batched stream along which we communicate to the scheduler
* **log**: `[(message)]`
  : A structured and queryable log.  See `Worker.story`

**Volatile State**

These attributes track the progress of tasks that this worker is trying to
complete. In the descriptions below a `key` is the name of a task that
we want to compute and `dep` is the name of a piece of dependent data
that we want to collect from others.

* **threads**: `{key: int}`
  : The ID of the thread on which the task ran
* **active_threads**: `{int: key}`
  : The keys currently running on active threads
* **state**: `WorkerState`
  : Encapsulated state machine. See
    [`BaseWorker`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker) and
    [`WorkerState`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.WorkerState)

* **Parameters:**
  **scheduler_ip: str, optional**

  **scheduler_port: int, optional**

  **scheduler_file: str, optional**

  **host: str, optional**

  **data: MutableMapping, type, None**
  : The object to use for storage, builds a disk-backed LRU dict by default.
    <br/>
    If a callable to construct the storage object is provided, it
    will receive the worker’s attr:`local_directory` as an
    argument if the calling signature has an argument named
    `worker_local_directory`.

  **nthreads: int, optional**

  **local_directory: str, optional**
  : Directory where we place local resources

  **name: str, optional**

  **memory_limit: int, float, string**
  : Number of bytes of memory that this worker should use.
    Set to zero for no limit.  Set to ‘auto’ to calculate
    as system.MEMORY_LIMIT \* min(1, nthreads / total_cores)
    Use strings or numbers like 5GB or 5e9

  **memory_target_fraction: float or False**
  : Fraction of memory to try to stay beneath
    (default: read from config key distributed.worker.memory.target)

  **memory_spill_fraction: float or False**
  : Fraction of memory at which we start spilling to disk
    (default: read from config key distributed.worker.memory.spill)

  **memory_pause_fraction: float or False**
  : Fraction of memory at which we stop running new tasks
    (default: read from config key distributed.worker.memory.pause)

  **max_spill: int, string or False**
  : Limit of number of bytes to be spilled on disk.
    (default: read from config key distributed.worker.memory.max-spill)

  **executor: concurrent.futures.Executor, dict[str, concurrent.futures.Executor], “offload”**
  : The executor(s) to use. Depending on the type, it has the following meanings:
    : - Executor instance: The default executor.
      - Dict[str, Executor]: mapping names to Executor instances. If the
        “default” key isn’t in the dict, a “default” executor will be created
        using `ThreadPoolExecutor(nthreads)`.
      - Str: The string “offload”, which refer to the same thread pool used for
        offloading communications. This results in the same thread being used
        for deserialization and computation.

  **resources: dict**
  : Resources that this worker has like `{'GPU': 2}`

  **nanny: str**
  : Address on which to contact nanny, if it exists

  **lifetime: str**
  : Amount of time like “1 hour” after which we gracefully shut down the worker.
    This defaults to None, meaning no explicit shutdown time.

  **lifetime_stagger: str**
  : Amount of time like “5 minutes” to stagger the lifetime value
    The actual lifetime will be selected uniformly at random between
    lifetime +/- lifetime_stagger

  **lifetime_restart: bool**
  : Whether or not to restart a worker after it has reached its lifetime
    Default False

  **kwargs: optional**
  : Additional parameters to ServerNode constructor

#### SEE ALSO
[`distributed.scheduler.Scheduler`](#distributed.Scheduler)

[`distributed.nanny.Nanny`](#distributed.Nanny)

### Examples

Use the command line to start a worker:

```default
$ dask scheduler
Start scheduler at 127.0.0.1:8786

$ dask worker 127.0.0.1:8786
Start worker at:               127.0.0.1:1234
Registered with scheduler at:  127.0.0.1:8786
```

<!-- !! processed by numpydoc !! -->

#### batched_send(msg: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]) → [None](https://docs.python.org/3/library/constants.html#None)

Implements BaseWorker abstract method.

Send a fire-and-forget message to the scheduler through bulk comms.

If we’re not currently connected to the scheduler, the message will be silently
dropped!

#### SEE ALSO
[`distributed.worker_state_machine.BaseWorker.batched_send`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker.batched_send)

<!-- !! processed by numpydoc !! -->

#### *async* close(timeout: [float](https://docs.python.org/3/library/functions.html#float) = 30, executor_wait: [bool](https://docs.python.org/3/library/functions.html#bool) = True, nanny: [bool](https://docs.python.org/3/library/functions.html#bool) = True, reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'worker-close') → [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None)

Close the worker

Close asynchronous operations running on the worker, stop all executors and
comms. If requested, this also closes the nanny.

* **Parameters:**
  **timeout**
  : Timeout in seconds for shutting down individual instructions

  **executor_wait**
  : If True, shut down executors synchronously, otherwise asynchronously

  **nanny**
  : If True, close the nanny

  **reason**
  : Reason for closing the worker
* **Returns:**
  str | None
  : None if worker already in closing state or failed, “OK” otherwise

<!-- !! processed by numpydoc !! -->

#### *async* close_gracefully(restart=None, reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'worker-close-gracefully')

Gracefully shut down a worker

This first informs the scheduler that we’re shutting down, and asks it
to move our data elsewhere. Afterwards, we close as normal

<!-- !! processed by numpydoc !! -->

#### *property* data *: [MutableMapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [object](https://docs.python.org/3/library/functions.html#object)]*

{task key: task payload} of all completed tasks, whether they were computed
on this Worker or computed somewhere else and then transferred here over the
network.

When using the default configuration, this is a zict buffer that automatically
spills to disk whenever the target threshold is exceeded.
If spilling is disabled, it is a plain dict instead.
It could also be a user-defined arbitrary dict-like passed when initialising
the Worker or the Nanny.
Worker logic should treat this opaquely and stick to the MutableMapping API.

#### NOTE
This same collection is also available at `self.state.data` and
`self.memory_manager.data`.

<!-- !! processed by numpydoc !! -->

#### digest_metric(name: [Hashable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable), value: [float](https://docs.python.org/3/library/functions.html#float)) → [None](https://docs.python.org/3/library/constants.html#None)

Implement BaseWorker.digest_metric by calling Server.digest_metric

<!-- !! processed by numpydoc !! -->

#### *async* execute(key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], , stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [StateMachineEvent](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.StateMachineEvent)

Execute a task. Implements BaseWorker abstract method.

#### SEE ALSO
[`distributed.worker_state_machine.BaseWorker.execute`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker.execute)

<!-- !! processed by numpydoc !! -->

#### *async* gather(who_has: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)]]) → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [object](https://docs.python.org/3/library/functions.html#object)]

Endpoint used by Scheduler.rebalance() and Scheduler.replicate()

<!-- !! processed by numpydoc !! -->

#### *async* gather_dep(worker: [str](https://docs.python.org/3/library/stdtypes.html#str), to_gather: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], total_nbytes: [int](https://docs.python.org/3/library/functions.html#int), , stimulus_id: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [StateMachineEvent](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.StateMachineEvent)

Implements BaseWorker abstract method

#### SEE ALSO
[`distributed.worker_state_machine.BaseWorker.gather_dep`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker.gather_dep)

<!-- !! processed by numpydoc !! -->

#### get_current_task() → [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...]

Get the key of the task we are currently running

This only makes sense to run within a task

#### SEE ALSO
`get_worker`

### Examples

```pycon
>>> from dask.distributed import get_worker
>>> def f():
...     return get_worker().get_current_task()
```

```pycon
>>> future = client.submit(f)
>>> future.result()
'f-1234'
```

<!-- !! processed by numpydoc !! -->

#### handle_stimulus(\*stims: [StateMachineEvent](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.StateMachineEvent)) → [None](https://docs.python.org/3/library/constants.html#None)

Override BaseWorker method for added validation

#### SEE ALSO
[`distributed.worker_state_machine.BaseWorker.handle_stimulus`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker.handle_stimulus)

[`distributed.worker_state_machine.WorkerState.handle_stimulus`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.WorkerState.handle_stimulus)

<!-- !! processed by numpydoc !! -->

#### log_event(topic: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)], msg: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [None](https://docs.python.org/3/library/constants.html#None)

Log an event under a given topic

* **Parameters:**
  **topic**
  : Name of the topic under which to log an event. To log the same
    event under multiple topics, pass a list of topic names.

  **msg**
  : Event message to log. Note this must be msgpack serializable.

#### SEE ALSO
[`Client.log_event`](futures.md#distributed.Client.log_event)

<!-- !! processed by numpydoc !! -->

#### *async* retry_busy_worker_later(worker: [str](https://docs.python.org/3/library/stdtypes.html#str)) → [StateMachineEvent](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.StateMachineEvent)

Wait some time, then take a peer worker out of busy state.
Implements BaseWorker abstract method.

#### SEE ALSO
[`distributed.worker_state_machine.BaseWorker.retry_busy_worker_later`](https://distributed.dask.org/en/latest/worker-state.html#distributed.worker_state_machine.BaseWorker.retry_busy_worker_later)

<!-- !! processed by numpydoc !! -->

#### *async* start_unsafe()

Attempt to start the server. This is not idempotent and not protected against concurrent startup attempts.

This is intended to be overwritten or called by subclasses. For a safe
startup, please use `Server.start` instead.

If `death_timeout` is configured, we will require this coroutine to
finish before this timeout is reached. If the timeout is reached we will
close the instance and raise an `asyncio.TimeoutError`

<!-- !! processed by numpydoc !! -->

#### transfer_outgoing_bytes *: [int](https://docs.python.org/3/library/functions.html#int)*

Current total size of open data transfers to other workers

<!-- !! processed by numpydoc !! -->

#### transfer_outgoing_bytes_total *: [int](https://docs.python.org/3/library/functions.html#int)*

Total size of data transfers to other workers (including in-progress and failed transfers)

<!-- !! processed by numpydoc !! -->

#### transfer_outgoing_count *: [int](https://docs.python.org/3/library/functions.html#int)*

Current number of open data transfers to other workers

<!-- !! processed by numpydoc !! -->

#### transfer_outgoing_count_total *: [int](https://docs.python.org/3/library/functions.html#int)*

Total number of data transfers to other workers since the worker was started

<!-- !! processed by numpydoc !! -->

#### trigger_profile() → [None](https://docs.python.org/3/library/constants.html#None)

Get a frame from all actively computing threads

Merge these frames into existing profile counts

<!-- !! processed by numpydoc !! -->

#### *property* worker_address

For API compatibility with Nanny

<!-- !! processed by numpydoc !! -->

### Nanny

### *class* distributed.Nanny(scheduler_ip=None, scheduler_port=None, scheduler_file=None, worker_port: [int](https://docs.python.org/3/library/functions.html#int) | [str](https://docs.python.org/3/library/stdtypes.html#str) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[int](https://docs.python.org/3/library/functions.html#int)] | [None](https://docs.python.org/3/library/constants.html#None) = 0, nthreads=None, local_directory=None, services=None, name=None, memory_limit='auto', reconnect=True, validate=False, quiet=False, resources=None, silence_logs=None, death_timeout=None, preload=None, preload_argv=None, preload_nanny=None, preload_nanny_argv=None, security=None, contact_address=None, listen_address=None, worker_class=None, env=None, interface=None, host=None, port: [int](https://docs.python.org/3/library/functions.html#int) | [str](https://docs.python.org/3/library/stdtypes.html#str) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[int](https://docs.python.org/3/library/functions.html#int)] | [None](https://docs.python.org/3/library/constants.html#None) = None, protocol=None, config=None, \*\*worker_kwargs)

A process to manage worker processes

The nanny spins up Worker processes, watches them, and kills or restarts
them as necessary. It is necessary if you want to use the
`Client.restart` method, or to restart the worker automatically if
it gets to the terminate fraction of its memory limit.

The parameters for the Nanny are mostly the same as those for the Worker
with exceptions listed below.

* **Parameters:**
  **env: dict, optional**
  : Environment variables set at time of Nanny initialization will be
    ensured to be set in the Worker process as well. This argument allows to
    overwrite or otherwise set environment variables for the Worker. It is
    also possible to set environment variables using the option
    `distributed.nanny.environ`. Precedence as follows
    > 1. Nanny arguments
    > 2. Existing environment variables
    > 3. Dask configuration
    <br/>
    #### NOTE
    Some environment variables, like `OMP_NUM_THREADS`, must be set before
    importing numpy to have effect. Others, like `MALLOC_TRIM_THRESHOLD_` (see
    [Memory not released back to the OS](https://distributed.dask.org/en/latest/worker-memory.html#memtrim)), must be set before starting the Linux process. Such
    variables would be ineffective if set here or in
    `distributed.nanny.environ`; they must be set in
    `distributed.nanny.pre-spawn-environ` so that they are set before spawning
    the subprocess, even if this means poisoning the process running the Nanny.
    <br/>
    For the same reason, be warned that changing
    `distributed.worker.multiprocessing-method` from `spawn` to `fork` or
    `forkserver` may inhibit some environment variables; if you do, you should
    set the variables yourself in the shell before you start `dask worker`.

#### SEE ALSO
[`Worker`](#distributed.Worker)

<!-- !! processed by numpydoc !! -->

#### *async* close(timeout: [float](https://docs.python.org/3/library/functions.html#float) = 5, reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'nanny-close') → [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['OK']

Close the worker process, stop all comms.

<!-- !! processed by numpydoc !! -->

#### close_gracefully(reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'nanny-close-gracefully') → [None](https://docs.python.org/3/library/constants.html#None)

A signal that we shouldn’t try to restart workers if they go away

This is used as part of the cluster shutdown process.

<!-- !! processed by numpydoc !! -->

#### *async* instantiate() → Status

Start a local worker process

Blocks until the process is up and the scheduler is properly informed

<!-- !! processed by numpydoc !! -->

#### *async* kill(timeout: [float](https://docs.python.org/3/library/functions.html#float) = 5, reason: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'nanny-kill') → [None](https://docs.python.org/3/library/constants.html#None)

Kill the local worker process

Blocks until both the process is down and the scheduler is properly
informed

<!-- !! processed by numpydoc !! -->

#### log_event(topic, msg)

Log an event under a given topic

* **Parameters:**
  **topic**
  : Name of the topic under which to log an event. To log the same
    event under multiple topics, pass a list of topic names.

  **msg**
  : Event message to log. Note this must be msgpack serializable.

#### SEE ALSO
[`Client.log_event`](futures.md#distributed.Client.log_event)

<!-- !! processed by numpydoc !! -->

#### *async* start_unsafe()

Start nanny, start local process, start watching

<!-- !! processed by numpydoc !! -->


# deploying-python.html.md

# Python API

You can create a `dask.distributed` scheduler by importing and creating a
`Client` with no arguments.  This overrides whatever default was previously
set.

```python
from dask.distributed import Client
client = Client()
```

You can navigate to `http://localhost:8787/status` to see the diagnostic
dashboard if you have Bokeh installed.

## Client

You can trivially set up a local cluster on your machine by instantiating a Dask
Client with no arguments

```python
from dask.distributed import Client
client = Client()
```

This sets up a scheduler in your local process along with a number of workers and
threads per worker related to the number of cores in your machine.

If you want to run workers in your same process, you can pass the
`processes=False` keyword argument.

```python
client = Client(processes=False)
```

This is sometimes preferable if you want to avoid inter-worker communication
and your computations release the GIL.  This is common when primarily using
NumPy or Dask Array.

## LocalCluster

The `Client()` call described above is shorthand for creating a LocalCluster
and then passing that to your client.

```python
from dask.distributed import Client, LocalCluster
cluster = LocalCluster()
client = Client(cluster)
```

This is equivalent, but somewhat more explicit.

You may want to look at the
keyword arguments available on `LocalCluster` to understand the options available
to you on handling the mixture of threads and processes, like specifying explicit
ports, and so on.

To create a local cluster with all workers running in dedicated subprocesses,
`dask.distributed` also offers the experimental `SubprocessCluster`.

## Cluster manager features

Instantiating a cluster manager class like `LocalCluster` and then passing it to the
`Client` is a common pattern. Cluster managers also provide useful utilities to help
you understand what is going on.

For example, you can retrieve the Dashboard URL.

```python
>>> cluster.dashboard_link
'http://127.0.0.1:8787/status'
```

You can retrieve logs from cluster components.

```python
>>> cluster.get_logs()
{'Cluster': '',
'Scheduler': "distributed.scheduler - INFO - Clear task state\ndistributed.scheduler - INFO -   S...
```

If you are using a cluster manager that supports scaling you can modify the number of workers manually
or automatically based on workload.

```python
>>> cluster.scale(10)  # Sets the number of workers to 10

>>> cluster.adapt(minimum=1, maximum=10)  # Allows the cluster to auto scale to 10 when tasks are computed
```

## Reference

### *class* distributed.deploy.local.LocalCluster(name=None, n_workers=None, threads_per_worker=None, processes=None, loop=None, start=None, host=None, ip=None, scheduler_port=0, silence_logs=30, dashboard_address=':8787', worker_dashboard_address=None, services=None, worker_services=None, service_kwargs=None, asynchronous=False, security=None, protocol=None, blocked_handlers=None, interface=None, worker_class=None, scheduler_kwargs=None, scheduler_sync_interval=1, \*\*worker_kwargs)

Create local Scheduler and Workers

This creates a “cluster” of a scheduler and workers running on the local
machine.

* **Parameters:**
  **n_workers: int**
  : Number of workers to start

  **memory_limit: str, float, int, or None, default “auto”**
  : Sets the memory limit *per worker*.
    <br/>
    Notes regarding argument data type:
    * If None or 0, no limit is applied.
    * If “auto”, the total system memory is split evenly between the workers.
    * If a float, that fraction of the system memory is used *per worker*.
    * If a string giving a number of bytes (like `"1GiB"`), that amount is used *per worker*.
    * If an int, that number of bytes is used *per worker*.
    <br/>
    Note that the limit will only be enforced when `processes=True`, and the limit is only
    enforced on a best-effort basis — it’s still possible for workers to exceed this limit.

  **processes: bool**
  : Whether to use processes (True) or threads (False).  Defaults to True, unless
    worker_class=Worker, in which case it defaults to False.

  **threads_per_worker: int**
  : Number of threads per each worker

  **scheduler_port: int**
  : Port of the scheduler. Use 0 to choose a random port (default). 8786 is a common choice.

  **silence_logs: logging level**
  : Level of logs to print out to stdout.  `logging.WARN` by default.
    Use a falsey value like False or None for no change.

  **host: string**
  : Host address on which the scheduler will listen, defaults to only localhost

  **dashboard_address: str**
  : Address on which to listen for the Bokeh diagnostics server like
    ‘localhost:8787’ or ‘0.0.0.0:8787’.  Defaults to ‘:8787’.
    Set to `None` to disable the dashboard.
    Use ‘:0’ for a random port.
    When specifying only a port like ‘:8787’, the dashboard will bind to the given interface from the `host` parameter.
    If `host` is empty, binding will occur on all interfaces ‘0.0.0.0’.
    To avoid firewall issues when deploying locally, set `host` to ‘localhost’.

  **worker_dashboard_address: str**
  : Address on which to listen for the Bokeh worker diagnostics server like
    ‘localhost:8787’ or ‘0.0.0.0:8787’.  Defaults to None which disables the dashboard.
    Use ‘:0’ for a random port.

  **asynchronous: bool (False by default)**
  : Set to True if using this cluster within async/await functions or within
    Tornado gen.coroutines.  This should remain False for normal use.

  **blocked_handlers: List[str]**
  : A list of strings specifying a blocklist of handlers to disallow on the
    Scheduler, like `['feed', 'run_function']`

  **service_kwargs: Dict[str, Dict]**
  : Extra keywords to hand to the running services

  **security**
  : Configures communication security in this cluster. Can be a security
    object, or True. If True, temporary self-signed credentials will
    be created automatically.

  **protocol: str (optional)**
  : Protocol to use like `tcp://`, `tls://`, `inproc://`
    This defaults to sensible choice given other keyword arguments like
    `processes` and `security`

  **interface: str (optional)**
  : Network interface to use.  Defaults to lo/localhost

  **worker_class: Worker**
  : Worker class used to instantiate workers from. Defaults to Worker if
    processes=False and Nanny if processes=True or omitted.

  **\*\*worker_kwargs:**
  : Extra worker arguments. Any additional keyword arguments will be passed
    to the `Worker` class constructor.

### Examples

```pycon
>>> cluster = LocalCluster()  # Create a local cluster
>>> cluster
LocalCluster("127.0.0.1:8786", workers=8, threads=8)
```

```pycon
>>> c = Client(cluster)  # connect to local cluster
```

Scale the cluster to three workers

```pycon
>>> cluster.scale(3)
```

Pass extra keyword arguments to Bokeh

```pycon
>>> LocalCluster(service_kwargs={'dashboard': {'prefix': '/foo'}})
```

<!-- !! processed by numpydoc !! -->


# deploying-ssh.html.md

# SSH

It is easy to set up Dask on informally managed networks of machines using SSH.
This can be done manually using SSH and the
Dask [command-line interface](deploying-cli.md),
or automatically using either the [`dask.distributed.SSHCluster`](#dask.distributed.SSHCluster) Python *cluster manager* or the
`dask-ssh` command line tool. This document describes both of these options.

#### NOTE
Before instaniating a `SSHCluster` it is recommended to configure keyless SSH
for your local machine and other machines. For example, on a Mac to SSH into
localhost (local machine) you need to ensure the Remote Login option is set in
System Preferences -> Sharing. In addition, `id_rsa.pub` should be in
`authorized_keys` for keyless login.

## Python Interface

### dask.distributed.SSHCluster(hosts: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, connect_options: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [list](https://docs.python.org/3/library/stdtypes.html#list)[[dict](https://docs.python.org/3/library/stdtypes.html#dict)] | [None](https://docs.python.org/3/library/constants.html#None) = None, worker_options: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, scheduler_options: [dict](https://docs.python.org/3/library/stdtypes.html#dict) | [None](https://docs.python.org/3/library/constants.html#None) = None, worker_class: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'distributed.Nanny', remote_python: [str](https://docs.python.org/3/library/stdtypes.html#str) | [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → [SpecCluster](https://distributed.dask.org/en/latest/api.html#distributed.SpecCluster)

Deploy a Dask cluster using SSH

The SSHCluster function deploys a Dask Scheduler and Workers for you on a
set of machine addresses that you provide.  The first address will be used
for the scheduler while the rest will be used for the workers (feel free to
repeat the first hostname if you want to have the scheduler and worker
co-habitate one machine.)

You may configure the scheduler and workers by passing
`scheduler_options` and `worker_options` dictionary keywords.  See the
`dask.distributed.Scheduler` and `dask.distributed.Worker` classes for
details on the available options, but the defaults should work in most
situations.

You may configure your use of SSH itself using the `connect_options`
keyword, which passes values to the `asyncssh.connect` function.  For
more information on these see the documentation for the `asyncssh`
library [https://asyncssh.readthedocs.io](https://asyncssh.readthedocs.io) .

* **Parameters:**
  **hosts**
  : List of hostnames or addresses on which to launch our cluster.
    The first will be used for the scheduler and the rest for workers.

  **connect_options**
  : Keywords to pass through to [`asyncssh.connect()`](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.connect).
    This could include things such as `port`, `username`, `password`
    or `known_hosts`. See docs for [`asyncssh.connect()`](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.connect) and
    [`asyncssh.SSHClientConnectionOptions`](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.SSHClientConnectionOptions) for full information.
    If a list it must have the same length as `hosts`.

  **worker_options**
  : Keywords to pass on to workers.

  **scheduler_options**
  : Keywords to pass on to scheduler.

  **worker_class**
  : The python class to use to create the worker(s).

  **remote_python**
  : Path to Python on remote nodes.

#### SEE ALSO
`dask.distributed.Scheduler`

`dask.distributed.Worker`

[`asyncssh.connect`](https://asyncssh.readthedocs.io/en/latest/api.html#asyncssh.connect)

### Examples

Create a cluster with one worker:

```pycon
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(["localhost", "localhost"])
>>> client = Client(cluster)
```

Create a cluster with three workers, each with two threads
and host the dashboard on port 8797:

```pycon
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
...     ["localhost", "localhost", "localhost", "localhost"],
...     connect_options={"known_hosts": None},
...     worker_options={"nthreads": 2},
...     scheduler_options={"port": 0, "dashboard_address": ":8797"}
... )
>>> client = Client(cluster)
```

Create a cluster with two workers on each host:

```pycon
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
...     ["localhost", "localhost", "localhost", "localhost"],
...     connect_options={"known_hosts": None},
...     worker_options={"nthreads": 2, "n_workers": 2},
...     scheduler_options={"port": 0, "dashboard_address": ":8797"}
... )
>>> client = Client(cluster)
```

An example using a different worker class, in particular the
`CUDAWorker` from the `dask-cuda` project:

```pycon
>>> from dask.distributed import Client, SSHCluster
>>> cluster = SSHCluster(
...     ["localhost", "hostwithgpus", "anothergpuhost"],
...     connect_options={"known_hosts": None},
...     scheduler_options={"port": 0, "dashboard_address": ":8797"},
...     worker_class="dask_cuda.CUDAWorker")
>>> client = Client(cluster)
```

<!-- !! processed by numpydoc !! -->

## Command Line

The convenience script `dask-ssh` opens several SSH connections to your
target computers and initializes the network accordingly. You can
give it a list of hostnames or IP addresses:

```default
$ dask-ssh 192.168.0.1 192.168.0.2 192.168.0.3 192.168.0.4
```

Or you can use normal UNIX grouping:

```default
$ dask-ssh 192.168.0.{1,2,3,4}
```

Or you can specify a hostfile that includes a list of hosts:

```default
$ cat hostfile.txt
192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4

$ dask-ssh --hostfile hostfile.txt
```

#### NOTE
The command line documentation here may differ depending on your installed
version. We recommend referring to the output of `dask-ssh --help`.

### dask-ssh

Launch a Dask cluster over SSH. A ‘dask scheduler’ process will run on the
first host specified in [HOSTNAMES] or in the hostfile, unless –scheduler is specified
explicitly. One or more ‘dask worker’ processes will be run on each host. Use the flag
–nworkers to adjust how many dask worker processes are run on each host and the flag
–nthreads to adjust how many CPUs are used by each dask worker process.

### Usage

```shell
dask-ssh [OPTIONS] [HOSTNAMES]...
```

### Options

### --scheduler <scheduler>

Specify scheduler node.  Defaults to first address.

### --scheduler-port <scheduler_port>

Specify scheduler port number.

* **Default:**
  `8786`

### --nthreads <nthreads>

Number of threads per worker process. Defaults to number of cores divided by the number of processes per host.

### --nworkers <n_workers>

Number of worker processes per host.

* **Default:**
  `1`

### --hostfile <hostfile>

Textfile with hostnames/IP addresses

### --ssh-username <ssh_username>

Username to use when establishing SSH connections.

### --ssh-port <ssh_port>

Port to use for SSH connections.

* **Default:**
  `22`

### --ssh-private-key <ssh_private_key>

Private key file to use for SSH connections.

### --nohost

Do not pass the hostname to the worker.

### --log-directory <log_directory>

Directory to use on all cluster nodes for the output of dask scheduler and dask worker commands.

### --local-directory <local_directory>

Directory to use on all cluster nodes to place workers files.

### --remote-python <remote_python>

Path to Python on remote nodes.

### --memory-limit <memory_limit>

Bytes of memory that the worker can use. This can be an integer (bytes), float (fraction of total system memory), string (like 5GB or 5000M), ‘auto’, or zero for no memory management

* **Default:**
  `'auto'`

### --worker-port <worker_port>

Serving computation port, defaults to random

### --nanny-port <nanny_port>

Serving nanny port, defaults to random

### --remote-dask-worker <remote_dask_worker>

Worker to run.

* **Default:**
  `'distributed.cli.dask_worker'`

### --version

Show the version and exit.

### Arguments

### HOSTNAMES

Optional argument(s)


# deploying.html.md

# Deploy Dask Clusters

Dask works well at many scales ranging from a single machine to clusters of
many machines.  This page describes the many ways to deploy and run Dask, including the following:

- [Python API](deploying-python.md)
- [Cloud](deploying-cloud.md)
- [High Performance Computers](deploying-hpc.md)
- [Kubernetes](deploying-kubernetes.md)

![image](images/dask-cluster-manager.svg)

<a id="deployment-single-machine"></a>

## Local Machine

You can run Dask without any setup.  Dask will use threads
on your local machine by default.

```python
import dask.dataframe as dd
df = dd.read_csv(...)
df.x.sum().compute()  # This uses threads on your local machine
```

Alternatively, you can set up a fully-featured multi-process Dask cluster on
your local machine.  This gives you access to multi-process computation and
diagnostic dashboards.

```python
from dask.distributed import LocalCluster
cluster = LocalCluster()          # Fully-featured local Dask cluster
client = cluster.get_client()

# Dask works as normal and leverages the infrastructure defined above
df.x.sum().compute()
```

The `LocalCluster` cluster manager defined above is easy to use and works
well on a single machine.  It follows the same interface as all other Dask
cluster managers, and so it’s easy to swap out when you’re ready to scale up.

```python
# You can swap out LocalCluster for other cluster types

from dask.distributed import LocalCluster
from dask_kubernetes import KubeCluster

# cluster = LocalCluster()
cluster = KubeCluster()  # example, you can swap out for Kubernetes

client = cluster.get_client()
```

<a id="deployment-options"></a>

The following resources explain how to set up Dask on a variety of local and distributed hardware.

<a id="cloud-deployment-options"></a>

## Cloud

Deploying on commercial cloud like AWS, GCP, or Azure is convenient because you can quickly scale out to many machines for just a few minutes, but also challenging because you need to navigate awkward cloud APIs, manage remote software environments with Docker, send data access credentials, make sure that costly resources are cleaned up, etc.  The following solutions help with this process.

- [**Coiled (recommended)**](https://docs.coiled.io/user_guide/index.html?utm_source=dask-docs&utm_medium=deploying): this commercial SaaS product handles most of the deployment pain Dask users
  encounter, is easy to use, and quite robust.  The free tier is large enough
  for most individual users, even for those who don’t want to engage with a
  commercial company.  The API looks like the following.
  ```python
  import coiled
  cluster = coiled.Cluster(
      n_workers=100,
      region="us-east-2",
      worker_memory="16 GiB",
      spot_policy="spot_with_fallback",
  )
  client = cluster.get_client()
  ```
- [Dask Cloud Provider](https://cloudprovider.dask.org/en/latest/): a pure and simple OSS solution that sets up Dask workers on cloud VMs, supporting AWS, GCP, Azure, and also other commercial clouds like Hetzner, Digital Ocean and Nebius.
- [Dask-Yarn](https://yarn.dask.org): deploys Dask on legacy YARN clusters, such as those that can be set up with AWS EMR or Google Cloud Dataproc.

See [Cloud](deploying-cloud.md) for more details.

## High Performance Computing

Dask runs on traditional HPC systems that use a resource manager like SLURM,
PBS, SGE, LSF, or similar systems, and a network file system.  This is an easy
way to dual-purpose large-scale hardware for analytics use cases.  Dask can
deploy either directly through the resource manager or through
`mpirun`/`mpiexec` and tends to use the NFS to distribute data and
software.

- [**Dask-Jobqueue (recommended)**](https://jobqueue.dask.org): interfaces directly with the
  resource manager (SLURM, PBS, SGE, LSF, and others) to launch many Dask
  workers as batch jobs.  It generates batch job scripts and submits them
  automatically to the user’s queue.  This approach operates entirely with user
  permissions (no IT support required) and enables interactive and adaptive use
  on large HPC systems.  It looks a little like the following:
  ```python
  from dask_jobqueue import PBSCluster
  cluster = PBSCluster(
      cores=24,
      memory="100GB",
      queue="regular",
      account="my-account",
  )
  cluster.scale(jobs=100)
  client = cluster.get_client()
  ```
- [Dask-MPI](http://mpi.dask.org/en/latest/): deploys Dask on top of any system that supports MPI using `mpirun`.  It is helpful for batch processing jobs where you want to ensure a fixed and stable number of workers.
- [Dask Gateway for Jobqueue](https://gateway.dask.org/install-jobqueue.html): Multi-tenant, secure clusters. Once configured, users can launch clusters without direct access to the underlying HPC backend.

See [High Performance Computers](deploying-hpc.md) for more details.

## Kubernetes

Dask runs natively on Kubernetes clusters.  This is a convenient choice when a
company already has dedicated Kubernetes infrastructure set up for running
other services.  When running Dask on Kubernetes users should also have a plan
to distribute software environments (probably with Docker), user credentials,
quota management, etc. In larger organizations with mature Kubernetes
deployments this is often handled by other Kubernetes services.

- [**Dask Kubernetes Operator (recommended)**](https://kubernetes.dask.org/en/latest/operator.html): The Dask Kubernetes
  Operator makes the most sense for fast moving or ephemeral deployments.  It
  is the most Kubernetes-native solution, and should be comfortable for K8s
  enthusiasts.  It looks a little like this:
  ```python
  from dask_kubernetes.operator import KubeCluster
  cluster = KubeCluster(
      name="my-dask-cluster",
      image="ghcr.io/dask/dask:latest",
      resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "64Gi"}},
  )
  cluster.scale(10)
  client = cluster.get_client()
  ```
- [Dask Gateway for Kubernetes](https://gateway.dask.org/install-kube.html): Multi-tenant, secure clusters. Once configured, users can launch clusters without direct access to the underlying Kubernetes backend.
- [Single Cluster Helm Chart](https://artifacthub.io/packages/helm/dask/dask): Single Dask cluster and (optionally) Jupyter on deployed with Helm.

See [Kubernetes](deploying-kubernetes.md) for more details.

<a id="managed-cluster-solutions"></a>

## Manual deployments (not recommended)

You can set up Dask clusters by hand, or with tools like SSH.

- [Manual Setup](deploying-cli.md): The command line interface to set up `dask-scheduler` and `dask-worker` processes.
- [SSH](deploying-ssh.md): Use SSH to set up Dask across an un-managed cluster.
- [Python API (advanced)](deploying-python-advanced.md): Create `Scheduler` and `Worker`   objects from Python as part of a distributed Tornado TCP application.

However, we don’t recommend this path.  Instead, we recommend that you use
some common resource manager to help you manage your machines, and then deploy
Dask on that system.  Those options are described above.

## Advanced Understanding

There are additional concepts to understand if you want to improve your
deployment. [This guide](deployment-considerations.md) covers the main topics to consider in addition to running Dask.


# deployment-considerations.html.md

# Deployment Considerations

Thanks to the efforts of the open-source community, there are tools to deploy Dask [pretty much anywhere](deploying.md#deployment-options)—if you can get computers to talk to each other, you can probably turn them into a Dask cluster.

**However, getting Dask running is often not the last step, but the first step.** This document attempts to cover some of the things *outside of Dask* you may have to think about when managing a Dask deployment.

These challenges are especially relevant when managing Dask for a team or organization, or transitioning into Dask for production (automated) use, but many will also come up for individual Dask users on distributed systems.

## Consistent software environments

For Dask to function properly, the same set of Python packages, at the same versions, need to be installed on the scheduler and workers as on the client. One of the most common stumbling points in deploying Dask on multiple machines is keeping what’s installed on the cluster up to date with what’s on the client, especially if they run on different systems (a laptop and a cloud provider, for example). For possible approaches, see [Manage Environments](software-environments.md).

Some ways of maintaining consistent environments may also require extra infrastructure. For example, using Docker images is common in cloud deployments, but then you need somewhere to store the images, such as [DockerHub](https://hub.docker.com/), [AWS ECR](https://aws.amazon.com/ecr/), [GCP Container Registry](https://cloud.google.com/container-registry), etc. As use matures, you’ll also want to version-control your Dockerfile and automatically build and publish new images when it changes, using a CI/CD system like [GitHub Actions](https://github.com/marketplace/actions/build-and-push-docker-images), [Google Cloud Build](https://cloud.google.com/build/docs/build-push-docker-image), or others.

Software environments can be particularly challenging when managing Dask for multiple users. Sometimes, using the same locked-down set of packages is sufficient for a team, but often individuals will need different packages, or different versions of the same package (including Dask itself!), to be productive. In these cases, you might end up giving end-users access to the build system (requiring them to build and maintain Docker images themselves, for example), or if that isn’t allowable in your organization, creating custom infrastructure, or resorting to workarounds like the [`PipInstall`](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.PipInstall) or [`UploadDirectory`](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.UploadDirectory) plugins.

Additional challenges can include getting local packages or scripts onto the cluster (and ensuring they’re up to date), as well as packages installed from private Git or PyPI repos.

Environment management options without additional infrastructure:

* [`PipInstall`](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.PipInstall) plugin
* [`UploadDirectory`](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.UploadDirectory) plugin
* Coiled’s [package sync](https://docs.coiled.io/user_guide/software/sync.html?utm_source=dask-docs&utm_medium=deployment-considerations) automatically replicates a local environment onto a cluster, including local packages and Git dependencies.

## Logging

When clusters break, logs may be your only tool to debug them. At a minimum, you should have a way to retain logs from all the machines in the cluster. You may also want to ingest these logs into a dedicated logging system, so you can search the logs, or view logs from multiple systems interleaved by time. (If deploying on a cloud provider, this might already be set up for you with the provider’s logging system, though be aware of potential charges for log storage.)

When managing dask for a team, you’ll also have to figure out how individual (and potentially less-technical) users can access logs and metrics for their own clusters. In a large organization, this may even include preventing users from accessing logs from other users’ clusters.

## Getting credentials on the cluster

Your laptop may have access to connect to an S3 bucket or a database, but depending on how your cluster is deployed, the workers may not. This can lead to errors where code that works locally fails with authentication errors when run on the cluster. The [Connect to remote data](how-to/connect-to-remote-data.md) page has more discussion of this.

Especially when used by teams, mature Dask deployments will want to avoid users passing their own credentials to the cluster directly in code, using strategies such as generating temporary tokens for users, authenticating workers under service accounts, etc.

## Controlling access

Most non-commercial deployment libraries rely on the user launching the cluster to have access to the underlying system the cluster will run on (such as a cloud provider, an HPC cluster, Kubernetes, etc.). When enabling a team to use Dask, this may not be the case: you want to let users launch clusters on demand, which might create cloud VMs or Kubernetes pods, without actually giving them permission to create cloud VMs or Kubernetes pods directly. [Dask-gateway](https://gateway.dask.org/) is a common way to do this, but that does require additional administration.

Most Dask deployment options set reasonable defaults for access (i.e. not making your cluster accessible to anyone on the Internet), but you still should make sure that your clusters (or your users’ clusters) aren’t accessible to unauthorized users.

Additionally, if you’re connecting to a cluster over the Internet, you should ensure that that connection is encrypted, since sensitive information, such as credentials or proprietary data, may flow over it. You might do this by port-forwarding your connection over SSH, using TLS, or using [Dask-gateway](https://gateway.dask.org/) or a commercial offering that manages this automatically.

## Controlling cost

It’s easy to forget to shut down a cluster and run up an expensive bill over the weekend. Some deployment libraries also may not always be able to fully clean up a cluster—for example, [dask-cloudprovider](https://cloudprovider.dask.org/) won’t clean up cloud resources if the client Python process (or machine!) shuts down abruptly.

Therefore, when launching clusters automatically in production, or enabling many team members to launch them, you should be confident that all resources will be cleaned up, or shut down if they exceed a cost threshold.

When managing Dask for a team, you may also want a way to limit how much individual users can spend, to prevent accidental overruns.

## Monitoring cost

It’s good to be able to answer questions such as:

- How much are we spending on Dask?
- What are we spending it on? (machines, machines that should have been turned off, network egress that shouldn’t have happened, etc.)
- Who/what is responsible?

Most deployment tools don’t build in this sort of monitoring. Organizations that need it either end up building their own tools, or turning to commercial deployment offerings.

## Managing networking

The Dask client needs to be able to talk to the scheduler, which is potentially on a different system. Users like to be able to access the [dashboard](dashboard.md) from a Web browser. The machines in the cluster need to be able to talk to each other. Typically, whatever [deployment system](deploying.md#deployment-options) you use will handle this for you. Sometimes, though, there can be additional considerations around what type of networking to use for best performance. Networking also can have costs associated—cloud providers may charge fixed or usage-based rates for certain types of networking configurations, for example.

You may also have other systems on restricted networks that workers need to access to read and write data, or call APIs. Connecting to those networks could add additional complexity.

Some organizations may have additional network security policies, such as requiring all traffic to be encrypted. Dask supports this with TLS. Some deployment systems enable this automatically using self-signed certificates; others may require additional configuration, especially if using certificates from your organization.

## Observability

The [dashboard](dashboard.md) is a powerful tool for monitoring live clusters. But once the cluster stops (or breaks), the dashboard is gone, so it’s invaluable for debugging to record information that lasts longer than the cluster. This is especially important when running automated jobs.

Dask provides [Prometheus metrics](prometheus.md), which offer close to dashboard-level detail, but can persist long after the cluster has shut down, making them especially valuable for monitoring and debugging production workarounds. They also can be aggregated, which is helpful when running many clusters at once, or even used to trigger automated alerts. Using these metrics requires deploying and managing [Prometheus](https://prometheus.io) (or a Prometheus-compatible service), configuring networking so it can access the machines in the cluster, and typically also deploying [Grafana](https://grafana.com/) to visualize metrics and create dashboards.

## Storing local data off the local machine

If you’re deploying Dask on a cluster, most data is probably already stored remotely, since a major reason for deploying Dask instead of [running locally](deploying.md#deployment-single-machine) is to run workers closer to the data. However, it can be common to also have some smaller, auxiliary data files locally.

In that case, you may need somewhere to store those auxiliary files remotely, where workers can access them. Depending on your deployment system, there are many options, from network file systems to cloud object stores like S3. Regardless, this can be another piece of infrastructure to manage and secure.

## Note on managed Dask offerings

As shown, setting up and managing a mature Dask deployment, especially for team or production use, can involve a fair amount of complexity outside of Dask itself. Addressing these challenges is generally out of scope for the open-source Dask deployment tools, but there are other projects as well as commercial Dask deployment services that handle many of these considerations. In alphabetical order:

- [Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=deployment-considerations) handles the creation and management of Dask clusters on cloud computing environments (AWS, Azure, and GCP).
- [Saturn Cloud](https://saturncloud.io/) lets users create Dask clusters in a hosted platform or within their own AWS accounts.


# develop.html.md

# Development Guidelines

Dask is a community maintained project.  We welcome contributions in the form
of bug reports, documentation, code, design proposals, and more.
This page provides resources on how best to contribute.

#### NOTE
Dask strives to be a welcoming community of individuals with diverse
backgrounds. For more information on our values, please see our
[code of conduct](https://github.com/dask/governance/blob/main/code-of-conduct.md)
and
[diversity statement](https://github.com/dask/governance/blob/main/diversity.md)

## Where to ask for help

Dask conversation happens in the following places:

1. [Dask Discourse forum](https://dask.discourse.group): for usage questions and general discussion
2. [Stack Overflow #dask tag](https://stackoverflow.com/questions/tagged/dask): for usage questions
3. [GitHub Issue Tracker](https://github.com/dask/dask/issues/): for discussions around new features or established bugs
4. [Dask Community Slack](https://join.slack.com/t/dask/shared_invite/zt-mfmh7quc-nIrXL6ocgiUH2haLYA914g): for real-time discussion

For usage questions and bug reports we prefer the use of Discourse, Stack Overflow
and GitHub issues over Slack chat.  Discourse, GitHub and Stack Overflow are more easily
searchable by future users, so conversations had there can be useful to many more people
than just those directly involved.

## Separate Code Repositories

Dask maintains code and documentation in a few git repositories hosted on the
GitHub `dask` organization, [https://github.com/dask](https://github.com/dask).  This includes the primary
repository and several other repositories for different components.  A
non-exhaustive list follows:

* [https://github.com/dask/dask](https://github.com/dask/dask): The main code repository holding parallel
  algorithms, the single-machine scheduler, and most documentation
* [https://github.com/dask/distributed](https://github.com/dask/distributed): The distributed memory scheduler
* [https://github.com/dask/dask-ml](https://github.com/dask/dask-ml): Machine learning algorithms
* [https://github.com/dask/s3fs](https://github.com/dask/s3fs): S3 Filesystem interface
* [https://github.com/dask/gcsfs](https://github.com/dask/gcsfs): GCS Filesystem interface
* [https://github.com/dask/hdfs3](https://github.com/dask/hdfs3): Hadoop Filesystem interface
* …

Git and GitHub can be challenging at first.  Fortunately good materials exist
on the internet.  Rather than repeat these materials here, we refer you to
pandas’ documentation and links on this subject at
[https://pandas.pydata.org/docs/dev/development/contributing.html](https://pandas.pydata.org/docs/dev/development/contributing.html)

## Issues

The community discusses and tracks known bugs and potential features in the
[GitHub Issue Tracker](https://github.com/dask/dask/issues/).  If you have a new idea or have identified a bug, then
you should raise it there to start public discussion.

If you are looking for an introductory issue to get started with development,
then check out the [“good first issue” label](https://github.com/dask/dask/labels/good%20first%20issue), which contains issues that are good
for starting developers.  Generally, familiarity with Python, NumPy, pandas, and
some parallel computing are assumed. These issues often spell out exactly what needs
to be done and are a great way to start to get familiar with the codebase and
contribution process. As these issues are intended to be learning oriented we ask
that you do not solve these with automated tools.

We strongly encourage discussion of issues before work is done on them. We generally follow
lazy consensus when implementing issues to avoid bottlenecks, but gathering some feedback and
giving opportunity for discussion is important. Iterating on a design before beginning
implementation can help save time when it comes to code review and make it more likely a
Pull Request will be accepted.

## Development Environment

### Download code

Make a fork of the main [Dask repository](https://github.com/dask/dask) and
clone the fork:

```default
git clone https://github.com/<your-github-username>/dask.git
cd dask
```

You should also pull the latest git tags (this ensures `pip`’s dependency resolver
can successfully install Dask):

```default
git remote add upstream https://github.com/dask/dask.git
git pull upstream main --tags
```

Contributions to Dask can then be made by submitting pull requests on GitHub.

<a id="develop-install"></a>

### Install

From the top level of your cloned Dask repository you can deploy and test a
local version of Dask, along with all necessary dependencies, using
[pixi](https://pixi.prefix.dev/).

Pixi uses lockfiles to freeze the installed version of all dependencies.
To update the lockfile:

```default
pixi update
```

### Run Tests

Dask uses [pytest](https://docs.pytest.org/en/latest/) for testing. You can run tests from the main dask directory
as follows:

```default
pixi run test
```

You can pass arbitrary pytest parameters to the command; e.g.:

```default
pixi run test dask/tests/test_base.py -k persist
```

pytest-xdist can be used to run tests in parallel:

```default
pixi run test -n auto
```

Test in parallel, with coverage, and including slow tests:

```default
pixi run test-ci
```

Generate a local coverage report after running `test-ci`:

```default
pixi run coverage html
```

Run doctests:

```default
pixi run doctest
```

There are several variant environments for testing, against obsolete but
still supported versions of dependencies, as well as against variant and
experimental configurations:

```default
pixi run -e mindeps-non-optional test-ci
pixi run -e mindeps-optional test-ci
pixi run -e mindeps-array test-ci
pixi run -e mindeps-dataframe test-ci
pixi run -e mindeps-distributed test-ci
pixi run -e py310 test-ci
pixi run -e py311 test-ci
pixi run -e py312 test-ci
pixi run -e py313 test-ci
pixi run -e py314 test-ci
pixi run -e py314t test-ci
pixi run -e nightly test-ci
```

Note that, besides Python versions, these variant environments also test a
matrix of different versions of NumPy, Pandas, and PyArrow. See `pixi.toml`
for details.

There are also specialty test tasks:

```default
pixi run test-spark
pixi run -e <any environment with NumPy> test-array-expr
```

## Contributing to Code

Dask maintains development standards that are similar to most PyData projects. These
standards include language support, testing, documentation, and style.

### Python Versions

Dask supports Python versions 3.10 to 3.14.
Name changes are handled by the `dask/compatibility.py` file.

<a id="develop-test"></a>

### Test

Dask employs extensive unit tests to ensure correctness of code both for today
and for the future.  Test coverage is expected for all code contributions.

Tests are written in a py.test style with bare functions:

```python
def test_fibonacci():
    assert fib(0) == 0
    assert fib(1) == 0
    assert fib(10) == 55
    assert fib(8) == fib(7) + fib(6)

    for x in [-3, 'cat', 1.5]:
        with pytest.raises(ValueError):
            fib(x)
```

These tests should compromise well between covering all branches and fail cases
and running quickly (slow test suites get run less often).

Tests run automatically on GitHub Actions on every push to every pull
request on GitHub.

Tests are organized within the various modules’ subdirectories:

```default
dask/array/tests/test_*.py
dask/bag/tests/test_*.py
dask/bytes/tests/test_*.py
dask/dataframe/tests/test_*.py
dask/diagnostics/tests/test_*.py
```

For the Dask collections like Dask Array and Dask DataFrame, behavior is
typically tested directly against the NumPy or pandas libraries using the
`assert_eq` functions:

```python
import numpy as np
import dask.array as da
from dask.array.utils import assert_eq

def test_aggregations():
    rng = np.random.default_rng()
    nx = rng.random(100)
    dx = da.from_array(nx, chunks=(10,))

    assert_eq(nx.sum(), dx.sum())
    assert_eq(nx.min(), dx.min())
    assert_eq(nx.max(), dx.max())
    ...
```

This technique helps to ensure compatibility with upstream libraries and tends
to be simpler than testing correctness directly.  Additionally, by passing Dask
collections directly to the `assert_eq` function rather than call compute
manually, the testing suite is able to run a number of checks on the lazy
collections themselves.

### Docstrings

User facing functions should roughly follow the [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard) standard, including
sections for `Parameters`, `Examples`, and general explanatory prose.

By default, examples will be doc-tested.  Reproducible examples in documentation
is valuable both for testing and, more importantly, for communication of common
usage to the user.  Documentation trumps testing in this case and clear
examples should take precedence over using the docstring as testing space.
To skip a test in the examples add the comment `# doctest: +SKIP` directly
after the line.

```python
def fib(i):
    """ A single line with a brief explanation

    A more thorough description of the function, consisting of multiple
    lines or paragraphs.

    Parameters
    ----------
    i: int
         A short description of the argument if not immediately clear

    Examples
    --------
    >>> fib(4)
    3
    >>> fib(5)
    5
    >>> fib(6)
    8
    >>> fib(-1)  # Robust to bad inputs
    ValueError(...)
    """
```

Docstrings are tested under Python 3.14 on GitHub Actions. You can test
docstrings with pytest as follows:

```default
py.test dask --doctest-modules
```

Docstring testing requires `graphviz` to be installed. This can be done via:

```default
conda install -y graphviz
```

### Code Formatting

Dask uses several code linters (ruff, black, mypy), which are enforced by CI. Developers
should run them locally before they submit a PR, through the single command:

```default
pixi run lint
```

This makes sure that linter versions and options are aligned for all developers.

Optionally, you may wish to setup the [pre-commit hooks](https://pre-commit.com/) to
run automatically when you make a git commit. This can be done by running:

```default
pixi run -e lint pre-commit install
```

from the root of the Dask repository. Now the code linters will be run each time you
commit changes. You can skip these checks with `git commit --no-verify` or with the
short version `git commit -n`.

## Contributing to Documentation

Dask uses [Sphinx](https://www.sphinx-doc.org/) for documentation, hosted on [https://readthedocs.org](https://readthedocs.org) .
Documentation is maintained in the RestructuredText markup language (`.rst`
files) in `dask/docs/source`.  The documentation consists both of prose
and API documentation.

The documentation is automatically built, and a live preview is available,
for each pull request submitted to Dask. Additionally, you may also
build the documentation yourself locally by following the instructions outlined
below.

### How to build the Dask documentation

To build the documentation locally, make a fork of the main
[Dask repository](https://github.com/dask/dask), clone the fork:

```default
git clone https://github.com/<your-github-username>/dask.git
cd dask/docs
```

Install the packages in `requirements-docs.txt`.

Optionally create and activate a `conda` environment first:

```default
conda create -n daskdocs -c conda-forge python=3.14
conda activate daskdocs
```

Install the dependencies with `pip`:

```default
python -m pip install -r requirements-docs.txt
```

Then build the documentation with `make`:

```default
make html
```

The resulting HTML files end up in the `build/html` directory.

You can now make edits to rst files and run `make html` again to update
the affected pages.

## Dask CI Infrastructure

### Github Actions

Dask uses Github Actions for Continuous Integration (CI) testing for each PR.
These CI builds will run the test suite across a variety of Python versions, operating
systems, and package dependency versions.

The CI workflows for Github Actions are defined in
[.github/workflows](https://github.com/dask/dask/tree/main/.github/workflows)
with additional scripts and metadata located in [continuous_integration](https://github.com/dask/dask/tree/main/continuous_integration).
CI is heavily driven by pixi, which is configured by
[pixi.toml](https://github.com/dask/dask/blob/main/pixi.toml).

## Making Pull Requests

### Pull Request Etiquette

When opening a Pull Request you are beginning a dialog with maintainers. This is a bidirectional
relationship where you are asking for the reviewer’s time to look at your contribution, and
the reviewer will likely ask for your input and engage you in discussion around the changes.

Please do not propose code that you are not willing to stand behind and discuss.
Be prepared to respond to review feedback, apply critical thinking and iterate on your contributions.

We ask that you fill out all sections of PR templates and provide reasoning behind your changes,
ideally with a linked issue that has been discussed by the community.

### Automated Contributions and AI Policy

We encourage the use of AI and automated tools to assist in code development,
documentation, and testing. However, we ask that contributors disclose these tools and
use them in a way that aligns with Dask’s community guidelines. In particular:

- do not use tools to think or speak for you in discussions, code reviews, or any other
  interactions within the Dask community.
- Before you open a PR, you (the human) must fully review, understand, and approve
  everything that the AI agent wrote.


# diagnostics-distributed.html.md

# Diagnostics (distributed)

The [Dask distributed scheduler](scheduling.md) provides live feedback in two
forms:

1. An interactive dashboard containing many plots and tables with live
   information
2. A progress bar suitable for interactive use in consoles or notebooks

## Dashboard

For information on the Dask dashboard see [Dashboard Diagnostics](dashboard.md).

## Capture diagnostics

| [`get_task_stream`](#dask.distributed.get_task_stream)([client, plot, filename])   | Collect task stream within a context block                  |
|------------------------------------------------------------------------------------|-------------------------------------------------------------|
| `Client.profile`([key, start, stop, workers, ...])                                 | Collect statistical profiling information about recent work |
| `performance_report`([filename, stacklevel, ...])                                  | Gather performance report                                   |

You can capture some of the same information that the dashboard presents for
offline processing using the `get_task_stream` and `Client.profile`
functions.  These capture the start and stop time of every task and transfer,
as well as the results of a statistical profiler.

```python
with get_task_stream(plot='save', filename="task-stream.html") as ts:
    x.compute()

client.profile(filename="dask-profile.html")

history = ts.data
```

Additionally, Dask can save many diagnostics dashboards at once including the
task stream, worker profiles, bandwidths, etc. with the `performance_report`
context manager:

```python
from dask.distributed import performance_report

with performance_report(filename="dask-report.html"):
    ## some dask computation
```

The following video demonstrates the `performance_report` context manager in greater
detail:

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/nTMGbkS761Q"
        frameborder="0"
        allow="autoplay; encrypted-media"
        allowfullscreen>
</iframe>

## Progress bar

| [`progress`](#dask.distributed.progress)(\*futures[, notebook, multi, ...])   | Track progress of futures   |
|-------------------------------------------------------------------------------|-----------------------------|

The `dask.distributed` progress bar differs from the `ProgressBar` used for
[local diagnostics](diagnostics-local.md).
The `progress` function takes a Dask object that is executing in the background:

```python
# Progress bar on a single-machine scheduler
from dask.diagnostics import ProgressBar

with ProgressBar():
    x.compute()

# Progress bar with the distributed scheduler
from dask.distributed import Client, progress

client = Client()  # use dask.distributed by default

x = x.persist()  # start computation in the background
progress(x)      # watch progress

x.compute()      # convert to final result when done if desired
```

## Connecting to the Dashboard

Some computer networks may restrict access to certain ports or only allow
access from certain machines.  If you are unable to access the dashboard then
you may want to contact your IT administrator.

Some common problems and solutions follow:

### Specify an accessible port

Some clusters restrict the ports that are visible to the outside world.  These
ports may include the default port for the web interface, `8787`.  There are
a few ways to handle this:

1. Open port `8787` to the outside world.  Often this involves asking your
   cluster administrator.
2. Use a different port that is publicly accessible using the
   `--dashboard-address :8787` option on the `dask-scheduler` command.
3. Use fancier techniques, like [Port Forwarding]()

### Port Forwarding

If you have SSH access then one way to gain access to a blocked port is through
SSH port forwarding. A typical use case looks like the following:

```bash
local$ ssh -L 8000:localhost:8787 user@remote
remote$ dask-scheduler  # now, the web UI is visible at localhost:8000
remote$ # continue to set up dask if needed -- add workers, etc
```

It is then possible to go to `localhost:8000` and see Dask Web UI. This same approach is
not specific to dask.distributed, but can be used by any service that operates over a
network, such as Jupyter notebooks. For example, if we chose to do this we could
forward port 8888 (the default Jupyter port) to port 8001 with
`ssh -L 8001:localhost:8888 user@remote`.

### Required Packages

Bokeh must be installed in your scheduler’s environment to run the dashboard. If it’s not the dashboard page will instruct you to install it.

Depending on your configuration, you might also need to install `jupyter-server-proxy` to access the dashboard.

## API

### dask.distributed.progress(\*futures, notebook=None, multi=True, complete=True, group_by='prefix', \*\*kwargs)

Track progress of futures

This operates differently in the notebook and the console

* Notebook:  This returns immediately, leaving an IPython widget on screen
* Console:  This blocks until the computation completes

* **Parameters:**
  **futures**
  : A list of futures or keys to track

  **notebook**
  : Running in the notebook or not (defaults to guess)

  **multi**
  : Track different functions independently (defaults to True)

  **complete**
  : Track all keys (True) or only keys that have not yet run (False)
    (defaults to True)

  **group_by**
  : Use spans instead of task key names for grouping tasks
    (defaults to “prefix”)

### Notes

In the notebook, the output of progress must be the last statement
in the cell. Typically, this means calling progress at the end of a
cell.

### Examples

```pycon
>>> progress(futures)
[########################################] | 100% Completed |  1.7s
```

<!-- !! processed by numpydoc !! -->

### dask.distributed.get_task_stream(client: [Client](futures.md#distributed.Client) | [None](https://docs.python.org/3/library/constants.html#None) = None, plot: [bool](https://docs.python.org/3/library/functions.html#bool) | Literal['save'] = False, filename: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'task-stream.html')

Collect task stream within a context block

This provides diagnostic information about every task that was run during
the time when this block was active.

This must be used as a context manager.

* **Parameters:**
  **plot: boolean, str**
  : If true then also return a Bokeh figure
    If plot == ‘save’ then save the figure to a file

  **filename: str (optional)**
  : The filename to save to if you set `plot='save'`

#### SEE ALSO
`Client.get_task_stream`
: Function version of this context manager

### Examples

```pycon
>>> with get_task_stream() as ts:
...     x.compute()
>>> ts.data
[...]
```

Get back a Bokeh figure and optionally save to a file

```pycon
>>> with get_task_stream(plot='save', filename='task-stream.html') as ts:
...    x.compute()
>>> ts.figure
<Bokeh Figure>
```

To share this file with others you may wish to upload and serve it online.
A common way to do this is to upload the file as a gist, and then serve it
on [https://raw.githack.com](https://raw.githack.com)

```default
$ python -m pip install gist
$ gist task-stream.html
https://gist.github.com/8a5b3c74b10b413f612bb5e250856ceb
```

You can then navigate to that site, click the “Raw” button to the right of
the `task-stream.html` file, and then provide that URL to
[https://raw.githack.com](https://raw.githack.com) .  This process should provide a sharable link that
others can use to see your task stream plot.

<!-- !! processed by numpydoc !! -->


# diagnostics-local.html.md

# Diagnostics (local)

Profiling parallel code can be challenging, but `dask.diagnostics` provides
functionality to aid in profiling and inspecting execution with the
[local task scheduler](scheduling.md).

This page describes the following few built-in options:

1. ProgressBar
2. Profiler
3. ResourceProfiler
4. CacheProfiler

Furthermore, this page then provides instructions on how to build your own custom diagnostic.

## Progress Bar

| [`ProgressBar`](#dask.diagnostics.ProgressBar)([minimum, width, dt, out])   | A progress bar for dask.   |
|-----------------------------------------------------------------------------|----------------------------|

The `ProgressBar` class builds on the scheduler callbacks described above to
display a progress bar in the terminal or notebook during computation. This can
give a nice feedback during long running graph execution. It can be used as a
context manager around calls to `get` or `compute` to profile the
computation:

```python
>>> import dask.array as da
>>> from dask.diagnostics import ProgressBar
>>> a = da.random.default_rng().normal(size=(10000, 10000), chunks=(1000, 1000))
>>> res = a.dot(a.T).mean(axis=0)

>>> with ProgressBar():
...     out = res.compute()
[########################################] | 100% Completed | 17.1 s
```

or registered globally using the `register` method:

```python
>>> pbar = ProgressBar()
>>> pbar.register()
>>> out = res.compute()
[########################################] | 100% Completed | 17.1 s
```

To unregister from the global callbacks, call the `unregister` method:

```python
>>> pbar.unregister()
```

## Profiler

| [`Profiler`](#dask.diagnostics.Profiler)()   | A profiler for dask execution at the task level.   |
|----------------------------------------------|----------------------------------------------------|

Dask provides a few tools for profiling execution. As with the `ProgressBar`,
they each can be used as context managers or registered globally.

The `Profiler` class is used to profile Dask’s execution at the task level.
During execution, it records the following information for each task:

1. Key
2. Task
3. Start time in seconds since the epoch
4. Finish time in seconds since the epoch
5. Worker id

## ResourceProfiler

| [`ResourceProfiler`](#dask.diagnostics.ResourceProfiler)([dt])   | A profiler for resource use.   |
|------------------------------------------------------------------|--------------------------------|

The `ResourceProfiler` class is used to profile Dask’s execution at the
resource level. During execution, it records the following information
for each timestep:

1. Time in seconds since the epoch
2. Memory usage in MB
3. % CPU usage

The default timestep is 1 second, but can be set manually using the `dt`
keyword:

```python
>>> from dask.diagnostics import ResourceProfiler
>>> rprof = ResourceProfiler(dt=0.5)
```

## CacheProfiler

| [`CacheProfiler`](#dask.diagnostics.CacheProfiler)([metric, metric_name])   | A profiler for dask execution at the scheduler cache level.   |
|-----------------------------------------------------------------------------|---------------------------------------------------------------|

The `CacheProfiler` class is used to profile Dask’s execution at the scheduler
cache level. During execution, it records the following information for each
task:

1. Key
2. Task
3. Size metric
4. Cache entry time in seconds since the epoch
5. Cache exit time in seconds since the epoch

Here the size metric is the output of a function called on the result of each
task. The default metric is to count each task (`metric` is 1 for all tasks).
Other functions may be used as a metric instead through the `metric` keyword.
For example, the `nbytes` function found in `cachey` can be used to measure
the number of bytes in the scheduler cache:

```python
>>> from dask.diagnostics import CacheProfiler
>>> from cachey import nbytes
>>> cprof = CacheProfiler(metric=nbytes)
```

## Example

As an example to demonstrate using the diagnostics, we’ll profile some linear
algebra done with Dask Array. We’ll create a random array, take its QR
decomposition, and then reconstruct the initial array by multiplying the Q and
R components together. Note that since the profilers (and all diagnostics) are
just context managers, multiple profilers can be used in a with block:

```python
>>> import dask.array as da
>>> from dask.diagnostics import Profiler, ResourceProfiler, CacheProfiler
>>> a = da.random.random(size=(10000, 1000), chunks=(1000, 1000))
>>> q, r = da.linalg.qr(a)
>>> a2 = q.dot(r)

>>> with Profiler() as prof, ResourceProfiler(dt=0.25) as rprof,
...         CacheProfiler() as cprof:
...     out = a2.compute()
```

The results of each profiler are stored in their `results` attribute as a
list of `namedtuple` objects:

```python
>>> prof.results[0]
TaskData(key=('tsqr-8d16e396b237bf7a731333130d310cb9_QR_st1', 5, 0),
         task=(qr, (_apply_random, 'random_sample', 1060164455, (1000, 1000), (), {})),
         start_time=1454368444.493292,
         end_time=1454368444.902987,
         worker_id=4466937856)

>>> rprof.results[0]
ResourceData(time=1454368444.078748, mem=74.100736, cpu=0.0)

>>> cprof.results[0]
CacheData(key=('tsqr-8d16e396b237bf7a731333130d310cb9_QR_st1', 7, 0),
          task=(qr, (_apply_random, 'random_sample', 1310656009, (1000, 1000), (), {})),
          metric=1,
          cache_time=1454368444.49662,
          free_time=1454368446.769452)
```

These can be analyzed separately or viewed in a bokeh plot using the provided
`visualize` method on each profiler:

```python
>>> prof.visualize()
```

<iframe src="_static/profile.html"
        marginwidth="0" marginheight="0" scrolling="no"
        width="650" height="300" style="border:none"></iframe>

To view multiple profilers at the same time, the [`dask.diagnostics.visualize()`](#dask.diagnostics.visualize)
function can be used. This takes a list of profilers and creates a vertical
stack of plots aligned along the x-axis:

```python
>>> from dask.diagnostics import visualize
>>> visualize([prof, rprof, cprof])
```

<iframe src="_static/stacked_profile.html"
        marginwidth="0" marginheight="0" scrolling="no"
        width="650" height="700" style="border:none"></iframe>

Looking at the above figure, from top to bottom:

1. The results from the `Profiler` object: This shows the execution time for
   each task as a rectangle, organized along the y-axis by worker (in this case
   threads). Similar tasks are grouped by color and, by hovering over each task,
   one can see the key and task that each block represents.
2. The results from the `ResourceProfiler` object: This shows two lines, one
   for total CPU percentage used by all the workers, and one for total memory
   usage.
3. The results from the `CacheProfiler` object: This shows a line for each
   task group, plotting the sum of the current `metric` in the cache against
   time. In this case it’s the default metric (count) and the lines represent
   the number of each object in the cache at time. Note that the grouping and
   coloring is the same as for the `Profiler` plot, and that the task
   represented by each line can be found by hovering over the line.

From these plots we can see that the initial tasks (calls to
`numpy.random.random` and `numpy.linalg.qr` for each chunk) are run
concurrently, but only use slightly more than 100% CPU. This is because the
call to `numpy.linalg.qr` currently doesn’t release the Global Interpreter
Lock (GIL), so those calls can’t truly be done in parallel. Next, there’s a reduction
step where all the blocks are combined. This requires all the results from the
first step to be held in memory, as shown by the increased number of results in
the cache, and increase in memory usage. Immediately after this task ends, the
number of elements in the cache decreases, showing that they were only needed
for this step. Finally, there’s an interleaved set of calls to `dot` and
`sum`. Looking at the CPU plot, it shows that these run both concurrently and in
parallel, as the CPU percentage spikes up to around 350%.

## Custom Callbacks

| [`Callback`](#dask.diagnostics.Callback)([start, start_state, pretask, ...])   | Base class for using the callback mechanism   |
|--------------------------------------------------------------------------------|-----------------------------------------------|

Schedulers based on `dask.local.get_async` (currently
`dask.get`, `dask.threaded.get`, and `dask.multiprocessing.get`)
accept five callbacks, allowing for inspection of scheduler execution.

The callbacks are:

1. `start(dsk)`: Run at the beginning of execution, right before the
state is initialized.  Receives the Dask graph

2. `start_state(dsk, state)`: Run at the beginning of execution, right
after the state is initialized.  Receives the Dask graph and scheduler state

3. `pretask(key, dsk, state)`: Run every time a new task is started.
Receives the key of the task to be run, the Dask graph, and the scheduler state

4. `posttask(key, result, dsk, state, id)`: Run every time a task is finished.
Receives the key of the task that just completed, the result, the Dask graph,
the scheduler state, and the id of the worker that ran the task

5. `finish(dsk, state, errored)`: Run at the end of execution, right before the
result is returned. Receives the Dask graph, the scheduler state, and a boolean
indicating whether or not the exit was due to an error

Custom diagnostics can be created either by instantiating the `Callback`
class with the some of the above methods as keywords or by subclassing the
`Callback` class.
Here we create a class that prints the name of every key as it’s computed:

```python
from dask.callbacks import Callback
class PrintKeys(Callback):
    def _pretask(self, key, dask, state):
        """Print the key of every task as it's started"""
        print("Computing: {0}!".format(repr(key)))
```

This can now be used as a context manager during computation:

```python
>>> from operator import add, mul
>>> dsk = {'a': (add, 1, 2), 'b': (add, 3, 'a'), 'c': (mul, 'a', 'b')}

>>> with PrintKeys():
...     get(dsk, 'c')
Computing 'a'!
Computing 'b'!
Computing 'c'!
```

Alternatively, functions may be passed in as keyword arguments to `Callback`:

```python
>>> def printkeys(key, dask, state):
...    print("Computing: {0}!".format(repr(key)))

>>> with Callback(pretask=printkeys):
...     get(dsk, 'c')
Computing 'a'!
Computing 'b'!
Computing 'c'!
```

## API

| [`CacheProfiler`](#dask.diagnostics.CacheProfiler)([metric, metric_name])          | A profiler for dask execution at the scheduler cache level.   |
|------------------------------------------------------------------------------------|---------------------------------------------------------------|
| [`Callback`](#dask.diagnostics.Callback)([start, start_state, pretask, ...])       | Base class for using the callback mechanism                   |
| [`Profiler`](#dask.diagnostics.Profiler)()                                         | A profiler for dask execution at the task level.              |
| [`ProgressBar`](#dask.diagnostics.ProgressBar)([minimum, width, dt, out])          | A progress bar for dask.                                      |
| [`ResourceProfiler`](#dask.diagnostics.ResourceProfiler)([dt])                     | A profiler for resource use.                                  |
| [`visualize`](#dask.diagnostics.visualize)(profilers[, filename, show, save, ...]) | Visualize the results of profiling in a bokeh plot.           |

### dask.diagnostics.ProgressBar(minimum=0, width=40, dt=0.1, out=None)

A progress bar for dask.

* **Parameters:**
  **minimum**
  : Minimum time threshold in seconds before displaying a progress bar.
    Default is 0 (always display)

  **width**
  : Width of the bar

  **dt**
  : Update resolution in seconds, default is 0.1 seconds

  **out**
  : File object to which the progress bar will be written
    It can be `sys.stdout`, `sys.stderr` or any other file object able to write `str` objects
    Default is `sys.stdout`

### Examples

Below we create a progress bar with a minimum threshold of 1 second before
displaying. For cheap computations nothing is shown:

```pycon
>>> with ProgressBar(minimum=1.0):
...     out = some_fast_computation.compute()
```

But for expensive computations a full progress bar is displayed:

```pycon
>>> with ProgressBar(minimum=1.0):
...     out = some_slow_computation.compute()
[########################################] | 100% Completed | 10.4 s
```

The duration of the last computation is available as an attribute

```pycon
>>> pbar = ProgressBar()
>>> with pbar:
...     out = some_computation.compute()
[########################################] | 100% Completed | 10.4 s
>>> pbar.last_duration
10.4
```

You can also register a progress bar so that it displays for all
computations:

```pycon
>>> pbar = ProgressBar()
>>> pbar.register()
>>> some_slow_computation.compute()
[########################################] | 100% Completed | 10.4 s
```

<!-- !! processed by numpydoc !! -->

### dask.diagnostics.Profiler()

A profiler for dask execution at the task level.

Records the following information for each task:
: 1. Key
  2. Task
  3. Start time in seconds since the epoch
  4. Finish time in seconds since the epoch
  5. Worker id

### Examples

```pycon
>>> from operator import add, mul
>>> from dask.threaded import get
>>> from dask.diagnostics import Profiler
>>> dsk = {'x': 1, 'y': (add, 'x', 10), 'z': (mul, 'y', 2)}
>>> with Profiler() as prof:
...     get(dsk, 'z')
22
```

```pycon
>>> prof.results
[TaskData(key='y', task=(add, 'x', 10), start_time=..., end_time=..., worker_id=...),
 TaskData(key='z', task=(mul, 'y', 2), start_time=..., end_time=..., worker_id=...)]
```

These results can be visualized in a bokeh plot using the `visualize`
method. Note that this requires bokeh to be installed.

```pycon
>>> prof.visualize()
```

You can activate the profiler globally

```pycon
>>> prof.register()
```

If you use the profiler globally you will need to clear out old results
manually.

```pycon
>>> prof.clear()
>>> prof.unregister()
```

<!-- !! processed by numpydoc !! -->

### dask.diagnostics.ResourceProfiler(dt=1)

A profiler for resource use.

Records the following each timestep
: 1. Time in seconds since the epoch
  2. Memory usage in MB
  3. % CPU usage

### Examples

```pycon
>>> from operator import add, mul
>>> from dask.threaded import get
>>> dsk = {'x': 1, 'y': (add, 'x', 10), 'z': (mul, 'y', 2)}
>>> with ResourceProfiler() as prof:
...     get(dsk, 'z')
22
```

These results can be visualized in a bokeh plot using the `visualize`
method. Note that this requires bokeh to be installed.

```pycon
>>> prof.visualize()
```

You can activate the profiler globally

```pycon
>>> prof.register()
```

If you use the profiler globally you will need to clear out old results
manually.

```pycon
>>> prof.clear()
```

Note that when used as a context manager data will be collected throughout
the duration of the enclosed block. In contrast, when registered globally
data will only be collected while a dask scheduler is active.

```pycon
>>> prof.unregister()
```

<!-- !! processed by numpydoc !! -->

### dask.diagnostics.CacheProfiler(metric=None, metric_name=None)

A profiler for dask execution at the scheduler cache level.

Records the following information for each task:
: 1. Key
  2. Task
  3. Size metric
  4. Cache entry time in seconds since the epoch
  5. Cache exit time in seconds since the epoch

### Examples

```pycon
>>> from operator import add, mul
>>> from dask.threaded import get
>>> from dask.diagnostics import CacheProfiler
>>> dsk = {'x': 1, 'y': (add, 'x', 10), 'z': (mul, 'y', 2)}
>>> with CacheProfiler() as prof:
...     get(dsk, 'z')
22
```

```pycon
>>> prof.results
[CacheData(key='y', task=(add, 'x', 10), metric=1, cache_time=..., free_time=...),
 CacheData(key='z', task=(mul, 'y', 2), metric=1, cache_time=..., free_time=...)]
```

The default is to count each task (`metric` is 1 for all tasks). Other
functions may used as a metric instead through the `metric` keyword. For
example, the `nbytes` function found in `cachey` can be used to measure
the number of bytes in the cache.

```pycon
>>> from cachey import nbytes
>>> with CacheProfiler(metric=nbytes) as prof:
...     get(dsk, 'z')
22
```

The profiling results can be visualized in a bokeh plot using the
`visualize` method. Note that this requires bokeh to be installed.

```pycon
>>> prof.visualize()
```

You can activate the profiler globally

```pycon
>>> prof.register()
```

If you use the profiler globally you will need to clear out old results
manually.

```pycon
>>> prof.clear()
>>> prof.unregister()
```

<!-- !! processed by numpydoc !! -->

### dask.diagnostics.Callback(start=None, start_state=None, pretask=None, posttask=None, finish=None)

Base class for using the callback mechanism

Create a callback with functions of the following signatures:

```pycon
>>> def start(dsk):
...     pass
>>> def start_state(dsk, state):
...     pass
>>> def pretask(key, dsk, state):
...     pass
>>> def posttask(key, result, dsk, state, worker_id):
...     pass
>>> def finish(dsk, state, failed):
...     pass
```

You may then construct a callback object with any number of them

```pycon
>>> cb = Callback(pretask=pretask, finish=finish)
```

And use it either as a context manager over a compute/get call

```pycon
>>> with cb:
...     x.compute()
```

Or globally with the `register` method

```pycon
>>> cb.register()
>>> cb.unregister()
```

Alternatively subclass the `Callback` class with your own methods.

```pycon
>>> class PrintKeys(Callback):
...     def _pretask(self, key, dask, state):
...         print("Computing: {0}!".format(repr(key)))
```

```pycon
>>> with PrintKeys():
...     x.compute()
```

<!-- !! processed by numpydoc !! -->

### dask.diagnostics.visualize(profilers, filename='profile.html', show=True, save=None, mode=None, \*\*kwargs)

Visualize the results of profiling in a bokeh plot.

If multiple profilers are passed in, the plots are stacked vertically.

* **Parameters:**
  **profilers**
  : Profiler or list of profilers.

  **filename**
  : Name of the plot output file.

  **show**
  : If True (default), the plot is opened in a browser.

  **save**
  : If True (default when not in notebook), the plot is saved to disk.

  **mode**
  : Mode passed to bokeh.output_file()

  **\*\*kwargs**
  : Other keyword arguments, passed to bokeh.figure. These will override
    all defaults set by visualize.
* **Returns:**
  The completed bokeh plot object.

<!-- !! processed by numpydoc !! -->


# ecosystem.html.md

<!-- this page is referenced from the topbar which comes from the theme -->

# Ecosystem

There are a number of open source projects that extend the Dask interface and provide different
mechanisms for deploying Dask clusters. This is likely an incomplete list so if you spot something
missing - please [suggest a fix](https://github.com/dask/dask/edit/main/docs/source/ecosystem.rst)!

## Building on Dask

Many packages include built-in support for Dask collections or wrap Dask collections internally
to enable parallelization.

### Array

- [xarray](https://xarray.pydata.org):  Wraps Dask
  Array, offering the same scalability, but with axis labels which add convenience when
  dealing with complex datasets.
- [cupy](https://docs.cupy.dev/en/stable): Part of the Rapids project, GPU-enabled arrays
  can be used as the blocks of Dask Arrays. See the section [GPUs](gpu.md) for more information.
- [sparse](https://github.com/pydata/sparse): Implements sparse arrays of arbitrary dimension
  on top of `numpy` and `scipy.sparse`.
- [pint](https://pint.readthedocs.io): Allows arithmetic operations between them and conversions
  from and to different units.
- [HyperSpy](https://hyperspy.org): Uses dask to allow for scalability on multi-dimensional datasets
  where navigation and signal axes can be separated (e.g. hyperspectral images).

### DataFrame

- [cudf](https://docs.rapids.ai/api/cudf/stable/): Part of the Rapids project, implements
  GPU-enabled dataframes which can be used as partitions in Dask Dataframes.
- [dask-geopandas](https://github.com/geopandas/dask-geopandas): Early-stage subproject of
  geopandas, enabling parallelization of geopandas dataframes.

### SQL

- [blazingSQL](https://docs.blazingsql.com/): Part of the Rapids project, implements SQL queries using `cuDF`
  and Dask, for execution on CUDA/GPU-enabled hardware, including referencing
  externally-stored data.
- [dask-sql](https://dask-sql.readthedocs.io/en/latest/): Adds a SQL query layer on top of Dask.
  The API matches blazingSQL but it uses CPU instead of GPU. It still under development
  and not ready for a production use-case.
- [fugue-sql](https://fugue-tutorials.readthedocs.io/en/latest/tutorials/fugue_sql/index.html): Adds an abstract layer that makes code portable between across differing
  computing frameworks such as Pandas, Spark and Dask.

### Machine Learning

- [dask-ml](https://ml.dask.org): Implements distributed versions of common machine learning algorithms.
- [scikit-learn](https://scikit-learn.org/stable/): Provide ‘dask’ to the joblib backend to parallelize
  scikit-learn algorithms with dask as the processor.
- [xgboost](https://xgboost.readthedocs.io): Powerful and popular library for gradient boosted trees;
  includes native support for distributed training using dask.
- [lightgbm](https://lightgbm.readthedocs.io): Similar to XGBoost; lightgbm also natively supplies native
  distributed training for decision trees.

## Deploying Dask

There are many different implementations of the Dask distributed cluster.

- [dask-jobqueue](https://jobqueue.dask.org): Deploy Dask on job queuing systems like PBS, Slurm, MOAB, SGE, LSF, and HTCondor.
- [dask-kubernetes](https://kubernetes.dask.org): Deploy Dask workers on Kubernetes from within a Python script or interactive session.
- [dask-helm](https://helm.dask.org): Deploy Dask and (optionally) Jupyter or JupyterHub on Kubernetes easily using Helm.
- [dask-yarn / Hadoop](https://yarn.dask.org): Deploy Dask on YARN clusters, such as are found in traditional Hadoop
  installations.
- [dask-cloudprovider](https://cloudprovider.dask.org): Deploy Dask on various cloud platforms such as AWS, Azure, and GCP
  leveraging cloud native APIs.
- [dask-gateway](https://gateway.dask.org): Secure, multi-tenant server for managing Dask clusters. Launch and use Dask
  clusters in a shared, centrally managed cluster environment, without requiring users to have direct access to the underlying
  cluster backend.
- [dask-cuda](https://github.com/rapidsai/dask-cuda): Construct a Dask cluster which resembles `LocalCluster`  and is specifically
  optimized for GPUs.

### Commercial Dask Deployment Options

- You can use [Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=ecosystem) to handle the creation and management of Dask clusters on cloud computing environments (AWS and GCP).


# expr-system-internals.html.md

# Query planning with Expression system

#### NOTE
This document is intended for Dask developers and contributors. It is not
intended for end-users.
For a high level user guide, see [Optimizer](dataframe-optimizer.md).

The expression system was originally developed for Dask DataFrames, as implemented in the [dask-expr](https://github.com/dask/dask-expr) project.

## Expr objects

The expression system is built around the Expr class. This class is used to represent a computation that can be performed on a Dask DataFrame. The Expr class is designed to be subclassed, and each subclass represents a specific type of computation. For example, there are subclasses for arithmetic operations, logical operations, and so on.

### Construction

The expression system centers around the Expr class, which represents a computation on a Dask DataFrame. This class is designed for subclassing; each subclass corresponds to a specific computation type (e.g., arithmetic, logical operations, filtering, joins).

Notably, custom initializers (`__init__`) are disallowed, both in the base class and its subclasses. This design decision reflects concerns around performance, as expression objects may be created and recreated frequently, and custom logic in constructors could introduce unnecessary overhead.

Instead, expression classes use a dataclass-like interface defined by two attributes:

* `_parameters`: List of parameter names
* `_defaults`: Dictionary of default values for optional parameters

Arguments passed to the constructor are stored in the operands attribute, with minimal input validation. Here’s an example:

```python
>>> class MyExpr(Expr):
        _parameters = ["param1", "param2"]
        _defaults = {"param2": None}

>>> expr = MyExpr(1, 2, 3)
>>> expr.param1
1

>>> expr.param2
2

>>> expr.operands
[1, 2, 3]
```

### Names and Tokens

Every expression is uniquely identified by a name, composed of:

* A prefix (typically the class name or a variant)
* A token generated by hashing its operands via `^dask.base.tokenize()`

This tokenization enables:

* Deduplication of equivalent expressions in the graph
* Detection of changes across optimization steps
* Singleton enforcement for certain expression types

Expressions that subclass `^dask.dataframe._expr.SingletonExpr` (the default for most DataFrame expressions) are guaranteed to be unique by name. However, this tokenization system introduces several challenges:

* Performance: Tokenization is slow due to recursive traversal and Dask’s dispatch mechanisms.
* Determinism: Objects without a registered \_\_dask_tokenize_\_ fallback to (cloud)pickling, which can be slow and non-deterministic.
* Cross-interpreter behavior: Tokens are not consistent across interpreters or machines, complicating client-scheduler interactions.

To address this, each expression computes and caches its name and token upon construction. These values are stored and serialized to ensure pickle roundtrip stability. The token is accessible via `_determ_token` or `deterministic_token`.

### Caching and Singletons

Despite efforts to keep expressions stateless, in practice many attributes are computed on demand and cached via `functools.cached_property`. This defers computation but complicates reasoning about when and how state is evaluated.

Cached properties are typically serialized with the expression, unless `_pickle_functools_cache` is set to False.

To preserve cache values during repeated optimization (which recreates expressions), most classes subclass `^dask.dataframe._expr.SingletonExpr`. This ensures that instances with the same name return a previously created, cached version. This makes expressions effectively immutable singletons — and must not be mutated in-place.

## Optimization Procedure

Expressions form a directed graph structure: when one expression is passed as an operand to another, it becomes a dependency. While this starts as a tree, deduplication by name quickly transforms it into a directed acyclic graph (DAG) — a key property for optimization.

The optimizer currently performs the following five steps:

* Simplify
* Rewrite / Tune
* Lower
* Simplify (again)
* Fuse

### Simplify

Simplification rewrites expressions into more optimal but semantically equivalent forms. A common example is pushing projections or filters down the graph to reduce computation earlier.

Constraints for simplification (not enforced at runtime):

* The number of partitions (npartitions) must not increase.
* No computations with side effects (e.g., computing divisions) may occur.

### Rewrite / Tune

This step implements performance tuning based on heuristics. Typically this targets a more efficient intermediate partitioning.

Two examples:

* Fuse I/O operations based on column projection (e.g., `FusedIO`)
* Choose an appropriate `split_out` to balance partitioning

This step does not alter the logical meaning of expressions but adjusts execution-related parameters.

### Lowering

At this stage, abstract operations are transformed into concrete execution strategies.

For example:

A logical `Merge` might become a `BlockwiseMerge` if the input DataFrames are already partitioned appropriately.

In less favorable cases, a general `HashJoinP2P` might be selected instead.

This marks the transition from logical to physical plan, akin to traditional query planners.

After lowering, a second simplification step is applied to the resulting expressions.

### Fuse

Linear chains of blockwise tasks are combined into a single task, minimizing scheduler overhead.

## Walking the Graph During Optimization

Each optimizer step traverses the expression graph until no further changes are found. The traversal typically follows this pattern (using Expr.simplify as an example):

1. Call `simplify_once`, which:
2. Calls `_simplify_down` on self (the current expression). This downward pass:
   : * Only has access to the current node and its operands
     * May return a new (optimized) expression or None
3. If a new expression is returned, check whether its name has changed (as unchanged names imply no effective change).
4. Then call `_simplify_up` on each dependency, passing in the parent node and a map of dependents. This upward pass:
   : * Has access to context across branches (e.g., siblings, shared parents)
     * Returns a replacement for the parent expression

Finally, the traversal recurses into dependencies by calling `simplify_once` on each.

#### NOTE
Convergence and Memoization

Without safeguards, this recursive traversal could loop indefinitely or cause exponential blowups. Protections include:

* Memoization by expression name
* Detection of repeated subgraphs

Despite these, pathological cases occasionally arise (e.g. [dask-expr#835](https://github.com/dask/dask-expr/issues/835)).

### Expressions as the Client-Scheduler Interface

Instead of transmitting low-level task graphs, we submit expressions directly to the scheduler. This reduces overhead but introduces complications:

* The distributed.Client requires final task keys before submission.
* Tokenization is non-deterministic across interpreters.

Optimization changes keys — so it must be run before submission to lock in key names and populate caches.

Some expressions (e.g., ReadParquet) require I/O to gather metadata like partition statistics. These steps must occur client-side, not on the scheduler, and are currently handled during lowering.

### Legacy HighLevelGraph (HLG) Support

HighLevelGraph is a legacy representation still used by Dask Arrays, Bags, and Delayed objects. Despite its goal of deferring graph materialization, many code paths trigger unintended conversion to low-level graphs.

Key issues:

* Low-level optimization often forces premature materialization.
* HLG lacks knowledge of the collection type and required optimizations.
* HLG does not encode postcompute behavior (e.g., how to combine partitions).

To bridge this gap, the `HLGExpr` class wraps an HLG and implements the full expression interface. Materialization is delayed until the scheduler explicitly calls `__dask_graph__`, at which point low-level optimization occurs. This ensures materialization and execution stay decoupled.

## Custom Expressions and Collections


# extend-sizeof.html.md

# Extend sizeof

When Dask needs to compute the size of an object in bytes, e.g. to determine which objects to spill to disk, it uses the `dask.sizeof.sizeof` registration mechanism. Users who need to define a `sizeof` implementation for their own objects can use `sizeof.register`:

```python
>>> import numpy as np
>>> from dask.sizeof import sizeof
>>> @sizeof.register(np.ndarray)
>>> def sizeof_numpy_like(array):
...     return array.nbytes
```

This code can be executed in order to register the implementation with Dask by placing it in one of the library’s modules e.g. `__init__.py`. However, this introduces a maintenance burden on the developers of these libraries, and must be manually imported on all workers in the event that these libraries do not accept the patch.

Therefore, Dask also exposes an [entrypoint](https://packaging.python.org/specifications/entry-points/) under the group `dask.sizeof` to enable third-party libraries to develop and maintain these `sizeof` implementations.

For a fictitious library `numpy_sizeof_dask.py`, the necessary `setup.cfg` configuration would be as follows:

```ini
[options.entry_points]
dask.sizeof =
   numpy = numpy_sizeof_dask:sizeof_plugin
```

whilst `numpy_sizeof_dask.py` would contain

```python
>>> import numpy as np
>>> def sizeof_plugin(sizeof):
...    @sizeof.register(np.ndarray)
...    def sizeof_numpy_like(array):
...        return array.nbytes
```

Upon the first import of dask.sizeof, Dask calls the entrypoint (`sizeof_plugin`) with the `dask.sizeof.sizeof` object, which can then be used to register a sizeof implementation.


# faq.html.md

# FAQ

**Question**: *Is Dask appropriate for adoption within a larger institutional context?*

**Answer**: *Yes.* Dask is used within the world’s largest banks, national labs,
retailers, technology companies, and government agencies.  It is used in highly
secure environments.  It is used in conservative institutions as well as fast
moving ones.

This page contains Frequently Asked Questions and concerns from institutions and users
when they first investigate Dask.

> * [For Management](#for-management)
>   * [Briefly, what problem does Dask solve for us?](#briefly-what-problem-does-dask-solve-for-us)
>   * [Is Dask mature?  Why should we trust it?](#is-dask-mature-why-should-we-trust-it)
>   * [Who else uses Dask?](#who-else-uses-dask)
>   * [How does Dask compare with Apache Spark?](#how-does-dask-compare-with-apache-spark)
>   * [Are there companies that we can get support from?](#are-there-companies-that-we-can-get-support-from)
> * [For IT](#for-it)
>   * [How would I set up Dask on institutional hardware?](#how-would-i-set-up-dask-on-institutional-hardware)
>   * [Is Dask secure?](#is-dask-secure)
>   * [Do I need to purchase a new cluster?](#do-i-need-to-purchase-a-new-cluster)
>   * [How do I manage users?](#how-do-i-manage-users)
>   * [How do I manage software environments?](#how-do-i-manage-software-environments)
>   * [How does Dask communicate data between machines?](#how-does-dask-communicate-data-between-machines)
>   * [Are deployments long running, or ephemeral?](#are-deployments-long-running-or-ephemeral)
> * [For Users](#for-users)
>   * [Will Dask “just work” on our existing code?](#will-dask-just-work-on-our-existing-code)
>   * [How well does Dask scale?  What are Dask’s limitations?](#how-well-does-dask-scale-what-are-dask-s-limitations)
>   * [Is Dask resilient?  What happens when a machine goes down?](#is-dask-resilient-what-happens-when-a-machine-goes-down)
>   * [Is the API exactly the same as NumPy/Pandas/Scikit-Learn?](#is-the-api-exactly-the-same-as-numpy-pandas-scikit-learn)
>   * [How much performance tuning does Dask require?](#how-much-performance-tuning-does-dask-require)
>   * [What Data formats does Dask support?](#what-data-formats-does-dask-support)
>   * [Does Dask have a SQL interface?](#does-dask-have-a-sql-interface)
>   * [Does Dask work on GPUs?](#does-dask-work-on-gpus)
> * [For Marketing](#for-marketing)
>   * [Where can I find logos?](#where-can-i-find-logos)

## For Management

### Briefly, what problem does Dask solve for us?

Dask is a general purpose parallel programming solution.
As such it is used in *many* different ways.

However, the most common problem that Dask solves is connecting Python analysts
to distributed hardware, particularly for data science and machine learning
workloads.  The institutions for whom Dask has the greatest
impact are those who have a large body of Python users who are accustomed to
libraries like NumPy, Pandas, Jupyter, Scikit-Learn and others, but want to
scale those workloads across a cluster.  Often they also have distributed
computing resources that are going underused.

Dask removes both technological and cultural barriers to connect Python users
to computing resources in a way that is native to both the users and IT.

“*Help me scale my notebook onto the cluster*” is a common pain point for
institutions today, and it is a common entry point for Dask usage.

### Is Dask mature?  Why should we trust it?

Yes.  While Dask itself is relatively new (it began in 2015) it is built by the
NumPy, Pandas, Jupyter, Scikit-Learn developer community, which is well trusted.
Dask is a relatively thin wrapper on top of these libraries and,
as a result, the project can be relatively small and simple.
It doesn’t reinvent a whole new system.

Additionally, this tight integration with the broader technology stack
gives substantial benefits long term.  For example:

- Because Pandas maintainers also maintain Dask,
  when Pandas issues a new releases Dask issues a release at the same time to
  ensure continuity and compatibility.
- Because Scikit-Learn maintainers maintain and use Dask when they train on large clusters,
  you can be assured that Dask-ML focuses on pragmatic and important
  solutions like XGBoost integration, and hyper-parameter selection,
  and that the integration between the two feels natural for novice and
  expert users alike.
- Because Jupyter maintainers also maintain Dask,
  powerful Jupyter technologies like JupyterHub and JupyterLab are designed
  with Dask’s needs in mind, and new features are pushed quickly to provide a
  first class and modern user experience.

Additionally, Dask is maintained both by a broad community of maintainers,
as well as substantial institutional support (several full-time employees each)
by both Anaconda, the company behind the leading data science distribution, and
NVIDIA, the leading hardware manufacturer of GPUs.  Despite large corporate
support, Dask remains a community governed project, and is fiscally sponsored by
NumFOCUS, the same 501c3 that fiscally sponsors NumPy, Pandas, Jupyter, and many others.

### Who else uses Dask?

Dask is used by individual researchers in practically every field today.  It
has millions of downloads per month, and is integrated into many PyData
software packages today.

On an *institutional* level Dask is used by analytics and research groups in a
similarly broad set of domains across both energetic startups as well as large
conservative household names.  A web search shows articles by Capital One,
Barclays, Walmart, NASA, Los Alamos National Laboratories, and hundreds of
other similar institutions.

### How does Dask compare with Apache Spark?

*This question has longer and more technical coverage* [here](spark.md)

Dask and Apache Spark are similar in that they both …

- Promise easy parallelism for data science Python users
- Provide Dataframe and ML APIs for ETL, data science, and machine learning
- Scale out to similar scales, around 1-1000 machines

Dask differs from Apache Spark in a few ways:

- Dask is more Python native, Spark is Scala/JVM native with Python bindings.

  Python users may find Dask more comfortable,
  but Dask is only useful for Python users,
  while Spark can also be used from JVM languages.
- Dask is one component in the broader Python ecosystem alongside libraries
  like Numpy, Pandas, and Scikit-Learn,
  while Spark is an all-in-one system that re-invents much of the Python world
  in a single package.

  This means that it’s often easier to compose Dask with new problem domains,
  but also that you need to install multiple things (like Dask and Pandas or
  Dask and Numpy) rather than just having everything in an all-in-one solution.
- Apache Spark focuses strongly on traditional business intelligence workloads,
  like ETL, SQL queries, and then some lightweight machine learning,
  while Dask is more general purpose.

  This means that Dask is much more flexible and can handle other problem
  domains like multi-dimensional arrays, GIS, advanced machine learning, and
  custom systems, but that it is less focused and less tuned on typical SQL
  style computations.

  If you mostly want to focus on SQL queries then Spark is probably a better
  bet.  If you want to support a wide variety of custom workloads then Dask
  might be more natural.

See the section [Comparison to Spark](spark.md).

### Are there companies that we can get support from?

There are several companies that offer support for dask in different capacities. See
[Paid support](https://docs.dask.org/en/latest/support.html#paid-support) for a full list.

## For IT

### How would I set up Dask on institutional hardware?

You already have cluster resources.
Dask can run on them today without significant change.

Most institutional clusters today have a resource manager.
This is typically managed by IT, with some mild permissions given to users to
launch jobs.  Dask works with all major resource managers today, including
those on Hadoop, HPC, Kubernetes, and Cloud clusters.

1. **Hadoop/Spark**: If you have a Hadoop/Spark cluster, such as one purchased
   through Cloudera/Hortonworks/MapR then you will likely want to deploy Dask
   with YARN, the resource manager that deploys services like Hadoop, Spark,
   Hive, and others.

   To help with this, you’ll likely want to use [Dask-Yarn](https://yarn.dask.org).
2. **HPC**: If you have an HPC machine that runs resource managers like SGE,
   SLURM, PBS, LSF, Torque, Condor, or other job batch queuing systems, then
   users can launch Dask on these systems today using either:
   - [Dask Jobqueue](https://jobqueue.dask.org) , which uses typical
     `qsub`, `sbatch`, `bsub` or other submission tools in interactive
     settings.
   - [Dask MPI](https://mpi.dask.org) which uses MPI for deployment in
     batch settings

   For more information see [High Performance Computers](deploying-hpc.md)
3. **Kubernetes/Cloud**: Newer clusters may employ Kubernetes for deployment.
   This is particularly commonly used today on major cloud providers,
   all of which provide hosted Kubernetes as a service.  People today use Dask
   on Kubernetes using either of the following:
   - **Helm**: an easy way to stand up a long-running Dask cluster and
     Jupyter notebook
   - **Dask-Kubernetes**: for native Kubernetes integration for fast moving
     or ephemeral deployments.

   For more information see [Kubernetes](deploying-kubernetes.md)
4. **Commercial Dask deployment:**
   - You can use [Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=faq) to handle the creation and management of Dask clusters on cloud computing environments (AWS and GCP).
   - [Domino Data Lab](https://www.dominodatalab.com/) Lets users create Dask clusters in a hosted platform.
   - [Saturn Cloud](https://saturncloud.io/) Lets users create Dask clusters in a hosted platform or within their own AWS accounts.

### Is Dask secure?

Dask is deployed today within highly secure institutions,
including major financial, healthcare, and government agencies.

That being said it’s worth noting that, by its very nature, Dask enables the
execution of arbitrary user code on a large set of machines. Care should be
taken to isolate, authenticate, and govern access to these machines.  Fortunately,
your institution likely already does this and uses standard technologies like
SSL/TLS, Kerberos, and other systems with which Dask can integrate.

### Do I need to purchase a new cluster?

No.  It is easy to run Dask today on most clusters.
If you have a pre-existing HPC or Spark/Hadoop cluster then that will be fine
to start running Dask.

You can start using Dask without any capital expenditure.

### How do I manage users?

Dask doesn’t manage users, you likely have existing systems that do this well.
In a large institutional setting we assume that you already have a resource
manager like Yarn (Hadoop), Kubernetes, or PBS/SLURM/SGE/LSF/…, each of which
have excellent user management capabilities, which are likely preferred by your
IT department anyway.

Dask is designed to operate with user-level permissions, which means that
your data science users should be able to ask those systems mentioned above for
resources, and have their processes tracked accordingly.

However, there are institutions where analyst-level users aren’t given direct access to
the cluster.  This is particularly common in Cloudera/Hortonworks Hadoop/Spark deployments.
In these cases some level of explicit indirection may be required.  For this, we
recommend the [Dask Gateway project](https://gateway.dask.org), which uses IT-level
permissions to properly route authenticated users into secure resources.

You may also want to consider a managed cluster solution (see [Manual deployments (not recommended)](deploying.md#managed-cluster-solutions)).

### How do I manage software environments?

This depends on your cluster resource manager:

- Most HPC users use their network file system
- Hadoop/Spark/Yarn users package their environment into a tarball and ship it
  around with HDFS (Dask-Yarn integrates with [Conda Pack](https://conda.github.io/conda-pack/) for this capability)
- Kubernetes or Cloud users use Docker images

In each case Dask integrates with existing processes and technologies
that are well understood and familiar to the institution.

### How does Dask communicate data between machines?

Dask usually communicates over TCP, using msgpack for small administrative
messages, and its own protocol for efficiently passing around large data.
The scheduler and each worker host their own TCP server, making Dask a
distributed peer-to-peer network that uses point-to-point communication.
We do not use Spark-style shuffle systems.  We do not use MPI-style
collectives.  Everything is direct point-to-point.

For high performance networks you can use either TCP-over-Infiniband for about
1 GB/s bandwidth, or UCX (experimental) for full speed communication.

### Are deployments long running, or ephemeral?

We see both, but ephemeral deployments are more common.

Most Dask use today is about enabling data science or data engineering users to
scale their interactive workloads across the cluster.
These are typically either interactive sessions with Jupyter, or batch scripts
that run at a pre-defined time.  In both cases, the user asks the resource
manager for a bunch of machines, does some work, and then gives up those
machines.

Some institutions also use Dask in an always-on fashion, either handling
real-time traffic in a scalable way, or responding to a broad set of
interactive users with large datasets that it keeps resident in memory.

## For Users

### Will Dask “just work” on our existing code?

No, you will need to make modifications,
but these modifications are usually small.

The vast majority of lines of business logic within your institution
will not have to change, assuming that they are in Python and use tooling like
Numpy, Pandas and Scikit-Learn.

### How well does Dask scale?  What are Dask’s limitations?

The largest Dask deployments that we see today are on around 1000 multi-core
machines, perhaps 20,000 cores in total, but these are rare.
Most institutional-level problems (1-100 TB) are well solved by deployments of 10-50 nodes.

Technically, the back-of-the-envelope number to keep in mind is that each task
(an individual Python function call) in Dask has an overhead of around *200
microseconds*.  So if these tasks take 1 second each, then Dask can saturate
around 5000 cores before scheduling overhead dominates costs.  As workloads
reach this limit they are encouraged to use larger chunk sizes to compensate.
The *vast majority* of institutional users though do not reach this limit.
For more information you may want to peruse our [best practices](best-practices.md)

### Is Dask resilient?  What happens when a machine goes down?

Yes, Dask is resilient to the failure of worker nodes.  It knows how it came to
any result, and can replay the necessary work on other machines if one goes
down.

If Dask’s centralized scheduler goes down then you would need to resubmit the
computation.  This is a fairly standard level of resiliency today, shared with
other tooling like Apache Spark, Flink, and others.

The resource managers that host Dask, like Yarn or Kubernetes, typically
provide long-term 24/7 resilience for always-on operation.

### Is the API exactly the same as NumPy/Pandas/Scikit-Learn?

No, but it’s very close.  That being said your data scientists will still
have to learn some things.

What we find is that the Numpy/Pandas/Scikit-Learn APIs aren’t the challenge
when institutions adopt Dask.  When API inconsistencies do exist, even
modestly skilled programmers are able to understand why and work around them
without much pain.

Instead, the challenge is building intuition around parallel performance.
We’ve all built up a mental model for what is fast and slow on a single
machine.  This model changes when we factor in network communication and
parallel algorithms, and the performance that we get for familiar operations
can be surprising.

Our main solution to build this intuition, other than
accumulated experience, is Dask’s [Diagnostic Dashboard](dashboard.md).
The dashboard delivers a ton of visual feedback to users as they are running
their computation to help them understand what is going on.  This both helps
them to identify and resolve immediate bottlenecks, and also builds up that
parallel performance intuition surprisingly quickly.

### How much performance tuning does Dask require?

*Some other systems are notoriously hard to tune for optimal performance.
What is Dask’s story here?  How many knobs are there that we need to be aware
of?*

Like the rest of the Python software tools, Dask puts a lot of effort into
having sane defaults.  Dask workers automatically detect available memory and
cores, and choose sensible defaults that are decent in most situations.  Dask
algorithms similarly provide decent choices by default, and informative warnings
when tricky situations arise, so that, in common cases, things should be fine.

The most common knobs to tune include the following:

- The thread/process mixture to deal with GIL-holding computations (which are
  rare in Numpy/Pandas/Scikit-Learn workflows)
- Partition size, like if should you have 100 MB chunks or 1 GB chunks

That being said, almost no institution’s needs are met entirely by the common
case, and given the variety of problems that people throw at Dask,
exceptional problems are commonplace.
In these cases we recommend watching the dashboard during execution to see what
is going on.  It can commonly inform you what’s going wrong, so that you can
make changes to your system.

### What Data formats does Dask support?

Because Dask builds on NumPy and Pandas, it supports most formats that they
support, which is most formats.
That being said, not all formats are well suited for
parallel access.  In general people using the following formats are usually
pretty happy:

- **Tabular:** Parquet, ORC, CSV, Line Delimited JSON, Avro, text
- **Arrays:** HDF5, NetCDF, Zarr, GRIB

More generally, if you have a Python function that turns a chunk of your stored
data into a Pandas dataframe or Numpy array then Dask can probably call that
function many times without much effort.

For groups looking for advice on which formats to use, we recommend Parquet
for tables and Zarr or HDF5 for arrays.

### Does Dask have a SQL interface?

Dask supports various ways to communicate with SQL databases, some
requiring extra packages to be installed; see the section
[Dask Dataframe and SQL](dataframe-sql.md).

### Does Dask work on GPUs?

Yes! Dask works with GPUs in a few ways.

The [RAPIDS](https://rapids.ai) libraries provide a GPU-accelerated
Pandas-like library,
[cuDF](https://github.com/rapidsai/cudf),
which interoperates well and is tested against Dask DataFrame.

[Chainer’s CuPy](https://cupy.chainer.org/) library provides a GPU
accelerated NumPy-like library that interoperates nicely with Dask Array.

For custom workflows people use Dask alongside GPU-accelerated libraries like PyTorch and
TensorFlow to manage workloads across several machines.  They typically use
Dask’s custom APIs, notably [Delayed](delayed.md) and [Futures](futures.md).

See the section [GPUs](gpu.md).

## For Marketing

There is a special subsite dedicated to addressing marketing concerns. You can
find it at [dask.org/brand-guide](https://dask.org/brand-guide).

### Where can I find logos?

Yes! You can find them at [Images and Logos](logos.md).


# futures.html.md

# Futures

Dask supports a real-time task framework that extends Python’s
[concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html)
interface. Dask futures allow you to scale generic Python workflows across
a Dask cluster with minimal code changes.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/07EiCpdhtDE"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

This interface is good for arbitrary task scheduling like
[dask.delayed](delayed.md), but is immediate rather than lazy, which
provides some more flexibility in situations where the computations may evolve
over time. These features depend on the second generation task scheduler found in
[dask.distributed](https://distributed.dask.org/en/latest) (which,
despite its name, runs very well on a single machine).

## Examples

Visit [https://examples.dask.org/futures.html](https://examples.dask.org/futures.html) to see and run examples
using futures with Dask.

## Start Dask Client

You must start a `Client` to use the futures interface.  This tracks state
among the various worker processes or threads:

```python
from dask.distributed import Client

client = Client()  # start local workers as processes
# or
client = Client(processes=False)  # start local workers as threads
```

If you have [Bokeh](https://docs.bokeh.org) installed, then this starts up a
diagnostic dashboard at `http://localhost:8787` .

## Submit Tasks

| [`Client.submit`](#distributed.Client.submit)(func, \*args[, key, workers, ...])   | Submit a function application to the scheduler                    |
|------------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`Client.map`](#distributed.Client.map)(func, \*iterables[, key, workers, ...])    | Map a function on a sequence of arguments                         |
| [`Future.result`](#distributed.Future.result)([timeout])                           | Wait until computation completes, gather result to local process. |

You can submit individual tasks using the `submit` method:

```python
def inc(x):
    return x + 1

def add(x, y):
    return x + y

a = client.submit(inc, 10)  # calls inc(10) in background thread or process
b = client.submit(inc, 20)  # calls inc(20) in background thread or process
```

The `submit` function returns a `Future`, which refers to a remote result.  This result may
not yet be completed:

```python
>>> a
<Future: status: pending, key: inc-b8aaf26b99466a7a1980efa1ade6701d>
```

Eventually it will complete.  The result stays in the remote
thread/process/worker until you ask for it back explicitly:

```python
>>> a
<Future: status: finished, type: int, key: inc-b8aaf26b99466a7a1980efa1ade6701d>

>>> a.result()  # blocks until task completes and data arrives
11
```

You can pass futures as inputs to submit.  Dask automatically handles dependency
tracking; once all input futures have completed, they will be moved onto a
single worker (if necessary), and then the computation that depends on them
will be started.  You do not need to wait for inputs to finish before
submitting a new task; Dask will handle this automatically:

```python
c = client.submit(add, a, b)  # calls add on the results of a and b
```

Similar to Python’s `map`, you can use `Client.map` to call the same
function and many inputs:

```python
futures = client.map(inc, range(1000))
```

However, note that each task comes with about 1ms of overhead.  If you want to
map a function over a large number of inputs, then you might consider
[dask.bag](bag.md) or [dask.dataframe](dataframe.md) instead.

<!-- note: See `this page <https://docs.dask.org/en/latest/graphs.html>`_ for
restrictions on what functions you use with Dask. -->

## Move Data

| [`Future.result`](#distributed.Future.result)([timeout])                         | Wait until computation completes, gather result to local process.   |
|----------------------------------------------------------------------------------|---------------------------------------------------------------------|
| [`Client.gather`](#distributed.Client.gather)(futures[, errors, direct, ...])    | Gather futures from distributed memory                              |
| [`Client.scatter`](#distributed.Client.scatter)(data[, workers, broadcast, ...]) | Scatter data into distributed memory                                |

Given any future, you can call the `.result` method to gather the result.
This will block until the future is done computing and then transfer the result
back to your local process if necessary:

```python
>>> c.result()
32
```

You can gather many results concurrently using the `Client.gather` method.
This can be more efficient than calling `.result()` on each future
sequentially:

```python
>>> # results = [future.result() for future in futures]
>>> results = client.gather(futures)  # this can be faster
```

If you have important local data that you want to include in your computation,
you can either include it as a normal input to a submit or map call:

```python
>>> df = pd.read_csv('training-data.csv')
>>> future = client.submit(my_function, df)
```

Or you can `scatter` it explicitly.  Scattering moves your data to a worker
and returns a future pointing to that data:

```python
>>> remote_df = client.scatter(df)
>>> remote_df
<Future: status: finished, type: DataFrame, key: bbd0ca93589c56ea14af49cba470006e>

>>> future = client.submit(my_function, remote_df)
```

Both of these accomplish the same result, but using scatter can sometimes be
faster.  This is especially true if you use processes or distributed workers
(where data transfer is necessary) and you want to use `df` in many
computations.  Scattering the data beforehand avoids excessive data movement.

Calling scatter on a list scatters all elements individually.  Dask will spread
these elements evenly throughout workers in a round-robin fashion:

```python
>>> client.scatter([1, 2, 3])
[<Future: status: finished, type: int, key: c0a8a20f903a4915b94db8de3ea63195>,
 <Future: status: finished, type: int, key: 58e78e1b34eb49a68c65b54815d1b158>,
 <Future: status: finished, type: int, key: d3395e15f605bc35ab1bac6341a285e2>]
```

## References, Cancellation, and Exceptions

| [`Future.cancel`](#distributed.Future.cancel)([reason, msg])                | Cancel the request to run this future                                                                                                   |
|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| [`Future.exception`](#distributed.Future.exception)([timeout])              | Return the exception of a failed task                                                                                                   |
| [`Future.traceback`](#distributed.Future.traceback)([timeout])              | Return the traceback of a failed task                                                                                                   |
| [`Client.cancel`](#distributed.Client.cancel)(futures[, asynchronous, ...]) | Cancel running futures This stops future tasks from being scheduled if they have not yet run and deletes them if they have already run. |

Dask will only compute and hold onto results for which there are active
futures.  In this way, your local variables define what is active in Dask.  When
a future is garbage collected by your local Python session, Dask will feel free
to delete that data or stop ongoing computations that were trying to produce
it:

```python
>>> del future  # deletes remote data once future is garbage collected
```

You can also explicitly cancel a task using the `Future.cancel` or
`Client.cancel` methods:

```python
>>> future.cancel()  # deletes data even if other futures point to it
```

If a future fails, then Dask will raise the remote exceptions and tracebacks if
you try to get the result:

```python
def div(x, y):
    return x / y

>>> a = client.submit(div, 1, 0)  # 1 / 0 raises a ZeroDivisionError
>>> a
<Future: status: error, key: div-3601743182196fb56339e584a2bf1039>

>>> a.result()
      1 def div(x, y):
----> 2     return x / y

ZeroDivisionError: division by zero
```

All futures that depend on an erred future also err with the same exception:

```python
>>> b = client.submit(inc, a)
>>> b
<Future: status: error, key: inc-15e2e4450a0227fa38ede4d6b1a952db>
```

You can collect the exception or traceback explicitly with the
`Future.exception` or `Future.traceback` methods.

## Waiting on Futures

| [`as_completed`](#distributed.as_completed)([futures, loop, with_results, ...])   | Return futures in the order in which they complete   |
|-----------------------------------------------------------------------------------|------------------------------------------------------|
| [`wait`](#distributed.wait)(fs[, timeout, return_when])                           | Wait until all/any futures are finished              |

You can wait on a future or collection of futures using the `wait` function:

```python
from dask.distributed import wait

>>> wait(futures)
```

This blocks until all futures are finished or have erred.

You can also iterate over the futures as they complete using the
`as_completed` function:

```python
from dask.distributed import as_completed

futures = client.map(score, x_values)

best = -1
for future in as_completed(futures):
   y = future.result()
   if y > best:
       best = y
```

For greater efficiency, you can also ask `as_completed` to gather the results
in the background:

```python
for future, result in as_completed(futures, with_results=True):
    # y = future.result()  # don't need this
   ...
```

Or collect all futures in batches that had arrived since the last iteration:

```python
for batch in as_completed(futures, with_results=True).batches():
   for future, result in batch:
       ...
```

Additionally, for iterative algorithms, you can add more futures into the
`as_completed` iterator *during* iteration:

```python
seq = as_completed(futures)

for future in seq:
    y = future.result()
    if condition(y):
        new_future = client.submit(...)
        seq.add(new_future)  # add back into the loop
```

or use `seq.update(futures)` to add multiple futures at once.

## Fire and Forget

| [`fire_and_forget`](#distributed.fire_and_forget)(obj)   | Run tasks at least once, even if we release the futures   |
|----------------------------------------------------------|-----------------------------------------------------------|

Sometimes we don’t care about gathering the result of a task, and only care
about side effects that it might have like writing a result to a file:

```python
>>> a = client.submit(load, filename)
>>> b = client.submit(process, a)
>>> c = client.submit(write, b, out_filename)
```

As noted above, Dask will stop work that doesn’t have any active futures.  It
thinks that because no one has a pointer to this data that no one cares.  You
can tell Dask to compute a task anyway, even if there are no active futures,
using the `fire_and_forget` function:

```python
from dask.distributed import fire_and_forget

>>> fire_and_forget(c)
```

This is particularly useful when a future may go out of scope, for example, as
part of a function:

```python
def process(filename):
    out_filename = 'out-' + filename
    a = client.submit(load, filename)
    b = client.submit(process, a)
    c = client.submit(write, b, out_filename)
    fire_and_forget(c)
    return  # here we lose the reference to c, but that's now ok

for filename in filenames:
    process(filename)
```

## Submit task and retrieve results from a different process

Sometimes we care about retrieving a result but not necessarily from the same process.

```python
from distributed import Variable

var = Variable("my-result")
fut = client.submit(...)
var.set(fut)
```

Using a `Variable` instructs dask to remember the result of this task under
the given name so that it can be retrieved later without having to keep the
Client alive in the meantime.

```python
var = Variable("my-result")
fut = var.get()
result = fut.result()
```

## Submit Tasks from Tasks

| [`get_client`](#distributed.get_client)([address, timeout, resolve_address])   | Get a client while within a task.                   |
|--------------------------------------------------------------------------------|-----------------------------------------------------|
| [`rejoin`](#distributed.rejoin)()                                              | Have this thread rejoin the ThreadPoolExecutor      |
| [`secede`](#distributed.secede)()                                              | Have this task secede from the worker's thread pool |

*This is an advanced feature and is rarely necessary in the common case.*

Tasks can launch other tasks by getting their own client.  This enables complex
and highly dynamic workloads:

```python
from dask.distributed import get_client

def my_function(x):
    ...

    # Get locally created client
    client = get_client()

    # Do normal client operations, asking cluster for computation
    a = client.submit(...)
    b = client.submit(...)
    a, b = client.gather([a, b])

    return a + b
```

It also allows you to set up long running tasks that watch other resources like
sockets or physical sensors:

```python
def monitor(device):
   client = get_client()
   while True:
       data = device.read_data()
       future = client.submit(process, data)
       fire_and_forget(future)

for device in devices:
    fire_and_forget(client.submit(monitor))
```

However, each running task takes up a single thread, and so if you launch many
tasks that launch other tasks, then it is possible to deadlock the system if you
are not careful.  You can call the `secede` function from within a task to
have it remove itself from the dedicated thread pool into an administrative
thread that does not take up a slot within the Dask worker:

```python
from dask.distributed import get_client, secede

def monitor(device):
   client = get_client()
   secede()  # remove this task from the thread pool
   while True:
       data = device.read_data()
       future = client.submit(process, data)
       fire_and_forget(future)
```

If you intend to do more work in the same thread after waiting on client work,
you may want to explicitly block until the thread is able to *rejoin* the
thread pool.  This allows some control over the number of threads that are
created and stops too many threads from being active at once, over-saturating
your hardware:

```python
def f(n):  # assume that this runs as a task
   client = get_client()

   secede()  # secede while we wait for results to come back
   futures = client.map(func, range(n))
   results = client.gather(futures)

   rejoin()  # block until a slot is open in the thread pool
   result = analyze(results)
   return result
```

Alternatively, you can just use the normal `compute` function *within* a
task.  This will automatically call `secede` and `rejoin` appropriately:

```python
def f(name, fn):
    df = dd.read_csv(fn)  # note that this is a dask collection
    result = df[df.name == name].count()

    # This calls secede
    # Then runs the computation on the cluster (including this worker)
    # Then blocks on rejoin, and finally delivers the answer
    result = result.compute()

    return result
```

## Coordination Primitives

| [`Queue`](#distributed.Queue)([name, client, maxsize])                        | Distributed Queue                                                                                                                                                           |
|-------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`Variable`](#distributed.Variable)([name, client])                           | Distributed Global Variable                                                                                                                                                 |
| [`Lock`](#distributed.Lock)([name, scheduler_rpc, loop])                      | Distributed Centralized Lock                                                                                                                                                |
| [`Event`](#distributed.Event)([name, client])                                 | Distributed Centralized Event equivalent to asyncio.Event                                                                                                                   |
| [`Semaphore`](#distributed.Semaphore)([max_leases, name, scheduler_rpc, ...]) | This [semaphore](https://en.wikipedia.org/wiki/Semaphore_(programming)) will track leases on the scheduler which can be acquired and released by an instance of this class. |
<!-- note: These are advanced features and are rarely necessary in the common case. -->

Sometimes situations arise where tasks, workers, or clients need to coordinate
with each other in ways beyond normal task scheduling with futures.  In these
cases Dask provides additional primitives to help in complex situations.

Dask provides distributed versions of coordination primitives like locks, events,
queues, and global variables that, where appropriate, match
their in-memory counterparts.  These can be used to control access to external
resources, track progress of ongoing computations, or share data in
side-channels between many workers, clients, and tasks sensibly.

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/Q-Y3BR1u7c0"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

These features are rarely necessary for common use of Dask.  We recommend that
beginning users stick with using the simpler futures found above (like
`Client.submit` and `Client.gather`) rather than embracing needlessly
complex techniques.

### Queues

| [`Queue`](#distributed.Queue)([name, client, maxsize])   | Distributed Queue   |
|----------------------------------------------------------|---------------------|

Dask queues follow the API for the standard Python Queue, but now move futures
or small messages between clients.  Queues serialize sensibly and reconnect
themselves on remote clients if necessary:

```python
from dask.distributed import Queue

def load_and_submit(filename):
    data = load(filename)
    client = get_client()
    future = client.submit(process, data)
    queue.put(future)

client = Client()

queue = Queue()

for filename in filenames:
    future = client.submit(load_and_submit, filename)
    fire_and_forget(future)

while True:
    future = queue.get()
    print(future.result())
```

Queues can also send small pieces of information, anything that is msgpack
encodable (ints, strings, bools, lists, dicts, etc.).  This can be useful to
send back small scores or administrative messages:

```python
def func(x):
    try:
       ...
    except Exception as e:
        error_queue.put(str(e))

error_queue = Queue()
```

Queues are mediated by the central scheduler, and so they are not ideal for
sending large amounts of data (everything you send will be routed through a
central point).  They are well suited to move around small bits of metadata, or
futures.  These futures may point to much larger pieces of data safely:

```python
>>> x = ... # my large numpy array

# Don't do this!
>>> q.put(x)

# Do this instead
>>> future = client.scatter(x)
>>> q.put(future)

# Or use futures for metadata
>>> q.put({'status': 'OK', 'stage=': 1234})
```

### Global Variables

| [`Variable`](#distributed.Variable)([name, client])   | Distributed Global Variable   |
|-------------------------------------------------------|-------------------------------|

Variables are like Queues in that they communicate futures and small data
between clients.  However, variables hold only a single value.  You can get or
set that value at any time:

```python
>>> var = Variable('stopping-criterion')
>>> var.set(False)

>>> var.get()
False
```

This is often used to signal stopping criteria or current parameters
between clients.

If you want to share large pieces of information, then scatter the data first:

```python
>>> parameters = np.array(...)
>>> future = client.scatter(parameters)
>>> var.set(future)
```

### Locks

| [`Lock`](#distributed.Lock)([name, scheduler_rpc, loop])   | Distributed Centralized Lock   |
|------------------------------------------------------------|--------------------------------|

You can also hold onto cluster-wide locks using the `Lock` object.
Dask Locks have the same API as normal `threading.Lock` objects, except that
they work across the cluster:

```python
from dask.distributed import Lock
lock = Lock()

with lock:
    # access protected resource
```

You can manage several locks at the same time.  Lock can either be given a
consistent name or you can pass the lock object around itself.

Using a consistent name is convenient when you want to lock some known named resource:

```python
from dask.distributed import Lock

def load(fn):
    with Lock('the-production-database'):
        # read data from filename using some sensitive source
        return ...

futures = client.map(load, filenames)
```

Passing around a lock works as well and is easier when you want to create short-term
locks for a particular situation:

```python
from dask.distributed import Lock
lock = Lock()

def load(fn, lock=None):
    with lock:
        # read data from filename using some sensitive source
        return ...

futures = client.map(load, filenames, lock=lock)
```

This can be useful if you want to control concurrent access to some external
resource like a database or un-thread-safe library.

### Events

| [`Event`](#distributed.Event)([name, client])   | Distributed Centralized Event equivalent to asyncio.Event   |
|-------------------------------------------------|-------------------------------------------------------------|

Dask Events mimic `asyncio.Event` objects, but on a cluster scope.
They hold a single flag which can be set or cleared.
Clients can wait until the event flag is set.
Different from a `Lock`, every client can set or clear the flag and there
is no “ownership” of an event.

You can use events to e.g. synchronize multiple clients:

```python
# One one client
from dask.distributed import Event

event = Event("my-event-1")
event.wait()
```

The call to wait will block until the event is set, e.g. in another client

```python
# In another client
from dask.distributed import Event

event = Event("my-event-1")

# do some work

event.set()
```

Events can be set, cleared and waited on multiple times.
Every waiter referencing the same event name will be notified on event set
(and not only the first one as in the case of a lock):

```python
from dask.distributed import Event

def wait_for_event(x):
   event = Event("my-event")

   event.wait()
   # at this point, all function calls
   # are in sync once the event is set

futures = client.map(wait_for_event, range(10))

Event("my-event").set()
client.gather(futures)
```

### Semaphore

| [`Semaphore`](#distributed.Semaphore)([max_leases, name, scheduler_rpc, ...])   |    |
|---------------------------------------------------------------------------------|----|

Similar to the single-valued `Lock` it is also possible to use a cluster-wide
semaphore to coordinate and limit access to a sensitive resource like a
database.

```python
from dask.distributed import Semaphore

sem = Semaphore(max_leases=2, name="database")

def access_limited(val, sem):
   with sem:
      # Interact with the DB
      return

futures = client.map(access_limited, range(10), sem=sem)
client.gather(futures)
sem.close()
```

## Actors

Actors allow workers to manage rapidly changing state without coordinating with
the central scheduler.  This has the advantage of reducing latency
(worker-to-worker roundtrip latency is around 1ms), reducing pressure on the
centralized scheduler (workers can coordinate actors entirely among each other),
and also enabling workflows that require stateful or in-place memory
manipulation.

However, these benefits come at a cost.  The scheduler is unaware of actors and
so they don’t benefit from diagnostics, load balancing, or resilience.  Once an
actor is running on a worker it is forever tied to that worker.  If that worker
becomes overburdened or dies, then there is no opportunity to recover the
workload.

*Because Actors avoid the central scheduler they can be high-performing, but not resilient.*

### Example: Counter

An actor is a class containing both state and methods that is submitted to a
worker:

```python
class Counter:
    n = 0

    def __init__(self):
        self.n = 0

    def increment(self):
        self.n += 1
        return self.n

from dask.distributed import Client

if __name__ == '__main__':
    client = Client()

    future = client.submit(Counter, actor=True)
    counter = future.result()

>>> counter
<Actor: Counter, key=Counter-afa1cdfb6b4761e616fa2cfab42398c8>
```

Method calls on this object produce `ActorFutures`, which are similar to
normal Futures, but interact only with the worker holding the Actor:

```python
>>> future = counter.increment()
>>> future
<ActorFuture>

>>> future.result()
1
```

Attribute access is synchronous and blocking:

```python
>>> counter.n
1
```

### Example: Parameter Server

This example will perform the following minimization with a parameter server:

$$
\min_{p\in\mathbb{R}^{1000}} \sum_{i=1}^{1000} (p_i - 1)^2
$$

This is a simple minimization that will serve as an illustrative example.

The Dask Actor will serve as the parameter server that will hold the model.
The client will calculate the gradient of the loss function above.

```python
import numpy as np

from dask.distributed import Client
client = Client(processes=False)

class ParameterServer:
    def __init__(self):
        self.data = dict()

    def put(self, key, value):
        self.data[key] = value

    def get(self, key):
        return self.data[key]

def train(params, lr=0.1):
    grad = 2 * (params - 1)  # gradient of (params - 1)**2
    new_params = params - lr * grad
    return new_params

ps_future = client.submit(ParameterServer, actor=True)
ps = ps_future.result()

ps.put('parameters', np.random.default_rng().random(1000))
for k in range(20):
    params = ps.get('parameters').result()
    new_params = train(params)
    ps.put('parameters', new_params)
    print(new_params.mean())
    # k=0: "0.5988202981316124"
    # k=10: "0.9569236575164062"
```

This example works, and the loss function is minimized. The (simple) equation
above is minimize, so each $p_i$ converges to 1. If desired, this example
could be adapted to machine learning with a more complex function to minimize.

### Asynchronous Operation

All operations that require talking to the remote worker are awaitable:

```python
async def f():
    future = client.submit(Counter, actor=True)
    counter = await future  # gather actor object locally

    counter.increment()  # send off a request asynchronously
    await counter.increment()  # or wait until it was received

    n = await counter.n  # attribute access also must be awaited
```

Generally, all I/O operations that trigger computations (e.g. `to_parquet`) should be done using the `compute=False`
parameter to avoid asynchronous blocking:

```python
await client.compute(ddf.to_parquet('/tmp/some.parquet', compute=False))
```

## API

**Client**

| [`Client`](#distributed.Client)([address, loop, timeout, ...])                           | Connect to and submit computation to a Dask cluster                                                                                     |
|------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| [`Client.cancel`](#distributed.Client.cancel)(futures[, asynchronous, ...])              | Cancel running futures This stops future tasks from being scheduled if they have not yet run and deletes them if they have already run. |
| [`Client.compute`](#distributed.Client.compute)(collections[, sync, ...])                | Compute dask collections on cluster                                                                                                     |
| [`Client.gather`](#distributed.Client.gather)(futures[, errors, direct, ...])            | Gather futures from distributed memory                                                                                                  |
| [`Client.get`](#distributed.Client.get)(dsk, keys[, workers, ...])                       | Compute dask graph                                                                                                                      |
| [`Client.get_dataset`](#distributed.Client.get_dataset)(name[, default])                 | Get named dataset from the scheduler if present.                                                                                        |
| [`Client.get_executor`](#distributed.Client.get_executor)(\*\*kwargs)                    | Return a concurrent.futures Executor for submitting tasks on this Client                                                                |
| [`Client.has_what`](#distributed.Client.has_what)([workers])                             | Which keys are held by which workers                                                                                                    |
| [`Client.list_datasets`](#distributed.Client.list_datasets)(\*\*kwargs)                  | List named datasets available on the scheduler                                                                                          |
| [`Client.map`](#distributed.Client.map)(func, \*iterables[, key, workers, ...])          | Map a function on a sequence of arguments                                                                                               |
| [`Client.ncores`](#distributed.Client.ncores)([workers])                                 | The number of threads/cores available on each worker node                                                                               |
| [`Client.persist`](#distributed.Client.persist)(collections[, ...])                      | Persist dask collections on cluster                                                                                                     |
| [`Client.profile`](#distributed.Client.profile)([key, start, stop, workers, ...])        | Collect statistical profiling information about recent work                                                                             |
| [`Client.publish_dataset`](#distributed.Client.publish_dataset)()                        | Publish named datasets to scheduler                                                                                                     |
| [`Client.rebalance`](#distributed.Client.rebalance)([futures, workers])                  | Rebalance data within network                                                                                                           |
| [`Client.replicate`](#distributed.Client.replicate)(futures[, n, workers, ...])          | Set replication of futures within network                                                                                               |
| [`Client.restart`](#distributed.Client.restart)([timeout, wait_for_workers])             | Restart all workers.                                                                                                                    |
| [`Client.run`](#distributed.Client.run)(function, \*args[, workers, wait, ...])          | Run a function on all workers outside of task scheduling system                                                                         |
| [`Client.run_on_scheduler`](#distributed.Client.run_on_scheduler)(function, \*args, ...) | Run a function on the scheduler process                                                                                                 |
| [`Client.scatter`](#distributed.Client.scatter)(data[, workers, broadcast, ...])         | Scatter data into distributed memory                                                                                                    |
| [`Client.shutdown`](#distributed.Client.shutdown)()                                      | Shut down the connected scheduler and workers                                                                                           |
| [`Client.scheduler_info`](#distributed.Client.scheduler_info)([n_workers])               | Basic information about the workers in the cluster                                                                                      |
| [`Client.submit`](#distributed.Client.submit)(func, \*args[, key, workers, ...])         | Submit a function application to the scheduler                                                                                          |
| [`Client.unpublish_dataset`](#distributed.Client.unpublish_dataset)(name, \*\*kwargs)    | Remove named datasets from scheduler                                                                                                    |
| [`Client.upload_file`](#distributed.Client.upload_file)(filename[, load])                | Upload local package to scheduler and workers                                                                                           |
| [`Client.who_has`](#distributed.Client.who_has)([futures])                               | The workers storing each future's data                                                                                                  |

**Future**

| [`Future`](#distributed.Future)(key[, client, state, \_id])             | A remotely running computation                                    |
|-------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`Future.add_done_callback`](#distributed.Future.add_done_callback)(fn) | Call callback on future when future has finished                  |
| [`Future.cancel`](#distributed.Future.cancel)([reason, msg])            | Cancel the request to run this future                             |
| [`Future.cancelled`](#distributed.Future.cancelled)()                   | Returns True if the future has been cancelled                     |
| [`Future.done`](#distributed.Future.done)()                             | Returns whether or not the computation completed.                 |
| [`Future.exception`](#distributed.Future.exception)([timeout])          | Return the exception of a failed task                             |
| [`Future.result`](#distributed.Future.result)([timeout])                | Wait until computation completes, gather result to local process. |
| [`Future.traceback`](#distributed.Future.traceback)([timeout])          | Return the traceback of a failed task                             |

**Functions**

| [`as_completed`](#distributed.as_completed)([futures, loop, with_results, ...])   | Return futures in the order in which they complete                                                                      |
|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [`fire_and_forget`](#distributed.fire_and_forget)(obj)                            | Run tasks at least once, even if we release the futures                                                                 |
| [`get_client`](#distributed.get_client)([address, timeout, resolve_address])      | Get a client while within a task.                                                                                       |
| [`secede`](#distributed.secede)()                                                 | Have this task secede from the worker's thread pool                                                                     |
| [`rejoin`](#distributed.rejoin)()                                                 | Have this thread rejoin the ThreadPoolExecutor                                                                          |
| [`wait`](#distributed.wait)(fs[, timeout, return_when])                           | Wait until all/any futures are finished                                                                                 |
| [`print`](#distributed.print)(\*args[, sep, end, file, flush])                    | A drop-in replacement of the built-in `print` function for remote printing from workers to clients.                     |
| [`warn`](#distributed.warn)(message[, category, stacklevel, source])              | A drop-in replacement of the built-in `warnings.warn()` function for issuing warnings remotely from workers to clients. |

### distributed.as_completed(futures=None, loop=None, with_results=False, raise_errors=True, , timeout=None)

Return futures in the order in which they complete

This returns an iterator that yields the input future objects in the order
in which they complete.  Calling `next` on the iterator will block until
the next future completes, irrespective of order.

Additionally, you can also add more futures to this object during
computation with the `.add` method

* **Parameters:**
  **futures: Collection of futures**
  : A list of Future objects to be iterated over in the order in which they
    complete

  **with_results: bool (False)**
  : Whether to wait and include results of futures as well;
    in this case `as_completed` yields a tuple of (future, result)

  **raise_errors: bool (True)**
  : Whether we should raise when the result of a future raises an
    exception; only affects behavior when `with_results=True`.

  **timeout: int (optional)**
  : The returned iterator raises a `dask.distributed.TimeoutError`
    if `__next__()` or `__anext__()` is called and the result
    isn’t available after timeout seconds from the original call to
    `as_completed()`. If timeout is not specified or `None`, there is no limit
    to the wait time.

### Examples

```pycon
>>> x, y, z = client.map(inc, [1, 2, 3])
>>> for future in as_completed([x, y, z]):
...     print(future.result())
3
2
4
```

Add more futures during computation

```pycon
>>> x, y, z = client.map(inc, [1, 2, 3])
>>> ac = as_completed([x, y, z])
>>> for future in ac:
...     print(future.result())
...     if random.random() < 0.5:
...         ac.add(c.submit(double, future))
4
2
8
3
6
12
24
```

Optionally wait until the result has been gathered as well

```pycon
>>> ac = as_completed([x, y, z], with_results=True)
>>> for future, result in ac:
...     print(result)
2
4
3
```

<!-- !! processed by numpydoc !! -->

### distributed.fire_and_forget(obj)

Run tasks at least once, even if we release the futures

Under normal operation Dask will not run any tasks for which there is not
an active future (this avoids unnecessary work in many situations).
However sometimes you want to just fire off a task, not track its future,
and expect it to finish eventually.  You can use this function on a future
or collection of futures to ask Dask to complete the task even if no active
client is tracking it.

The results will not be kept in memory after the task completes (unless
there is an active future) so this is only useful for tasks that depend on
side effects.

* **Parameters:**
  **obj**
  : The futures that you want to run at least once

### Examples

```pycon
>>> fire_and_forget(client.submit(func, *args))
```

<!-- !! processed by numpydoc !! -->

### distributed.get_client(address=None, timeout=None, resolve_address=True) → [Client](#distributed.Client)

Get a client while within a task.

This client connects to the same scheduler to which the worker is connected

* **Parameters:**
  **address**
  : The address of the scheduler to connect to. Defaults to the scheduler
    the worker is connected to.

  **timeout**
  : Timeout (in seconds) for getting the Client. Defaults to the
    `distributed.comm.timeouts.connect` configuration value.

  **resolve_address**
  : Whether to resolve address to its canonical form.
* **Returns:**
  Client

#### SEE ALSO
[`get_worker`](https://distributed.dask.org/en/latest/api.html#distributed.get_worker)

[`worker_client`](https://distributed.dask.org/en/latest/api.html#distributed.worker_client)

[`secede`](#distributed.secede)

### Examples

```pycon
>>> def f():
...     client = get_client(timeout="10s")
...     futures = client.map(lambda x: x + 1, range(10))  # spawn many tasks
...     results = client.gather(futures)
...     return sum(results)
```

```pycon
>>> future = client.submit(f)
>>> future.result()
55
```

<!-- !! processed by numpydoc !! -->

### distributed.secede()

Have this task secede from the worker’s thread pool

This opens up a new scheduling slot and a new thread for a new task. This
enables the client to schedule tasks on this node, which is
especially useful while waiting for other jobs to finish (e.g., with
`client.gather`).

#### SEE ALSO
[`get_client`](#distributed.get_client)

[`get_worker`](https://distributed.dask.org/en/latest/api.html#distributed.get_worker)

### Examples

```pycon
>>> def mytask(x):
...     # do some work
...     client = get_client()
...     futures = client.map(...)  # do some remote work
...     secede()  # while that work happens, remove ourself from the pool
...     return client.gather(futures)  # return gathered results
```

<!-- !! processed by numpydoc !! -->

### distributed.rejoin()

Have this thread rejoin the ThreadPoolExecutor

This will block until a new slot opens up in the executor.  The next thread
to finish a task will leave the pool to allow this one to join.

#### SEE ALSO
[`secede`](#distributed.secede)
: leave the thread pool

<!-- !! processed by numpydoc !! -->

### distributed.wait(fs, timeout=None, return_when='ALL_COMPLETED')

Wait until all/any futures are finished

* **Parameters:**
  **fs**

  **timeout**
  : Time after which to raise a `dask.distributed.TimeoutError`.
    Can be a string like `"10 minutes"` or a number of seconds to wait.

  **return_when**
  : One of ALL_COMPLETED or FIRST_COMPLETED
* **Returns:**
  Named tuple of completed, not completed

<!-- !! processed by numpydoc !! -->

### distributed.print(\*args, sep: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = ' ', end: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = '\\n', file: [TextIO](https://docs.python.org/3/library/typing.html#typing.TextIO) | [None](https://docs.python.org/3/library/constants.html#None) = None, flush: [bool](https://docs.python.org/3/library/functions.html#bool) = False) → [None](https://docs.python.org/3/library/constants.html#None)

A drop-in replacement of the built-in `print` function for remote printing
from workers to clients. If called from outside a dask worker, its arguments
are passed directly to `builtins.print()`. If called by code running on a
worker, then in addition to printing locally, any clients connected
(possibly remotely) to the scheduler managing this worker will receive an
event instructing them to print the same output to their own standard output
or standard error streams. For example, the user can perform simple
debugging of remote computations by including calls to this `print`
function in the submitted code and inspecting the output in a local Jupyter
notebook or interpreter session.

All arguments behave the same as those of `builtins.print()`, with the
exception that the `file` keyword argument, if specified, must either be
`sys.stdout` or `sys.stderr`; arbitrary file-like objects are not
allowed.

All non-keyword arguments are converted to strings using `str()` and
written to the stream, separated by `sep` and followed by `end`. Both
`sep` and `end` must be strings; they can also be `None`, which means
to use the default values. If no objects are given, `print()` will just
write `end`.

* **Parameters:**
  **sep**
  : String inserted between values, default a space.

  **end**
  : String appended after the last value, default a newline.

  **file**
  : Defaults to the current sys.stdout.

  **flush**
  : Whether to forcibly flush the stream.

### Examples

```pycon
>>> from dask.distributed import Client, print
>>> client = distributed.Client(...)
>>> def worker_function():
...     print("Hello from worker!")
>>> client.submit(worker_function)
<Future: finished, type: NoneType, key: worker_function-...>
Hello from worker!
```

<!-- !! processed by numpydoc !! -->

### distributed.warn(message: str | Warning, category: type[Warning] | None = <class 'UserWarning'>, stacklevel: int = 1, source: ~typing.Any = None) → [None](https://docs.python.org/3/library/constants.html#None)

A drop-in replacement of the built-in `warnings.warn()` function for
issuing warnings remotely from workers to clients.

If called from outside a dask worker, its arguments are passed directly to
`warnings.warn()`. If called by code running on a worker, then in addition
to emitting a warning locally, any clients connected (possibly remotely) to
the scheduler managing this worker will receive an event instructing them to
emit the same warning (subject to their own local filters, etc.). When
implementing computations that may run on a worker, the user can call this
`warn` function to ensure that any remote client sessions will see their
warnings, for example in a Jupyter output cell.

While all of the arguments are respected by the locally emitted warning
(with same meanings as in `warnings.warn()`), `stacklevel` and
`source` are ignored by clients because they would not be meaningful in
the client’s thread.

### Examples

```pycon
>>> from dask.distributed import Client, warn
>>> client = Client()
>>> def do_warn():
...    warn("A warning from a worker.")
>>> client.submit(do_warn).result()
/path/to/distributed/client.py:678: UserWarning: A warning from a worker.
```

<!-- !! processed by numpydoc !! -->

### *class* distributed.Client(address=None, loop=None, timeout=<no_default>, set_as_default=True, scheduler_file=None, security=None, asynchronous=False, name=None, heartbeat_interval=None, serializers=None, deserializers=None, extensions={}, direct_to_workers=None, connection_limit=512, \*\*kwargs)

Connect to and submit computation to a Dask cluster

The Client connects users to a Dask cluster.  It provides an asynchronous
user interface around functions and futures.  This class resembles
executors in `concurrent.futures` but also allows `Future` objects
within `submit/map` calls.  When a Client is instantiated it takes over
all `dask.compute` and `dask.persist` calls by default.

It is also common to create a Client without specifying the scheduler
address , like `Client()`.  In this case the Client creates a
`LocalCluster` in the background and connects to that.  Any extra
keywords are passed from Client to LocalCluster in this case.  See the
LocalCluster documentation for more information.

* **Parameters:**
  **address: string, or Cluster**
  : This can be the address of a `Scheduler` server like a string
    `'127.0.0.1:8786'` or a cluster object like `LocalCluster()`

  **loop**
  : The event loop

  **timeout: int (defaults to configuration \`\`distributed.comm.timeouts.connect\`\`)**
  : Timeout duration for initial connection to the scheduler

  **set_as_default: bool (True)**
  : Use this Client as the global dask scheduler

  **scheduler_file: string (optional)**
  : Path to a file with scheduler information if available

  **security: Security or bool, optional**
  : Optional security information. If creating a local cluster can also
    pass in `True`, in which case temporary self-signed credentials will
    be created automatically.

  **asynchronous: bool (False by default)**
  : Set to True if using this client within async/await functions or within
    Tornado gen.coroutines.  Otherwise this should remain False for normal
    use.

  **name: string (optional)**
  : Gives the client a name that will be included in logs generated on
    the scheduler for matters relating to this client

  **heartbeat_interval: int (optional)**
  : Time in milliseconds between heartbeats to scheduler

  **serializers**
  : Iterable of approaches to use when serializing the object.
    See [Serialization](https://distributed.dask.org/en/latest/serialization.html#serialization) for more.

  **deserializers**
  : Iterable of approaches to use when deserializing the object.
    See [Serialization](https://distributed.dask.org/en/latest/serialization.html#serialization) for more.

  **extensions**
  : The extensions

  **direct_to_workers: bool (optional)**
  : Whether or not to connect directly to the workers, or to ask
    the scheduler to serve as intermediary.

  **connection_limit**
  : The number of open comms to maintain at once in the connection pool

  **\*\*kwargs:**
  : If you do not pass a scheduler address, Client will create a
    `LocalCluster` object, passing any extra keyword arguments.

#### SEE ALSO
[`distributed.scheduler.Scheduler`](deploying-python-advanced.md#distributed.Scheduler)
: Internal scheduler

[`distributed.LocalCluster`](https://distributed.dask.org/en/latest/api.html#distributed.LocalCluster)

### Examples

Provide cluster’s scheduler node address on initialization:

```pycon
>>> client = Client('127.0.0.1:8786')
```

Use `submit` method to send individual computations to the cluster

```pycon
>>> a = client.submit(add, 1, 2)
>>> b = client.submit(add, 10, 20)
```

Continue using submit or map on results to build up larger computations

```pycon
>>> c = client.submit(add, a, b)
```

Gather results with the `gather` method.

```pycon
>>> client.gather(c)
33
```

You can also call Client with no arguments in order to create your own
local cluster.

```pycon
>>> client = Client()  # makes your own local "cluster"
```

Extra keywords will be passed directly to LocalCluster

```pycon
>>> client = Client(n_workers=2, threads_per_worker=4)
```

<!-- !! processed by numpydoc !! -->

#### *property* amm

Convenience accessors for the active_memory_manager

<!-- !! processed by numpydoc !! -->

#### as_current()

Thread-local, Task-local context manager that causes the Client.current
class method to return self. Any Future objects deserialized inside this
context manager will be automatically attached to this Client.

<!-- !! processed by numpydoc !! -->

#### benchmark_hardware() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)

Run a benchmark on the workers for memory, disk, and network bandwidths

* **Returns:**
  result: dict
  : A dictionary mapping the names “disk”, “memory”, and “network” to
    dictionaries mapping sizes to bandwidths.  These bandwidths are
    averaged over many workers running computations across the cluster.

<!-- !! processed by numpydoc !! -->

#### call_stack(futures=None, keys=None)

The actively running call stack of all relevant keys

You can specify data of interest either by providing futures or
collections in the `futures=` keyword or a list of explicit keys in
the `keys=` keyword.  If neither are provided then all call stacks
will be returned.

* **Parameters:**
  **futures**
  : List of futures, defaults to all data

  **keys**
  : List of key names, defaults to all data

### Examples

```pycon
>>> df = dd.read_parquet(...).persist()
>>> client.call_stack(df)  # call on collections
```

```pycon
>>> client.call_stack()  # Or call with no arguments for all activity
```

<!-- !! processed by numpydoc !! -->

#### cancel(futures, asynchronous=None, force=False, reason=None, msg=None)

Cancel running futures
This stops future tasks from being scheduled if they have not yet run
and deletes them if they have already run.  After calling, this result
and all dependent results will no longer be accessible

* **Parameters:**
  **futures**
  : The list of Futures

  **asynchronous: bool**
  : If True the client is in asynchronous mode

  **force**
  : Cancel this future even if other clients desire it

  **reason: str**
  : Reason for cancelling the futures

  **msg**
  : Message that will be attached to the cancelled future

<!-- !! processed by numpydoc !! -->

#### close(timeout=<no_default>)

Close this client

Clients will also close automatically when your Python session ends

If you started a client without arguments like `Client()` then this
will also close the local cluster that was started at the same time.

* **Parameters:**
  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`

#### SEE ALSO
[`Client.restart`](#distributed.Client.restart)

<!-- !! processed by numpydoc !! -->

#### compute(collections, sync=False, optimize_graph=True, workers=None, allow_other_workers=False, resources=None, retries=0, priority=0, fifo_timeout='60s', actors=None, traverse=True, \*\*kwargs)

Compute dask collections on cluster

* **Parameters:**
  **collections**
  : Collections like dask.array or dataframe or dask.value objects

  **sync**
  : Returns Futures if False (default) or concrete values if True

  **optimize_graph**
  : Whether or not to optimize the underlying graphs

  **workers**
  : A set of worker hostnames on which computations may be performed.
    Leave empty to default to all workers (common case)

  **allow_other_workers**
  : Used with workers. Indicates whether or not the computations
    may be performed on workers that are not in the workers set(s).

  **retries**
  : Number of allowed automatic retries if computing a result fails

  **priority**
  : Optional prioritization of task.  Zero is default.
    Higher priorities take precedence

  **fifo_timeout**
  : Allowed amount of time between calls to consider the same priority

  **traverse**
  : By default dask traverses builtin python collections looking for
    dask objects passed to `compute`. For large collections this can
    be expensive. If none of the arguments contain any dask objects,
    set `traverse=False` to avoid doing this traversal.

  **resources**
  : Defines the resources each instance of this mapped task requires
    on the worker; e.g. `{'GPU': 2}`.
    See worker resources for details on defining
    resources.

  **actors**
  : Whether these tasks should exist on the worker as stateful actors.
    Specified on a global (True/False) or per-task (`{'x': True,
    'y': False}`) basis. See actors for additional details.

  **\*\*kwargs**
  : Options to pass to the graph optimize calls
* **Returns:**
  List of Futures if input is a sequence, or a single future otherwise

#### SEE ALSO
[`Client.get`](#distributed.Client.get)
: Normal synchronous dask.get function

### Examples

```pycon
>>> from dask import delayed
>>> from operator import add
>>> x = delayed(add)(1, 2)
>>> y = delayed(add)(x, x)
>>> xx, yy = client.compute([x, y])
>>> xx
<Future: status: finished, key: add-8f6e709446674bad78ea8aeecfee188e>
>>> xx.result()
3
>>> yy.result()
6
```

Also support single arguments

```pycon
>>> xx = client.compute(x)
```

<!-- !! processed by numpydoc !! -->

#### *classmethod* current(allow_global=True)

When running within the context of as_client, return the context-local
current client. Otherwise, return the latest initialised Client.
If no Client instances exist, raise ValueError.
If allow_global is set to False, raise ValueError if running outside of
the as_client context manager.

* **Parameters:**
  **allow_global**
  : If True returns the default client
* **Returns:**
  Client
  : The current client
* **Raises:**
  ValueError
  : If there is no client set, a ValueError is raised

#### SEE ALSO
`default_client`

<!-- !! processed by numpydoc !! -->

#### *property* dashboard_link

Link to the scheduler’s dashboard.

* **Returns:**
  str
  : Dashboard URL.

### Examples

Opening the dashboard in your default web browser:

```pycon
>>> import webbrowser
>>> from distributed import Client
>>> client = Client()
>>> webbrowser.open(client.dashboard_link)
```

<!-- !! processed by numpydoc !! -->

#### dump_cluster_state(filename: [str](https://docs.python.org/3/library/stdtypes.html#str) = 'dask-cluster-dump', write_from_scheduler: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = None, exclude: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)] = (), format: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['msgpack', 'yaml'] = 'msgpack', \*\*storage_options)

Extract a dump of the entire cluster state and persist to disk or a URL.
This is intended for debugging purposes only.

Warning: Memory usage on the scheduler (and client, if writing the dump locally)
can be large. On a large or long-running cluster, this can take several minutes.
The scheduler may be unresponsive while the dump is processed.

Results will be stored in a dict:

```default
{
    "scheduler": {...},  # scheduler state
    "workers": {
        worker_addr: {...},  # worker state
        ...
    }
    "versions": {
        "scheduler": {...},
        "workers": {
            worker_addr: {...},
            ...
        }
    }
}
```

* **Parameters:**
  **filename:**
  : The path or URL to write to. The appropriate file suffix (`.msgpack.gz` or
    `.yaml`) will be appended automatically.
    <br/>
    Must be a path supported by [`fsspec.open()`](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open) (like `s3://my-bucket/cluster-dump`,
    or `cluster-dumps/dump`). See `write_from_scheduler` to control whether
    the dump is written directly to `filename` from the scheduler, or sent
    back to the client over the network, then written locally.

  **write_from_scheduler:**
  : If None (default), infer based on whether `filename` looks like a URL
    or a local path: True if the filename contains `://` (like
    `s3://my-bucket/cluster-dump`), False otherwise (like `local_dir/cluster-dump`).
    <br/>
    If True, write cluster state directly to `filename` from the scheduler.
    If `filename` is a local path, the dump will be written to that
    path on the *scheduler’s* filesystem, so be careful if the scheduler is running
    on ephemeral hardware. Useful when the scheduler is attached to a network
    filesystem or persistent disk, or for writing to buckets.
    <br/>
    If False, transfer cluster state from the scheduler back to the client
    over the network, then write it to `filename`. This is much less
    efficient for large dumps, but useful when the scheduler doesn’t have
    access to any persistent storage.

  **exclude:**
  : A collection of attribute names which are supposed to be excluded
    from the dump, e.g. to exclude code, tracebacks, logs, etc.
    <br/>
    Defaults to exclude `run_spec`, which is the serialized user code.
    This is typically not required for debugging. To allow serialization
    of this, pass an empty tuple.

  **format:**
  : Either `"msgpack"` or `"yaml"`. If msgpack is used (default),
    the output will be stored in a gzipped file as msgpack.
    <br/>
    To read:
    ```default
    import gzip, msgpack
    with gzip.open("filename") as fd:
        state = msgpack.unpack(fd)
    ```
    <br/>
    or:
    ```default
    import yaml
    try:
        from yaml import CLoader as Loader
    except ImportError:
        from yaml import Loader
    with open("filename") as fd:
        state = yaml.load(fd, Loader=Loader)
    ```

  **\*\*storage_options:**
  : Any additional arguments to [`fsspec.open()`](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open) when writing to a URL.

<!-- !! processed by numpydoc !! -->

#### forward_logging(logger_name=None, level=0)

Begin forwarding the given logger (by default the root) and all loggers
under it from worker tasks to the client process. Whenever the named
logger handles a LogRecord on the worker-side, the record will be
serialized, sent to the client, and handled by the logger with the same
name on the client-side.

Note that worker-side loggers will only handle LogRecords if their level
is set appropriately, and the client-side logger will only emit the
forwarded LogRecord if its own level is likewise set appropriately. For
example, if your submitted task logs a DEBUG message to logger “foo”,
then in order for `forward_logging()` to cause that message to be
emitted in your client session, you must ensure that the logger “foo”
have its level set to DEBUG (or lower) in the worker process *and* in the
client process.

* **Parameters:**
  **logger_name**
  : The name of the logger to begin forwarding. The usual rules of the
    `logging` module’s hierarchical naming system apply. For example,
    if `name` is `"foo"`, then not only `"foo"`, but also
    `"foo.bar"`, `"foo.baz"`, etc. will be forwarded. If `name` is
    `None`, this indicates the root logger, and so *all* loggers will
    be forwarded.
    <br/>
    Note that a logger will only forward a given LogRecord if the
    logger’s level is sufficient for the LogRecord to be handled at all.

  **level**
  : Optionally restrict forwarding to LogRecords of this level or
    higher, even if the forwarded logger’s own level is lower.

### Examples

For purposes of the examples, suppose we configure client-side logging
as a user might: with a single StreamHandler attached to the root logger
with an output level of INFO and a simple output format:

```default
import logging
import distributed
import io, yaml

TYPICAL_LOGGING_CONFIG = '''
version: 1
handlers:
  console:
    class : logging.StreamHandler
    formatter: default
    level   : INFO
formatters:
  default:
    format: '%(asctime)s %(levelname)-8s [worker %(worker)s] %(name)-15s %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'
root:
  handlers:
    - console
'''
config = yaml.safe_load(io.StringIO(TYPICAL_LOGGING_CONFIG))
logging.config.dictConfig(config)
```

Now create a client and begin forwarding the root logger from workers
back to our local client process.

```pycon
>>> client = distributed.Client()
>>> client.forward_logging()  # forward the root logger at any handled level
```

Then submit a task that does some error logging on a worker. We see
output from the client-side StreamHandler.

```pycon
>>> def do_error():
...     logging.getLogger("user.module").error("Hello error")
...     return 42
>>> client.submit(do_error).result()
2022-11-09 03:43:25 ERROR    [worker tcp://127.0.0.1:34783] user.module     Hello error
42
```

Note how an attribute `"worker"` is also added by dask to the
forwarded LogRecord, which our custom formatter uses. This is useful for
identifying exactly which worker logged the error.

One nuance worth highlighting: even though our client-side root logger
is configured with a level of INFO, the worker-side root loggers still
have their default level of ERROR because we haven’t done any explicit
logging configuration on the workers. Therefore worker-side INFO logs
will *not* be forwarded because they never even get handled in the first
place.

```pycon
>>> def do_info_1():
...     # no output on the client side
...     logging.getLogger("user.module").info("Hello info the first time")
...     return 84
>>> client.submit(do_info_1).result()
84
```

It is necessary to set the client-side logger’s level to INFO before the info
message will be handled and forwarded to the client. In other words, the
“effective” level of the client-side forwarded logging is the maximum of each
logger’s client-side and worker-side levels.

```pycon
>>> def do_info_2():
...     logger = logging.getLogger("user.module")
...     logger.setLevel(logging.INFO)
...     # now produces output on the client side
...     logger.info("Hello info the second time")
...     return 84
>>> client.submit(do_info_2).result()
2022-11-09 03:57:39 INFO     [worker tcp://127.0.0.1:42815] user.module     Hello info the second time
84
```

<!-- !! processed by numpydoc !! -->

#### futures_of(futures)

Wrapper method of futures_of

* **Parameters:**
  **futures**
  : The futures

<!-- !! processed by numpydoc !! -->

#### gather(futures, errors='raise', direct=None, asynchronous=None)

Gather futures from distributed memory

Accepts a future, nested container of futures, iterator, or queue.
The return type will match the input type.

* **Parameters:**
  **futures**
  : This can be a possibly nested collection of Future objects.
    Collections can be lists, sets, or dictionaries

  **errors**
  : Either ‘raise’ or ‘skip’ if we should raise if a future has erred
    or skip its inclusion in the output collection

  **direct**
  : Whether or not to connect directly to the workers, or to ask
    the scheduler to serve as intermediary.  This can also be set when
    creating the Client.

  **asynchronous: bool**
  : If True the client is in asynchronous mode
* **Returns:**
  results: a collection of the same type as the input, but now with

  gathered results rather than futures

#### SEE ALSO
[`Client.scatter`](#distributed.Client.scatter)
: Send data out to cluster

### Examples

```pycon
>>> from operator import add
>>> c = Client('127.0.0.1:8787')
>>> x = c.submit(add, 1, 2)
>>> c.gather(x)
3
>>> c.gather([x, [x], x])  # support lists and dicts
[3, [3], 3]
```

<!-- !! processed by numpydoc !! -->

#### get(dsk, keys, workers=None, allow_other_workers=None, resources=None, sync=True, asynchronous=None, direct=None, retries=None, priority=0, fifo_timeout='60s', actors=None, \*\*kwargs)

Compute dask graph

* **Parameters:**
  **dsk**

  **keys**

  **workers**
  : A set of worker addresses or hostnames on which computations may be
    performed. Leave empty to default to all workers (common case)

  **allow_other_workers**
  : Used with `workers`. Indicates whether or not the computations
    may be performed on workers that are not in the workers set(s).

  **resources**
  : Defines the `resources` each instance of this mapped task
    requires on the worker; e.g. `{'GPU': 2}`.
    See worker resources for details on defining
    resources.

  **sync**
  : Returns Futures if False or concrete values if True (default).

  **asynchronous: bool**
  : If True the client is in asynchronous mode

  **direct**
  : Whether or not to connect directly to the workers, or to ask
    the scheduler to serve as intermediary.  This can also be set when
    creating the Client.

  **retries**
  : Number of allowed automatic retries if computing a result fails

  **priority**
  : Optional prioritization of task.  Zero is default.
    Higher priorities take precedence

  **fifo_timeout**
  : Allowed amount of time between calls to consider the same priority

  **actors**
  : Whether these tasks should exist on the worker as stateful actors.
    Specified on a global (True/False) or per-task (`{'x': True,
    'y': False}`) basis. See actors for additional details.
* **Returns:**
  results
  : If ‘sync’ is True, returns the results. Otherwise, returns the
    known data packed
    If ‘sync’ is False, returns the known data. Otherwise, returns
    the results

#### SEE ALSO
[`Client.compute`](#distributed.Client.compute)
: Compute asynchronous collections

### Examples

```pycon
>>> from operator import add
>>> c = Client('127.0.0.1:8787')
>>> c.get({'x': (add, 1, 2)}, 'x')
3
```

<!-- !! processed by numpydoc !! -->

#### get_dataset(name: str | int | float | tuple[str | int | float | tuple[Key, ...], ...] | list[str | int | float | tuple[str | int | float | tuple[Key, ...], ...]], default=<no_default>, \*\*kwargs)

Get named dataset from the scheduler if present.
Return the default or raise a KeyError if not present.

* **Parameters:**
  **name**
  : name(s) of the dataset(s) to retrieve

  **default**
  : optional, not set by default
    If set, do not raise a KeyError if the name is not present but
    return this default
* **Returns:**
  The dataset from the scheduler, if present.

  If name is a list of keys, return a list of datasets in the same order.

#### SEE ALSO
[`Client.publish_dataset`](#distributed.Client.publish_dataset)

[`Client.unpublish_dataset`](#distributed.Client.unpublish_dataset)

[`Client.list_datasets`](#distributed.Client.list_datasets)

<!-- !! processed by numpydoc !! -->

#### get_events(topic: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None)

Retrieve structured topic logs

* **Parameters:**
  **topic**
  : Name of topic log to retrieve events for. If no `topic` is
    provided, then logs for all topics will be returned.

<!-- !! processed by numpydoc !! -->

#### get_executor(\*\*kwargs)

Return a concurrent.futures Executor for submitting tasks on this
Client

* **Parameters:**
  **\*\*kwargs**
  : Any submit()- or map()- compatible arguments, such as
    workers or resources.
* **Returns:**
  ClientExecutor
  : An Executor object that’s fully compatible with the
    concurrent.futures API.

<!-- !! processed by numpydoc !! -->

#### get_metadata(keys, default=<no_default>)

Get arbitrary metadata from scheduler

See set_metadata for the full docstring with examples

* **Parameters:**
  **keys**
  : Key to access.  If a list then gets within a nested collection

  **default**
  : If the key does not exist then return this value instead.
    If not provided then this raises a KeyError if the key is not
    present

#### SEE ALSO
[`Client.set_metadata`](#distributed.Client.set_metadata)

<!-- !! processed by numpydoc !! -->

#### get_scheduler_logs(n=None)

Get logs from scheduler

* **Parameters:**
  **n**
  : Number of logs to retrieve.  Maxes out at 10000 by default,
    configurable via the `distributed.admin.log-length`
    configuration value.
* **Returns:**
  Logs in reversed order (newest first)

<!-- !! processed by numpydoc !! -->

#### get_task_stream(start=None, stop=None, count=None, plot=False, filename='task-stream.html', bokeh_resources=None)

Get task stream data from scheduler

This collects the data present in the diagnostic “Task Stream” plot on
the dashboard.  It includes the start, stop, transfer, and
deserialization time of every task for a particular duration.

Note that the task stream diagnostic does not run by default.  You may
wish to call this function once before you start work to ensure that
things start recording, and then again after you have completed.

* **Parameters:**
  **start**
  : When you want to start recording
    If a number it should be the result of calling time()
    If a string then it should be a time difference before now,
    like ’60s’ or ‘500 ms’

  **stop**
  : When you want to stop recording

  **count**
  : The number of desired records, ignored if both start and stop are
    specified

  **plot**
  : If true then also return a Bokeh figure
    If plot == ‘save’ then save the figure to a file

  **filename**
  : The filename to save to if you set `plot='save'`

  **bokeh_resources**
  : Specifies if the resource component is INLINE or CDN
* **Returns:**
  L: List[Dict]

#### SEE ALSO
[`get_task_stream`](#distributed.Client.get_task_stream)
: a context manager version of this method

### Examples

```pycon
>>> client.get_task_stream()  # prime plugin if not already connected
>>> x.compute()  # do some work
>>> client.get_task_stream()
[{'task': ...,
  'type': ...,
  'thread': ...,
  ...}]
```

Pass the `plot=True` or `plot='save'` keywords to get back a Bokeh
figure

```pycon
>>> data, figure = client.get_task_stream(plot='save', filename='myfile.html')
```

Alternatively consider the context manager

```pycon
>>> from dask.distributed import get_task_stream
>>> with get_task_stream() as ts:
...     x.compute()
>>> ts.data
[...]
```

<!-- !! processed by numpydoc !! -->

#### get_versions(check: [bool](https://docs.python.org/3/library/functions.html#bool) = False, packages: [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None) → VersionsDict | [Coroutine](https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine)[[Any](https://docs.python.org/3/library/typing.html#typing.Any), [Any](https://docs.python.org/3/library/typing.html#typing.Any), VersionsDict]

Return version info for the scheduler, all workers and myself

* **Parameters:**
  **check**
  : raise ValueError if all required & optional packages
    do not match

  **packages**
  : Extra package names to check

### Examples

```pycon
>>> c.get_versions()
```

```pycon
>>> c.get_versions(packages=['sklearn', 'geopandas'])
```

<!-- !! processed by numpydoc !! -->

#### get_worker_logs(n=None, workers=None, nanny=False)

Get logs from workers

* **Parameters:**
  **n**
  : Number of logs to retrieve.  Maxes out at 10000 by default,
    configurable via the `distributed.admin.log-length`
    configuration value.

  **workers**
  : List of worker addresses to retrieve.  Gets all workers by default.

  **nanny**
  : Whether to get the logs from the workers (False) or the nannies
    (True). If specified, the addresses in workers should still be
    the worker addresses, not the nanny addresses.
* **Returns:**
  Dictionary mapping worker address to logs.

  Logs are returned in reversed order (newest first)

<!-- !! processed by numpydoc !! -->

#### has_what(workers=None, \*\*kwargs)

Which keys are held by which workers

This returns the keys of the data that are held in each worker’s
memory.

* **Parameters:**
  **workers**
  : A list of worker addresses, defaults to all

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.who_has`](#distributed.Client.who_has)

[`Client.nthreads`](#distributed.Client.nthreads)

[`Client.processing`](#distributed.Client.processing)

### Examples

```pycon
>>> x, y, z = c.map(inc, [1, 2, 3])
>>> wait([x, y, z])
>>> c.has_what()
{'192.168.1.141:46784': ['inc-1c8dd6be1c21646c71f76c16d09304ea',
                         'inc-fd65c238a7ea60f6a01bf4c8a5fcf44b',
                         'inc-1e297fc27658d7b67b3a758f16bcf47a']}
```

<!-- !! processed by numpydoc !! -->

#### list_datasets(\*\*kwargs)

List named datasets available on the scheduler

#### SEE ALSO
[`Client.publish_dataset`](#distributed.Client.publish_dataset)

[`Client.unpublish_dataset`](#distributed.Client.unpublish_dataset)

[`Client.get_dataset`](#distributed.Client.get_dataset)

<!-- !! processed by numpydoc !! -->

#### log_event(topic: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection)[[str](https://docs.python.org/3/library/stdtypes.html#str)], msg: [Any](https://docs.python.org/3/library/typing.html#typing.Any))

Log an event under a given topic

* **Parameters:**
  **topic**
  : Name of the topic under which to log an event. To log the same
    event under multiple topics, pass a list of topic names.

  **msg**
  : Event message to log. Note this must be msgpack serializable.

### Examples

```pycon
>>> from time import time
>>> client.log_event("current-time", time())
```

<!-- !! processed by numpydoc !! -->

#### map(func: [Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[...], \_T], \*iterables: [Collection](https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection), key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [list](https://docs.python.org/3/library/stdtypes.html#list) | [None](https://docs.python.org/3/library/constants.html#None) = None, workers: [str](https://docs.python.org/3/library/stdtypes.html#str) | [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, retries: [int](https://docs.python.org/3/library/functions.html#int) | [None](https://docs.python.org/3/library/constants.html#None) = None, resources: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)] | [None](https://docs.python.org/3/library/constants.html#None) = None, priority: [int](https://docs.python.org/3/library/functions.html#int) = 0, allow_other_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = False, fifo_timeout: [str](https://docs.python.org/3/library/stdtypes.html#str) = '100 ms', actor: [bool](https://docs.python.org/3/library/functions.html#bool) = False, actors: [bool](https://docs.python.org/3/library/functions.html#bool) = False, pure: [bool](https://docs.python.org/3/library/functions.html#bool) = True, batch_size=None, \*\*kwargs) → [list](https://docs.python.org/3/library/stdtypes.html#list)[[Future](#distributed.Future)[\_T]]

Map a function on a sequence of arguments

Arguments can be normal objects or Futures

* **Parameters:**
  **func**
  : Callable to be scheduled for execution. If `func` returns a coroutine, it
    will be run on the main event loop of a worker. Otherwise `func` will be
    run in a worker’s task executor pool (see `Worker.executors` for more
    information.)

  **iterables**
  : List-like objects to map over.  They should have the same length.

  **key**
  : Prefix for task names if string.  Explicit names if list.

  **workers**
  : A set of worker hostnames on which computations may be performed.
    Leave empty to default to all workers (common case)

  **retries**
  : Number of allowed automatic retries if a task fails

  **resources**
  : Defines the resources each instance of this mapped task requires
    on the worker; e.g. `{'GPU': 2}`.
    See worker resources for details on defining
    resources.

  **priority**
  : Optional prioritization of task.  Zero is default.
    Higher priorities take precedence

  **allow_other_workers**
  : Used with workers. Indicates whether or not the computations
    may be performed on workers that are not in the workers set(s).

  **fifo_timeout**
  : Allowed amount of time between calls to consider the same priority

  **actor**
  : Whether these tasks should exist on the worker as stateful actors.
    See actors for additional details.

  **actors**
  : Alias for actor

  **pure**
  : Whether or not the function is pure.  Set `pure=False` for
    impure functions like `np.random.random`. Note that if both
    `actor` and `pure` kwargs are set to True, then the value
    of `pure` will be reverted to False, since an actor is stateful.
    See [Pure Functions by Default](https://distributed.dask.org/en/latest/client.html#pure-functions) for more details.

  **batch_size**
  : Submit tasks to the scheduler in batches of (at most)
    `batch_size`.
    The tradeoff in batch size is that large batches avoid more per-batch overhead,
    but batches that are too big can take a long time to submit and unreasonably delay
    the cluster from starting its processing.

  **\*\*kwargs**
  : Extra keyword arguments to send to the function.
    Large values will be included explicitly in the task graph.
* **Returns:**
  List, iterator, or Queue of futures, depending on the type of the

  inputs.

#### SEE ALSO
[`Client.submit`](#distributed.Client.submit)
: Submit a single function

### Notes

The current implementation of a task graph resolution searches for occurrences of `key`
and replaces it with a corresponding `Future` result. That can lead to unwanted
substitution of strings passed as arguments to a task if these strings match some `key`
that already exists on a cluster. To avoid these situations it is required to use unique
values if a `key` is set manually.
See [https://github.com/dask/dask/issues/9969](https://github.com/dask/dask/issues/9969) to track progress on resolving this issue.

### Examples

```pycon
>>> L = client.map(func, sequence)
```

<!-- !! processed by numpydoc !! -->

#### nbytes(keys=None, summary=True, \*\*kwargs)

The bytes taken up by each key on the cluster

This is as measured by `sys.getsizeof` which may not accurately
reflect the true cost.

* **Parameters:**
  **keys**
  : A list of keys, defaults to all keys

  **summary**
  : Summarize keys into key types

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.who_has`](#distributed.Client.who_has)

### Examples

```pycon
>>> x, y, z = c.map(inc, [1, 2, 3])
>>> c.nbytes(summary=False)
{'inc-1c8dd6be1c21646c71f76c16d09304ea': 28,
 'inc-1e297fc27658d7b67b3a758f16bcf47a': 28,
 'inc-fd65c238a7ea60f6a01bf4c8a5fcf44b': 28}
```

```pycon
>>> c.nbytes(summary=True)
{'inc': 84}
```

<!-- !! processed by numpydoc !! -->

#### ncores(workers=None, \*\*kwargs)

The number of threads/cores available on each worker node

* **Parameters:**
  **workers**
  : A list of workers that we care about specifically.
    Leave empty to receive information about all workers.

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.who_has`](#distributed.Client.who_has)

[`Client.has_what`](#distributed.Client.has_what)

### Examples

```pycon
>>> c.nthreads()
{'192.168.1.141:46784': 8,
 '192.167.1.142:47548': 8,
 '192.167.1.143:47329': 8,
 '192.167.1.144:37297': 8}
```

<!-- !! processed by numpydoc !! -->

#### normalize_collection(collection)

Replace collection’s tasks by already existing futures if they exist

This normalizes the tasks within a collections task graph against the
known futures within the scheduler.  It returns a copy of the
collection with a task graph that includes the overlapping futures.

* **Parameters:**
  **collection**
  : Collection like dask.array or dataframe or dask.value objects
* **Returns:**
  **collection**
  : Collection with its tasks replaced with any existing futures.

#### SEE ALSO
[`Client.persist`](#distributed.Client.persist)
: trigger computation of collection’s tasks

### Examples

```pycon
>>> len(x.__dask_graph__())  # x is a dask collection with 100 tasks
100
>>> set(client.futures).intersection(x.__dask_graph__())  # some overlap exists
10
```

```pycon
>>> x = client.normalize_collection(x)
>>> len(x.__dask_graph__())  # smaller computational graph
20
```

<!-- !! processed by numpydoc !! -->

#### nthreads(workers=None, \*\*kwargs)

The number of threads/cores available on each worker node

* **Parameters:**
  **workers**
  : A list of workers that we care about specifically.
    Leave empty to receive information about all workers.

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.who_has`](#distributed.Client.who_has)

[`Client.has_what`](#distributed.Client.has_what)

### Examples

```pycon
>>> c.nthreads()
{'192.168.1.141:46784': 8,
 '192.167.1.142:47548': 8,
 '192.167.1.143:47329': 8,
 '192.167.1.144:37297': 8}
```

<!-- !! processed by numpydoc !! -->

#### persist(collections, optimize_graph=True, workers=None, allow_other_workers=None, resources=None, retries=None, priority=0, fifo_timeout='60s', actors=None, \*\*kwargs)

Persist dask collections on cluster

Starts computation of the collection on the cluster in the background.
Provides a new dask collection that is semantically identical to the
previous one, but now based off of futures currently in execution.

* **Parameters:**
  **collections**
  : Collections like dask.array or dataframe or dask.value objects

  **optimize_graph**
  : Whether or not to optimize the underlying graphs

  **workers**
  : A set of worker hostnames on which computations may be performed.
    Leave empty to default to all workers (common case)

  **allow_other_workers**
  : Used with workers. Indicates whether or not the computations
    may be performed on workers that are not in the workers set(s).

  **retries**
  : Number of allowed automatic retries if computing a result fails

  **priority**
  : Optional prioritization of task.  Zero is default.
    Higher priorities take precedence

  **fifo_timeout**
  : Allowed amount of time between calls to consider the same priority

  **resources**
  : Defines the resources each instance of this mapped task requires
    on the worker; e.g. `{'GPU': 2}`.
    See worker resources for details on defining
    resources.

  **actors**
  : Whether these tasks should exist on the worker as stateful actors.
    Specified on a global (True/False) or per-task (`{'x': True,
    'y': False}`) basis. See actors for additional details.
* **Returns:**
  List of collections, or single collection, depending on type of input.

#### SEE ALSO
[`Client.compute`](#distributed.Client.compute)

### Examples

```pycon
>>> xx = client.persist(x)
>>> xx, yy = client.persist([x, y])
```

<!-- !! processed by numpydoc !! -->

#### processing(workers=None)

The tasks currently running on each worker

* **Parameters:**
  **workers**
  : A list of worker addresses, defaults to all

#### SEE ALSO
[`Client.who_has`](#distributed.Client.who_has)

[`Client.has_what`](#distributed.Client.has_what)

[`Client.nthreads`](#distributed.Client.nthreads)

### Examples

```pycon
>>> x, y, z = c.map(inc, [1, 2, 3])
>>> c.processing()
{'192.168.1.141:46784': ['inc-1c8dd6be1c21646c71f76c16d09304ea',
                         'inc-fd65c238a7ea60f6a01bf4c8a5fcf44b',
                         'inc-1e297fc27658d7b67b3a758f16bcf47a']}
```

<!-- !! processed by numpydoc !! -->

#### profile(key=None, start=None, stop=None, workers=None, merge_workers=True, plot=False, filename=None, server=False, scheduler=False)

Collect statistical profiling information about recent work

* **Parameters:**
  **key**
  : Key prefix to select, this is typically a function name like ‘inc’
    Leave as None to collect all data

  **start**

  **stop**

  **workers**
  : List of workers to restrict profile information

  **server**
  : If true, return the profile of the worker’s administrative thread
    rather than the worker threads.
    This is useful when profiling Dask itself, rather than user code.

  **scheduler**
  : If true, return the profile information from the scheduler’s
    administrative thread rather than the workers.
    This is useful when profiling Dask’s scheduling itself.

  **plot**
  : Whether or not to return a plot object

  **filename**
  : Filename to save the plot

### Examples

```pycon
>>> client.profile()  # call on collections
>>> client.profile(filename='dask-profile.html')  # save to html file
```

<!-- !! processed by numpydoc !! -->

#### publish_dataset(\*args: [Any](https://docs.python.org/3/library/typing.html#typing.Any), name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], override: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs)

#### publish_dataset(\*args: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)], override: [bool](https://docs.python.org/3/library/functions.html#bool) = False, \*\*kwargs)

Publish named datasets to scheduler

This stores a named reference to one or more dask collections or futures
on the scheduler.  These references are available to other Clients
which can download the collections or futures with `get_dataset`.

Datasets are not immediately computed. You should call `persist` prior to
publishing a dataset. Any unpersisted keys will be stored on the scheduler
uncomputed and returned as-is to the user when calling `get_dataset`.

* **Parameters:**
  **args**
  : Alternatively, a single dict of {name: object} pairs.

  **name**
  : Name to publish args under

  **override: bool**
  : False (default) to raise KeyError if a dataset with the same name already
    exists on the scheduler; True to overwrite it.

  **kwargs**
  : named collections to publish on the scheduler
* **Returns:**
  None

#### SEE ALSO
[`Client.list_datasets`](#distributed.Client.list_datasets)

[`Client.get_dataset`](#distributed.Client.get_dataset)

[`Client.unpublish_dataset`](#distributed.Client.unpublish_dataset)

[`Client.persist`](#distributed.Client.persist)

### Examples

Publishing client:

```pycon
>>> df = dd.read_csv('s3://...')
>>> df = c.persist(df)
>>> c.publish_dataset({"my_dataset": df})
```

Alternative invocation
>>> c.publish_dataset(my_dataset=df)

Alternative invocation
>>> c.publish_dataset(df, name=’my_dataset’)

Receiving client:

```pycon
>>> c.list_datasets()
['my_dataset']
>>> df2 = c.get_dataset('my_dataset')
```

<!-- !! processed by numpydoc !! -->

#### rebalance(futures=None, workers=None, \*\*kwargs)

Rebalance data within network

Move data between workers to roughly balance memory burden.  This
either affects a subset of the keys/workers or the entire network,
depending on keyword arguments.

For details on the algorithm and configuration options, refer to the matching
scheduler-side method [`rebalance()`](https://distributed.dask.org/en/latest/scheduling-state.html#distributed.scheduler.Scheduler.rebalance).

#### WARNING
This operation is generally not well tested against normal operation of the
scheduler. It is not recommended to use it while waiting on computations.

* **Parameters:**
  **futures**
  : A list of futures to balance, defaults all data

  **workers**
  : A list of workers on which to balance, defaults to all workers

  **\*\*kwargs**
  : Optional keyword arguments for the function

<!-- !! processed by numpydoc !! -->

#### register_plugin(plugin: [NannyPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.NannyPlugin) | [SchedulerPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.SchedulerPlugin) | [WorkerPlugin](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.WorkerPlugin), name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = None)

Register a plugin.

See [https://distributed.readthedocs.io/en/latest/plugins.html](https://distributed.readthedocs.io/en/latest/plugins.html)

* **Parameters:**
  **plugin**
  : A nanny, scheduler, or worker plugin to register.

  **name**
  : Name for the plugin; if None, a name is taken from the
    plugin instance or automatically generated if not present.

<!-- !! processed by numpydoc !! -->

#### register_worker_callbacks(setup=None)

Registers a setup callback function for all current and future workers.

This registers a new setup function for workers in this cluster. The
function will run immediately on all currently connected workers. It
will also be run upon connection by any workers that are added in the
future. Multiple setup functions can be registered - these will be
called in the order they were added.

If the function takes an input argument named `dask_worker` then
that variable will be populated with the worker itself.

* **Parameters:**
  **setup**
  : Function to register and run on all workers

<!-- !! processed by numpydoc !! -->

#### replicate(futures, n=None, workers=None, branching_factor=2, \*\*kwargs)

Set replication of futures within network

Copy data onto many workers.  This helps to broadcast frequently
accessed data and can improve resilience.

This performs a tree copy of the data throughout the network
individually on each piece of data.  This operation blocks until
complete.  It does not guarantee replication of data to future workers.

#### NOTE
This method is incompatible with the Active Memory Manager’s
[ReduceReplicas](https://distributed.dask.org/en/latest/active_memory_manager.html#reducereplicas) policy. If you wish to use it, you must first disable
the policy or disable the AMM entirely.

* **Parameters:**
  **futures**
  : Futures we wish to replicate

  **n**
  : Number of processes on the cluster on which to replicate the data.
    Defaults to all.

  **workers**
  : Workers on which we want to restrict the replication.
    Defaults to all.

  **branching_factor**
  : The number of workers that can copy data in each generation

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.rebalance`](#distributed.Client.rebalance)

### Examples

```pycon
>>> x = c.submit(func, *args)
>>> c.replicate([x])  # send to all workers
>>> c.replicate([x], n=3)  # send to three workers
>>> c.replicate([x], workers=['alice', 'bob'])  # send to specific
>>> c.replicate([x], n=1, workers=['alice', 'bob'])  # send to one of specific workers
>>> c.replicate([x], n=1)  # reduce replications
```

<!-- !! processed by numpydoc !! -->

#### restart(timeout: str | int | float | ~typing.Literal[_NoDefault.no_default] = <no_default>, wait_for_workers: bool = True)

Restart all workers. Reset local state. Optionally wait for workers to return.

Workers without nannies are shut down, hoping an external deployment system
will restart them. Therefore, if not using nannies and your deployment system
does not automatically restart workers, `restart` will just shut down all
workers, then time out!

After `restart`, all connected workers are new, regardless of whether `TimeoutError`
was raised. Any workers that failed to shut down in time are removed, and
may or may not shut down on their own in the future.

* **Parameters:**
  **timeout:**
  : How long to wait for workers to shut down and come back, if `wait_for_workers`
    is True, otherwise just how long to wait for workers to shut down.
    Raises `asyncio.TimeoutError` if this is exceeded.

  **wait_for_workers:**
  : Whether to wait for all workers to reconnect, or just for them to shut down
    (default True). Use `restart(wait_for_workers=False)` combined with
    [`Client.wait_for_workers()`](#distributed.Client.wait_for_workers) for granular control over how many workers to
    wait for.

#### SEE ALSO
[`Scheduler.restart`](deploying-python-advanced.md#distributed.Scheduler.restart)

[`Client.restart_workers`](#distributed.Client.restart_workers)

<!-- !! processed by numpydoc !! -->

#### restart_workers(workers: list[str], timeout: str | int | float | ~typing.Literal[_NoDefault.no_default] = <no_default>, raise_for_error: bool = True)

Restart a specified set of workers

#### NOTE
Only workers being monitored by a [`distributed.Nanny`](deploying-python-advanced.md#distributed.Nanny) can be restarted.
See `Nanny.restart` for more details.

* **Parameters:**
  **workers**
  : Workers to restart. This can be a list of worker addresses, names, or a both.

  **timeout**
  : Number of seconds to wait

  **raise_for_error: bool (default True)**
  : Whether to raise a [`TimeoutError`](https://docs.python.org/3/library/exceptions.html#TimeoutError) if restarting worker(s) doesn’t
    finish within `timeout`, or another exception caused from restarting
    worker(s).
* **Returns:**
  dict[str, “OK” | “removed” | “timed out”]
  : Mapping of worker and restart status, the keys will match the original
    values passed in via `workers`.

#### SEE ALSO
[`Client.restart`](#distributed.Client.restart)

### Notes

This method differs from [`Client.restart()`](#distributed.Client.restart) in that this method
simply restarts the specified set of workers, while `Client.restart`
will restart all workers and also reset local state on the cluster
(e.g. all keys are released).

Additionally, this method does not gracefully handle tasks that are
being executed when a worker is restarted. These tasks may fail or have
their suspicious count incremented.

### Examples

You can get information about active workers using the following:

```pycon
>>> workers = client.scheduler_info()['workers']
```

From that list you may want to select some workers to restart

```pycon
>>> client.restart_workers(workers=['tcp://address:port', ...])
```

<!-- !! processed by numpydoc !! -->

#### retire_workers(workers: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, close_workers: [bool](https://docs.python.org/3/library/functions.html#bool) = True, \*\*kwargs)

Retire certain workers on the scheduler

See [`distributed.Scheduler.retire_workers()`](deploying-python-advanced.md#distributed.Scheduler.retire_workers) for the full docstring.

* **Parameters:**
  **workers**

  **close_workers**

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
`dask.distributed.Scheduler.retire_workers`

### Examples

You can get information about active workers using the following:

```pycon
>>> workers = client.scheduler_info()['workers']
```

From that list you may want to select some workers to close

```pycon
>>> client.retire_workers(workers=['tcp://address:port', ...])
```

<!-- !! processed by numpydoc !! -->

#### retry(futures, asynchronous=None)

Retry failed futures

* **Parameters:**
  **futures**
  : The list of Futures

  **asynchronous: bool**
  : If True the client is in asynchronous mode

<!-- !! processed by numpydoc !! -->

#### run(function, \*args, workers: [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str)] | [None](https://docs.python.org/3/library/constants.html#None) = None, wait: [bool](https://docs.python.org/3/library/functions.html#bool) = True, nanny: [bool](https://docs.python.org/3/library/functions.html#bool) = False, on_error: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['raise', 'return', 'ignore'] = 'raise', \*\*kwargs)

Run a function on all workers outside of task scheduling system

This calls a function on all currently known workers immediately,
blocks until those results come back, and returns the results
asynchronously as a dictionary keyed by worker address.  This method
is generally used for side effects such as collecting diagnostic
information or installing libraries.

If your function takes an input argument named `dask_worker` then
that variable will be populated with the worker itself.

* **Parameters:**
  **function**
  : The function to run

  **\*args**
  : Optional arguments for the remote function

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

  **workers**
  : Workers on which to run the function. Defaults to all known
    workers.

  **wait**
  : If the function is asynchronous whether or not to wait until that
    function finishes.

  **nanny**
  : Whether to run `function` on the nanny. By default, the function
    is run on the worker process.  If specified, the addresses in
    `workers` should still be the worker addresses, not the nanny addresses.

  **on_error: “raise” | “return” | “ignore”**
  : If the function raises an error on a worker:
    <br/>
    raise
    : (default) Re-raise the exception on the client.
      The output from other workers will be lost.
    <br/>
    return
    : Return the Exception object instead of the function output for
      the worker
    <br/>
    ignore
    : Ignore the exception and remove the worker from the result dict

### Examples

```pycon
>>> c.run(os.getpid)
{'192.168.0.100:9000': 1234,
 '192.168.0.101:9000': 4321,
 '192.168.0.102:9000': 5555}
```

Restrict computation to particular workers with the `workers=`
keyword argument.

```pycon
>>> c.run(os.getpid, workers=['192.168.0.100:9000',
...                           '192.168.0.101:9000'])
{'192.168.0.100:9000': 1234,
 '192.168.0.101:9000': 4321}
```

```pycon
>>> def get_status(dask_worker):
...     return dask_worker.status
```

```pycon
>>> c.run(get_status)
{'192.168.0.100:9000': 'running',
 '192.168.0.101:9000': 'running}
```

Run asynchronous functions in the background:

```pycon
>>> async def print_state(dask_worker):
...    while True:
...        print(dask_worker.status)
...        await asyncio.sleep(1)
```

```pycon
>>> c.run(print_state, wait=False)
```

<!-- !! processed by numpydoc !! -->

#### run_on_scheduler(function, \*args, \*\*kwargs)

Run a function on the scheduler process

This is typically used for live debugging.  The function should take a
keyword argument `dask_scheduler=`, which will be given the scheduler
object itself.

* **Parameters:**
  **function**
  : The function to run on the scheduler process

  **\*args**
  : Optional arguments for the function

  **\*\*kwargs**
  : Optional keyword arguments for the function

#### SEE ALSO
[`Client.run`](#distributed.Client.run)
: Run a function on all workers

### Examples

```pycon
>>> def get_number_of_tasks(dask_scheduler=None):
...     return len(dask_scheduler.tasks)
```

```pycon
>>> client.run_on_scheduler(get_number_of_tasks)
100
```

Run asynchronous functions in the background:

```pycon
>>> async def print_state(dask_scheduler):
...    while True:
...        print(dask_scheduler.status)
...        await asyncio.sleep(1)
```

```pycon
>>> c.run(print_state, wait=False)
```

<!-- !! processed by numpydoc !! -->

#### scatter(data, workers=None, broadcast=False, direct=None, hash=True, timeout=<no_default>, asynchronous=None)

Scatter data into distributed memory

This moves data from the local client process into the workers of the
distributed scheduler.  Note that it is often better to submit jobs to
your workers to have them load the data rather than loading data
locally and then scattering it out to them.

* **Parameters:**
  **data**
  : Data to scatter out to workers.  Output type matches input type.

  **workers**
  : Optionally constrain locations of data.
    Specify workers as hostname/port pairs, e.g.
    `('127.0.0.1', 8787)`.

  **broadcast**
  : Whether to send each data element to all workers.
    By default we round-robin based on number of cores.
    <br/>
    #### NOTE
    Setting this flag to True is incompatible with the Active Memory
    Manager’s [ReduceReplicas](https://distributed.dask.org/en/latest/active_memory_manager.html#reducereplicas) policy. If you wish to use it, you must
    first disable the policy or disable the AMM entirely.

  **direct**
  : Whether or not to connect directly to the workers, or to ask
    the scheduler to serve as intermediary.  This can also be set when
    creating the Client.

  **hash**
  : Whether or not to hash data to determine key.
    If False then this uses a random key

  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`

  **asynchronous: bool**
  : If True the client is in asynchronous mode
* **Returns:**
  List, dict, iterator, or queue of futures matching the type of input.

#### SEE ALSO
[`Client.gather`](#distributed.Client.gather)
: Gather data back to local process

### Notes

Scattering a dictionary uses `dict` keys to create `Future` keys.
The current implementation of a task graph resolution searches for occurrences of `key`
and replaces it with a corresponding `Future` result. That can lead to unwanted
substitution of strings passed as arguments to a task if these strings match some `key`
that already exists on a cluster. To avoid these situations it is required to use unique
values if a `key` is set manually.
See [https://github.com/dask/dask/issues/9969](https://github.com/dask/dask/issues/9969) to track progress on resolving this issue.

### Examples

```pycon
>>> c = Client('127.0.0.1:8787')
>>> c.scatter(1)
<Future: status: finished, key: c0a8a20f903a4915b94db8de3ea63195>
```

```pycon
>>> c.scatter([1, 2, 3])
[<Future: status: finished, key: c0a8a20f903a4915b94db8de3ea63195>,
 <Future: status: finished, key: 58e78e1b34eb49a68c65b54815d1b158>,
 <Future: status: finished, key: d3395e15f605bc35ab1bac6341a285e2>]
```

```pycon
>>> c.scatter({'x': 1, 'y': 2, 'z': 3})
{'x': <Future: status: finished, key: x>,
 'y': <Future: status: finished, key: y>,
 'z': <Future: status: finished, key: z>}
```

Constrain location of data to subset of workers

```pycon
>>> c.scatter([1, 2, 3], workers=[('hostname', 8788)])
```

Broadcast data to all workers

```pycon
>>> [future] = c.scatter([element], broadcast=True)
```

Send scattered data to parallelized function using client futures
interface

```pycon
>>> data = c.scatter(data, broadcast=True)
>>> res = [c.submit(func, data, i) for i in range(100)]
```

<!-- !! processed by numpydoc !! -->

#### scheduler_info(n_workers: [int](https://docs.python.org/3/library/functions.html#int) = -1, \*\*kwargs: [Any](https://docs.python.org/3/library/typing.html#typing.Any)) → SchedulerInfo

Basic information about the workers in the cluster

* **Parameters:**
  **n_workers: int**
  : The number of workers for which to fetch information. Defaults to
    `-1`, which fetches all workers. Pass a positive integer to limit
    the number of workers returned.
    <br/>
    Note: this argument is only honored for synchronous clients. For
    asynchronous clients this method returns the most recently cached
    value without a fresh fetch. That cache carries only cluster-wide
    totals and no per-worker information (`"workers"` is empty); use
    `await client.scheduler.identity(n_workers=-1)` to fetch worker
    information on demand.

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

### Examples

```pycon
>>> c.scheduler_info()
{'id': '2de2b6da-69ee-11e6-ab6a-e82aea155996',
 'services': {},
 'type': 'Scheduler',
 'workers': {'127.0.0.1:40575': {'active': 0,
                                 'last-seen': 1472038237.4845693,
                                 'name': '127.0.0.1:40575',
                                 'services': {},
                                 'stored': 0,
                                 'time-delay': 0.0061032772064208984}}}
```

<!-- !! processed by numpydoc !! -->

#### set_metadata(key, value)

Set arbitrary metadata in the scheduler

This allows you to store small amounts of data on the central scheduler
process for administrative purposes.  Data should be msgpack
serializable (ints, strings, lists, dicts)

If the key corresponds to a task then that key will be cleaned up when
the task is forgotten by the scheduler.

If the key is a list then it will be assumed that you want to index
into a nested dictionary structure using those keys.  For example if
you call the following:

```default
>>> client.set_metadata(['a', 'b', 'c'], 123)
```

Then this is the same as setting

```pycon
>>> scheduler.task_metadata['a']['b']['c'] = 123
```

The lower level dictionaries will be created on demand.

#### SEE ALSO
[`get_metadata`](#distributed.Client.get_metadata)

### Examples

```pycon
>>> client.set_metadata('x', 123)
>>> client.get_metadata('x')
123
```

```pycon
>>> client.set_metadata(['x', 'y'], 123)
>>> client.get_metadata('x')
{'y': 123}
```

```pycon
>>> client.set_metadata(['x', 'w', 'z'], 456)
>>> client.get_metadata('x')
{'y': 123, 'w': {'z': 456}}
```

```pycon
>>> client.get_metadata(['x', 'w'])
{'z': 456}
```

<!-- !! processed by numpydoc !! -->

#### shutdown()

Shut down the connected scheduler and workers

Note, this may disrupt other clients that may be using the same
scheduler and workers.

#### SEE ALSO
[`Client.close`](#distributed.Client.close)
: close only this client

<!-- !! processed by numpydoc !! -->

#### start(\*\*kwargs)

Start scheduler running in separate thread

<!-- !! processed by numpydoc !! -->

#### story(\*keys_or_stimuli, on_error='raise')

Returns a cluster-wide story for the given keys or stimulus_id’s

<!-- !! processed by numpydoc !! -->

#### submit(func: [Callable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[...], \_T], \*args, key=None, workers=None, resources=None, retries=None, priority=0, fifo_timeout='100 ms', allow_other_workers=False, actor=False, actors=False, pure=True, \*\*kwargs) → [Future](#distributed.Future)[\_T]

Submit a function application to the scheduler

* **Parameters:**
  **func**
  : Callable to be scheduled as `func(*args **kwargs)`. If `func` returns a
    coroutine, it will be run on the main event loop of a worker. Otherwise
    `func` will be run in a worker’s task executor pool (see
    `Worker.executors` for more information.)

  **\*args**
  : Optional positional arguments

  **key**
  : Unique identifier for the task.  Defaults to function-name and hash

  **workers**
  : A set of worker addresses or hostnames on which computations may be
    performed. Leave empty to default to all workers (common case)

  **resources**
  : Defines the `resources` each instance of this mapped task
    requires on the worker; e.g. `{'GPU': 2}`.
    See worker resources for details on defining
    resources.

  **retries**
  : Number of allowed automatic retries if the task fails

  **priority**
  : Optional prioritization of task.  Zero is default.
    Higher priorities take precedence

  **fifo_timeout**
  : Allowed amount of time between calls to consider the same priority

  **allow_other_workers**
  : Used with `workers`. Indicates whether or not the computations
    may be performed on workers that are not in the workers set(s).

  **actor**
  : Whether this task should exist on the worker as a stateful actor.
    See actors for additional details.

  **actors**
  : Alias for actor

  **pure**
  : Whether or not the function is pure.  Set `pure=False` for
    impure functions like `np.random.random`. Note that if both
    `actor` and `pure` kwargs are set to True, then the value
    of `pure` will be reverted to False, since an actor is stateful.
    See [Pure Functions by Default](https://distributed.dask.org/en/latest/client.html#pure-functions) for more details.

  **\*\*kwargs**
* **Returns:**
  Future
  : If running in asynchronous mode, returns the future. Otherwise
    returns the concrete value
* **Raises:**
  TypeError
  : If ‘func’ is not callable, a TypeError is raised

  ValueError
  : If ‘allow_other_workers’is True and ‘workers’ is None, a
    ValueError is raised

#### SEE ALSO
[`Client.map`](#distributed.Client.map)
: Submit on many arguments at once

### Notes

The current implementation of a task graph resolution searches for occurrences of `key`
and replaces it with a corresponding `Future` result. That can lead to unwanted
substitution of strings passed as arguments to a task if these strings match some `key`
that already exists on a cluster. To avoid these situations it is required to use unique
values if a `key` is set manually.
See [https://github.com/dask/dask/issues/9969](https://github.com/dask/dask/issues/9969) to track progress on resolving this issue.

### Examples

```pycon
>>> c = client.submit(add, a, b)
```

<!-- !! processed by numpydoc !! -->

#### subscribe_topic(topic, handler)

Subscribe to a topic and execute a handler for every received event

* **Parameters:**
  **topic: str**
  : The topic name

  **handler: callable or coroutine function**
  : A handler called for every received event. The handler must accept a
    single argument event which is a tuple (timestamp, msg) where
    timestamp refers to the clock on the scheduler.

#### SEE ALSO
`dask.distributed.Client.unsubscribe_topic`

`dask.distributed.Client.get_events`

`dask.distributed.Client.log_event`

### Examples

```pycon
>>> import logging
>>> logger = logging.getLogger("myLogger")  # Log config not shown
>>> client.subscribe_topic("topic-name", lambda: logger.info)
```

<!-- !! processed by numpydoc !! -->

#### unforward_logging(logger_name=None)

Stop forwarding the given logger (default root) from worker tasks to the
client process.

<!-- !! processed by numpydoc !! -->

#### unpublish_dataset(name: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...] | [list](https://docs.python.org/3/library/stdtypes.html#list)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]], \*\*kwargs)

Remove named datasets from scheduler

* **Parameters:**
  **name**
  : Name(s) of the dataset(s) to unpublish.

#### SEE ALSO
[`Client.publish_dataset`](#distributed.Client.publish_dataset)

[`Client.list_datasets`](#distributed.Client.list_datasets)

[`Client.get_dataset`](#distributed.Client.get_dataset)

### Examples

```pycon
>>> c.list_datasets()
['foo', 'bar', 'baz']
>>> c.unpublish_dataset('foo')
>>> c.list_datasets()
['bar', 'baz']
>>> c.unpublish_dataset(['bar', 'baz'])
>>> c.list_datasets()
[]
```

<!-- !! processed by numpydoc !! -->

#### unregister_scheduler_plugin(name: [str](https://docs.python.org/3/library/stdtypes.html#str))

Unregisters a scheduler plugin

See [https://distributed.readthedocs.io/en/latest/plugins.html#scheduler-plugins](https://distributed.readthedocs.io/en/latest/plugins.html#scheduler-plugins)

* **Parameters:**
  **name**
  : Name of the plugin to unregister. See the `Client.register_scheduler_plugin()`
    docstring for more information.

#### SEE ALSO
`register_scheduler_plugin`

### Examples

```pycon
>>> class MyPlugin(SchedulerPlugin):
...     def __init__(self, *args, **kwargs):
...         pass  # the constructor is up to you
...     async def start(self, scheduler: Scheduler) -> None:
...         pass
...     async def before_close(self) -> None:
...         pass
...     async def close(self) -> None:
...         pass
...     def restart(self, scheduler: Scheduler) -> None:
...         pass
```

```pycon
>>> plugin = MyPlugin(1, 2, 3)
>>> client.register_plugin(plugin, name='foo')
>>> client.unregister_scheduler_plugin(name='foo')
```

<!-- !! processed by numpydoc !! -->

#### unregister_worker_plugin(name, nanny=None)

Unregisters a lifecycle worker plugin

This unregisters an existing worker plugin. As part of the unregistration process
the plugin’s `teardown` method will be called.

* **Parameters:**
  **name**
  : Name of the plugin to unregister. See the [`Client.register_plugin()`](#distributed.Client.register_plugin)
    docstring for more information.

#### SEE ALSO
[`register_plugin`](#distributed.Client.register_plugin)

### Examples

```pycon
>>> class MyPlugin(WorkerPlugin):
...     def __init__(self, *args, **kwargs):
...         pass  # the constructor is up to you
...     def setup(self, worker: dask.distributed.Worker):
...         pass
...     def teardown(self, worker: dask.distributed.Worker):
...         pass
...     def transition(self, key: str, start: str, finish: str, **kwargs):
...         pass
...     def release_key(self, key: str, state: str, cause: str | None, reason: None, report: bool):
...         pass
```

```pycon
>>> plugin = MyPlugin(1, 2, 3)
>>> client.register_plugin(plugin, name='foo')
>>> client.unregister_worker_plugin(name='foo')
```

<!-- !! processed by numpydoc !! -->

#### unsubscribe_topic(topic)

Unsubscribe from a topic and remove event handler

#### SEE ALSO
`dask.distributed.Client.subscribe_topic`

`dask.distributed.Client.get_events`

`dask.distributed.Client.log_event`

<!-- !! processed by numpydoc !! -->

#### upload_file(filename, load: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Upload local package to scheduler and workers

This sends a local file up to the scheduler and all worker nodes.
This file is placed into the working directory of each node, see config option
`temporary-directory` (defaults to [`tempfile.gettempdir()`](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)).

This directory will be added to the Python’s system path so any `.py`,
`.egg` or `.zip`  files will be importable.

* **Parameters:**
  **filename**
  : Filename of `.py`, `.egg`, or `.zip` file to send to workers

  **load**
  : Whether or not to import the module as part of the upload process.
    Defaults to `True`.

### Examples

```pycon
>>> client.upload_file('mylibrary.egg')
>>> from mylibrary import myfunc
>>> L = client.map(myfunc, seq)
>>>
>>> # Where did that file go? Use `dask_worker.local_directory`.
>>> def where_is_mylibrary(dask_worker):
>>>     path = pathlib.Path(dask_worker.local_directory) / 'mylibrary.egg'
>>>     assert path.exists()
>>>     return str(path)
>>>
>>> client.run(where_is_mylibrary)
```

<!-- !! processed by numpydoc !! -->

#### wait_for_workers(n_workers: [int](https://docs.python.org/3/library/functions.html#int), timeout: [float](https://docs.python.org/3/library/functions.html#float) | [None](https://docs.python.org/3/library/constants.html#None) = None) → [None](https://docs.python.org/3/library/constants.html#None)

Blocking call to wait for n workers before continuing

* **Parameters:**
  **n_workers**
  : The number of workers

  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`

<!-- !! processed by numpydoc !! -->

#### who_has(futures=None, \*\*kwargs)

The workers storing each future’s data

* **Parameters:**
  **futures**
  : A list of futures, defaults to all data

  **\*\*kwargs**
  : Optional keyword arguments for the remote function

#### SEE ALSO
[`Client.has_what`](#distributed.Client.has_what)

[`Client.nthreads`](#distributed.Client.nthreads)

### Examples

```pycon
>>> x, y, z = c.map(inc, [1, 2, 3])
>>> wait([x, y, z])
>>> c.who_has()
{'inc-1c8dd6be1c21646c71f76c16d09304ea': ['192.168.1.141:46784'],
 'inc-1e297fc27658d7b67b3a758f16bcf47a': ['192.168.1.141:46784'],
 'inc-fd65c238a7ea60f6a01bf4c8a5fcf44b': ['192.168.1.141:46784']}
```

```pycon
>>> c.who_has([x, y])
{'inc-1c8dd6be1c21646c71f76c16d09304ea': ['192.168.1.141:46784'],
 'inc-1e297fc27658d7b67b3a758f16bcf47a': ['192.168.1.141:46784']}
```

<!-- !! processed by numpydoc !! -->

#### write_scheduler_file(scheduler_file)

Write the scheduler information to a json file.

This facilitates easy sharing of scheduler information using a file
system. The scheduler file can be used to instantiate a second Client
using the same scheduler.

* **Parameters:**
  **scheduler_file**
  : Path to a write the scheduler file.

### Examples

```pycon
>>> client = Client()
>>> client.write_scheduler_file('scheduler.json')
# connect to previous client's scheduler
>>> client2 = Client(scheduler_file='scheduler.json')
```

<!-- !! processed by numpydoc !! -->

### *class* distributed.Future(key: [str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], client: [Client](#distributed.Client) | [None](https://docs.python.org/3/library/constants.html#None) = None, state: FutureState | [None](https://docs.python.org/3/library/constants.html#None) = None, \_id: [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str), [int](https://docs.python.org/3/library/functions.html#int)] | [None](https://docs.python.org/3/library/constants.html#None) = None)

A remotely running computation

A Future is a local proxy to a result running on a remote worker.  A user
manages future objects in the local Python process to determine what
happens in the larger cluster.

#### NOTE
Users should not instantiate futures manually. This can lead to state
corruption and deadlocking clusters.

* **Parameters:**
  **key: str, or tuple**
  : Key of remote data to which this future refers

  **client: Client**
  : Client that should own this future.  Defaults to \_get_global_client()

  **inform: bool**
  : Do we inform the scheduler that we need an update on this future

  **state: FutureState**
  : The state of the future

#### SEE ALSO
[`Client`](#distributed.Client)
: Creates futures

### Examples

Futures typically emerge from Client computations

```pycon
>>> my_future = client.submit(add, 1, 2)
```

We can track the progress and results of a future

```pycon
>>> my_future
<Future: status: finished, key: add-8f6e709446674bad78ea8aeecfee188e>
```

We can get the result or the exception and traceback from the future

```pycon
>>> my_future.result()
```

<!-- !! processed by numpydoc !! -->

#### add_done_callback(fn)

Call callback on future when future has finished

The callback `fn` should take the future as its only argument.  This
will be called regardless of if the future completes successfully,
errs, or is cancelled

The callback is executed in a separate thread.

* **Parameters:**
  **fn**
  : The method or function to be called

<!-- !! processed by numpydoc !! -->

#### cancel(reason=None, msg=None, \*\*kwargs)

Cancel the request to run this future

#### SEE ALSO
[`Client.cancel`](#distributed.Client.cancel)

<!-- !! processed by numpydoc !! -->

#### cancelled()

Returns True if the future has been cancelled

* **Returns:**
  bool
  : True if the future was ‘cancelled’, otherwise False

<!-- !! processed by numpydoc !! -->

#### done()

Returns whether or not the computation completed.

* **Returns:**
  bool
  : True if the computation is complete, otherwise False

<!-- !! processed by numpydoc !! -->

#### exception(timeout=None, \*\*kwargs)

Return the exception of a failed task

* **Parameters:**
  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`

  **\*\*kwargs**
  : Optional keyword arguments for the function
* **Returns:**
  Exception
  : The exception that was raised
    If *timeout* seconds are elapsed before returning, a
    `dask.distributed.TimeoutError` is raised.

#### SEE ALSO
[`Future.traceback`](#distributed.Future.traceback)

<!-- !! processed by numpydoc !! -->

#### *property* executor

Returns the executor, which is the client.

* **Returns:**
  Client
  : The executor

<!-- !! processed by numpydoc !! -->

#### release()

### Notes

This method can be called from different threads
(see e.g. Client.get() or Future._\_del_\_())

<!-- !! processed by numpydoc !! -->

#### result(timeout=None) → \_T

Wait until computation completes, gather result to local process.

* **Parameters:**
  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`
* **Returns:**
  result
  : The result of the computation. Or a coroutine if the client is asynchronous.
* **Raises:**
  dask.distributed.TimeoutError
  : If *timeout* seconds are elapsed before returning, a
    `dask.distributed.TimeoutError` is raised.

<!-- !! processed by numpydoc !! -->

#### retry(\*\*kwargs)

Retry this future if it has failed

#### SEE ALSO
[`Client.retry`](#distributed.Client.retry)

<!-- !! processed by numpydoc !! -->

#### *property* status *: [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['pending', 'cancelled', 'finished', 'lost', 'error'] | [None](https://docs.python.org/3/library/constants.html#None)*

Returns the status

* **Returns:**
  str or None
  : The status of the future. Possible values:
    - “pending”: The future is waiting to be computed
    - “finished”: The future has completed successfully
    - “error”: The future encountered an error during computation
    - “cancelled”: The future was cancelled
    - “lost”: The future’s data was lost from memory
    - None: The future is not yet bound to a client

<!-- !! processed by numpydoc !! -->

#### traceback(timeout=None, \*\*kwargs)

Return the traceback of a failed task

This returns a traceback object.  You can inspect this object using the
`traceback` module.  Alternatively if you call `future.result()`
this traceback will accompany the raised exception.

* **Parameters:**
  **timeout**
  : Time in seconds after which to raise a
    `dask.distributed.TimeoutError`
    If *timeout* seconds are elapsed before returning, a
    `dask.distributed.TimeoutError` is raised.
* **Returns:**
  traceback
  : The traceback object. Or a coroutine if the client is asynchronous.

#### SEE ALSO
[`Future.exception`](#distributed.Future.exception)

### Examples

```pycon
>>> import traceback
>>> tb = future.traceback()
>>> traceback.format_tb(tb)
[...]
```

<!-- !! processed by numpydoc !! -->

#### *property* type

Returns the type

<!-- !! processed by numpydoc !! -->

### *class* distributed.Queue(name=None, client=None, maxsize=0)

Distributed Queue

This allows multiple clients to share futures or small bits of data between
each other with a multi-producer/multi-consumer queue.  All metadata is
sequentialized through the scheduler.

Elements of the Queue must be either Futures or msgpack-encodable data
(ints, strings, lists, dicts).  All data is sent through the scheduler so
it is wise not to send large objects.  To share large objects scatter the
data and share the future instead.

#### WARNING
This object is experimental

* **Parameters:**
  **name: string (optional)**
  : Name used by other clients and the scheduler to identify the queue. If
    not given, a random name will be generated.

  **client: Client (optional)**
  : Client used for communication with the scheduler.
    If not given, the default global client will be used.

  **maxsize: int (optional)**
  : Number of items allowed in the queue. If 0 (the default), the queue
    size is unbounded.

#### SEE ALSO
[`Variable`](#distributed.Variable)
: shared variable between clients

### Examples

```pycon
>>> from dask.distributed import Client, Queue
>>> client = Client()
>>> queue = Queue('x')
>>> future = client.submit(f, x)
>>> queue.put(future)
```

<!-- !! processed by numpydoc !! -->

#### get(timeout=None, batch=False, \*\*kwargs)

Get data from the queue

* **Parameters:**
  **timeout**
  : Time in seconds to wait before timing out.
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.

  **batch**
  : If True then return all elements currently waiting in the queue.
    If an integer than return that many elements from the queue
    If False (default) then return one item at a time

<!-- !! processed by numpydoc !! -->

#### put(value, timeout=None, \*\*kwargs)

Put data into the queue

* **Parameters:**
  **timeout**
  : Time in seconds to wait before timing out.
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.

<!-- !! processed by numpydoc !! -->

#### qsize(\*\*kwargs)

Current number of elements in the queue

<!-- !! processed by numpydoc !! -->

### *class* distributed.Variable(name=None, client=None)

Distributed Global Variable

This allows multiple clients to share futures and data between each other
with a single mutable variable.  All metadata is sequentialized through the
scheduler.  Race conditions can occur.

Values must be either Futures or msgpack-encodable data (ints, lists,
strings, etc..)  All data will be kept and sent through the scheduler, so
it is wise not to send too much.  If you want to share a large amount of
data then `scatter` it and share the future instead.

* **Parameters:**
  **name: string (optional)**
  : Name used by other clients and the scheduler to identify the variable.
    If not given, a random name will be generated.

  **client: Client (optional)**
  : Client used for communication with the scheduler.
    If not given, the default global client will be used.

#### SEE ALSO
[`Queue`](#distributed.Queue)
: shared multi-producer/multi-consumer queue between clients

### Examples

```pycon
>>> from dask.distributed import Client, Variable
>>> client = Client()
>>> x = Variable('x')
>>> x.set(123)
>>> x.get()
123
>>> future = client.submit(f, x)
>>> x.set(future)
```

<!-- !! processed by numpydoc !! -->

#### delete()

Delete this variable

Caution, this affects all clients currently pointing to this variable.

<!-- !! processed by numpydoc !! -->

#### get(timeout=None, \*\*kwargs)

Get the value of this variable

* **Parameters:**
  **timeout**
  : Time in seconds to wait before timing out.
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.

<!-- !! processed by numpydoc !! -->

#### set(value, timeout='30 s', \*\*kwargs)

Set the value of this variable

* **Parameters:**
  **value**
  : Must be either a Future or a msgpack-encodable value

<!-- !! processed by numpydoc !! -->

### *class* distributed.Lock(name=None, scheduler_rpc=None, loop=None)

Distributed Centralized Lock

#### WARNING
This is using the `distributed.Semaphore` as a backend, which is
susceptible to lease overbooking. For the Lock this means that if a
lease is timing out, two or more instances could acquire the lock at the
same time. To disable lease timeouts, set
`distributed.scheduler.locks.lease-timeout` to inf, e.g.

```python
with dask.config.set({"distributed.scheduler.locks.lease-timeout": "inf"}):
    lock = Lock("x")
    ...
```

Note, that without lease timeouts, the Lock may deadlock in case of
cluster downscaling or worker failures.

* **Parameters:**
  **name: string (optional)**
  : Name of the lock to acquire.  Choosing the same name allows two
    disconnected processes to coordinate a lock.  If not given, a random
    name will be generated.

### Examples

```pycon
>>> lock = Lock('x')
>>> lock.acquire(timeout=1)
>>> # do things with protected resource
>>> lock.release()
```

<!-- !! processed by numpydoc !! -->

#### acquire(blocking=True, timeout=None)

Acquire the lock

* **Parameters:**
  **blocking**
  : If false, don’t wait on the lock in the scheduler at all.

  **timeout**
  : Seconds to wait on the lock in the scheduler.  This does not
    include local coroutine time, network transfer time, etc..
    It is forbidden to specify a timeout when blocking is false.
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.
* **Returns:**
  True or False whether or not it successfully acquired the lock

### Examples

```pycon
>>> lock = Lock('x')
>>> lock.acquire(timeout="1s")
```

<!-- !! processed by numpydoc !! -->

### *class* distributed.Event(name=None, client=None)

Distributed Centralized Event equivalent to asyncio.Event

An event stores a single flag, which is set to false on start.
The flag can be set to true (using the set() call) or back to false
(with the clear() call).
Every call to wait() blocks until the event flag is set to true.

* **Parameters:**
  **name: string (optional)**
  : Name of the event.  Choosing the same name allows two
    disconnected processes to coordinate an event.
    If not given, a random name will be generated.

  **client: Client (optional)**
  : Client to use for communication with the scheduler.
    If not given, the default global client will be used.

### Examples

```pycon
>>> event_1 = Event('a')
>>> event_1.wait(timeout=1)
>>> # in another process
>>> event_2 = Event('a')
>>> event_2.set()
>>> # now event_1 will stop waiting
```

<!-- !! processed by numpydoc !! -->

#### clear()

Clear the event (set its flag to false).

All waiters will now block.

<!-- !! processed by numpydoc !! -->

#### is_set()

Check if the event is set

<!-- !! processed by numpydoc !! -->

#### set()

Set the event (set its flag to True).

All waiters will now be released.

<!-- !! processed by numpydoc !! -->

#### wait(timeout=None)

Wait until the event is set.

* **Parameters:**
  **timeout**
  : Seconds to wait on the event in the scheduler.  This does not
    include local coroutine time, network transfer time, etc..
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.
* **Returns:**
  True if the event was set; False if a timeout happened

### Examples

```pycon
>>> event = Event('a')
>>> event.wait(timeout="1s")
```

<!-- !! processed by numpydoc !! -->

### *class* distributed.Semaphore(max_leases=1, name=None, scheduler_rpc=None, loop=None)

This [semaphore](https://en.wikipedia.org/wiki/Semaphore_(programming))
will track leases on the scheduler which can be acquired and
released by an instance of this class. If the maximum amount of leases are
already acquired, it is not possible to acquire more and the caller waits
until another lease has been released.

The lifetime or leases are controlled using a timeout. This timeout is
refreshed in regular intervals by the `Client` of this instance and
provides protection from deadlocks or resource starvation in case of worker
failure.
The timeout can be controlled using the configuration option
`distributed.scheduler.locks.lease-timeout` and the interval in which the
scheduler verifies the timeout is set using the option
`distributed.scheduler.locks.lease-validation-interval`.

A noticeable difference to the Semaphore of the python standard library is
that this implementation does not allow to release more often than it was
acquired. If this happens, a warning is emitted but the internal state is
not modified.

#### WARNING
This implementation is susceptible to lease overbooking in case of
lease timeouts. It is advised to monitor log information and adjust
above configuration options to suitable values for the user application.

* **Parameters:**
  **max_leases: int (optional)**
  : The maximum amount of leases that may be granted at the same time. This
    effectively sets an upper limit to the amount of parallel access to a specific resource.
    Defaults to 1.

  **name: string (optional)**
  : Name of the semaphore to acquire.  Choosing the same name allows two
    disconnected processes to coordinate.  If not given, a random
    name will be generated.

  **register: bool**
  : If True, register the semaphore with the scheduler. This needs to be
    done before any leases can be acquired. If not done during
    initialization, this can also be done by calling the register method of
    this class.
    When registering, this needs to be awaited.

  **scheduler_rpc: ConnectionPool**
  : The ConnectionPool to connect to the scheduler. If None is provided, it
    uses the worker or client pool. This parameter is mostly used for
    testing.

  **loop: IOLoop**
  : The event loop this instance is using. If None is provided, reuse the
    loop of the active worker or client.

### Notes

If a client attempts to release the semaphore but doesn’t have a lease acquired, this will raise an exception.

dask executes functions by default assuming they are pure, when using semaphore acquire/releases inside
such a function, it must be noted that there *are* in fact side-effects, thus, the function can no longer be
considered pure. If this is not taken into account, this may lead to unexpected behavior.

### Examples

```pycon
>>> from distributed import Semaphore
... sem = Semaphore(max_leases=2, name='my_database')
...
... def access_resource(s, sem):
...     # This automatically acquires a lease from the semaphore (if available) which will be
...     # released when leaving the context manager.
...     with sem:
...         pass
...
... futures = client.map(access_resource, range(10), sem=sem)
... client.gather(futures)
... # Once done, close the semaphore to clean up the state on scheduler side.
... sem.close()
```

<!-- !! processed by numpydoc !! -->

#### acquire(timeout=None)

Acquire a semaphore.

If the internal counter is greater than zero, decrement it by one and return True immediately.
If it is zero, wait until a release() is called and return True.

* **Parameters:**
  **timeout**
  : Seconds to wait on acquiring the semaphore.  This does not
    include local coroutine time, network transfer time, etc..
    Instead of number of seconds, it is also possible to specify
    a timedelta in string format, e.g. “200ms”.

<!-- !! processed by numpydoc !! -->

#### get_value()

Return the number of currently registered leases.

<!-- !! processed by numpydoc !! -->

#### release()

Release the semaphore.

* **Returns:**
  bool
  : This value indicates whether a lease was released immediately or not. Note that a user should  *not* retry
    this operation. Under certain circumstances (e.g. scheduler overload) the lease may not be released
    immediately, but it will always be automatically released after a specific interval configured using
    “distributed.scheduler.locks.lease-validation-interval” and “distributed.scheduler.locks.lease-timeout”.

<!-- !! processed by numpydoc !! -->


# gpu.html.md

# GPUs

Dask works with GPUs in a few ways.

## Custom Computations

Many people use Dask alongside GPU-accelerated libraries like PyTorch and
TensorFlow to manage workloads across several machines.  They typically use
Dask’s custom APIs, notably [Delayed](delayed.md) and [Futures](futures.md).

Dask doesn’t need to know that these functions use GPUs.  It just runs Python
functions.  Whether or not those Python functions use a GPU is orthogonal to
Dask.  It will work regardless.

As a worked example, you may want to view this talk:

<video width="560" height="315" controls>
    <source src="https://developer.download.nvidia.com/video/gputechconf/gtc/2019/video/S9198/s9198-dask-and-v100s-for-fast-distributed-batch-scoring-of-computer-vision-workloads.mp4"
            type="video/mp4">
</video>

## High Level Collections

Dask can also help to scale out large array and dataframe computations by
combining the Dask Array and DataFrame collections with a GPU-accelerated
array or dataframe library.

Recall that [Dask Array](array.md) creates a large array out of many NumPy
arrays and [Dask DataFrame](dataframe.md) creates a large dataframe out of
many Pandas dataframes.  We can use these same systems with GPUs if we swap out
the NumPy/Pandas components with GPU-accelerated versions of those same
libraries, as long as the GPU accelerated version looks enough like
NumPy/Pandas in order to interoperate with Dask.

Fortunately, libraries that mimic NumPy, Pandas, and Scikit-Learn on the GPU do
exist.

### DataFrames

The [RAPIDS](https://rapids.ai) libraries provide a GPU accelerated
Pandas-like library,
[cuDF](https://github.com/rapidsai/cudf),
which interoperates well and is tested against Dask DataFrame.

If you have cuDF installed then you should be able to convert a Pandas-backed
Dask DataFrame to a cuDF-backed Dask DataFrame as follows:

```python
import cudf

df = df.map_partitions(cudf.from_pandas)  # convert pandas partitions into cudf partitions
```

However, cuDF does not support the entire Pandas interface, and so a variety of
Dask DataFrame operations will not function properly. Check the
[cuDF API Reference](https://docs.rapids.ai/api/cudf/stable/)
for currently supported interface.

### Arrays

#### NOTE
Dask’s integration with CuPy relies on features recently added to
NumPy and CuPy, particularly in version `numpy>=1.17` and `cupy>=6`

[Chainer’s CuPy](https://cupy.chainer.org/) library provides a GPU
accelerated NumPy-like library that interoperates nicely with Dask Array.

If you have CuPy installed then you should be able to convert a NumPy-backed
Dask Array into a CuPy backed Dask Array as follows:

```python
import cupy

x = x.map_blocks(cupy.asarray)
```

CuPy is fairly mature and adheres closely to the NumPy API.  However, small
differences do exist and these can cause Dask Array operations to function
improperly. Check the
[CuPy Reference Manual](https://docs-cupy.chainer.org/en/stable/reference/index.html)
for API compatibility.

### Scikit-Learn

There are a variety of GPU accelerated machine learning libraries that follow
the Scikit-Learn Estimator API of fit, transform, and predict.  These can
generally be used within [Dask-ML’s](https://ml.dask.org) meta estimators,
such as [hyper parameter optimization](https://ml.dask.org/hyper-parameter-search.html).

Some of these include:

- [Skorch](https://skorch.readthedocs.io/)
- [cuML](https://rapidsai.github.io/projects/cuml/en/latest/)
- [LightGBM](https://github.com/Microsoft/LightGBM)
- [XGBoost](https://xgboost.readthedocs.io/en/latest/)
- [Thunder SVM](https://github.com/Xtra-Computing/thundersvm)
- [Thunder GBM](https://github.com/Xtra-Computing/thundergbm)

## Setup

From the examples above we can see that the user experience of using Dask with
GPU-backed libraries isn’t very different from using it with CPU-backed
libraries.  However, there are some changes you might consider making when
setting up your cluster.

### Restricting Work

By default Dask allows as many tasks as you have CPU cores to run concurrently.
However if your tasks primarily use a GPU then you probably want far fewer
tasks running at once.  There are a few ways to limit parallelism here:

- Limit the number of threads explicitly on your workers using the
  `--nthreads` keyword in the CLI or the `ncores=` keyword the
  Cluster constructor.
- Use [worker resources](https://distributed.dask.org/en/latest/resources.html) and tag certain
  tasks as GPU tasks so that the scheduler will limit them, while leaving the
  rest of your CPU cores for other work

### Specifying GPUs per Machine

Some configurations may have many GPU devices per node.  Dask is often used to
balance and coordinate work between these devices.

In these situations it is common to start one Dask worker per device, and use
the CUDA environment variable `CUDA_VISIBLE_DEVICES` to pin each worker to
prefer one device.

```bash
# If we have four GPUs on one machine
CUDA_VISIBLE_DEVICES=0 dask-worker ...
CUDA_VISIBLE_DEVICES=1 dask-worker ...
CUDA_VISIBLE_DEVICES=2 dask-worker ...
CUDA_VISIBLE_DEVICES=3 dask-worker ...
```

The [Dask CUDA](https://github.com/rapidsai/dask-cuda) project contains some
convenience CLI and Python utilities to automate this process.

## Work in Progress

GPU computing is a quickly moving field today and as a result the information
in this page is likely to go out of date quickly.  We encourage interested
readers to check out [Dask’s Blog](https://blog.dask.org) which has more
timely updates on ongoing work.


# graph_manipulation.html.md

<a id="graph-manipulation"></a>

# Advanced graph manipulation

There are some situations where computations with Dask collections will result in
suboptimal memory usage (e.g. an entire Dask DataFrame is loaded into memory).
This may happen when Dask’s scheduler doesn’t automatically delay the computation of
nodes in a task graph to avoid occupying memory with their output for prolonged periods
of time, or in scenarios where recalculating nodes is much cheaper than holding their
output in memory.

This page highlights a set of graph manipulation utilities which can be used to help
avoid these scenarios. In particular, the utilities described below rewrite the
underlying Dask graph for Dask collections, producing equivalent collections with
different sets of keys.

Consider the following example:

```python
>>> import dask.array as da
>>> x = da.random.default_rng().normal(size=500_000_000, chunks=100_000)
>>> x_mean = x.mean()
>>> y = (x - x_mean).max().compute()
```

The above example computes the largest value of a distribution after removing its bias.
This involves loading the chunks of `x` into memory in order to compute `x_mean`.
However, since the `x` array is needed later in the computation to compute `y`, the
entire `x` array is kept in memory. For large Dask Arrays this can be very
problematic.

To alleviate the need for the entire `x` array to be kept in memory, one could rewrite
the last line as follows:

```python
>>> from dask.graph_manipulation import bind
>>> xb = bind(x, x_mean)
>>> y = (xb - x_mean).max().compute()
```

Here we use [`bind()`](#dask.graph_manipulation.bind) to create a new Dask Array, `xb`,
which produces exactly the same output as `x`, but whose underlying Dask graph has
different keys than `x`, and will only be computed after `x_mean` has been
calculated.

This results in the chunks of `x` being computed and immediately individually reduced
by `mean`; then recomputed and again immediately pipelined into the subtraction
followed by reduction with `max`. This results in a much smaller peak memory usage as
the full `x` array is no longer loaded into memory. However, the tradeoff is that the
compute time increases as `x` is computed twice.

## API

| [`checkpoint`](#dask.graph_manipulation.checkpoint)(\*collections[, split_every])     | Build a [Dask Delayed](delayed.md) which waits until all chunks of the input collection(s) have been computed before returning None.   |
|---------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| [`wait_on`](#dask.graph_manipulation.wait_on)(\*collections[, split_every])           | Ensure that all chunks of all input collections have been computed before computing the dependents of any of the chunks.               |
| [`bind`](#dask.graph_manipulation.bind)(children, parents, \*[, omit, seed, ...])     | Make `children` collection(s), optionally omitting sub-collections, dependent on `parents` collection(s).                              |
| [`clone`](#dask.graph_manipulation.clone)(\*collections[, omit, seed, assume_layers]) | Clone dask collections, returning equivalent collections that are generated from independent calculations.                             |

### Definitions

### dask.graph_manipulation.checkpoint(\*collections, split_every: [float](https://docs.python.org/3/library/functions.html#float) | [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)[False] | [None](https://docs.python.org/3/library/constants.html#None) = None) → [Delayed](delayed-api.md#dask.delayed.Delayed)

Build a [Dask Delayed](delayed.md) which waits until all chunks of the input collection(s)
have been computed before returning None.

* **Parameters:**
  **collections**
  : Zero or more Dask collections or nested data structures containing zero or more
    collections

  **split_every: int >= 2 or False, optional**
  : Determines the depth of the recursive aggregation. If greater than the number of
    input keys, the aggregation will be performed in multiple steps; the depth of
    the aggregation graph will be $\log_\text{split_every}(\text{input keys})$.
    Setting to a low value can reduce cache size and network transfers, at the cost
    of more CPU and a larger dask graph.
    <br/>
    Set to False to disable. Defaults to 8.
* **Returns:**
  [Dask Delayed](delayed.md) yielding None

<!-- !! processed by numpydoc !! -->

### dask.graph_manipulation.wait_on(\*collections, split_every: [float](https://docs.python.org/3/library/functions.html#float) | [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)[False] | [None](https://docs.python.org/3/library/constants.html#None) = None)

Ensure that all chunks of all input collections have been computed before
computing the dependents of any of the chunks.

The following example creates a dask array `u` that, when used in a computation,
will only proceed when all chunks of the array `x` have been computed, but
otherwise matches `x`:

```pycon
>>> import dask.array as da
>>> x = da.ones(10, chunks=5)
>>> u = wait_on(x)
```

The following example will create two arrays `u` and `v` that, when used in a
computation, will only proceed when all chunks of the arrays `x` and `y` have
been computed but otherwise match `x` and `y`:

```pycon
>>> x = da.ones(10, chunks=5)
>>> y = da.zeros(10, chunks=5)
>>> u, v = wait_on(x, y)
```

* **Parameters:**
  **collections**
  : Zero or more Dask collections or nested structures of Dask collections

  **split_every**
  : See [`checkpoint()`](#dask.graph_manipulation.checkpoint)
* **Returns:**
  Same as `collections`
  : Dask collection of the same type as the input, which computes to the same value,
    or a nested structure equivalent to the input where the original collections
    have been replaced.
    The keys of the regenerated nodes of the new collections will be different from
    the original ones, so that they can be used within the same graph.

<!-- !! processed by numpydoc !! -->

### dask.graph_manipulation.bind(children: T, parents, , omit=None, seed: [Hashable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable) | [None](https://docs.python.org/3/library/constants.html#None) = None, assume_layers: [bool](https://docs.python.org/3/library/functions.html#bool) = True, split_every: [float](https://docs.python.org/3/library/functions.html#float) | [Literal](https://docs.python.org/3/library/typing.html#typing.Literal)[False] | [None](https://docs.python.org/3/library/constants.html#None) = None) → T

Make `children` collection(s), optionally omitting sub-collections, dependent on
`parents` collection(s). Two examples follow.

The first example creates an array `b2` whose computation first computes an array
`a` completely and then computes `b` completely, recomputing `a` in the
process:

```pycon
>>> import dask
>>> import dask.array as da
>>> a = da.ones(4, chunks=2)
>>> b = a + 1
>>> b2 = bind(b, a)
>>> len(b2.dask)
9
>>> b2.compute()
array([2., 2., 2., 2.])
```

The second example creates arrays `b3` and `c3`, whose computation first
computes an array `a` and then computes the additions, this time not
recomputing `a` in the process:

```pycon
>>> c = a + 2
>>> b3, c3 = bind((b, c), a, omit=a)
>>> len(b3.dask), len(c3.dask)
(7, 7)
>>> dask.compute(b3, c3)
(array([2., 2., 2., 2.]), array([3., 3., 3., 3.]))
```

* **Parameters:**
  **children**
  : Dask collection or nested structure of Dask collections

  **parents**
  : Dask collection or nested structure of Dask collections

  **omit**
  : Dask collection or nested structure of Dask collections

  **seed**
  : Hashable used to seed the key regeneration. Omit to default to a random number
    that will produce different keys at every call.

  **assume_layers**
  : True
    : Use a fast algorithm that works at layer level, which assumes that all
      collections in `children` and `omit`
      1. use [`HighLevelGraph`](high-level-graphs.md#dask.highlevelgraph.HighLevelGraph),
      2. define the `__dask_layers__()` method, and
      3. never had their graphs squashed and rebuilt between the creation of the
         `omit` collections and the `children` collections; in other words if
         the keys of the `omit` collections can be found among the keys of the
         `children` collections, then the same must also hold true for the
         layers.
    <br/>
    False
    : Use a slower algorithm that works at keys level, which makes none of the
      above assumptions.

  **split_every**
  : See [`checkpoint()`](#dask.graph_manipulation.checkpoint)
* **Returns:**
  Same as `children`
  : Dask collection or structure of dask collection equivalent to `children`,
    which compute to the same values. All nodes of `children` will be regenerated,
    up to and excluding the nodes of `omit`. Nodes immediately above `omit`, or
    the leaf nodes if the collections in `omit` are not found, are prevented from
    computing until all collections in `parents` have been fully computed.
    The keys of the regenerated nodes will be different from the original ones, so
    that they can be used within the same graph.

<!-- !! processed by numpydoc !! -->

### dask.graph_manipulation.clone(\*collections, omit=None, seed: [Hashable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable) = None, assume_layers: [bool](https://docs.python.org/3/library/functions.html#bool) = True)

Clone dask collections, returning equivalent collections that are generated from
independent calculations.

* **Parameters:**
  **collections**
  : Zero or more Dask collections or nested structures of Dask collections

  **omit**
  : Dask collection or nested structure of Dask collections which will not be cloned

  **seed**
  : See [`bind()`](#dask.graph_manipulation.bind)

  **assume_layers**
  : See [`bind()`](#dask.graph_manipulation.bind)
* **Returns:**
  Same as `collections`
  : Dask collections of the same type as the inputs, which compute to the same
    value, or nested structures equivalent to the inputs, where the original
    collections have been replaced.
    The keys of the regenerated nodes in the new collections will be different from
    the original ones, so that they can be used within the same graph.

### Examples

(tokens have been simplified for the sake of brevity)

```pycon
>>> import dask.array as da
>>> x_i = da.asarray([1, 1, 1, 1], chunks=2)
>>> y_i = x_i + 1
>>> z_i = y_i + 2
>>> dict(z_i.dask)
{('array-1', 0): array([1, 1]),
 ('array-1', 1): array([1, 1]),
 ('add-2', 0): (<function operator.add>, ('array-1', 0), 1),
 ('add-2', 1): (<function operator.add>, ('array-1', 1), 1),
 ('add-3', 0): (<function operator.add>, ('add-2', 0), 1),
 ('add-3', 1): (<function operator.add>, ('add-2', 1), 1)}
>>> w_i = clone(z_i, omit=x_i)
>>> w_i.compute()
array([4, 4, 4, 4])
>>> dict(w_i.dask)
{('array-1', 0): array([1, 1]),
 ('array-1', 1): array([1, 1]),
 ('add-4', 0): (<function operator.add>, ('array-1', 0), 1),
 ('add-4', 1): (<function operator.add>, ('array-1', 1), 1),
 ('add-5', 0): (<function operator.add>, ('add-4', 0), 1),
 ('add-5', 1): (<function operator.add>, ('add-4', 1), 1)}
```

The typical usage pattern for clone() is the following:

```pycon
>>> x = cheap_computation_with_large_output()
>>> y = expensive_and_long_computation(x)
>>> z = wrap_up(clone(x), y)
```

In the above code, the chunks of x will be forgotten as soon as they are consumed by
the chunks of y, and then they’ll be regenerated from scratch at the very end of the
computation. Without clone(), x would only be computed once and then kept in memory
throughout the whole computation of y, needlessly consuming memory.

<!-- !! processed by numpydoc !! -->


# graphs.html.md

<a id="graphs"></a>

# Task Graphs

Internally, Dask encodes algorithms as task graphs which are typically expressed as dictionaries. This graph format can be used in isolation from the
dask collections. Working directly with dask graphs is rare, though, unless you intend
to develop new modules with Dask.  Even then, [dask.delayed](delayed.md) is
often a better choice. If you are a *core developer*, then you should start here.

* [Specification](spec.md)
* [Custom Graphs](custom-graphs.md)
* [Optimization](optimize.md)
* [Advanced graph manipulation](graph_manipulation.md)
* [Custom Collections](custom-collections.md)
* [High Level Graphs](high-level-graphs.md)

## Motivation

Normally, humans write programs and then compilers/interpreters interpret them
(for example, `python`, `javac`, `clang`).  Sometimes humans disagree with how
these compilers/interpreters choose to interpret and execute their programs.
In these cases, humans often bring the analysis, optimization, and execution of
code into the code itself.

Commonly a desire for parallel execution causes this shift of responsibility
from compiler to human developer.  In these cases, we often represent the
structure of our program explicitly as data within the program itself.

A common approach to parallel execution in user-space is *task scheduling*.  In
task scheduling we break our program into many medium-sized tasks or units of
computation, often a function call on a non-trivial amount of data.  We
represent these tasks as nodes in a graph with edges between nodes if one task
depends on data produced by another.  We call upon a *task scheduler* to
execute this graph in a way that respects these data dependencies and leverages
parallelism where possible, so multiple independent tasks can be run
simultaneously.

<br/>
![image](images/map-reduce-task-scheduling.svg)
<br/>

Many solutions exist.  This is a common approach in parallel execution
frameworks.  Often task scheduling logic hides within other larger frameworks
(e.g. Luigi, Storm, Spark, IPython Parallel, etc.) and so is often reinvented.

## Example

Consider the following simple program:

```python
def inc(i):
    return i + 1

def add(a, b):
    return a + b

x = 1
y = inc(x)
z = add(y, 10)
```

We encode this as a dictionary in the following way:

```python
d = {'x': DataNode(None, 1),
     'y': Task('y', inc, TaskRef('x')),
     'z': Task('z', add, TaskRef('y'), 10)}
```

Which is represented by the following Dask graph:

![A simple dask dictionary](_static/dask-simple.png)
<br/>

While less pleasant than our original code, this representation can be analyzed
and executed by other Python code, not just the CPython interpreter.  We don’t
recommend that users write code in this way, but rather that it is an
appropriate target for automated systems.  Also, in non-toy examples, the
execution times are likely much larger than for `inc` and `add`, warranting
the extra complexity.

## Schedulers

The Dask library currently contains a few schedulers to execute these
graphs.  Each scheduler works differently, providing different performance
guarantees and operating in different contexts.  These implementations are not
special and others can write different schedulers better suited to other
applications or architectures easily.  Systems that emit dask graphs (like
Dask Array, Dask Bag, and so on) may leverage the appropriate scheduler for
the application and hardware.

## Task Expectations

When a task is submitted to Dask for execution, there are a number of assumptions
that are made about that task.

### Don’t Modify Data In-Place

In general, tasks with side-effects that alter the state of a future in-place
are not recommended. Modifying data that is stored in Dask in-place can have
unintended consequences. For example, consider a workflow involving a Numpy
array:

```python
from dask.distributed import Client
import numpy as np

client = Client()
x = client.submit(np.arange, 10)  # [0, 1, 2, 3, ...]

def f(arr):
    arr[arr > 5] = 0  # modifies input directly without making a copy
    arr += 1          # modifies input directly without making a copy
    return arr

y = client.submit(f, x)
```

In the example above Dask will update the values of the Numpy array
`x` in-place.  While efficient, this behavior can have unintended consequences,
particularly if other tasks need to use `x`, or if Dask needs to rerun this
computation multiple times because of worker failure.

### Avoid Holding the GIL

Some Python functions that wrap external C/C++ code can hold onto the GIL,
which stops other Python code from running in the background.  This is
troublesome because while Dask workers run your function, they also need to
communicate to each other in the background.

If you wrap external code then please try to release the GIL.  This is usually
easy to do if you are using any of the common solutions to code-wrapping like
Cython, Numba, ctypes or others.


# graphviz.html.md

# Visualize task graphs

| [`visualize`](api.md#dask.visualize)(\*args[, filename, traverse, ...])   | Visualize several dask graphs simultaneously.   |
|---------------------------------------------------------------------------|-------------------------------------------------|

Before executing your computation you might consider visualizing the underlying task graph.
By looking at the inter-connectedness of tasks
you can learn more about potential bottlenecks
where parallelism may not be possible,
or areas where many tasks depend on each other,
which may cause a great deal of communication.

## Visualize the low level graph

The `.visualize` method and `dask.visualize` function works like
the `.compute` method and `dask.compute` function,
except that rather than computing the result,
they produce an image of the task graph.
These images are written to files, and if you are within a Jupyter notebook
context they will also be displayed as cell outputs.

By default the task graph is rendered from top to bottom.
In the case that you prefer to visualize it from left to right, pass
`rankdir="LR"` as a keyword argument to `.visualize`.

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))

y = x + x.T

# y.compute()

# visualize the low level Dask graph
y.visualize(filename='transpose.svg')
```

![Dask low level task graph for adding an array to its transpose](images/transpose.svg)

It is often helpful to inspect the task graph before and after graph optimizations
are applied. You can do that by setting the `optimize_graph` keyword.
So the above example becomes:

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))

y = x + x.T

# visualize the low level Dask graph after optimizations
y.visualize(filename="transpose_opt.svg", optimize_graph=True)
```

![Dask low level task graph for adding an array to its transpose.
Compared to the unoptimized graph it is simpler, as a number of
the tasks have been fused together.](images/transpose_opt.svg)

The `visualize` function supports two different graph rendering engines: `graphviz`
(the default), and `cytoscape`. In order to change the engine that is used, pass the
name of the engine for the `engine` argument to `visualize`:

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))

y = x + x.T

# visualize the low level Dask graph using cytoscape
y.visualize(engine="cytoscape")
```

<iframe src="_static/transpose_cyto.html"
        marginwidth="0" marginheight="0" scrolling="no"
        width="100%" height="400" style="border:none"></iframe>

You can also set the default visualization engine by setting the `visualization.engine`
configuration option:

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))

y = x + x.T

with dask.config.set({"visualization.engine": "cytoscape"}):
    y.visualize()
```

Note that both visualization engines require optional dependencies to be installed.
The `graphviz` engine is powered by the [GraphViz](https://www.graphviz.org/)
system library. This library has a few considerations:

1. You must install both the graphviz system library (with tools like apt-get, yum, or brew)
   *and* the graphviz Python library.
   If you use Conda then you need to install `python-graphviz`,
   which will bring along the `graphviz` system library as a dependency.
2. Graphviz takes a while on graphs larger than about 100 nodes.
   For large computations you might have to simplify your computation a bit
   for the visualize method to work well.

The `cytoscape` engine uses the [Cytoscape](https://js.cytoscape.org/) javascript
library for rendering, and is driven on the Python side by the `ipycytoscape`
library. Because it doesn’t rely on any system libraries, this engine may be easier
to install than graphviz in some deployment settings.

## Visualize the high level graph

The low level Dask task graph can be overwhelming, especially for large computations.
A more concise alternative is to look at the Dask high level graph instead.
The high level graph can be visualized using `.dask.visualize()`.

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))
y = x + x.T

# visualize the high level Dask graph
y.dask.visualize(filename='transpose-hlg.svg')
```

![Dask high level task graph for adding an array to its transpose](images/transpose-hlg-hovertooltip.png)

Hovering your mouse above each high level graph label will bring up
a tooltip with more detailed information about that layer.
Note that if you save the graph to disk using the `filename=` keyword argument
in `visualize`, then the tooltips will only be preserved by the SVG image format.

## High level graph HTML representation

Dask high level graphs also have their own HTML representation,
which is useful if you like to work with Jupyter notebooks.

```python
import dask.array as da
x = da.ones((15, 15), chunks=(5, 5))
y = x + x.T

y.dask  # shows the HTML representation in a Jupyter notebook
```

![Dask high level graph HTML representation](images/transpose-hlg-html-repr.png)

You can click on any of the layer names to expand or collapse more detailed
information about each layer.


# high-level-graphs.html.md

<a id="high-level-graphs"></a>

# High Level Graphs

Dask graphs produced by collections like Arrays, Bags, and DataFrames have
high-level structure that can be useful for visualization and high-level
optimization.  The task graphs produced by these collections encode this
structure explicitly as `HighLevelGraph` objects.  This document describes
how to work with these in more detail.

## Motivation and Example

In full generality, Dask schedulers expect arbitrary task graphs where each
node is a single Python function call and each edge is a dependency between
two function calls.  These are usually stored in flat dictionaries.  Here is
some simple Dask DataFrame code and the task graph that it might generate:

```python
import dask.dataframe as dd

df = dd.read_csv('myfile.*.csv')
df = df + 100
df = df[df.name == 'Alice']
```

```python
{
 ('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
 ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
 ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
 ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv'),
 ('add', 0): (operator.add, ('read-csv', 0), 100),
 ('add', 1): (operator.add, ('read-csv', 1), 100),
 ('add', 2): (operator.add, ('read-csv', 2), 100),
 ('add', 3): (operator.add, ('read-csv', 3), 100),
 ('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
 ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
 ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
 ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3)),
}
```

The task graph is a dictionary that stores every Pandas-level function call
necessary to compute the final result.  We can see that there is some structure
to this dictionary if we separate out the tasks that were associated to each
high-level Dask DataFrame operation:

```python
{
 # From the dask.dataframe.read_csv call
 ('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
 ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
 ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
 ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv'),

 # From the df + 100 call
 ('add', 0): (operator.add, ('read-csv', 0), 100),
 ('add', 1): (operator.add, ('read-csv', 1), 100),
 ('add', 2): (operator.add, ('read-csv', 2), 100),
 ('add', 3): (operator.add, ('read-csv', 3), 100),

 # From the df[df.name == 'Alice'] call
 ('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
 ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
 ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
 ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3)),
}
```

By understanding this high-level structure we are able to understand our task
graphs more easily (this is more important for larger datasets when there are
thousands of tasks per layer) and how to perform high-level optimizations.  For
example, in the case above we may want to automatically rewrite our code to
filter our datasets before adding 100:

```python
# Before
df = dd.read_csv('myfile.*.csv')
df = df + 100
df = df[df.name == 'Alice']

# After
df = dd.read_csv('myfile.*.csv')
df = df[df.name == 'Alice']
df = df + 100
```

Dask’s high level graphs help us to explicitly encode this structure by storing
our task graphs in layers with dependencies between layers:

```python
>>> import dask.dataframe as dd

>>> df = dd.read_csv('myfile.*.csv')
>>> df = df + 100
>>> df = df[df.name == 'Alice']

>>> graph = df.__dask_graph__()
>>> graph.layers
{
 'read-csv': {('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
              ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
              ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
              ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv')},

 'add': {('add', 0): (operator.add, ('read-csv', 0), 100),
         ('add', 1): (operator.add, ('read-csv', 1), 100),
         ('add', 2): (operator.add, ('read-csv', 2), 100),
         ('add', 3): (operator.add, ('read-csv', 3), 100)}

 'filter': {('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
            ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
            ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
            ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3))}
}

>>> graph.dependencies
{
 'read-csv': set(),
 'add': {'read-csv'},
 'filter': {'add'}
}
```

While the DataFrame points to the output layers on which it depends directly:

```python
>>> df.__dask_layers__()
{'filter'}
```

## HighLevelGraphs

The `HighLevelGraph` object is a `Mapping` object composed of other
sub-`Mappings`, along with a high-level dependency mapping between them:

```python
class HighLevelGraph(Mapping):
    layers: Dict[str, Mapping]
    dependencies: Dict[str, Set[str]]
```

You can construct a HighLevelGraph explicitly by providing both to the
constructor:

```python
layers = {
   'read-csv': {('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
                ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
                ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
                ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv')},

   'add': {('add', 0): (operator.add, ('read-csv', 0), 100),
           ('add', 1): (operator.add, ('read-csv', 1), 100),
           ('add', 2): (operator.add, ('read-csv', 2), 100),
           ('add', 3): (operator.add, ('read-csv', 3), 100)},

   'filter': {('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
              ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
              ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
              ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3))}
}

dependencies = {'read-csv': set(),
                'add': {'read-csv'},
                'filter': {'add'}}

graph = HighLevelGraph(layers, dependencies)
```

This object satisfies the `Mapping` interface, and so operates as a normal
Python dictionary that is the semantic merger of the underlying layers:

```python
>>> len(graph)
12
>>> graph[('read-csv', 0)]
('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
```

## API

### *class* dask.highlevelgraph.HighLevelGraph(layers: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)]], dependencies: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str), [set](https://docs.python.org/3/library/stdtypes.html#set)[[str](https://docs.python.org/3/library/stdtypes.html#str)]], key_dependencies: [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [set](https://docs.python.org/3/library/stdtypes.html#set)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]] | [None](https://docs.python.org/3/library/constants.html#None) = None)

Task graph composed of layers of dependent subgraphs

This object encodes a Dask task graph that is composed of layers of
dependent subgraphs, such as commonly occurs when building task graphs
using high level collections like Dask array, bag, or dataframe.

Typically each high level array, bag, or dataframe operation takes the task
graphs of the input collections, merges them, and then adds one or more new
layers of tasks for the new operation.  These layers typically have at
least as many tasks as there are partitions or chunks in the collection.
The HighLevelGraph object stores the subgraphs for each operation
separately in sub-graphs, and also stores the dependency structure between
them.

* **Parameters:**
  **layers**
  : The subgraph layers, keyed by a unique name

  **dependencies**
  : The set of layers on which each layer depends

  **key_dependencies**
  : Mapping (some) keys in the high level graph to their dependencies. If
    a key is missing, its dependencies will be calculated on-the-fly.

#### SEE ALSO
[`HighLevelGraph.from_collections`](#dask.highlevelgraph.HighLevelGraph.from_collections)
: typically used by developers to make new HighLevelGraphs

### Examples

Here is an idealized example that shows the internal state of a
HighLevelGraph

```pycon
>>> import dask.dataframe as dd
```

```pycon
>>> df = dd.read_csv('myfile.*.csv')
>>> df = df + 100
>>> df = df[df.name == 'Alice']
```

```pycon
>>> graph = df.__dask_graph__()
>>> graph.layers
{
 'read-csv': {('read-csv', 0): (pandas.read_csv, 'myfile.0.csv'),
              ('read-csv', 1): (pandas.read_csv, 'myfile.1.csv'),
              ('read-csv', 2): (pandas.read_csv, 'myfile.2.csv'),
              ('read-csv', 3): (pandas.read_csv, 'myfile.3.csv')},
 'add': {('add', 0): (operator.add, ('read-csv', 0), 100),
         ('add', 1): (operator.add, ('read-csv', 1), 100),
         ('add', 2): (operator.add, ('read-csv', 2), 100),
         ('add', 3): (operator.add, ('read-csv', 3), 100)}
 'filter': {('filter', 0): (lambda part: part[part.name == 'Alice'], ('add', 0)),
            ('filter', 1): (lambda part: part[part.name == 'Alice'], ('add', 1)),
            ('filter', 2): (lambda part: part[part.name == 'Alice'], ('add', 2)),
            ('filter', 3): (lambda part: part[part.name == 'Alice'], ('add', 3))}
}
```

```pycon
>>> graph.dependencies
{
 'read-csv': set(),
 'add': {'read-csv'},
 'filter': {'add'}
}
```

<!-- !! processed by numpydoc !! -->

#### cull(keys: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]) → [HighLevelGraph](#dask.highlevelgraph.HighLevelGraph)

Return new HighLevelGraph with only the tasks required to calculate keys.

In other words, remove unnecessary tasks from dask.

* **Parameters:**
  **keys**
  : iterable of keys or nested list of keys such as the output of
    `__dask_keys__()`
* **Returns:**
  hlg: HighLevelGraph
  : Culled high level graph

<!-- !! processed by numpydoc !! -->

#### cull_layers(layers: [Iterable](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[str](https://docs.python.org/3/library/stdtypes.html#str)]) → [HighLevelGraph](#dask.highlevelgraph.HighLevelGraph)

Return a new HighLevelGraph with only the given layers and their
dependencies. Internally, layers are not modified.

This is a variant of [`HighLevelGraph.cull()`](#dask.highlevelgraph.HighLevelGraph.cull) which is much faster and does
not risk creating a collision between two layers with the same name and
different content when two culled graphs are merged later on.

* **Returns:**
  hlg: HighLevelGraph
  : Culled high level graph

<!-- !! processed by numpydoc !! -->

#### *classmethod* from_collections(name: [str](https://docs.python.org/3/library/stdtypes.html#str), layer: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)], dependencies: [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[DaskCollection](custom-collections.md#dask.typing.DaskCollection)] = ()) → [HighLevelGraph](#dask.highlevelgraph.HighLevelGraph)

Construct a HighLevelGraph from a new layer and a set of collections

This constructs a HighLevelGraph in the common case where we have a single
new layer and a set of old collections on which we want to depend.

This pulls out the `__dask_layers__()` method of the collections if
they exist, and adds them to the dependencies for this new layer.  It
also merges all of the layers from all of the dependent collections
together into the new layers for this graph.

* **Parameters:**
  **name**
  : The name of the new layer

  **layer**
  : The graph layer itself

  **dependencies**
  : A list of other dask collections (like arrays or dataframes) that
    have graphs themselves

### Examples

In typical usage we make a new task layer, and then pass that layer
along with all dependent collections to this method.

```pycon
>>> def add(self, other):
...     name = 'add-' + tokenize(self, other)
...     layer = {(name, i): (add, input_key, other)
...              for i, input_key in enumerate(self.__dask_keys__())}
...     graph = HighLevelGraph.from_collections(name, layer, dependencies=[self])
...     return new_collection(name, graph)
```

<!-- !! processed by numpydoc !! -->

#### get(k) → D[k] if k in D, else d.  d defaults to None.

<!-- !! processed by numpydoc !! -->

#### get_all_dependencies() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [set](https://docs.python.org/3/library/stdtypes.html#set)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]]

Get dependencies of all keys

This will in most cases materialize all layers, which makes
it an expensive operation.

* **Returns:**
  map: Mapping
  : A map that maps each key to its dependencies

<!-- !! processed by numpydoc !! -->

#### get_all_external_keys() → [set](https://docs.python.org/3/library/stdtypes.html#set)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...]]

Get all output keys of all layers

This will in most cases \_not_ materialize any layers, which makes
it a relative cheap operation.

* **Returns:**
  keys: set
  : A set of all external keys

<!-- !! processed by numpydoc !! -->

#### items() → a set-like object providing a view on D's items

<!-- !! processed by numpydoc !! -->

#### keys() → [KeysView](https://docs.python.org/3/library/collections.abc.html#collections.abc.KeysView)

Get all keys of all the layers.

This will in many cases materialize layers, which makes it a relatively
expensive operation. See [`get_all_external_keys()`](#dask.highlevelgraph.HighLevelGraph.get_all_external_keys) for a faster alternative.

<!-- !! processed by numpydoc !! -->

#### to_dict() → [dict](https://docs.python.org/3/library/stdtypes.html#dict)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[[str](https://docs.python.org/3/library/stdtypes.html#str) | [int](https://docs.python.org/3/library/functions.html#int) | [float](https://docs.python.org/3/library/functions.html#float) | [tuple](https://docs.python.org/3/library/stdtypes.html#tuple)[Key, ...], ...], [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Efficiently convert to plain dict. This method is faster than dict(self).

<!-- !! processed by numpydoc !! -->

#### values() → an object providing a view on D's values

<!-- !! processed by numpydoc !! -->
<!-- TODO: Fix graphviz dependency in docs build and remove ``visualize`` from
exclude-members in the above directive -->


# install.html.md

# Dask Installation

## How to Install Dask

You can install Dask with `conda`, with `pip`, or install from source.

### Conda

If you use the [Anaconda distribution](https://www.anaconda.com/download/), Dask will be installed by default.

You can also install or upgrade Dask using the
[conda install](https://docs.conda.io/projects/conda/en/latest/commands/install.html) command:

```default
conda install dask
```

This installs Dask and **all** common dependencies, including pandas and NumPy.
Dask packages are maintained both on the defaults channel and on
[conda-forge](https://conda-forge.github.io/).
You can select the channel with the `-c` flag:

```default
conda install dask -c conda-forge
```

Optionally, you can obtain a minimal Dask installation using the following command:

```default
conda install dask-core
```

This will install a minimal set of dependencies required to run Dask similar to (but not exactly the same as) `python -m pip install dask`.

### Pip

To install Dask with `pip` run the following:

```default
python -m pip install "dask[complete]"    # Install everything
```

This installs Dask, the distributed scheduler, and common dependencies
like pandas, Numpy, and others.

You can also install only the Dask library and no optional dependencies:

```default
python -m pip install dask                # Install only core parts of dask
```

Dask modules like `dask.array`, `dask.dataframe`, or
`dask.distributed` won’t work until you also install NumPy, pandas, or
Tornado, respectively.  This is uncommon for users but more common for
downstream library maintainers.

We also maintain other dependency sets for different subsets of functionality:

```default
python -m pip install "dask[array]"       # Install requirements for dask array
python -m pip install "dask[dataframe]"   # Install requirements for dask dataframe
python -m pip install "dask[diagnostics]" # Install requirements for dask diagnostics
python -m pip install "dask[distributed]" # Install requirements for distributed dask
```

We have these options so that users of the lightweight core Dask scheduler
aren’t required to download the more exotic dependencies of the collections
(Numpy, pandas, Tornado, etc.).

### Source

To install Dask from source, clone the repository from [GitHub](https://github.com/dask/dask):

```default
git clone https://github.com/dask/dask.git
cd dask
python -m pip install .
```

You can also install all dependencies as well:

```default
python -m pip install ".[complete]"
```

You can view the list of all dependencies within the `project.optional-dependencies` field
of `pyproject.toml`.

Or do a developer install by using the `-e` flag
(see the [Install section](develop.md#develop-install) in the Development Guidelines):

```default
python -m pip install -e .
```

## Distributed Deployment

To run Dask on a distributed cluster you will want to also install the Dask
cluster manager that matches your resource manager, like Kubernetes, SLURM, PBS,
LSF, AWS, GCP, Azure, or similar technology.

Read more on this topic at [Deploy Documentation](deploying.html)

## Optional dependencies

Specific functionality in Dask may require additional optional dependencies.
For example, reading from Amazon S3 requires [s3fs](https://s3fs.readthedocs.io/en/latest/).
These optional dependencies and their minimum supported versions are listed below.

| Dependency                                                               | Version      | Description                                                                                                                               |
|--------------------------------------------------------------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| [bokeh](https://bokeh.org/)                                              | `>=3.1.0`    | Generate profiles of Dask execution (required for `dask.diagnostics`)                                                                     |
| [bottleneck](https://bottleneck.readthedocs.io/en/latest/index.html)     | `>=1.3.7`    | Used for dask arrays `push` implementation                                                                                                |
| [cachey](https://github.com/dask/cachey)                                 | `>=0.1.1`    | Use caching for computation                                                                                                               |
| [cityhash](https://github.com/escherba/python-cityhash)                  | `>=0.2.4`    | Use CityHash and FarmHash hash functions for array hashing (~2x faster than MurmurHash)                                                   |
| [crick](https://github.com/dask/crick)                                   | `>=0.0.5`    | Use `tdigest` internal method for dataframe statistics computation                                                                        |
| [cytoolz](https://github.com/pytoolz/cytoolz)                            | `>=0.11.2`   | Faster cythonized implementation of internal iterators, functions, and dictionaries                                                       |
| [dask-ml](https://ml.dask.org/)                                          | `>=1.4.0`    | Common machine learning functions scaled with Dask                                                                                        |
| [fastavro](https://fastavro.readthedocs.io/en/latest/)                   | `>=1.1.0`    | Storing and reading data from Apache Avro files                                                                                           |
| [gcsfs](https://gcsfs.readthedocs.io/en/latest/)                         | `>=2021.9.0` | Storing and reading data located in Google Cloud Storage                                                                                  |
| [graphviz](https://graphviz.readthedocs.io/en/stable/)                   | `>=0.8.4`    | Graph visualization using the graphviz engine                                                                                             |
| [h5py](https://www.h5py.org/)                                            | `>=3.7.0`    | Storing array data in hdf5 files                                                                                                          |
| [ipycytoscape](https://ipycytoscape.readthedocs.io/en/master/index.html) | `>=1.0.1`    | Graph visualization using the cytoscape engine                                                                                            |
| [IPython](https://ipython.org/)                                          | `>=7.30.1`   | Write graph visualizations made with graphviz engine to file                                                                              |
| [jinja2](https://jinja.palletsprojects.com/)                             | `>=2.10.3`   | HTML representations of Dask objects in Jupyter notebooks (required for `dask.diagnostics`)                                               |
| [lz4](https://python-lz4.readthedocs.io/en/stable/index.html)            | `>=4.3.2`    | Transparent use of lz4 compression algorithm                                                                                              |
| [matplotlib](https://matplotlib.org/)                                    | `>=3.5.0`    | Color map support for graph visualization                                                                                                 |
| [mimesis](https://mimesis.name/en/master/)                               | `>=5.3.0`    | Random bag data generation with [`dask.datasets.make_people()`](api.md#dask.datasets.make_people)                                         |
| [mmh3](https://github.com/hajimes/mmh3)                                  | `>=3.0.0`    | Use MurmurHash hash functions for array hashing (~8x faster than SHA1)                                                                    |
| ```<br/>`numba`_<br/>```                                                 | `>=0.59.1`   | Can be used to write functions to run e.g. with `map_blocks` / `map_partitions`                                                           |
| ```<br/>`numbagg`_<br/>```                                               | `>=0.8.0`    | Accelerates some `dask.array` nan-aggregations if installed                                                                               |
| [numpy](https://numpy.org/)                                              | `>=1.24`     | Required for `dask.array`                                                                                                                 |
| [pandas](https://pandas.pydata.org/)                                     | `>=2.0`      | Required for `dask.dataframe`                                                                                                             |
| [psutil](https://psutil.readthedocs.io/en/latest/)                       | `>=5.8.0`    | Factor CPU affinity into CPU count, intelligently infer blocksize when reading CSV files                                                  |
| [pyarrow](https://arrow.apache.org/docs/python/index.html)               | `>=16.0`     | Support for Apache Arrow datatypes & engine when storing/reading Apache ORC or Parquet files                                              |
| [python-snappy](https://github.com/andrix/python-snappy)                 | `>=0.7.1`    | Snappy compression to bs used when storing/reading Avro or Parquet files                                                                  |
| [s3fs](https://s3fs.readthedocs.io/en/latest/)                           | `>=2021.9.0` | Storing and reading data located in Amazon S3                                                                                             |
| [scipy](https://scipy.org/)                                              | `>=1.10.0`   | Required for `dask.array.stats`, `dask.array.fft`, and [`dask.array.linalg.lu()`](generated/dask.array.linalg.lu.md#dask.array.linalg.lu) |
| [sparse](https://sparse.pydata.org/en/stable/)                           | `>=0.13.0`   | Use sparse arrays as backend for dask arrays                                                                                              |
| [sqlalchemy](https://www.sqlalchemy.org/)                                | `>=1.4.26`   | Writing and reading from SQL databases                                                                                                    |
| [tblib](https://python-tblib.readthedocs.io/en/latest/readme.html)       | `>=1.6.0`    | Serialization of worker traceback objects                                                                                                 |
| [tiledb](https://github.com/TileDB-Inc/TileDB-Py)                        | `>=0.27.0`   | Storing and reading data from TileDB files                                                                                                |
| [xxhash](https://github.com/ifduyue/python-xxhash)                       | `>=2.0.0`    | Use xxHash hash functions for array hashing (~2x faster than MurmurHash, slightly slower than CityHash)                                   |
| [zarr](https://zarr.readthedocs.io/en/stable/index.html)                 | `>=2.12.0`   | Storing and reading data from Zarr files                                                                                                  |

## Test

Test Dask with `py.test`:

```default
cd dask
py.test dask
```

Installing Dask naively may not install all requirements by default (see the `pip` section above).
You may choose to install the `dask[complete]` version which includes
all dependencies for all collections:

```default
python -m pip install "dask[complete]"
```

Alternatively, you may choose to test
only certain submodules depending on the libraries within your environment.
For example, to test only Dask core and Dask array we would run tests as
follows:

```default
py.test dask/tests dask/array/tests
```

See the [section on testing](develop.md#develop-test) in the Development Guidelines for more details.


# internals.html.md

# Dask Internals

This section is intended for contributors and power users who are interested in
learning more about how Dask works internally.

* [User Interfaces](user-interfaces.md)
* [Understanding Performance](understanding-performance.md)
* [Stages of Computation](phases-of-computation.md)
* [Ordering](order.md)
* [Opportunistic Caching](caching.md)
* [Shared Memory](shared.md)
* [Scheduling in Depth](scheduling-policy.md)
* [Query planning with Expression system](expr-system-internals.md)


# logos.html.md

# Images and Logos

Here are some commonly used Dask icons and logos
(see the [Dask style guide](https://www.dask.org/style-guide) for more details).

![Primary Dask icon.](images/dask_icon.svg)![Dask icon in black.](images/dask_icon_black.svg)![Dask icon in white.](images/dask_icon_white.svg)![Dask icon to use on a pink background.](images/dask_icon_on_pink.svg)![Primary Dask logo.](images/dask_horizontal.svg)![Dask logo in black.](images/dask_horizontal_black.svg)![Dask logo in white.](images/dask_horizontal_white.svg)![Dask logo to use on a pink background.](images/dask_horizontal_on_pink.svg)![Dask logo to use on a blue background.](images/dask_horizontal_on_blue.svg)


# maintainers.html.md

# Maintainer Guidelines

This page describes best practices for Dask maintainers.

Thank you for helping make Dask a useful library and fostering a
welcoming community.

## Merging pull requests

### Pull requests should be reviewed

Pull requests from non-maintainers should be reviewed and approved by at
least one maintainer before being merged.

Ideally, pull requests from maintainers should also be reviewed before
merging. However, because reviewer bandwidth is limited, maintainers will
sometimes self-merge their own pull requests if the content of the pull
request is viewed as uncontroversial (e.g. typo fix). If a maintainer has
a substantial pull request which hasn’t received a review, they should ping at
least one team or individual maintainer who might be interested in the proposed
change. If after some time there is no response and the maintainer is confident
in the proposed changes, they should post a final comment along the lines of
“Merging in 48 hours if no further response” and then self-merge if no further
comments are left within the specified time period. Please leave at least
48 hours to allow maintainers in other timezones to respond.

No maintainer should merge a pull request that they’re not comfortable
supporting in the future.

### CI should pass

Before merging a pull request, maintainers should make every effort to ensure
that CI passes. Often this will require looking into the logs of a failed run
to see what went wrong, and alerting the pull request author. Ideally, no
pull request should be merged if there are CI failures, as broken CI in `main`
can easily mask problems with other PRs, and a consistently broken CI can be
demoralizing for maintainers.

However, in practice, there are occasionally flaky tests, broken upstream
dependencies, and failures that are otherwise obviously not related to the PR
at hand. If that is the case, a maintainer may merge a PR with failing tests,
but they should be prepared to follow up with any failures that result from
such an unsafe operation.

### Squash merge pull requests

Use squash merging when merging pull requests, as opposed to other merging
strategies like rebase merging. To streamline this process, all non-squash
merging strategies have been disabled in repository GitHub settings.

### Use clean merge commits

Dask aims to have a straightforward, yet meaningful, `git log`. To
accomplish this, maintainers ensure that both the squashed merge commit
title and (optional) message meaningfully reflect the content of the pull
request before merging. For example, a merge commit title of “fix typo” should
be updated to something along the lines of “Fix typo in `Array.max` docstring”.

Note that when squash merging pull requests GitHub will, by default, pre-populate
the squashed commit message by concatenating all the individual commit messages
in the corresponding pull request. Depending on how careful pull request authors
are during development, this default message may be quite verbose and not very
descriptive. For example:

```default
* Fix DataFrame.head bug

* Handle merge conflicts

* Oops, fix typo
```

If this is the case, the maintainer merging the pull request should either:

1. Write their own merge commit message that describes the changes in pull request
   (if they have the available bandwidth).
2. Leave the merge commit message blank (again, making sure the commit title is
   meaningful).


# ml.html.md

# Machine Learning

Machine learning is a broad field involving many different workflows.  This
page lists a few of the more common ways in which Dask can help you with ML
workloads.

- [Hyperparameter Optimization](#hpo)
- [Gradient Boosted Trees](#boosted-trees)
- [Batch Prediction](#batch-prediction)

<a id="hpo"></a>

## Hyperparameter Optimization

### Optuna

For state of the art hyperparameter optimization (HPO) we recommend the
[Optuna](https://optuna.org/) library,
with the associated
[Dask integration](https://optuna-integration.readthedocs.io/en/latest/reference/generated/optuna_integration.DaskStorage.html).

In Optuna you construct an objective function that takes a trial object, which
generates parameters from distributions that you define in code.  Your
objective function eventually produces a score.  Optuna is smart about what
values from the distribution it suggests based on the scores it has received.

```python
def objective(trial):
    params = {
        "max_depth": trial.suggest_int("max_depth", 2, 10, step=1),
        "learning_rate": trial.suggest_float("learning_rate", 1e-8, 1.0, log=True),
        ...
    }
    model = train_model(train_data, **params)
    result = score(model, test_data)
    return result
```

Dask and Optuna are often used together by running many objective functions in
parallel and synchronizing the scores and parameter selection on the Dask
scheduler.  To do this, we use the `DaskStorage` object provided by the
[optuna-integration](https://optuna-integration.readthedocs.io/) package
(install it alongside Optuna with `pip install optuna optuna-integration`).

```python
import optuna
from optuna_integration import DaskStorage

storage = DaskStorage()

study = optuna.create_study(
    direction="maximize",
    storage=storage,  # This makes the study Dask-enabled
)
```

Then we run many optimize methods in parallel.

```python
from dask.distributed import LocalCluster, wait

cluster = LocalCluster(processes=False)  # replace this with some scalable cluster
client = cluster.get_client()

futures = [
    client.submit(study.optimize, objective, n_trials=1, pure=False) for _ in range(500)
]
wait(futures)

print(study.best_params)
```

For a more fully worked example see this [Optuna + XGBoost example](https://docs.coiled.io/user_guide/hpo.html?utm_source=dask-docs&utm_medium=ml).

### Dask Futures

Additionally, for simpler situations people often use [Dask Futures](futures.md) to
train the same model on lots of parameters.  Dask Futures are a general purpose
API that is used to run normal Python functions on various inputs.  An example
might look like the following:

```python
from dask.distributed import LocalCluster

cluster = LocalCluster(processes=False)  # replace this with some scalable cluster
client = cluster.get_client()

def train_and_score(params: dict) -> float:
    data = load_data()
    model = make_model(**params)
    train(model)
    score = evaluate(model)
    return score

params_list = [...]
futures = [
    client.submit(train_and_score, params) for params in params_list
]
scores = client.gather(futures)
best = max(scores)

best_params = params_list[scores.index(best)]
```

For a more fully worked example see [Futures Documentation](futures.html).

<a id="boosted-trees"></a>

## Gradient Boosted Trees

Popular GBT libraries, like XGBoost and LightGBM, have native Dask support which allows you to train models
on very large datasets in parallel.

- [XGBoost](https://xgboost.readthedocs.io/en/stable/tutorials/dask.html)
- [LightGBM](https://lightgbm.readthedocs.io/en/latest/Parallel-Learning-Guide.html#dask)

For example, using Dask DataFrame, XGBoost, and a local Dask cluster looks like the following:

```python
import dask.dataframe as dd
import xgboost as xgb
from dask.distributed import LocalCluster

df = dask.datasets.timeseries()  # Randomly generated data
# df = dd.read_parquet(...)      # In practice, you would probably read data though

train, test = df.random_split([0.80, 0.20])
X_train, y_train, X_test, y_test = ...

with LocalCluster() as cluster:
    with cluster.get_client() as client:
        d_train = xgb.dask.DaskDMatrix(client, X_train, y_train, enable_categorical=True)
        model = xgb.dask.train(
            ...
            d_train,
        )
        predictions = xgb.dask.predict(client, model, X_test)
```

For a more fully worked example see this [XGBoost example](https://docs.coiled.io/user_guide/xgboost.html?utm_source=dask-docs&utm_medium=ml).

<a id="batch-prediction"></a>

## Batch Prediction

Once a model is trained, it’s common to want to apply the model across
lots of data.  We see this done most often in two ways:

1. Using Dask Futures
2. Using [`DataFrame.map_partitions`](generated/dask.dataframe.DataFrame.map_partitions.md#dask.dataframe.DataFrame.map_partitions)
   or [`Array.map_blocks`](generated/dask.array.Array.map_blocks.md#dask.array.Array.map_blocks)

We’ll show examples of each approach below.

### Dask Futures

Dask Futures are a general purpose API that lets you run arbitrary Python
functions on Python data in parallel. It’s easy to apply this tool to solve the problem of
batch prediction.

For example, we often see this when people want to apply a model across many
data files.

```python
from dask.distributed import LocalCluster

cluster = LocalCluster(processes=False)  # replace this with some scalable cluster
client = cluster.get_client()

filenames = [...]

def predict(filename, model):
    data = load(filename)
    result = model.predict(data)
    return result

model = client.submit(load_model, path_to_model)
predictions = client.map(predict, filenames, model=model)
results = client.gather(predictions)
```

For a more fully worked example see [Batch Scoring for Computer Vision Workloads (video)](https://developer.download.nvidia.com/video/gputechconf/gtc/2019/video/S9198/s9198-dask-and-v100s-for-fast-distributed-batch-scoring-of-computer-vision-workloads.mp4).

### Dask DataFrame

Sometimes we want to process our model with a higher
level Dask API, like Dask DataFrame or Dask Array. This is more common with
record data, for example if we had a set of patient records and wanted to
see which patients were likely to become ill.

```python
import dask.dataframe as dd

df = dd.read_parquet("/path/to/my/data.parquet")

model = load_model("/path/to/my/model")

# pandas code
# predictions = model.predict(df)
# predictions.to_parquet("/path/to/results.parquet")

# Dask code
predictions = df.map_partitions(model.predict)
predictions.to_parquet("/path/to/results.parquet")
```

For more information see [Dask DataFrame documentation](dataframe.html).


# optimize.html.md

<a id="optimization"></a>

# Optimization

Performance can be significantly improved in different contexts by making
small optimizations on the Dask graph before calling the scheduler.

The `dask.optimization` module contains several functions to transform graphs
in a variety of useful ways. In most cases, users won’t need to interact with
these functions directly, as specialized subsets of these transforms are done
automatically in the Dask collections (`dask.array`, `dask.bag`, and
`dask.dataframe`). However, users working with custom graphs or computations
may find that applying these methods results in substantial speedups.

In general, there are two goals when doing graph optimizations:

1. Simplify computation
2. Improve parallelism

Simplifying computation can be done on a graph level by removing unnecessary
tasks (`cull`), or on a task level by replacing expensive operations with
cheaper ones (`RewriteRule`).

Parallelism can be improved by reducing
inter-task communication, whether by fusing many tasks into one (`fuse`), or
by inlining cheap operations (`inline`, `inline_functions`).

Below, we show an example walking through the use of some of these to optimize
a task graph.

## Example

Suppose you had a custom Dask graph for doing a word counting task:

```python
>>> def print_and_return(string):
...     print(string)
...     return string

>>> def format_str(count, val, nwords):
...     return (f'word list has {count} occurrences of '
...             f'{val}, out of {nwords} words')

>>> dsk = {'words': 'apple orange apple pear orange pear pear',
...        'nwords': (len, (str.split, 'words')),
...        'val1': 'orange',
...        'val2': 'apple',
...        'val3': 'pear',
...        'count1': (str.count, 'words', 'val1'),
...        'count2': (str.count, 'words', 'val2'),
...        'count3': (str.count, 'words', 'val3'),
...        'format1': (format_str, 'count1', 'val1', 'nwords'),
...        'format2': (format_str, 'count2', 'val2', 'nwords'),
...        'format3': (format_str, 'count3', 'val3', 'nwords'),
...        'print1': (print_and_return, 'format1'),
...        'print2': (print_and_return, 'format2'),
...        'print3': (print_and_return, 'format3')}
```

![The original non-optimized Dask task graph.](images/optimize_dask1.svg)

Here we are counting the occurrence of the words `'orange`, `'apple'`, and
`'pear'` in the list of words, formatting an output string reporting the
results, printing the output, and then returning the output string.

To perform the computation, we first remove unnecessary components from the
graph using the `cull` function and then pass the Dask graph and the desired
output keys to a scheduler `get` function:

```python
>>> from dask.threaded import get
>>> from dask.optimization import cull

>>> outputs = ['print1', 'print2']
>>> dsk1, dependencies = cull(dsk, outputs)  # remove unnecessary tasks from the graph

>>> results = get(dsk1, outputs)
word list has 2 occurrences of apple, out of 7 words
word list has 2 occurrences of orange, out of 7 words
```

As can be seen above, the scheduler computed only the requested outputs
(`'print3'` was never computed). This is because we called the
`dask.optimization.cull` function, which removes the unnecessary tasks from
the graph.

Culling is part of the default optimization pass of almost all collections.
Often you want to call it somewhat early to reduce the amount of work done in
later steps:

```python
>>> from dask.optimization import cull
>>> outputs = ['print1', 'print2']
>>> dsk1, dependencies = cull(dsk, outputs)
```

![The Dask task graph after culling tasks for optimization.](images/optimize_dask2.svg)

Looking at the task graph above, there are multiple accesses to constants such
as `'val1'` or `'val2'` in the Dask graph. These can be inlined into the
tasks to improve efficiency using the `inline` function. For example:

```python
>>> from dask.optimization import inline
>>> dsk2 = inline(dsk1, dependencies=dependencies)
>>> results = get(dsk2, outputs)
word list has 2 occurrences of apple, out of 7 words
word list has 2 occurrences of orange, out of 7 words
```

![The Dask task graph after inlining for optimization.](images/optimize_dask3.svg)

Now we have two sets of *almost* linear task chains. The only link between them
is the word counting function. For cheap operations like this, the
serialization cost may be larger than the actual computation, so it may be
faster to do the computation more than once, rather than passing the results to
all nodes. To perform this function inlining, the `inline_functions` function
can be used:

```python
>>> from dask.optimization import inline_functions
>>> dsk3 = inline_functions(dsk2, outputs, [len, str.split],
...                         dependencies=dependencies)
>>> results = get(dsk3, outputs)
word list has 2 occurrences of apple, out of 7 words
word list has 2 occurrences of orange, out of 7 words
```

![The Dask task graph after inlining functions for optimization.](images/optimize_dask4.svg)

Now we have a set of purely linear tasks. We’d like to have the scheduler run
all of these on the same worker to reduce data serialization between workers.
One option is just to merge these linear chains into one big task using the
`fuse` function:

```python
>>> from dask.optimization import fuse
>>> dsk4, dependencies = fuse(dsk3)
>>> results = get(dsk4, outputs)
word list has 2 occurrences of apple, out of 7 words
word list has 2 occurrences of orange, out of 7 words
```

![The Dask task graph after fusing tasks for optimization.](images/optimize_dask5.svg)

Putting it all together:

```python
>>> def optimize_and_get(dsk, keys):
...     dsk1, deps = cull(dsk, keys)
...     dsk2 = inline(dsk1, dependencies=deps)
...     dsk3 = inline_functions(dsk2, keys, [len, str.split],
...                             dependencies=deps)
...     dsk4, deps = fuse(dsk3)
...     return get(dsk4, keys)

>>> optimize_and_get(dsk, outputs)
word list has 2 occurrences of apple, out of 7 words
word list has 2 occurrences of orange, out of 7 words
```

In summary, the above operations accomplish the following:

1. Removed tasks unnecessary for the desired output using `cull`
2. Inlined constants using `inline`
3. Inlined cheap computations using `inline_functions`, improving parallelism
4. Fused linear tasks together to ensure they run on the same worker using `fuse`

As stated previously, these optimizations are already performed automatically
in the Dask collections. Users not working with custom graphs or computations
should rarely need to directly interact with them.

These are just a few of the optimizations provided in `dask.optimization`. For
more information, see the API below.

## Rewrite Rules

For context based optimizations, `dask.rewrite` provides functionality for
pattern matching and term rewriting. This is useful for replacing expensive
computations with equivalent, cheaper computations. For example, Dask Array
uses the rewrite functionality to replace series of array slicing operations
with a more efficient single slice.

The interface to the rewrite system consists of two classes:

1. `RewriteRule(lhs, rhs, vars)`
   > Given a left-hand-side (`lhs`), a right-hand-side (`rhs`), and a set of
   > variables (`vars`), a rewrite rule declaratively encodes the following
   > operation:

   > `lhs -> rhs if task matches lhs over variables`
2. `RuleSet(*rules)`
   > A collection of rewrite rules. The design of `RuleSet` class allows for
   > efficient “many-to-one” pattern matching, meaning that there is minimal
   > overhead for rewriting with multiple rules in a rule set.

### Example

Here we create two rewrite rules expressing the following mathematical transformations:

1. `a + a -> 2*a`
2. `a * a -> a**2`

where `'a'` is a variable:

```python
>>> from dask.rewrite import RewriteRule, RuleSet
>>> from operator import add, mul, pow

>>> variables = ('a',)

>>> rule1 = RewriteRule((add, 'a', 'a'), (mul, 'a', 2), variables)

>>> rule2 = RewriteRule((mul, 'a', 'a'), (pow, 'a', 2), variables)

>>> rs = RuleSet(rule1, rule2)
```

The `RewriteRule` objects describe the desired transformations in a
declarative way, and the `RuleSet` builds an efficient automata for applying
that transformation. Rewriting can then be done using the `rewrite` method:

```python
>>> rs.rewrite((add, 5, 5))
(mul, 5, 2)

>>> rs.rewrite((mul, 5, 5))
(pow, 5, 2)

>>> rs.rewrite((mul, (add, 3, 3), (add, 3, 3)))
(pow, (mul, 3, 2), 2)
```

The whole task is traversed by default. If you only want to apply a transform
to the top-level of the task, you can pass in `strategy='top_level'` as shown:

```python
# Transforms whole task
>>> rs.rewrite((sum, [(add, 3, 3), (mul, 3, 3)]))
(sum, [(mul, 3, 2), (pow, 3, 2)])

# Only applies to top level, no transform occurs
>>> rs.rewrite((sum, [(add, 3, 3), (mul, 3, 3)]), strategy='top_level')
(sum, [(add, 3, 3), (mul, 3, 3)])
```

The rewriting system provides a powerful abstraction for transforming
computations at a task level. Again, for many users, directly interacting with
these transformations will be unnecessary.

## Keyword Arguments

Some optimizations take optional keyword arguments.  To pass keywords from the
compute call down to the right optimization, prepend the keyword with the name
of the optimization.  For example, to send a `keys=` keyword argument to the
`fuse` optimization from a compute call, use the `fuse_keys=` keyword:

```python
def fuse(dsk, keys=None):
    ...

x.compute(fuse_keys=['x', 'y', 'z'])
```

## Customizing Optimization

Dask defines a default optimization strategy for each collection type (Array,
Bag, DataFrame, Delayed).  However, different applications may have different
needs.  To address this variability of needs, you can construct your own custom
optimization function and use it instead of the default.  An optimization
function takes in a task graph and list of desired keys and returns a new
task graph:

```python
def my_optimize_function(dsk, keys):
    new_dsk = {...}
    return new_dsk
```

You can then register this optimization class against whichever collection type
you prefer and it will be used instead of the default scheme:

```python
with dask.config.set(array_optimize=my_optimize_function):
    x, y = dask.compute(x, y)
```

You can register separate optimization functions for different collections, or
you can register `None` if you do not want particular types of collections to
be optimized:

```python
with dask.config.set(array_optimize=my_optimize_function,
                     dataframe_optimize=None,
                     delayed_optimize=my_other_optimize_function):
    ...
```

You do not need to specify all collections.  Collections will default to their
standard optimization scheme (which is usually a good choice).

## API

**Top level optimizations**

| [`cull`](#dask.optimization.cull)(dsk, keys)                                  | Return new dask with only the tasks required to calculate keys.   |
|-------------------------------------------------------------------------------|-------------------------------------------------------------------|
| [`fuse`](#dask.optimization.fuse)(dsk[, keys, dependencies, ave_width, ...])  | Fuse tasks that form reductions; more advanced than `fuse_linear` |
| [`inline`](#dask.optimization.inline)(dsk[, keys, inline_constants, ...])     | Return new dask with the given keys inlined with their values.    |
| [`inline_functions`](#dask.optimization.inline_functions)(dsk, output[, ...]) | Inline cheap functions into larger operations                     |

**Utility functions**

| [`functions_of`](#dask.optimization.functions_of)(task)   | Set of functions contained within nested task   |
|-----------------------------------------------------------|-------------------------------------------------|

**Rewrite Rules**

| [`RewriteRule`](#dask.rewrite.RewriteRule)(lhs, rhs[, vars])   | A rewrite rule.         |
|----------------------------------------------------------------|-------------------------|
| [`RuleSet`](#dask.rewrite.RuleSet)(\*rules)                    | A set of rewrite rules. |

### Definitions

### dask.optimization.cull(dsk, keys)

Return new dask with only the tasks required to calculate keys.

In other words, remove unnecessary tasks from dask.
`keys` may be a single key or list of keys.

* **Returns:**
  dsk: culled dask graph

  dependencies: Dict mapping {key: [deps]}.  Useful side effect to accelerate
  : other optimizations, notably fuse.

### Examples

```pycon
>>> def inc(x):
...     return x + 1
```

```pycon
>>> def add(x, y):
...     return x + y
```

```pycon
>>> d = {'x': 1, 'y': (inc, 'x'), 'out': (add, 'x', 10)}
>>> dsk, dependencies = cull(d, 'out')
>>> dsk
{'out': (<function add at ...>, 'x', 10), 'x': 1}
>>> dependencies
{'out': ['x'], 'x': []}
```

<!-- !! processed by numpydoc !! -->

### dask.optimization.fuse(dsk, keys=None, dependencies=None, ave_width=<default>, max_width=<default>, max_height=<default>, max_depth_new_edges=<default>, rename_keys=<default>)

Fuse tasks that form reductions; more advanced than `fuse_linear`

This trades parallelism opportunities for faster scheduling by making tasks
less granular.  It can replace `fuse_linear` in optimization passes.

This optimization applies to all reductions–tasks that have at most one
dependent–so it may be viewed as fusing “multiple input, single output”
groups of tasks into a single task.  There are many parameters to fine
tune the behavior, which are described below.  `ave_width` is the
natural parameter with which to compare parallelism to granularity, so
it should always be specified.  Reasonable values for other parameters
will be determined using `ave_width` if necessary.

* **Parameters:**
  **dsk: dict**
  : dask graph

  **keys: list or set, optional**
  : Keys that must remain in the returned dask graph

  **dependencies: dict, optional**
  : {key: [list-of-keys]}.  Must be a list to provide count of each key
    This optional input often comes from `cull`

  **ave_width: float (default 1)**
  : Upper limit for `width = num_nodes / height`, a good measure of
    parallelizability.
    dask.config key: `optimization.fuse.ave-width`

  **max_width: int (default infinite)**
  : Don’t fuse if total width is greater than this.
    dask.config key: `optimization.fuse.max-width`

  **max_height: int or None (default None)**
  : Don’t fuse more than this many levels. Set to None to dynamically
    adjust to `1.5 + ave_width * log(ave_width + 1)`.
    dask.config key: `optimization.fuse.max-height`

  **max_depth_new_edges: int or None (default None)**
  : Don’t fuse if new dependencies are added after this many levels.
    Set to None to dynamically adjust to ave_width \* 1.5.
    dask.config key: `optimization.fuse.max-depth-new-edges`

  **rename_keys: bool or func, optional (default True)**
  : Whether to rename the fused keys with `default_fused_keys_renamer`
    or not.  Renaming fused keys can keep the graph more understandable
    and comprehensive, but it comes at the cost of additional processing.
    If False, then the top-most key will be used.  For advanced usage, a
    function to create the new name is also accepted.
    dask.config key: `optimization.fuse.rename-keys`
* **Returns:**
  dsk
  : output graph with keys fused

  dependencies
  : dict mapping dependencies after fusion.  Useful side effect to accelerate other
    downstream optimizations.

<!-- !! processed by numpydoc !! -->

### dask.optimization.inline(dsk, keys=None, inline_constants=True, dependencies=None)

Return new dask with the given keys inlined with their values.

Inlines all constants if `inline_constants` keyword is True. Note that
the constant keys will remain in the graph, to remove them follow
`inline` with `cull`.

### Examples

```pycon
>>> def inc(x):
...     return x + 1
```

```pycon
>>> def add(x, y):
...     return x + y
```

```pycon
>>> d = {'x': 1, 'y': (inc, 'x'), 'z': (add, 'x', 'y')}
>>> inline(d)
{'x': 1, 'y': (<function inc at ...>, 1), 'z': (<function add at ...>, 1, 'y')}
```

```pycon
>>> inline(d, keys='y')
{'x': 1, 'y': (<function inc at ...>, 1), 'z': (<function add at ...>, 1, (<function inc at ...>, 1))}
```

```pycon
>>> inline(d, keys='y', inline_constants=False)
{'x': 1, 'y': (<function inc at ...>, 'x'), 'z': (<function add at ...>, 'x', (<function inc at ...>, 'x'))}
```

<!-- !! processed by numpydoc !! -->

### dask.optimization.inline_functions(dsk, output, fast_functions=None, inline_constants=False, dependencies=None)

Inline cheap functions into larger operations

### Examples

```pycon
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> double = lambda x: x * 2
>>> dsk = {'out': (add, 'i', 'd'),
...        'i': (inc, 'x'),
...        'd': (double, 'y'),
...        'x': 1, 'y': 1}
>>> inline_functions(dsk, [], [inc])
{'out': (add, (inc, 'x'), 'd'),
 'd': (double, 'y'),
 'x': 1, 'y': 1}
```

Protect output keys.  In the example below `i` is not inlined because it
is marked as an output key.

```pycon
>>> inline_functions(dsk, ['i', 'out'], [inc, double])
{'out': (add, 'i', (double, 'y')),
 'i': (inc, 'x'),
 'x': 1, 'y': 1}
```

<!-- !! processed by numpydoc !! -->

### dask.optimization.functions_of(task)

Set of functions contained within nested task

### Examples

```pycon
>>> inc = lambda x: x + 1
>>> add = lambda x, y: x + y
>>> mul = lambda x, y: x * y
>>> task = (add, (mul, 1, 2), (inc, 3))
>>> functions_of(task)
set([add, mul, inc])
```

<!-- !! processed by numpydoc !! -->

### dask.rewrite.RewriteRule(lhs, rhs, vars=())

A rewrite rule.

Expresses lhs -> rhs, for variables vars.

* **Parameters:**
  **lhs**
  : The left-hand-side of the rewrite rule.

  **rhs**
  : The right-hand-side of the rewrite rule. If it’s a task, variables in
    rhs will be replaced by terms in the subject that match the variables
    in lhs. If it’s a function, the function will be called with a dict
    of such matches.

  **vars: tuple, optional**
  : Tuple of variables found in the lhs. Variables can be represented as
    any hashable object; a good convention is to use strings. If there are
    no variables, this can be omitted.

### Examples

Here’s a RewriteRule to replace all nested calls to list, so that
(list, (list, ‘x’)) is replaced with (list, ‘x’), where ‘x’ is a
variable.

```pycon
>>> import dask.rewrite as dr
>>> lhs = (list, (list, 'x'))
>>> rhs = (list, 'x')
>>> variables = ('x',)
>>> rule = dr.RewriteRule(lhs, rhs, variables)
```

Here’s a more complicated rule that uses a callable right-hand-side. A
callable rhs takes in a dictionary mapping variables to their matching
values. This rule replaces all occurrences of (list, ‘x’) with ‘x’ if
‘x’ is a list itself.

```pycon
>>> lhs = (list, 'x')
>>> def repl_list(sd):
...     x = sd['x']
...     if isinstance(x, list):
...         return x
...     else:
...         return (list, x)
>>> rule = dr.RewriteRule(lhs, repl_list, variables)
```

<!-- !! processed by numpydoc !! -->

### dask.rewrite.RuleSet(\*rules)

A set of rewrite rules.

Forms a structure for fast rewriting over a set of rewrite rules. This
allows for syntactic matching of terms to patterns for many patterns at
the same time.

* **Attributes:**
  **rules**
  : A list of RewriteRule\`s included in the \`RuleSet.

### Examples

```pycon
>>> import dask.rewrite as dr
>>> def f(*args): pass
>>> def g(*args): pass
>>> def h(*args): pass
>>> from operator import add
```

```pycon
>>> rs = dr.RuleSet(
...         dr.RewriteRule((add, 'x', 0), 'x', ('x',)),
...         dr.RewriteRule((f, (g, 'x'), 'y'),
...                        (h, 'x', 'y'),
...                        ('x', 'y')))
```

```pycon
>>> rs.rewrite((add, 2, 0))
2
```

```pycon
>>> rs.rewrite((f, (g, 'a', 3)))
(<function h at ...>, 'a', 3)
```

```pycon
>>> dsk = {'a': (add, 2, 0),
...        'b': (f, (g, 'a', 3))}
```

```pycon
>>> from toolz import valmap
>>> valmap(rs.rewrite, dsk)
{'a': 2, 'b': (<function h at ...>, 'a', 3)}
```

<!-- !! processed by numpydoc !! -->


# order.html.md

<a id="order"></a>

# Ordering

#### NOTE
This is an advanced topic that most users won’t need to worry about.

When Dask is given a task graph to compute, it needs to choose an order to
execute the tasks in. We have some constraints: dependencies must be executed
before their dependents. But beyond that there’s a large space of options. We
want Dask to choose an ordering that maximizes parallelism while minimizing
the footprint necessary to run a computation.

At a high level, Dask has a policy that works towards *small goals* with *big steps*.

1. **Small goals**: prefer tasks that have few total dependents and whose final
   dependents have few total dependencies.

   We prefer to prioritize those tasks that help branches of computation that
   can terminate quickly.

   With more detail, we compute the total number of dependencies that each
   task depends on (both its own dependencies, the dependencies of its
   dependencies, and so on), and then we choose those tasks that drive towards
   results with a low number of total dependencies.  We choose to prioritize
   tasks that work towards finishing shorter computations first.
2. **Big steps**: prefer tasks with many dependents

   However, many tasks work towards the same final dependents.  Among those,
   we choose those tasks with the most work left to do.  We want to finish
   the larger portions of a sub-computation before we start on the smaller
   ones.

This is done with `dask.order.order()`. A more technical discussion is available
in [Scheduling in Depth](scheduling-policy.md#scheduling-policy). [https://distributed.dask.org/en/latest/scheduling-policies.html](https://distributed.dask.org/en/latest/scheduling-policies.html)
also discusses scheduling with a focus on the distributed scheduler, which includes
additional choices beyond the static ordering documented here.

## Debugging

Most of the time Dask’s ordering does well. But this is a genuinely hard problem
and there might be cases where you observe unexpectedly high memory usage or
communication, which may be a result of poor ordering. This section describes
how you would identify an ordering problem, and some steps you can take to
mitigate the problem.

Consider a computation that loads several chains of data from disk independently,
stacks pieces of them together, and does some reduction:

```python
>>> # create data on disk
>>> import dask.array as da
>>> x = da.zeros((12500, 10000), chunks=('10MB', -1))
>>> da.to_zarr(x, 'saved_x1.zarr', overwrite=True)
>>> da.to_zarr(x, 'saved_y1.zarr', overwrite=True)
>>> da.to_zarr(x, 'saved_x2.zarr', overwrite=True)
>>> da.to_zarr(x, 'saved_y2.zarr', overwrite=True)
```

We can load the data

```python
>>> # load the data.
>>> x1 = da.from_zarr('saved_x1.zarr')
>>> y1 = da.from_zarr('saved_x2.zarr')
>>> x2 = da.from_zarr('saved_y1.zarr')
>>> y2 = da.from_zarr('saved_y2.zarr')
```

And do some computation on it

```python
>>> def evaluate(x1, y1, x2, y2):
...     u = da.stack([x1, y1])
...     v = da.stack([x2, y2])
...     components = [u, v, u ** 2 + v ** 2]
...     return [
...         abs(c[0] - c[1]).mean(axis=-1)
...         for c in components
...     ]
>>> results = evaluate(x1, y1, x2, y2)
```

You can use [`dask.visualize()`](api.md#dask.visualize) with `color="order"` to visualize a
task graph with the static ordering included as node labels. As usual with
`dask.visualize`, you may need to trim down the problem to a smaller size,
so we’ll slice off a subset of the data. Make sure to include `optimize_graph=True`
to get a true representation of what order the tasks will be executed in.

```python
>>> import dask
>>> n = 125 * 4
>>> dask.visualize(evaluate(x1[:n], y1[:n], x2[:n], y2[:n]),
...                optimize_graph=True, color="order",
...                cmap="autumn", node_attr={"penwidth": "4"})
```

![Complex task graph of several vertical node chains at the output, and a few input sub-trees. In between these sections, there is a many-to-many area of crossing dependency arrows. The color coding of the output trees is interleaved without a clear progression.](images/order-failure.png)

In this visualization the nodes are colored by order of execution (from dark red
to light yellow) and the node labels are the order Dask’s assigned to each task.

It’s a bit hard to see, but there are actually four mostly independent “towers”
of execution here. We start at the middle-right array (label 1, bottom), move
up to the right (label 8, top-right) and then jump to a completely different
array (label 11, bottom-left). However, computing the first tower (downstream
of label 8, top-right) required loading some data from our second input array
(label 5, bottom-right). We’d much prefer to finish tasks downstream of it.

When Dask is executing that task graph, you might observe high memory usage.
The poor static ordering means we fail to complete tasks that would let us
release pieces of data. We load more pieces into memory at once, leading to
higher memory usage.

This specific ordering failure (which may be fixed) comes from the shared
dependencies (the boxes at the bottom of each task, which represent the input
Zarr arrays) at the bottom of each computation chain. We can inline those and
see the effect of ordering:

```python
>>> # load and profile data
>>> x1 = da.from_zarr('saved_x1.zarr', inline_array=True)
>>> y1 = da.from_zarr('saved_x2.zarr', inline_array=True)
>>> x2 = da.from_zarr('saved_y1.zarr', inline_array=True)
>>> y2 = da.from_zarr('saved_y2.zarr', inline_array=True)

>>> import dask
>>> n = 125 * 4
>>> dask.visualize(evaluate(x1[:n], y1[:n], x2[:n], y2[:n]),
...                optimize_graph=True, color="order",
...                cmap="autumn", node_attr={"penwidth": "4"})
```

![Complex task graph of several vertical node chains at the output, and a similar number of input blocks. The outputs and inputs are linked by simple nodes of a few inputs each, laid out without significant crossover between sections of the tree. The color coding of the output chains shows clear progression in the order of execution with each output color having a corresponding input of the same color.](images/order-success.png)

At a glance, we can see that this ordering is looks much more regular and
uniform. There’s fewer lines crossing, and the color of the ordering moves
smoothly from bottom to top, left to right. This shows that Dask is completing
one chain of computation before moving onto the next.

The lesson here is *not* “always use `inline_array=True`”. While the static
ordering looks better, there are other [Stages of Computation](phases-of-computation.md#phases-of-computation) to consider.
Whether the actual performance is better will depend on more factors than we
can consider here. See [`dask.array.from_array()`](generated/dask.array.from_array.md#dask.array.from_array) for more.

Instead, the lessons to take away here are:

1. What symptoms might lead you to diagnose Dask’s ordering as a problem (e.g.
   high memory usage)
2. How to generate and read task graphs with Dask’s ordering information
   included.


# phases-of-computation.html.md

<a id="phases-of-computation"></a>

# Stages of Computation

This page describes all of the parts of computation, some common causes of
slowness, and how to effectively profile.  This is intended for more advanced
users who are encountering slowdowns on larger computations.

## Graph Construction

Operations on Dask collections (array, dataframe, bag, delayed) build task
graphs.  These are dictionaries of Python functions that include an entry
every time some function needs to run on some chunk of data.  When these
dictionaries become large (millions of tasks) the overhead of constructing them
can become considerable.  Additionally the code that builds the graphs may
itself be inefficient.

Fortunately, this computation is all happening in normal Python right on your
own computer, so you can profile it just as you would any other Python code on
your computer using tools like the cProfile module, or the `%prun` or
`%snakeviz` IPython magics.

Assuming that no obvious cause comes up when profiling, a common solution to
this problem is to reduce your graph size by increasing your chunk size if
possible, or manually batching many operations into fewer functions.

## Graph Optimization

Just before you submit the graph to be executed, Dask sees if it can clean up
the graph a bit.  This helps to remove unnecessary work, and sometimes swaps
out more efficient operations.  As before though, if your graph is very large
(millions of tasks) then this can take some time.

Also as before, this is all happening in Python on your local machine.  You can
profile optimization separately from computation with the `dask.optimize`
function.

```python
# x, y = dask.compute(x, y)
x, y = dask.optimize(x, y)
```

It’s rare for people to change optimization.  It is rarely the main cause of
slowdown.

## Graph serialization

When you are using the distributed scheduler the graph must be sent to the
scheduler process, and from there to the workers.  To send this data it must
first be converted into bytes.  This serialization process can sometimes be
expensive either if the objects that you’re passing around are very complex, or
if they are very large.

The easiest way to profile this is to profile the `persist` call with the
distributed scheduler.  This will include both the optimization phase above, as
well as the serialization and some of the communication phase below
(serialization is often the largest component).  Fortunately `persist`
returns immediately after, not waiting for the computation to actually finish.

Most often the cause of long serialization times is placing large objects
like NumPy arrays or Pandas dataframes into your graph repeatedly.  Dask will
usually raise a warning when it notices this.  Often the best solution is to
read your data in as a task instead of including it directly, pre-scatter large
data, or wrap them in `dask.delayed`.  Sometimes serialization is caused by
other issues with complex objects.  These tend to be very library specific, and
so it is hard to provide general guidelines for them.

## Graph Communication

The graph must then be communicated to the scheduler.  You can watch the
`/system` tab of the dashboard to watch network communication to and from the
scheduler.  There is no good way to profile this.

## Scheduling

The scheduler now receives the graph, and must populate its internal data
structures to be able to efficiently schedule these tasks to the various
workers.

It is only after these data structures are populated that the dashboard will
show any activity.  All time between pressing `compute/persist` and seeing
activity is taken up in the stages above.

You can profile scheduling costs with the `/profile-server` page of the
dashboard.  However, this is rarely useful for users because, unless you’re
willing to dive into the scheduling code, it is hard to act here.  Still, the
interested user may find the profile of the inter workings of the scheduler of
interest.

If scheduling is expensive then the best you can do is to reduce your graph
size, often by increasing chunk size.

## Execution

Finally your workers get sent some tasks and get to run them.  Your code runs
on a thread of a worker, and does whatever it was told to do.

Dask’s Dashboard is a good tool to profile and investigate performance here,
particularly the `/status` and `/profile` pages.

Accelerating this phase is often up to the author of the tasks that you are
submitting.  This might be you if you are using custom code, or the NumPy or
Pandas developers.  We encourage you to consider efficient libraries like
Cython, Numba, or any other solution that is commonly used to accelerate Python
code.


# presentations.html.md

# Talks & Tutorials

<iframe width="560"
        height="315"
        src="https://www.youtube.com/embed/nnndxbr_Xq4"
        style="margin: 0 auto 20px auto; display: block;"
        frameborder="0"
        allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture"
        allowfullscreen></iframe>

## Dask Tutorial

[Dask Tutorial](https://tutorial.dask.org) provides an overview of Dask and is typically delivered in 3 hours.
See [Parallel and Distributed Computing in Python with Dask](https://www.youtube.com/watch?v=EybGGLbLipI) for the
latest Dask Tutorial recording from SciPy 2020.

## Dask YouTube channel

You can find lots of videos about Dask on the [Dask YouTube channel](https://www.youtube.com/c/dask-dev)


# prometheus.html.md

<!-- When modifying the contents of this page, please adjust the corresponding page in the dask.distributed documentation accordingly. -->

# Prometheus

[Prometheus](https://prometheus.io) is a widely popular tool for monitoring and alerting a wide variety of
systems. A distributed cluster offers a number of Prometheus metrics if the
[prometheus_client](https://github.com/prometheus/client_python) package is installed. The metrics are exposed in Prometheus’
text-based format at the `/metrics` endpoint on both schedulers and workers.

## Available metrics

Apart from the metrics exposed per default by the [prometheus_client](https://github.com/prometheus/client_python), schedulers and
workers expose a number of Dask-specific metrics.
See the [dask.distributed documentation](https://distributed.dask.org/en/latest/prometheus.html) for details.


# scheduler-overview.html.md

# Scheduler Overview

After we create a dask graph, we use a scheduler to run it. Dask currently
implements a few different schedulers:

- `dask.threaded.get`: a scheduler backed by a thread pool
- `dask.multiprocessing.get`: a scheduler backed by a process pool
- `dask.get`: a synchronous scheduler, good for debugging
- `distributed.Client.get`: a distributed scheduler for executing graphs
  on multiple machines.  This lives in the external [distributed](https://distributed.dask.org/en/latest/) project.

## The `get` function

The entry point for all schedulers is a `get` function. This takes a dask
graph, and a key or list of keys to compute:

```python
>>> from operator import add

>>> dsk = {'a': 1,
...        'b': 2,
...        'c': (add, 'a', 'b'),
...        'd': (sum, ['a', 'b', 'c'])}

>>> get(dsk, 'c')
3

>>> get(dsk, 'd')
6

>>> get(dsk, ['a', 'b', 'c'])
[1, 2, 3]
```

## Using `compute` methods

When working with dask collections, you will rarely need to
interact with scheduler `get` functions directly. Each collection has a
default scheduler, and a built-in `compute` method that calculates the output
of the collection:

```python
>>> import dask.array as da
>>> x = da.arange(100, chunks=10)
>>> x.sum().compute()
4950
```

The compute method takes a number of keywords:

- `scheduler`: the name of the desired scheduler as a string (`"threads"`, `"processes"`, `"single-threaded"`, etc.), a `get` function, or a `dask.distributed.Client` object.  Overrides the default for the collection.
- `**kwargs`: extra keywords to pass on to the scheduler `get` function.

See also: [Configuring the schedulers](#configuring-schedulers).

## The `compute` function

You may wish to compute results from multiple dask collections at once.
Similar to the `compute` method on each collection, there is a general
`compute` function that takes multiple collections and returns multiple
results. This merges the graphs from each collection, so intermediate results
are shared:

```python
>>> y = (x + 1).sum()
>>> z = (x + 1).mean()
>>> da.compute(y, z)    # Compute y and z, sharing intermediate results
(5050, 50.5)
```

Here the `x + 1` intermediate was only computed once, while calling
`y.compute()` and `z.compute()` would compute it twice. For large graphs
that share many intermediates, this can be a big performance gain.

The `compute` function works with any dask collection, and is found in
`dask.base`. For convenience it has also been imported into the top level
namespace of each collection.

```python
>>> from dask.base import compute
>>> compute is da.compute
True
```

<a id="configuring-schedulers"></a>

## Configuring the schedulers

The dask collections each have a default scheduler:

- `dask.array` and `dask.dataframe` use the threaded scheduler by default
- `dask.bag` uses the multiprocessing scheduler by default.

For most cases, the default settings are good choices. However, sometimes you
may want to use a different scheduler. There are two ways to do this.

1. Using the `scheduler` keyword in the `compute` method:
   > ```python
   > >>> x.sum().compute(scheduler='processes')
   > ```
2. Using `dask.config.set`. This can be used either as a context manager, or to
   set the scheduler globally:
   > ```python
   > # As a context manager
   > >>> with dask.config.set(scheduler='processes'):
   > ...     x.sum().compute()

   > # Set globally
   > >>> dask.config.set(scheduler='processes')
   > >>> x.sum().compute()
   > ```

Additionally, each scheduler may take a few extra keywords specific to that
scheduler. For example, the multiprocessing and threaded schedulers each take a
`num_workers` keyword, which sets the number of processes or threads to use
(defaults to number of cores). This can be set by passing the keyword when
calling `compute`:

```python
# Compute with 4 threads
>>> x.compute(num_workers=4)
```

Alternatively, the multiprocessing and threaded schedulers will check for a
global pool set with `dask.config.set`:

```python
>>> from concurrent.futures import ThreadPoolExecutor
>>> with dask.config.set(pool=ThreadPoolExecutor(4)):
...     x.compute()
```

The multiprocessing scheduler also supports [different contexts](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) (“spawn”,
“forkserver”, “fork”) which you can set with `dask.config.set`. The default
context is “spawn”, but you can set a different one:

```python
>>> with dask.config.set({"multiprocessing.context": "forkserver"}):
...     x.compute()
```

For more information on the individual options for each scheduler, see the
docstrings for each scheduler `get` function.

## Debugging the schedulers

Debugging parallel code can be difficult, as conventional tools such as `pdb`
don’t work well with multiple threads or processes. To get around this when
debugging, we recommend using the synchronous scheduler found at
`dask.get`. This runs everything serially, allowing it to work
well with `pdb`:

```python
>>> dask.config.set(scheduler='single-threaded')
>>> x.sum().compute()    # This computation runs serially instead of in parallel
```

The shared memory schedulers also provide a set of callbacks that can be used
for diagnosing and profiling. You can learn more about scheduler callbacks and
diagnostics [here](diagnostics-local.md).

## More Information

- See [Shared Memory](shared.md) for information on the design of the shared memory
  (threaded or multiprocessing) schedulers
- See [distributed](https://distributed.dask.org/en/latest/) for information on the distributed memory scheduler


# scheduling-policy.html.md

<a id="scheduling-policy"></a>

# Scheduling in Depth

*Note: this technical document is not optimized for user readability.*

The default shared memory scheduler used by most dask collections lives in
`dask/local.py`. This scheduler dynamically schedules tasks to new
workers as they become available.  It operates in a shared memory environment
without consideration to data locality, all workers have access to all data
equally.

We find that our workloads are best served by trying to minimize the memory
footprint.  This document talks about our policies to accomplish this in our
scheduling budget of one millisecond per task, irrespective of the number of
tasks.

Generally we are faced with the following situation:  A worker arrives with a
newly completed task.  We update our data structures of execution state and
have to provide a new task for that worker.  In general there are very many
available tasks, which should we give to the worker?

*Q: Which of our available tasks should we give to the newly ready worker?*

This question is simple and local and yet strongly impacts the performance of
our algorithm.  We want to choose a task that lets us free memory now and in
the future.  We need a clever and cheap way to break a tie between the set of
available tasks.

At this stage we choose the policy of “last in, first out.”  That is we choose
the task that was most recently made available, quite possibly by the worker
that just returned to us.  This encourages the general theme of finishing
things before starting new things.

We implement this with a stack.  When a worker arrives with its finished task
we figure out what new tasks we can now compute with the new data and put those
on top of the stack if any exist.  We pop an item off of the top of the stack
and deliver that to the waiting worker.

And yet if the newly completed task makes ready multiple newly ready tasks in
which order should we place them on the stack?  This is yet another opportunity
for a tie breaker.  This is particularly important at *the beginning* of
execution where we typically add a large number of leaf tasks onto the stack.
Our choice in this tie breaker also strongly affects performance in many cases.

We want to encourage depth first behavior where, if our computation is composed
of something like many trees we want to fully explore one subtree before moving
on to the next.  This encourages our workers to complete blocks/subtrees of our
graph before moving on to new blocks/subtrees.

And so to encourage this “depth first behavior” we do a depth first search and
number all nodes according to their number in the depth first search (DFS)
traversal.  We use this number to break ties when adding tasks on to the stack.
Please note that while we spoke of optimizing the many-distinct-subtree case
above this choice is entirely local and applies quite generally beyond this
case.  Anything that behaves even remotely like the many-distinct-subtree case
will benefit accordingly, and this case is quite common in normal workloads.

And yet we have glossed over another tie breaker. Performing the depth
first search, when we arrive at a node with many children we can choose the
order in which to traverse the children.  We resolve this tie breaker by
selecting those children whose result is depended upon by the most nodes.  This
dependence can be either direct for those nodes that take that data as input or
indirect for any ancestor node in the graph.  This emphasizing traversing first
those nodes that are parts of critical paths having long vertical chains that
rest on top of this node’s result, and nodes whose data is depended upon by
many nodes in the future.  We choose to dive down into these subtrees first in
our depth first search so that future computations don’t get stuck waiting for
them to complete.

And so we have three tie breakers

1. Q:  Which of these available tasks should I run?

   A:  Last in, first out
2. Q:  Which of these tasks should I put on the stack first?

   A:  Do a depth first search before the computation, use that ordering.
3. Q:  When performing the depth first search how should I choose between
   children?

   A:  Choose those children on whom the most data depends

We have found common workflow types that require each of these decisions.  We
have not yet run into a commonly occurring graph type in data analysis that is
not well handled by these heuristics for the purposes of minimizing memory use.


# scheduling.html.md

# Scheduling

All of the large-scale Dask collections like
[Dask Array](array.md), [Dask DataFrame](dataframe.md), and [Dask Bag](bag.md)
and the fine-grained APIs like [delayed](delayed.md) and [futures](futures.md)
generate task graphs where each node in the graph is a normal Python function
and edges between nodes are normal Python objects
that are created by one task as outputs and used as inputs in another task.
After Dask generates these task graphs, it needs to execute them on parallel hardware.
This is the job of a *task scheduler*.
Different task schedulers exist, and each will consume a task graph and compute the
same result, but with different performance characteristics.

Dask has two families of task schedulers:

1. **Single-machine scheduler**: This scheduler provides basic features on a
   local process or thread pool.  This scheduler was made first and is the
   default.  It is simple and cheap to use, although it can only be used on
   a single machine and does not scale
2. **Distributed scheduler**: This scheduler is more sophisticated, offers
   more features, but also requires a bit more effort to set up.  It can
   run locally or distributed across a cluster

<br/>
![Dask is composed of three parts. "Collections" create "Task Graphs" which are then sent to the "Scheduler" for execution. There are two types of schedulers that are described in more detail below.](images/dask-overview-schedulers.svg)
<br/>

For different computations you may find better performance with particular scheduler settings.
This document helps you understand how to choose between and configure different schedulers,
and provides guidelines on when one might be more appropriate.

<a id="threaded-scheduler"></a>

## Local Threads

```python
import dask
dask.config.set(scheduler='threads')  # overwrite default with threaded scheduler
```

The threaded scheduler executes computations with a local
`concurrent.futures.ThreadPoolExecutor`.
It is lightweight and requires no setup.
It introduces very little task overhead (around 50us per task)
and, because everything occurs in the same process,
it incurs no costs to transfer data between tasks.
However, due to Python’s Global Interpreter Lock (GIL),
this scheduler only provides parallelism when your computation is dominated by non-Python code,
as is primarily the case when operating on numeric data in NumPy arrays, Pandas DataFrames,
or using any of the other C/C++/Cython based projects in the ecosystem.

The threaded scheduler is the default choice for
[Dask Array](array.md), [Dask DataFrame](dataframe.md), and [Dask Delayed](delayed.md).
However, if your computation is dominated by processing pure Python objects
like strings, dicts, or lists,
then you may want to try one of the process-based schedulers below
(we currently recommend the distributed scheduler on a local machine).

## Local Processes

#### NOTE
The [distributed scheduler](#local-distributed) described below is often a better choice today.
We encourage readers to continue reading after this section.

```python
import dask
dask.config.set(scheduler='processes')  # overwrite default with multiprocessing scheduler
```

The multiprocessing scheduler executes computations with a local
`concurrent.futures.ProcessPoolExecutor`.
It is lightweight to use and requires no setup.
Every task and all of its dependencies are shipped to a local process,
executed, and then their result is shipped back to the main process.
This means that it is able to bypass issues with the GIL and provide parallelism
even on computations that are dominated by pure Python code,
such as those that process strings, dicts, and lists.

However, moving data to remote processes and back can introduce performance penalties,
particularly when the data being transferred between processes is large.
The multiprocessing scheduler is an excellent choice when workflows are relatively linear,
and so does not involve significant inter-task data transfer
as well as when inputs and outputs are both small, like filenames and counts.

This is common in basic data ingestion workloads,
such as those are common in [Dask Bag](bag.md),
where the multiprocessing scheduler is the default:

```python
>>> import dask.bag as db
>>> db.read_text('*.json').map(json.loads).pluck('name').frequencies().compute()
{'alice': 100, 'bob': 200, 'charlie': 300}
```

For more complex workloads,
where large intermediate results may be depended upon by multiple downstream tasks,
we generally recommend the use of the distributed scheduler on a local machine.
The distributed scheduler is more intelligent about moving around large intermediate results.

<a id="single-threaded-scheduler"></a>

## Single Thread

```python
import dask
dask.config.set(scheduler='synchronous')  # overwrite default with single-threaded scheduler
```

The single-threaded synchronous scheduler executes all computations in the local thread
with no parallelism at all.
This is particularly valuable for debugging and profiling,
which are more difficult when using threads or processes.

For example, when using IPython or Jupyter notebooks, the `%debug`, `%pdb`, or `%prun` magics
will not work well when using the parallel Dask schedulers
(they were not designed to be used in a parallel computing context).
However, if you run into an exception and want to step into the debugger,
you may wish to rerun your computation under the single-threaded scheduler
where these tools will function properly.

<a id="local-distributed"></a>

## Dask Distributed (local)

```python
from dask.distributed import Client
client = Client()
# or
client = Client(processes=False)
```

The Dask distributed scheduler can either be [setup on a cluster](deploying.md)
or run locally on a personal machine.  Despite having the name “distributed”,
it is often pragmatic on local machines for a few reasons:

1. It provides access to asynchronous API, notably [Futures](futures.md)
2. It provides a diagnostic dashboard that can provide valuable insight on
   performance and progress
3. It handles data locality with more sophistication, and so can be more
   efficient than the multiprocessing scheduler on workloads that require
   multiple processes

You can read more about using the Dask distributed scheduler on a single machine in
[these docs](deploying.md).

## Dask Distributed (Cluster)

You can also run Dask on a distributed cluster.
There are a variety of ways to set this up depending on your cluster.
We recommend referring to [how to deploy Dask clusters](deploying.md) for more information.

<a id="scheduling-configuration"></a>

## Configuration

You can configure the global default scheduler by using the `dask.config.set(scheduler...)` command.
This can be done globally:

```python
dask.config.set(scheduler='threads')

x.compute()
```

or as a context manager:

```python
with dask.config.set(scheduler='threads'):
    x.compute()
```

or within a single compute call:

```python
x.compute(scheduler='threads')
```

Each scheduler may support extra keywords specific to that scheduler. For example,
the pool-based single-machine scheduler allows you to provide custom pools or
specify the desired number of workers:

```python
from concurrent.futures import ThreadPoolExecutor
with dask.config.set(pool=ThreadPoolExecutor(4)):
    x.compute()

with dask.config.set(num_workers=4):
    x.compute()
```

Note that Dask also supports custom `concurrent.futures.Executor` subclasses,
such as the `ReusablePoolExecutor` from [loky](https://github.com/joblib/loky):

```python
from loky import get_reusable_executor
with dask.config.set(scheduler=get_reusable_executor()):
    x.compute()
```

Other libraries like [ipyparallel](https://ipyparallel.readthedocs.io/en/latest/examples/Futures.html#Executors) and [mpi4py](https://mpi4py.readthedocs.io/en/latest/mpi4py.futures.html) also supply
`concurrent.futures.Executor` subclasses that could be used as well.

## Standalone Python scripts

Some care needs to be taken when running Dask schedulers in a standalone Python script.
Specifically, when using the single-machine multiprocessing scheduler or the local distributed
scheduler, Dask will create additional Python processes. As part of Python’s normal subprocess
initialization, Python will import the contents of the script in every child process that is created
(this is true for any Python code where child processes are created – not just in Dask).
This import initialization can lead to subprocesses recursively creating other subprocesses
and eventually an error is raised.

### Common error encountered

```python
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:

   if __name__ == '__main__':
         freeze_support()
         ...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
```

To avoid this types of error, you should place any Dask code that create subprocesses
(for example, all `compute()` calls that use the multiprocessing scheduler, or when creating
a local distributed cluster) inside a `if __name__ == "__main__":` block. This ensures
subprocesses are only created when your script is run as the main program.

For example, running `python myscript.py` with the script below will raise an error:

```python
# myscript.py

from dask.distributed import Client
client = Client()  # Will raise an error when creating local subprocesses
```

Instead one should place the contents of the script inside a `if __name__ == "__main__":` block:

```python
# myscript.py

if __name__ == "__main__":  # This avoids infinite subprocess creation

   from dask.distributed import Client
   client = Client()
```

For more details on this topic see
[Python’s multiprocessing guidelines](https://docs.python.org/3/library/multiprocessing.html#programming-guidelines).


# selecting-the-collection-backend.html.md

# Selecting the collection backend

**Warning**: Backend-library dispatching at the collection level is still an experimental feature. Both the `DaskBackendEntrypoint` API and the set of “dispatchable” functions are expected to change.

## Changing the default backend library

The Dask-Dataframe and Dask-Array modules were originally designed with the Pandas and Numpy backend libraries in mind, respectively. However, other dataframe and array libraries can take advantage of the same collection APIs for out-of-core and parallel processing. For example, users with [cupy](https://cupy.dev/) installed can change their default Dask-Array backend to `cupy` with the `"array.backend"` configuration option:

```python
>>> import dask
>>> import dask.array as da
>>> with dask.config.set({"array.backend": "cupy"}):
...     darr = da.ones(10, chunks=(5,))  # Get cupy-backed collection
```

This code opts out of the default (`"numpy"`) backend for dispatchable Dask-Array creation functions, and uses the creation functions registered for `"cupy"` instead. The current set of dispatchable creation functions for Dask-Array is:

- `ones`
- `zeros`
- `empty`
- `full`
- `arange`

The Dask-Array API can also dispatch the backend `RandomState` class to be used for random-number generation. This means all creation functions in `dask.array.random` are also dispatchable.

The current set of dispatchable creation functions for Dask-Dataframe is:

- `from_dict`
- `read_parquet`
- `read_json`
- `read_orc`
- `read_csv`
- `read_hdf`

As the backend-library dispatching system becomes more mature, this set of dispatchable creation functions is likely to grow.

For an existing collection, the underlying data can be forcibly moved to a desired backend using the `to_backend` method:

```python
>>> import dask
>>> import dask.array as da
>>> darr = da.ones(10, chunks=(5,))  # Creates numpy-backed collection
>>> with dask.config.set({"array.backend": "cupy"}):
...     darr = darr.to_backend()  # Moves numpy data to cupy
```

## Defining a new collection backend

**Warning**: Defining a custom backend is **not** yet recommended for most users and down-stream libraries. The backend-entrypoint system should still be treated as experimental.

Dask currently exposes an [entrypoint](https://packaging.python.org/specifications/entry-points/) under the group `dask.array.backends` and `dask.dataframe.backends` to enable users and third-party libraries to develop and maintain backend implementations for Dask-Array and Dask-Dataframe. A custom Dask-Array backend should define a subclass of `DaskArrayBackendEntrypoint` (defined in `dask.array.backends`), while a custom Dask-DataFrame backend should define a subclass of `DataFrameBackendEntrypoint` (defined in `dask.dataframe.backends`).

For example, a cudf-based backend definition for Dask-Dataframe would look something like the CudfBackendEntrypoint definition below:

```python
from dask.dataframe.backends import DataFrameBackendEntrypoint
from dask.dataframe.dispatch import (
   ...
   make_meta_dispatch,
   ...
)
...

def make_meta_cudf(x, index=None):
   return x.head(0)
...

class CudfBackendEntrypoint(DataFrameBackendEntrypoint):

   def __init__(self):
      # Register compute-based dispatch functions
      # (e.g. make_meta_dispatch, sizeof_dispatch)
      ...
      make_meta_dispatch.register(
         (cudf.Series, cudf.DataFrame),
         func=make_meta_cudf,
      )
      # NOTE: Registration may also be outside __init__
      # if it is in the same module as this class
   ...

   @staticmethod
   def read_orc(*args, **kwargs):
      from .io import read_orc

      # Use dask_cudf version of read_orc
      return read_orc(*args, **kwargs)
   ...
```

In order to support pandas-to-cudf conversion with `DataFrame.to_backend`, this class also needs to implement the proper `to_backend` and `to_backend_dispatch` methods.

To expose this class as a `dask.dataframe.backends` entrypoint, the necessary `setup.cfg` configuration in `cudf` (or `dask_cudf`) would be as follows:

```ini
[options.entry_points]
dask.dataframe.backends =
   cudf = <module-path>:CudfBackendEntrypoint
```

### Compute dispatch

#### NOTE
The primary dispatching mechanism for array-like compute operations in both Dask-Array and Dask-DataFrame is the `__array_function__` protocol defined in [NEP-18](https://numpy.org/neps/nep-0018-array-function-protocol.html). For a custom collection backend to be functional, this protocol **must** cover many common numpy functions for the desired array backend. For example, the `cudf` backend for Dask-DataFrame depends on the `__array_function__` protocol being defined for both `cudf` and its complementary array backend (`cupy`). The compute-based dispatch functions discussed in this section correspond to functionality that is not already captured by NEP-18.

Notice that the `CudfBackendEntrypoint` definition must define a distinct method definition for each dispatchable creation routine, and register all non-creation (compute-based) dispatch functions within the `__init__` logic. These compute dispatch functions do not operate at the collection-API level, but at computation time (within a task). The list of all current “compute” dispatch functions are listed below.

Dask-Array compute-based dispatch functions (as defined in `dask.array.dispatch`, and defined for Numpy in `dask.array.backends`):

> - concatenate_lookup
> - divide_lookup
> - einsum_lookup
> - empty_lookup
> - nannumel_lookup
> - numel_lookup
> - percentile_lookup
> - tensordot_lookup
> - take_lookup

Dask-Dataframe compute-based dispatch functions (as defined in `dask.dataframe.dispatch`, and defined for Pandas in `dask.dataframe.backends`):

> - categorical_dtype_dispatch
> - concat_dispatch
> - get_collection_type
> - group_split_dispatch
> - grouper_dispatch
> - hash_object_dispatch
> - is_categorical_dtype_dispatch
> - make_meta_dispatch
> - make_meta_obj
> - meta_lib_from_array
> - meta_nonempty
> - pyarrow_schema_dispatch
> - tolist_dispatch
> - union_categoricals_dispatch

Note that the compute-based dispatching system is subject to change. Implementing a complete backend is still expected to require significant effort. However, the long-term goal is to bring further simplicity to this process.


# shared.html.md

# Shared Memory

The asynchronous scheduler accepts any `concurrent.futures.Executor`
instance. This includes instances of the `ThreadPoolExecutor` and
`ProcessPoolExecutor` defined in the Python standard library as well as any
other subclass from a third party library. Dask also defines its own
`SynchronousExecutor` for that simply runs functions on the main thread
(useful for debugging).

Full dask `get` functions exist in each of `dask.threaded.get`,
`dask.multiprocessing.get` and `dask.get` respectively.

## Policy

The asynchronous scheduler maintains indexed data structures that show which
tasks depend on which data, what data is available, and what data is waiting on
what tasks to complete before it can be released, and what tasks are currently
running.  It can update these in constant time relative to the number of total
and available tasks.  These indexed structures make the dask async scheduler
scalable to very many tasks on a single machine.

![Embarrassingly parallel dask flow](images/async-embarrassing.gif)

To keep the memory footprint small, we choose to keep ready-to-run tasks in a
last-in-first-out stack such that the most recently made available tasks get
priority.  This encourages the completion of chains of related tasks before new
chains are started.  This can also be queried in constant time.

## Performance

**tl;dr** The threaded scheduler overhead behaves roughly as follows:

* 200us overhead per task
* 10us startup time (if you wish to make a new ThreadPoolExecutor each time)
* Constant scaling with number of tasks
* Linear scaling with number of dependencies per task

Schedulers introduce overhead.  This overhead effectively limits the
granularity of our parallelism.  Below we measure overhead of the async
scheduler with different apply functions (threaded, sync, multiprocessing), and
under different kinds of load (embarrassingly parallel, dense communication).

The quickest/simplest test we can do it to use IPython’s `timeit` magic:

```ipython
In [1]: import dask.array as da

In [2]: x = da.ones(1000, chunks=(2,)).sum()

In [3]: len(x.dask)
Out[3]: 1168

In [4]: %timeit x.compute()
185 ms +- 14.2 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)
```

So this takes ~90 microseconds per task.  About 100ms of this is from overhead:

```ipython
In [5]: x = da.ones(1000, chunks=(1000,)).sum()

In [6]: %timeit x.compute()
2.38 ms +- 14.8 us per loop (mean +- std. dev. of 7 runs, 100 loops each)
```

There is some overhead from spinning up a ThreadPoolExecutor each time.
This may be mediated by using a global or contextual pool:

```python
>>> from concurrent.futures import ThreadPoolExecutor
>>> pool = ThreadPoolExecutor()
>>> dask.config.set(pool=pool)  # set global ThreadPoolExecutor

or

>>> with dask.config.set(pool=pool)  # use ThreadPoolExecutor throughout with block
...     ...
```

We now measure scaling the number of tasks and scaling the density of the
graph:

![Adding nodes](images/trivial.svg)

### Linear scaling with number of tasks

As we increase the number of tasks in a graph, we see that the scheduling
overhead grows linearly.  The asymptotic cost per task depends on the scheduler.
The schedulers that depend on some sort of asynchronous pool have costs of a few
milliseconds and the single threaded schedulers have costs of a few microseconds.

![Graph depicting how well Dask scales with the number of nodes in the task graph. Graph shows the duration in seconds on the y-axis versus number of edges per task on the x-axis. The time to schedule the entire graph is constant initially, followed by a linear increase after roughly 500 tasks for multiprocessing and threaded schedulers and 10 tasks for async and core schedulers. The inverse is true for the cost per task, with a linear cost decrease, followed by more or less constant cost.](images/scaling-nodes.png)![Adding edges](images/crosstalk.svg)

### Linear scaling with number of edges

As we increase the number of edges per task, the scheduling overhead
again increases linearly.

Note: Neither the naive core scheduler nor the multiprocessing scheduler
are good at workflows with non-trivial cross-task
communication; they have been removed from the plot.

![Graph depicting how well Dask scales with the number of edges in the task graph. Graph shows the duration in seconds on the y-axis versus number of edges per task on the x-axis. As the number of edges increases from 0 to 100, the time to schedule the entire graph using the threaded scheduler goes from 2 to 8 seconds whereas using the async scheduler goes from 0 to 3 seconds. The cost per edge decreases up until about 10 edges, after which the cost plateaus for both the threaded and async schedulers, with the async scheduler being consistently faster.](images/scaling-edges.png)

[Download scheduling script](https://github.com/dask/dask/tree/main/docs/source/scripts/scheduling.py)

## Known Limitations

The shared memory scheduler has some notable limitations:

1. It works on a single machine
2. The threaded scheduler is limited by the GIL on Python code, so if your
   operations are pure python functions, you should not expect a multi-core
   speedup
3. The multiprocessing scheduler must serialize functions between workers,
   which can fail
4. The multiprocessing scheduler must serialize data between workers and the
   central process, which can be expensive
5. The multiprocessing scheduler cannot transfer data directly between worker
   processes; all data routes through the main process.


# software-environments.html.md

# Manage Environments

It is critical that each of your dask workers uses the same set of
python packages and modules when executing your code, so that Dask
can function. Upon connecting with a distributed `Client`, Dask
will automatically check the versions of some critical packages
(including Dask itself) and warn you of any mismatch.

Most functions you will run on Dask will require imports. This
is even true for passing any object which is not a python builtin -
the pickle serialisation method will save references to imported modules
rather than trying to send all of your source code.

You therefore must ensure that workers have access to all of the modules
you will need, and ideally with exactly the same versions.

## Single-machine schedulers

If you are using the threaded scheduler, then you do not need to do
anything, since the workers are in the same process, and objects are
simply shared rather than serialised and deserialised.

Similarly, if you use the multiprocessing scheduler, new processes
will be copied from, or launched in the same way as the original process,
so you only need to make sure that you have not changed environment variables
related to starting up python and importing
code (such as PATH, PYTHONPATH, `sys.path`).

If you are using the distributed scheduler on a single machine, this is roughly
equivalent to using the multiprocessing scheduler, above, if you are launching
with `Client(...)` or `LocalCluster(...)`.

However, if you are launching your workers from the command line, then you must
ensure that you are running in the same environment (virtualenv, pipenv or conda).

The rest of this page concerns only distributed clusters.

## Maintain consistent environments

If you manage your environments yourself, then setting up module consistency
can be as simple as creating environments from the same pip or conda specification
on each machine. You should consult the documentation for `pip`, `pipenv`
and `conda`, whichever you normally use. You will normally want to be as specific
about package versions as possible, and distribute the same environment file to
workers before installation.

However, other common ways to distribute an environment directly, rather than build it
in-place, include:

- docker images, where the environment has been built into the image; this is the
  normal route when you are running on infrastructure enabled by docker, such as
  kubernetes
- [conda-pack](https://conda.github.io/conda-pack/) is a tool for bundling existing conda environments, so they can be
  relocated to other machines. This tool was specifically created for dask on YARN/hadoop
  clusters, but could be used elsewhere
- shared filesystem, e.g., NFS, that can be seen by all machines. Note that importing
  python modules is fairly IO intensive, so your server needs to be able to handle
  many requests
- cluster install method (e.g., [parcels](https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_parcels.html)): depending on your infrastructure, there may be
  ways to install specific binaries to all workers in a cluster.

## Temporary installations

The worker plugin [`distributed.diagnostics.plugin.PipInstall`](https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.PipInstall) allows you to
run pip installation commands on your workers, and optionally have them restart
upon success. Please read the plugin documentation to see how to use this.

## Objects in `__main__`

Objects that you create without any reference to modules, such as classes that
are defined right in the repl or notebook cells, are not pickled with reference to
any imported module. You can redefine such objects, and Dask will serialise them
completely, including the source code. This can be a good way to try new things
while working with a distributed setup.

## Send Source

Particularly during development, you may want to send files directly to workers
that are already running.

You should use `client.upload_file` in these cases.
For more detail, see the [API docs](https://distributed.readthedocs.io/en/latest/api.html#distributed.executor.Executor.upload_file) and a
StackOverflow question
[“Can I use functions imported from .py files in Dask/Distributed?”](http://stackoverflow.com/questions/39295200/can-i-use-functions-imported-from-py-files-in-dask-distributed)
This function supports both standalone file and setuptools’s `.egg` files
for larger modules.


# spark.html.md

# Comparison to Spark

[Apache Spark](https://spark.apache.org/) is a popular distributed computing
tool for tabular datasets that is growing to become a dominant name in Big Data
analysis today.  Dask has several elements that appear to intersect this space
and we are often asked, “How does Dask compare with Spark?”

Answering such comparison questions in an unbiased and informed way is hard,
particularly when the differences can be somewhat technical.  This document
tries to do this; we welcome any corrections.

## Summary

Generally Dask is smaller and lighter weight than Spark.  This means that it
has fewer features and, instead, is used in conjunction with other libraries,
particularly those in the numeric Python ecosystem.  It couples with libraries
like Pandas or Scikit-Learn to achieve high-level functionality.
Additionally, Dask is often faster and more robustly performant on [standard benchmarks](https://docs.coiled.io/blog/tpch.html?utm_source=dask-docs&utm_medium=spark-vs-dask#dask-vs-spark) than Spark.

### Language

- Spark is written in Scala with some support for Python and R.  It
  interoperates well with other JVM code.
- Dask is written in Python and only really supports Python.  It
  interoperates well with C/C++/Fortran/LLVM or other natively compiled
  code linked through Python.

### Ecosystem

- Spark is an all-in-one project that has inspired its own ecosystem.  It
  integrates well with many other Apache projects.
- Dask is a component of the larger Python ecosystem.  It couples with and
  enhances other libraries like NumPy, pandas, and Scikit-learn.

### Age and Trust

- Spark is older (since 2010) and has become a dominant and
  well-trusted tool in the Big Data enterprise world.
- Dask is younger (since 2014) and is an extension of the
  well trusted NumPy/pandas/Scikit-learn/Jupyter stack.

### Scope

- Spark is more focused on traditional business intelligence
  operations like SQL and lightweight machine learning.
- Dask is applied more generally both to business intelligence
  applications, as well as a number of scientific and custom situations.

### Internal Design

- Spark’s internal model is higher level, providing good high level
  optimizations on uniformly applied computations, but lacking flexibility
  for more complex algorithms or ad-hoc systems.  It is fundamentally an
  extension of the Map-Shuffle-Reduce paradigm.
- Dask’s internal model is lower level, and so lacks high level
  optimizations, but is able to implement more sophisticated algorithms and
  build more complex bespoke systems.  It is fundamentally based on generic
  task scheduling.

### Scale

- Spark scales from a single node to thousand-node clusters.
- Dask scales from a single node to thousand-node clusters.

### APIs

#### DataFrames

- Spark DataFrame has its own API and memory model.  It also
  implements a large subset of the SQL language.  Spark includes a
  high-level query optimizer for complex queries.
- Dask DataFrame reuses the Pandas API and memory model.  It implements
  neither SQL nor a query optimizer.  It is able to do random access,
  efficient time series operations, and other Pandas-style indexed
  operations.

#### Machine Learning

- Spark MLLib is a cohesive project with support for common operations
  that are easy to implement with Spark’s Map-Shuffle-Reduce style
  system.  People considering MLLib might also want to consider *other*
  JVM-based machine learning libraries like H2O, which may have better
  performance.
- Dask relies on and interoperates with existing libraries like
  Scikit-learn and XGBoost.  These can be more familiar or higher
  performance, but generally results in a less-cohesive whole.  See the
  [dask-ml](https://ml.dask.org) project for integrations.

#### Arrays

- Spark does not include support for multi-dimensional arrays natively
  (this would be challenging given their computation model), although
  some support for two-dimensional matrices may be found in MLLib.
  People may also want to look at the
  [Thunder](https://github.com/thunder-project/thunder) project, which
  combines Apache Spark with NumPy arrays.
- Dask fully supports the NumPy model for
  [scalable multi-dimensional arrays](array.md).

#### Streaming

- Spark’s support for streaming data is first-class and integrates well
  into their other APIs.  It follows a mini-batch approach.  This
  provides decent performance on large uniform streaming operations.
- Dask provides a [real-time futures interface](futures.md) that is
  lower-level than Spark streaming.  This enables more creative and
  complex use-cases, but requires more work than Spark streaming.

#### Graphs / complex networks

- Spark provides GraphX, a library for graph processing.
- Dask provides no such library.

#### Custom parallelism

- Spark generally expects users to compose computations out of their
  high-level primitives (map, reduce, groupby, join, …).  It is also
  possible to extend Spark through subclassing RDDs, although this is
  rarely done.
- Dask allows you to specify arbitrary task graphs for more complex and
  custom systems that are not part of the standard set of collections.

## Reasons you might choose Spark

- You prefer Scala or the SQL language
- You have mostly JVM infrastructure and legacy systems
- You want an established and trusted solution for business
- You are mostly doing business analytics with some lightweight machine learning
- You want an all-in-one solution

## Reasons you might choose Dask

- You prefer Python or native code, or have large legacy code bases that you
  do not want to entirely rewrite
- Your use case is complex or does not cleanly fit the Spark computing model
- You want a lighter-weight transition from local computing to cluster
  computing
- You want to interoperate with other technologies and don’t mind installing
  multiple packages

## Reasons to choose both

It is easy to use both Dask and Spark on the same data and on the same cluster.

They can both read and write common formats, like CSV, JSON, ORC, and Parquet,
making it easy to hand results off between Dask and Spark workflows.

They can both deploy on the same clusters.
Most clusters are designed to support many different distributed systems at the
same time, using resource managers like Kubernetes and YARN.  If you already
have a cluster on which you run Spark workloads, it’s likely easy to also run
Dask workloads on your current infrastructure and vice versa.

In particular, for users coming from traditional Hadoop/Spark clusters (such as
those sold by Cloudera/Hortonworks) you are using the Yarn resource
manager.  You can deploy Dask on these systems using the [Dask Yarn](https://yarn.dask.org) project, as well as other projects, like [JupyterHub
on Hadoop](https://jupyterhub-on-hadoop.readthedocs.io/en/latest/).

## Developer-Facing Differences

### Graph Granularity

Both Spark and Dask represent computations with directed acyclic graphs. These
graphs, however, represent computations at very different granularities.

One operation on a Spark RDD might add a node like `Map` and `Filter` to
the graph.  These are high-level operations that convey meaning and will
eventually be turned into many little tasks to execute on individual workers.
This many-little-tasks state is only available internally to the Spark
scheduler.

Dask graphs skip this high-level representation and go directly to the
many-little-tasks stage.  As such, one `map` operation on a Dask collection
will immediately generate and add possibly thousands of tiny tasks to the Dask
graph.

This difference in the scale of the underlying graph has implications on the
kinds of analysis and optimizations one can do and also on the generality that
one exposes to users.  Dask is unable to perform some optimizations that Spark
can because Dask schedulers do not have a top-down picture of the computation
they were asked to perform.  However, Dask is able to easily represent far more
[complex algorithms](http://matthewrocklin.com/blog/work/2015/06/26/Complex-Graphs) and expose the creation of these algorithms to normal users.

## Conclusion

- Spark is mature and all-inclusive.  If you want a single project that does
  everything and you’re already on Big Data hardware, then Spark is a safe bet,
  especially if your use cases are typical ETL + SQL and you’re already using
  Scala.
- Dask is lighter weight and is easier to integrate into existing code and hardware.
  If your problems vary beyond typical ETL + SQL and you want to add flexible
  parallelism to existing solutions, then Dask may be a good fit, especially if
  you are already using Python and associated libraries like NumPy and Pandas.

If you are looking to manage 100GB or less of tabular CSV or JSON data,
then you should forget both Spark and Dask and use [Postgres](https://www.postgresql.org/) or [MongoDB](https://www.mongodb.org/).


# spec.html.md

# Specification

Dask is a specification to encode a graph – specifically, a directed
acyclic graph of tasks with data dependencies – using ordinary Python data
structures, namely dicts, tuples, functions, and arbitrary Python
values.

## Definitions

A **Dask graph** is a dictionary mapping **keys** to **computation**:

### Tasks

```python
{'x': (x := DataNode(None, 1)),
'y': (y := DataNode(None, 2)),
'z': (z := Task("z", add, x.ref(), y.ref())),
'w': (w := Task("w", sum, List(x.ref(), y.ref(), z.ref()))),
'v': List(Task(None, sum, List(w.ref(), z.ref())), 2)}
```

### Legacy

```python
{'x': 1,
'y': 2,
'z': (add, 'y', 'x'),
'w': (sum, ['x', 'y', 'z']),
'v': [(sum, ['w', 'z']), 2]}
```

A **key** is a str, int, float, or tuple thereof:

```python
'x'
('x', 2, 3)
```

A **task** is a computation that Dask can perform and is expressed using the `dask.Task` class.  Tasks represent atomic
units of work meant to be run by a single worker.  Example:

```python
def add(x, y):
    return x + y

t = Task("t", add, 1, 2)
assert t() == 3

t2 = Task("t2", add, t.ref(), 2)
assert t2({"t": 3}) == 5
```

#### NOTE
The legacy representation of a task is a tuple with the first element being a
callable function.  The rest of the elements are arguments to that function.
The legacy representation is deprecated and will be removed in a future
version of Dask.  Use the `dask.Task` class instead.

A **computation** may be one of the following:

1. An `Alias` pointing to another key in the Dask graph like `Alias('new', 'x')`
2. A literal value as a `DataNode` like `DataNode(None, 1)`
3. A `Task` like `Task("t", add, 1, 2)`
4. A `List` of **computations**, like `List(1, TaskRef('x'), Task(None, inc, TaskRef("x"))]`

So all of the following are valid **computations**:

### Computations

```python
DataNode(None, np.array([...]))
Task("t", add, 1, 2)
Task("t", add, TaskRef('x'), 2)
Task("t", add, Task(None, inc, TaskRef('x')), 2)
Task("t", sum, [1, 2])
Task("t", sum, [TaskRef('x'), Task(None, inc, TaskRef('x'))])
Task("t", np.dot, np.array([...]), np.array([...]))
Task("t", sum, List(TaskRef('x'), TaskRef('y')), 'z')
```

### Legacy representation

```python
np.array([...])
(add, 1, 2)
(add, 'x', 2)
(add, (inc, 'x'), 2)
(sum, [1, 2])
(sum, ['x', (inc, 'x')])
(np.dot, np.array([...]), np.array([...]))
[(sum, ['x', 'y']), 'z']
```

## What functions should expect

In cases like `Task("t", add, TaskRef('x'), 2)`, functions like `add` receive concrete values instead of keys.  A Dask scheduler replaces task references (like `x`) with their computed values (like `1`) *before* calling the `add` function.
These references can be provided as either literal key references using
`TaskRef` or if a reference to the task is available, by
calling `ref()` on the task itself.

## Entry Point - The `get` function

The `get` function serves as entry point to computation for all
[schedulers](scheduler-overview.md).  This function gets the value
associated to the given key.  That key may refer to stored data, as is the case
with `'x'`, or to a task, as is the case with `'z'`.  In the latter case,
`get` should perform all necessary computation to retrieve the computed
value.

```python
>>> from dask.threaded import get

>>> from operator import add

>>> dsk = {'x': (x := DataNode(None, 1)),
...       'y': (y := DataNode(None, 2)),
...       'z': (z := Task("z", add, x.ref(), y.ref())),
...       'w': (w := Task("w", sum, List(x.ref(), y.ref(), z.ref())))}
```

```python
>>> get(dsk, 'x')
1

>>> get(dsk, 'z')
3

>>> get(dsk, 'w')
6
```

Additionally, if given a `list`, get should simultaneously acquire values for
multiple keys:

```python
>>> get(dsk, ['x', 'y', 'z'])
[1, 2, 3]
```

Because we accept lists of keys as keys, we support nested lists:

```python
>>> get(dsk, [['x', 'y'], ['z', 'w']])
[[1, 2], [3, 6]]
```

Internally `get` can be arbitrarily complex, calling out to distributed
computing, using caches, and so on.

## Why not tuples?

The tuples are objectively a more compact representation than the `Task` class so why did we choose to introduce this new representation?

As a tuple, the task is not self-describing and heavily context dependent. The
meaning of a tuple like `(func, "x", "y")` is depending on the graph it is
embedded in. The literals `x` and `y` could be either actual literals that
should be passed to the function or they could be references to other tasks.
Therefore, the \_interpretation_ of this task has to walk the tuple recursively
and compare every single encountered element with known keys in the graph.
Especially for large graphs or deeply nested tuple arguments, this can be a
performance bottleneck. For APIs that allow users to define their own key names
this can further cause false positives where intended literals are replaced by
pre-computed task results.

For the time being, both representation are supported. Legacy style tasks will
be automatically converted to new style tasks whenever dask encounters them. New
projects and algorithms are encouraged to use the new style tasks.


# support.html.md

<!-- this page is referenced from the topbar which comes from the theme -->

# Community

Dask is used and developed by individuals at a variety of institutions.  It
sits within the broader Python numeric ecosystem commonly referred to as PyData
or SciPy.

## Community Meeting

#### NOTE
We’ve combined the monthly Dask Demo Day and Dask Developer Meeting into a single, monthly Dask community meeting.

Join us for the monthly community meetings on the first Thursday of the month at
10:00 US Central Time. [Join via Zoom](https://dask.org/meeting-room).

Have something you’d like to share? Let us know by dropping a comment on [this GitHub issue](https://github.com/dask/community/issues/307).

Meeting notes are available in
[this Google doc](https://docs.google.com/document/d/1UqNAP87a56ERH_xkQsS5Q_0PKYybd5Lj2WANy_hRzI0/edit).

<iframe id="calendariframe" src="https://calendar.google.com/calendar/embed?ctz=local&amp;src=4l0vts0c1cgdbq5jhcogj55sfs%40group.calendar.google.com" style="border: 0" width="800" height="600" frameborder="0" scrolling="no"></iframe>
<script>document.getElementById("calendariframe").src = document.getElementById("calendariframe").src.replace("ctz=local", "ctz=" + Intl.DateTimeFormat().resolvedOptions().timeZone)</script>

You can subscribe to this calendar to be notified of changes:

* [Google Calendar](https://calendar.google.com/calendar/u/0?cid=NGwwdnRzMGMxY2dkYnE1amhjb2dqNTVzZnNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
* [iCal](https://calendar.google.com/calendar/ical/4l0vts0c1cgdbq5jhcogj55sfs%40group.calendar.google.com/public/basic.ics)

## Discussion

Conversation happens in the following places:

1. **Usage questions, requests for help, and general discussions** happen in the
   [Dask Discourse forum](https://dask.discourse.group). If your discussion topic is not a bug report
   or a feature request, this is the best place to start. It’s also a good
   place to show off cool things you have built using Dask and to get to know other
   community members.
2. **Usage questions** may also be directed to [Stack Overflow with the #dask tag](https://stackoverflow.com/questions/tagged/dask),
   which is monitored by Dask developers. However, the scope of what is considered
   a good Stack Overflow question can be narrow, so the Dask Discourse forum may
   be a better place to start.
3. **Bug reports and feature requests** are managed on the [GitHub issue
   tracker](https://github.com/dask/dask/issues/)
4. **Real-time chat** occurs on
   [https://dask.slack.com/](https://join.slack.com/t/dask/shared_invite/zt-mfmh7quc-nIrXL6ocgiUH2haLYA914g).
   Note that Slack chat is not easily searchable and indexed by search engines, so
   detailed discussion topics around bug reports or usage should go to GitHub issues or
   the Dask Discourse forum, respectively.

## Asking for help

We welcome usage questions and bug reports from all users, even those who are
new to using the project.  There are a few things you can do to improve the
likelihood of quickly getting a good answer.

1. **Ask questions in the right place**:  We strongly prefer the use
   of [Dask Discourse forum](https://dask.discourse.group) or [GitHub issues](https://github.com/dask/dask/issues/) over Slack chat.  Discourse and
   GitHub are more easily searchable by future users, and therefore can be
   useful to many more people than those directly involved.

   If you have a general question about how something should work or
   want best practices then use Discourse.  If you think you have found a
   bug then use GitHub
2. **Ask only in one place**: Please restrict yourself to posting your
   question in only one place (likely the Dask Discourse or GitHub) and don’t post
   in both
3. **Create a minimal example**:  It is ideal to create [minimal, complete,
   verifiable examples](https://stackoverflow.com/help/mcve).  This
   significantly reduces the time that answerers spend understanding your
   situation, resulting in higher quality answers more quickly.

   See also [this blogpost](http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)
   about crafting minimal bug reports.  These have a much higher likelihood of
   being answered

## Paid support

In addition to the previous options, paid support is available from the
following organizations (listed in alphabetical order):

- [Anaconda](https://www.anaconda.com/products/professional-services)
- [Coiled](https://coiled.io?utm_source=dask-docs&utm_medium=support)
- [Quansight](https://www.quansight.com/)


# understanding-performance.html.md

# Understanding Performance

The first step in making computations run quickly is to understand the costs involved.
In Python we often rely on tools like
the [CProfile module](https://docs.python.org/3/library/profile.html),
[%%prun IPython magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-prun),
[VMProf](https://vmprof.readthedocs.io/en/latest/), or
[snakeviz](https://jiffyclub.github.io/snakeviz/)
to understand the costs associated with our code.
However, few of these tools work well on multi-threaded or multi-process code,
and fewer still on computations distributed among many machines.
We also have new costs like data transfer, serialization, task scheduling overhead, and more
that we may not be accustomed to tracking.

Fortunately, the Dask schedulers come with diagnostics
to help you understand the performance characteristics of your computations.
By using these diagnostics and with some thought,
we can often identify the slow parts of troublesome computations.

The [single-machine and distributed schedulers](scheduling.md) come with *different* diagnostic tools.
These tools are deeply integrated into each scheduler,
so a tool designed for one will not transfer over to the other.

These pages provide four options for profiling parallel code:

1. [Visualize task graphs](graphviz.md)
2. [Single threaded scheduler and a normal Python profiler](scheduling.md#single-threaded-scheduler)
3. [Diagnostics for the single-machine scheduler](diagnostics-local.md)
4. [Diagnostics for the distributed scheduler and dashboard](diagnostics-distributed.md)

Additionally, if you are interested in understanding the various phases where
slowdown can occur, you may wish to read the following:

- [Phases of computation](phases-of-computation.md)


# user-interfaces.html.md

# User Interfaces

Dask supports several user interfaces:

- High-Level
  : - [Arrays](array.md): Parallel NumPy
    - [Bags](bag.md): Parallel lists
    - [DataFrames](dataframe.md): Parallel Pandas
    - [Machine Learning](https://ml.dask.org) : Parallel Scikit-Learn
    - Others from external projects, like [XArray](https://xarray.pydata.org)
- Low-Level
  : - [Delayed](delayed.md): Parallel function evaluation
    - [Futures](futures.md): Real-time parallel function evaluation

Each of these user interfaces employs the same underlying parallel computing
machinery, and so has the same scaling, diagnostics, resilience, and so on, but
each provides a different set of parallel algorithms and programming style.

This document helps you to decide which user interface best suits your needs,
and gives some general information that applies to all interfaces.
The pages linked above give more information about each interface in greater
depth.

## High-Level Collections

Many people who start using Dask are explicitly looking for a scalable version of
NumPy, Pandas, or Scikit-Learn.  For these situations, the starting point within
Dask is usually fairly clear.  If you want scalable NumPy arrays, then start with Dask
array; if you want scalable Pandas DataFrames, then start with Dask DataFrame, and so on.

These high-level interfaces copy the standard interface with slight variations.
These interfaces automatically parallelize over larger datasets for you for a
large subset of the API from the original project.

```python
# Arrays
import dask.array as da
rng = da.random.default_rng()
x = rng.uniform(low=0, high=10, size=(10000, 10000),  # normal numpy code
                chunks=(1000, 1000))  # break into chunks of size 1000x1000

y = x + x.T - x.mean(axis=0)  # Use normal syntax for high level algorithms

# DataFrames
import dask.dataframe as dd
df = dd.read_csv('2018-*-*.csv', parse_dates='timestamp',  # normal Pandas code
                 blocksize=64000000)  # break text into 64MB chunks

s = df.groupby('name').balance.mean()  # Use normal syntax for high level algorithms

# Bags / lists
import dask.bag as db
b = db.read_text('*.json').map(json.loads)
total = (b.filter(lambda d: d['name'] == 'Alice')
          .map(lambda d: d['balance'])
          .sum())
```

It is important to remember that, while APIs may be similar, some differences do
exist.  Additionally, the performance of some algorithms may differ from their
in-memory counterparts due to the advantages and disadvantages of parallel
programming.  Some thought and attention is still required when using Dask.

## Low-Level Interfaces

Often when parallelizing existing code bases or building custom algorithms, you
run into code that is parallelizable, but isn’t just a big DataFrame or array.
Consider the for-loopy code below:

```python
results = []
for a in A:
    for b in B:
        if a < b:
            c = f(a, b)
        else:
            c = g(a, b)
        results.append(c)
```

There is potential parallelism in this code (the many calls to `f` and `g`
can be done in parallel), but it’s not clear how to rewrite it into a big
array or DataFrame so that it can use a higher-level API.  Even if you could
rewrite it into one of these paradigms, it’s not clear that this would be a
good idea.  Much of the meaning would likely be lost in translation, and this
process would become much more difficult for more complex systems.

Instead, Dask’s lower-level APIs let you write parallel code one function call
at a time within the context of your existing for loops.  A common solution
here is to use [Dask delayed](delayed.md) to wrap individual function calls
into a lazily constructed task graph:

```python
import dask

lazy_results = []
for a in A:
    for b in B:
        if a < b:
            c = dask.delayed(f)(a, b)  # add lazy task
        else:
            c = dask.delayed(g)(a, b)  # add lazy task
        lazy_results.append(c)

results = dask.compute(*lazy_results)  # compute all in parallel
```

## Combining High- and Low-Level Interfaces

It is common to combine high- and low-level interfaces.
For example, you might use Dask array/bag/dataframe to load in data and do
initial pre-processing, then switch to Dask delayed for a custom algorithm that
is specific to your domain, then switch back to Dask array/dataframe to clean
up and store results.  Understanding both sets of user interfaces, and how
to switch between them, can be a productive combination.

```python
# Convert to a list of delayed Pandas dataframes
delayed_values = df.to_delayed()

# Manipulate delayed values arbitrarily as you like

# Convert many delayed Pandas DataFrames back to a single Dask DataFrame
df = dd.from_delayed(delayed_values)
```

## Laziness and Computing

Most Dask user interfaces are *lazy*, meaning that they do not evaluate until
you explicitly ask for a result using the `compute` method:

```python
# This array syntax doesn't cause computation
y = x + x.T - x.mean(axis=0)

# Trigger computation by explicitly calling the compute method
y = y.compute()
```

If you have multiple results that you want to compute at the same time, use the
`dask.compute` function.  This can share intermediate results and so be more
efficient:

```python
# compute multiple results at the same time with the compute function
min, max = dask.compute(y.min(), y.max())
```

Note that the `compute()` function returns in-memory results.  It converts
Dask DataFrames to Pandas DataFrames, Dask arrays to NumPy arrays, and Dask
bags to lists.  *You should only call compute on results that will fit
comfortably in memory*.  If your result does not fit in memory, then you might
consider writing it to disk instead.

```python
# Write larger results out to disk rather than store them in memory
my_dask_dataframe.to_parquet('myfile.parquet')
my_dask_array.to_hdf5('myfile.hdf5')
my_dask_bag.to_textfiles('myfile.*.txt')
```

## Persist into Distributed Memory

#### WARNING
persist will store the full dataset in memory. This has the disadvantage that
the available memory must actually exceed the size of the dataset. Use persist
only when interactively iterating on the same dataset over and over again and
avoid it in productive use-cases as much as possible.

Alternatively, if you are on a cluster, then you may want to trigger a
computation and store the results in distributed memory.  In this case you do
not want to call `compute`, which would create a single Pandas, NumPy, or
list result. Instead, you want to call `persist`, which returns a new Dask
object that points to actively computing, or already computed results spread
around your cluster’s memory.

```python
# Compute returns an in-memory non-Dask object
y = y.compute()

# Persist returns an in-memory Dask object that uses distributed storage if available
y = y.persist()
```

This is common to see after data loading an preprocessing steps, but before
rapid iteration, exploration, or complex algorithms.  For example, we might read
in a lot of data, filter down to a more manageable subset, and then persist
data into memory so that we can iterate quickly.

```python
import dask.dataframe as dd
df = dd.read_parquet('...')
df = df[df.name == 'Alice']  # select important subset of data
df = df.persist()  # trigger computation in the background

# These are all relatively fast now that the relevant data is in memory
df.groupby(df.id).balance.sum().compute()   # explore data quickly
df.groupby(df.id).balance.mean().compute()  # explore data quickly
df.id.nunique()                             # explore data quickly
```

## Lazy vs Immediate

As mentioned above, most Dask workloads are lazy, that is, they don’t start any
work until you explicitly trigger them with a call to `compute()`.
However, sometimes you *do* want to submit work as quickly as possible, track it
over time, submit new work or cancel work depending on partial results, and so
on.  This can be useful when tracking or responding to real-time events,
handling streaming data, or when building complex and adaptive algorithms.

For these situations, people typically turn to the [futures interface](futures.md) which is a low-level interface like Dask delayed, but operates
immediately rather than lazily.

Here is the same example with Dask delayed and Dask futures to illustrate the
difference.

### Delayed: Lazy

```python
@dask.delayed
def inc(x):
    return x + 1

@dask.delayed
def add(x, y):
    return x + y

a = inc(1)       # no work has happened yet
b = inc(2)       # no work has happened yet
c = add(a, b)    # no work has happened yet

c = c.compute()  # This triggers all of the above computations
```

### Futures: Immediate

```python
from dask.distributed import Client
client = Client()

def inc(x):
    return x + 1

def add(x, y):
    return x + y

a = client.submit(inc, 1)     # work starts immediately
b = client.submit(inc, 2)     # work starts immediately
c = client.submit(add, a, b)  # work starts immediately

c = c.result()                # block until work finishes, then gather result
```

You can also trigger work with the high-level collections using the
`persist` function.  This will cause work to happen in the background when
using the distributed scheduler.

## Combining Interfaces

There are established ways to combine the interfaces above:

1. The high-level interfaces (array, bag, dataframe) have a `to_delayed`
   method that can convert to a sequence (or grid) of Dask delayed objects
   ```python
   delayeds = df.to_delayed()
   ```
2. The high-level interfaces (array, bag, dataframe) have a `from_delayed`
   method that can convert from either Delayed *or* Future objects
   ```python
   df = dd.from_delayed(delayeds)
   df = dd.from_delayed(futures)
   ```
3. The `Client.compute` method converts Delayed objects into Futures
   ```python
   futures = client.compute(delayeds)
   ```
4. The `dask.distributed.futures_of` function gathers futures from
   persisted collections
   ```python
   from dask.distributed import futures_of

   df = df.persist()  # start computation in the background
   futures = futures_of(df)
   ```
5. The Dask.delayed object converts Futures into delayed objects
   ```python
   delayed_value = dask.delayed(future)
   ```

The approaches above should suffice to convert any interface into any other.
We often see some anti-patterns that do not work as well:

1. Calling low-level APIs (delayed or futures) on high-level objects (like
   Dask arrays or DataFrames). This downgrades those objects to their NumPy or
   Pandas equivalents, which may not be desired.
   Often people are looking for APIs like `dask.array.map_blocks` or
   `dask.dataframe.map_partitions` instead.
2. Calling `compute()` on Future objects.
   Often people want the `.result()` method instead.
3. Calling NumPy/Pandas functions on high-level Dask objects or
   high-level Dask functions on NumPy/Pandas objects

## Conclusion

Most people who use Dask start with only one of the interfaces above but
eventually learn how to use a few interfaces together.  This helps them
leverage the sophisticated algorithms in the high-level interfaces while also
working around tricky problems with the low-level interfaces.

For more information, see the documentation for the particular user interfaces
below:

- High Level
  : - [Arrays](array.md): Parallel NumPy
    - [Bags](bag.md): Parallel lists
    - [DataFrames](dataframe.md): Parallel Pandas
    - [Machine Learning](https://ml.dask.org) : Parallel Scikit-Learn
    - Others from external projects, like [XArray](https://xarray.pydata.org)
- Low Level
  : - [Delayed](delayed.md): Parallel function evaluation
    - [Futures](futures.md): Real-time parallel function evaluation


# why.html.md

<!-- this page is referenced from dask.org. It might move there some day. -->

# Why Dask?

This document gives high-level motivation on why people choose to adopt Dask.

> * [Python’s role in Data Science](#python-s-role-in-data-science)
> * [Dask has a Familiar API](#dask-has-a-familiar-api)
> * [Dask Scales out to Clusters](#dask-scales-out-to-clusters)
> * [Dask Scales Down to Single Computers](#dask-scales-down-to-single-computers)
> * [Dask Integrates Natively with Python Code](#dask-integrates-natively-with-python-code)
> * [Dask Supports Complex Applications](#dask-supports-complex-applications)
> * [Dask Delivers Responsive Feedback](#dask-delivers-responsive-feedback)
> * [Links and More Information](#links-and-more-information)

## Python’s role in Data Science

Python has grown to become the dominant language both in data analytics and
general programming:

![Graph showing the growth of major programming languages based on Stack Overflow’s question views in World Bank high-income countries. A line graph with time from 2012 to 2018 on the x-axis and percent of overall question views each month on the y-axis. Python’s question views increase from about 4% to about 11% from 2012 to 2018, reaching the popularity of Java and JavaScript.](images/growth_of_languages.png)

This is fueled both by computational libraries like Numpy, Pandas, and
Scikit-Learn and by a wealth of libraries for visualization, interactive
notebooks, collaboration, and so forth.

![Graph showing the growth of major python packages based on Stack Overflow's question views in World Bank high-income countries. A line graph with time on the x-axis from 2012 to 2018 and percent of overall question views each month on the y-axis. Pandas question views increased to about 0.9% in 2018, exceeding Django and NumPy.](images/growth_of_libraries.png)

However, these packages were not designed to scale beyond a single machine.
Dask was developed to scale these packages and the surrounding ecosystem.
It works with the existing Python ecosystem to scale it to multi-core
machines and distributed clusters.

*Image credit to Stack Overflow blogposts*
[#1](https://stackoverflow.blog/2017/09/06/incredible-growth-python)
*and*
[#2](https://stackoverflow.blog/2017/09/14/python-growing-quickly/).

## Dask has a Familiar API

Analysts often use tools like Pandas, Scikit-Learn, Numpy, and the rest of the
Python ecosystem to analyze data on their personal computer.  They like these
tools because they are efficient, intuitive, and widely trusted.  However, when
they choose to apply their analyses to larger datasets, they find that these
tools were not designed to scale beyond a single machine. And so, the analyst
rewrites their computation using a more scalable tool, often in another
language altogether.  This rewrite process slows down discovery and causes
frustration.

Dask provides ways to scale Pandas, Scikit-Learn, and Numpy workflows more
natively, with minimal rewriting.  It integrates well with these tools so that it copies
most of their API and uses their data structures internally.  Moreover, Dask is
co-developed with these libraries to ensure that they evolve consistently,
minimizing friction when transitioning from a local laptop,
to a multi-core workstation, and then to a distributed cluster.  Analysts
familiar with Pandas/Scikit-Learn/Numpy will be immediately familiar with their
Dask equivalents, and have much of their intuition carry over to a scalable
context.

## Dask Scales out to Clusters

As datasets and computations scale faster than CPUs and RAM, we need to find
ways to scale our computations across multiple machines.  This introduces many
new concerns:

- How to have computers talk to each other over the network?
- How and when to move data between machines?
- How to recover from machine failures?
- How to deploy on an in-house cluster?
- How to deploy on the cloud?
- How to deploy on an HPC super-computer?
- How to provide an API to this system that users find intuitive?
- …

While it is possible to build these systems in-house (and indeed, many exist),
many organizations increasingly depend on solutions developed within the
open source community.  These tend to be more robust, secure, and fully
featured without being tended by in-house staff.

Dask solves the problems above.  It figures out how to break up large
computations and route parts of them efficiently onto distributed hardware.
Dask is routinely run on thousand-machine clusters to process hundreds of
terabytes of data efficiently within secure environments.

Dask has utilities and documentation on how to deploy in-house, on
the cloud, or on HPC super-computers.  It supports encryption and
authentication using TLS/SSL certificates.  It is resilient and can handle the
failure of worker nodes gracefully and is elastic, and so can take advantage of
new nodes added on-the-fly.  Dask includes several user APIs that are used and
smoothed over by thousands of researchers across the globe working in different
domains.

## Dask Scales Down to Single Computers

*But a massive cluster is not always the right choice*

Today’s laptops and workstations are surprisingly powerful and, if used
correctly, can handle datasets and computations for which we previously
depended on clusters.  A modern laptop has a multi-core CPU, 32GB of RAM, and
flash-based hard drives that can stream through data several times faster than
HDDs or SSDs of even a year or two ago.

As a result, Dask can empower analysts to manipulate 100GB+ datasets on their
laptop or 1TB+ datasets on a workstation without bothering with the cluster at
all.  This can be preferable for the following reasons:

1. They can use their local software environment, rather than being
   constrained by what is available on the cluster or having to manage
   Docker images.
2. They can more easily work while in transit, at a coffee shop, or at home
   away from the corporate network
3. Debugging errors and analyzing performance is simpler and more pleasant on
   a single machine
4. Their iteration cycles can be faster
5. Their computations may be more efficient because all of the data is local
   and doesn’t need to flow through the network or between separate processes

Dask can enable efficient parallel computations on single machines by
leveraging their multi-core CPUs and streaming data efficiently from disk.
It *can* run on a distributed cluster, but it doesn’t *have* to.  Dask allows
you to swap out the cluster for single-machine schedulers which are surprisingly
lightweight, require no setup, and can run entirely within the same process as
the user’s session.

To avoid excess memory use, Dask is good at finding ways to evaluate
computations in a low-memory footprint when possible by pulling in chunks of
data from disk, doing the necessary processing, and throwing away intermediate
values as quickly as possible.  This lets analysts perform computations on
moderately large datasets (100GB+) even on relatively low-power laptops.
This requires no configuration and no setup, meaning that adding Dask to a
single-machine computation adds very little cognitive overhead.

Dask is installed by default with [Anaconda](https://anaconda.com)
and so is already deployed on most data science machines.

## Dask Integrates Natively with Python Code

Python includes computational libraries like Numpy, Pandas, and Scikit-Learn,
and many others for data access, plotting, statistics, image and
signal processing, and more.  These libraries work together seamlessly to
produce a cohesive *ecosystem* of packages that co-evolve to meet the needs of
analysts in most domains today.

This ecosystem is tied together by common standards and protocols to which
everyone adheres, which allows these packages to benefit each other in
surprising and delightful ways.

Dask evolved from within this ecosystem.  It abides by these standards and
protocols and actively engages in community efforts to push forward new ones.
This enables the rest of the ecosystem to benefit from parallel and distributed
computing with minimal coordination.  Dask does not seek to disrupt or displace
the existing ecosystem, but rather to complement and benefit it from within.

As a result, Dask development is pushed forward by developer communities
from Pandas, Numpy, Scikit-Learn, Scikit-Image, Jupyter, and others.  This
engagement from the broader community growth helps users to trust the project
and helps to ensure that the Python ecosystem will continue to evolve in a
smooth and sustainable manner.

## Dask Supports Complex Applications

Some parallel computations are simple and just apply the same routine onto many
inputs without any kind of coordination.  These are simple to parallelize with
any system.

Somewhat more complex computations can be expressed with the
map-shuffle-reduce pattern popularized by Hadoop and Spark.
This is often sufficient to do most data cleaning tasks,
database-style queries, and some lightweight machine learning algorithms.

However, more complex parallel computations exist which do not fit into these
paradigms, and so are difficult to perform with traditional big-data
technologies.  These include more advanced algorithms for statistics or machine
learning, time series or local operations, or bespoke parallelism often found
within the systems of large enterprises.

Many companies and institutions today have problems which are
clearly parallelizable, but not clearly transformable into a big DataFrame
computation.  Today these companies tend to solve their problems either by
writing custom code with low-level systems like MPI, ZeroMQ, or sockets and
complex queuing systems, or by shoving their problem into a standard big-data
technology like MapReduce or Spark, and hoping for the best.

Dask helps to resolve these situations by exposing low-level APIs to its
internal task scheduler which is capable of executing very advanced
computations.  This gives engineers within the institution the ability to build
their own parallel computing system using the same engine that powers Dask’s
arrays, DataFrames, and machine learning algorithms, but now with the
institution’s own custom logic.  This allows engineers to keep complex
business logic in-house while still relying on Dask to handle network
communication, load balancing, resilience, diagnostics, etc..

## Dask Delivers Responsive Feedback

Because everything happens remotely, interactive parallel computing can be
frustrating for users.  They don’t have a good sense of how computations are
progressing, what might be going wrong, or what parts of their code should they
focus on for performance.  The added distance between a user and their
computation can drastically affect how quickly they are able to identify and
resolve bugs and performance problems, which can drastically increase their
time to solution.

Dask keeps users informed with a suite of helpful diagnostic and
investigative tools including the following:

1. A [real-time and responsive dashboard](understanding-performance.md)
   that shows current progress, communication costs, memory use, and more,
   updated every 100ms
2. A statistical profiler installed on every worker that polls each thread
   every 10ms to determine which lines in your code are taking up the most
   time across your entire computation
3. An embedded IPython kernel in every worker and the scheduler, allowing
   users to directly investigate the state of their computation with a pop-up
   terminal
4. The ability to re-raise errors locally, so that they can use the traditional
   debugging tools to which they are accustomed, even when the error happens
   remotely

## Links and More Information

From here you may want to read about some of our more common introductory
content:

- [User Interfaces](user-interfaces.md)
- [Scheduling](scheduling.md)
- [Comparison to Spark](spark.md)
- [Slides](https://dask.org/slides.html)