Toggle navigation

.rst .pdf

Contents

Changelog

2023.11.0

Released on November 10, 2023

Highlights

Zero-copy P2P Array Rechunking

Users should see significant performance improvements when using in-memory P2P array rechunking. This is due to no longer copying underlying data buffers.

Below shows a simple example where we compare performance of different rechunking methods.

shape = (30_000, 6_000, 150) # 201.17 GiB
input_chunks = (60, -1, -1) # 411.99 MiB
output_chunks = (-1, 6, -1) # 205.99 MiB

arr = da.random.random(size, chunks=input_chunks)
with dask.config.set({
    "array.rechunk.method": "p2p",
    "distributed.p2p.disk": True,
}):
    (
      da.random.random(size, chunks=input_chunks)
      .rechunk(output_chunks)
      .sum()
      .compute()
    )

See GH#8282, GH#8318, GH#8321 from crusaderky and (GH#8322) from Hendrik Makait for details.

Deprecating PyArrow <14.0.1

pyarrow<14.0.1 usage is deprecated starting in this release. It’s recommended for all users to upgrade their version of pyarrow or install pyarrow-hotfix. See this CVE for full details.

See GH#10622 from Florian Jetter for details.

Improved PyArrow filesystem for Parquet

Using filesystem="arrow" when reading Parquet datasets now properly inferrs the correct cloud region when accessing remote, cloud-hosted data.

See GH#10590 from Richard (Rick) Zamora for details.

Improve Type Reconciliation in P2P Shuffling

See GH#8332 from Hendrik Makait for details.

Additional changes

• Fix sporadic failure of test_dataframe::test_quantile (GH#10625) Miles

• Bump minimum click to >=8.1 (GH#10623) Jacob Tomlinson

• Refactor test_quantile (GH#10620) Miles

• Avoid PerformanceWarning for fragmented DataFrame (GH#10621) Patrick Hoefler

• Generalize computation of NEW_*_VER in GPU CI updating workflow (GH#10610) Charles Blackmon-Luca

• Switch to newer GPU CI images (GH#10608) Charles Blackmon-Luca

• Remove double slash in fsspec tests (GH#10605) Mario Šaško

• Reenable test_ucx_config_w_env_var (GH#8272) Peter Andreas Entschev

• Don’t share host_array when receiving from network (GH#8308) crusaderky

• Generalize computation of NEW_*_VER in GPU CI updating workflow (GH#8319) Charles Blackmon-Luca

• Switch to newer GPU CI images (GH#8316) Charles Blackmon-Luca

• Minor updates to shuffle dashboard (GH#8315) Matthew Rocklin

• Don’t use bytearray().join (GH#8312) crusaderky

• Reuse identical shuffles in P2P hash join (GH#8306) Hendrik Makait

2023.10.1

Released on October 27, 2023

Highlights

Python 3.12

This release adds official support for Python 3.12.

See GH#10544 and GH#8223 from Thomas Grainger for details.

Additional changes

• Avoid splitting parquet files to row groups as aggressively (GH#10600) Matthew Rocklin

• Speed up normalize_chunks for common case (GH#10579) Martin Durant

• Use Python 3.11 for upstream and doctests CI build (GH#10596) Thomas Grainger

• Bump actions/checkout from 4.1.0 to 4.1.1 (GH#10592)

• Switch to PyTables HEAD (GH#10580) Thomas Grainger

• Remove numpy.core warning filter, link to issue on pyarrow caused BlockManager warning (GH#10571) Thomas Grainger

• Unignore and fix deprecated freq aliases (GH#10577) Thomas Grainger

• Move register_assert_rewrite earlier in conftest to fix warnings (GH#10578) Thomas Grainger

• Upgrade versioneer to 0.29 (GH#10575) Thomas Grainger

• change test_concat_categorical to be non-strict (GH#10574) Thomas Grainger

• Enable SciPy tests with NumPy 2.0 Thomas Grainger

• Enable tests for scikit-image with NumPy 2.0 (GH#10569) Thomas Grainger

• Fix upstream build (GH#10549) Thomas Grainger

• Add optimized code paths for drop_duplicates (GH#10542) Richard (Rick) Zamora

• Support cudf backend in dd.DataFrame.sort_values (GH#10551) Richard (Rick) Zamora

• Rename “GIL Contention” to just GIL in chart labels (GH#8305) Matthew Rocklin

• Bump actions/checkout from 4.1.0 to 4.1.1 (GH#8299)

• Fix dashboard (GH#8293) Hendrik Makait

• @log_errors for async tasks (GH#8294) crusaderky

• Annotations and better tests for serialize_bytes (GH#8300) crusaderky

• Temporarily xfail test_decide_worker_coschedule_order_neighbors to unblock CI (GH#8298) James Bourbeau

• Skip xdist and matplotlib in code samples (GH#8290) Matthew Rocklin

• Use numpy._core on numpy>=2.dev0 (GH#8291) Thomas Grainger

• Fix calculation of MemoryShardsBuffer.bytes_read (GH#8289) crusaderky

• Allow P2P to store data in-memory (GH#8279) Hendrik Makait

• Upgrade versioneer to 0.29 (GH#8288) Thomas Grainger

• Allow ResourceLimiter to be unlimited (GH#8276) Hendrik Makait

• Run pre-commit autoupdate (GH#8281) Thomas Grainger

• Annotate instance variables for P2P layers (GH#8280) Hendrik Makait

• Remove worker gracefully should not mark tasks as suspicious (GH#8234) Thomas Grainger

• Add signal handling to dask spec (GH#8261) Thomas Grainger

• Add typing for sync (GH#8275) Hendrik Makait

• Better annotations for shuffle offload (GH#8277) crusaderky

• Test minimum versions for p2p shuffle (GH#8270) crusaderky

• Run coverage on test failures (GH#8269) crusaderky

• Use aiohttp with extensions (GH#8274) Thomas Grainger

2023.10.0

Released on October 13, 2023

Highlights

Reduced memory pressure for multi array reductions

This release contains major updates to Dask’s task graph scheduling logic. The updates here significantly reduce memory pressure on array reductions. We anticipate this will have a strong impact on the array computing community.

See GH#10535 from Florian Jetter for details.

Improved P2P shuffling robustness

There are several updates (listed below) that make P2P shuffling much more robust and less likely to fail.

See GH#8262, GH#8264, GH#8242, GH#8244, and GH#8235 from Hendrik Makait and GH#8124 from Charles Blackmon-Luca for details.

Reduced scheduler CPU load for large graphs

Users should see reduced CPU load on their scheduler when computing large task graphs.

See GH#8238 and GH#10547 from Florian Jetter and GH#8240 from crusaderky for details.

Additional changes

• Dispatch the partd.Encode class used for disk-based shuffling (GH#10552) Richard (Rick) Zamora

• Add documentation for hive partitioning (GH#10454) Richard (Rick) Zamora

• Add typing to dask.order (GH#10553) Florian Jetter

• Allow passing index_col=False in dd.read_csv (GH#9961) Michael Leslie

• Tighten HighLevelGraph annotations (GH#10524) crusaderky

• Support for latest ipykernel/ipywidgets (GH#8253) crusaderky

• Check minimal pyarrow version for P2P merge (GH#8266) Hendrik Makait

• Support for Python 3.12 (GH#8223) Thomas Grainger

• Use memoryview.nbytes when warning on large graph send (GH#8268) crusaderky

• Run tests without gilknocker (GH#8263) crusaderky

• Disable ipv6 on MacOS CI (GH#8254) crusaderky

• Clean up redundant minimum versions (GH#8251) crusaderky

• Clean up use of BARRIER_PREFIX in scheduler plugin (GH#8252) crusaderky

• Improve shuffle run handling in P2P’s worker plugin (GH#8245) Hendrik Makait

• Explicitly set charset=utf-8 (GH#8250) crusaderky

• Typing tweaks to GH#8239 (GH#8247) crusaderky

• Simplify scheduler assertion (GH#8246) crusaderky

• Improve typing (GH#8239) Hendrik Makait

• Respect cgroups v2 “low” memory limit (GH#8243) Samantha Hughes

• Fix PackageInstall by making it a scheduler plugin (GH#8142) Hendrik Makait

• Xfail test_ucx_config_w_env_var (GH#8241) crusaderky

• SpecCluster resilience to broken workers (GH#8233) crusaderky

• Suppress SpillBuffer stack traces for cancelled tasks (GH#8232) crusaderky

• Update annotations after stringification changes (GH#8195) crusaderky

• Reduce max recursion depth of profile (GH#8224) crusaderky

• Offload deeply nested objects (GH#8214) crusaderky

• Fix flaky test_close_connections (GH#8231) crusaderky

• Fix flaky test_popen_timeout (GH#8229) crusaderky

• Fix flaky test_adapt_then_manual (GH#8228) crusaderky

• Prevent collisions in SpillBuffer (GH#8226) crusaderky

• Allow retire_workers to run concurrently (GH#8056) Florian Jetter

• Fix HTML repr for TaskState objects (GH#8188) Florian Jetter

• Fix AttributeError for builtin_function_or_method in profile.py (GH#8181) Florian Jetter

• Fix flaky test_spans (v2) (GH#8222) crusaderky

2023.9.3

Released on September 29, 2023

Highlights

Restore previous configuration override behavior

The 2023.9.2 release introduced an unintentional breaking change in how configuration options are overriden in dask.config.get with the override_with= keyword (see GH#10519). This release restores the previous behavior.

See GH#10521 from crusaderky for details.

Complex dtypes in Dask Array reductions

This release includes improved support for using common reductions in Dask Array (e.g. var, std, moment) with complex dtypes.

See GH#10009 from wkrasnicki for details.

Additional changes

• Bump actions/checkout from 4.0.0 to 4.1.0 (GH#10532)

• Match pandas reverting apply deprecation (GH#10531) James Bourbeau

• Update gpuCI RAPIDS_VER to 23.12 (GH#10526)

• Temporarily skip failing tests with fsspec==2023.9.1 (GH#10520) James Bourbeau

2023.9.2

Released on September 15, 2023

Highlights

P2P shuffling now raises when outdated PyArrow is installed

Previously the default shuffling method would silently fallback from P2P to task-based shuffling if an older version of pyarrow was installed. Now we raise an informative error with the minimum required pyarrow version for P2P instead of silently falling back.

See GH#10496 from Hendrik Makait for details.

Deprecation cycle for admin.traceback.shorten

The 2023.9.0 release modified the admin.traceback.shorten configuration option without introducing a deprecation cycle. This resulted in failures to create Dask clusters in some cases. This release introduces a deprecation cycle for this configuration change.

See GH#10509 from crusaderky for details.

Additional changes

• Avoid materializing all iterators in delayed tasks (GH#10498) James Bourbeau

• Overhaul deprecations system in dask.config (GH#10499) crusaderky

• Remove unnecessary check in timeseries (GH#10447) Patrick Hoefler

• Use register_plugin in tests (GH#10503) James Bourbeau

• Make preserve_index explicit in pyarrow_schema_dispatch (GH#10501) Hendrik Makait

• Add **kwargs support for pyarrow_schema_dispatch (GH#10500) Hendrik Makait

• Centralize and type no_default (GH#10495) crusaderky

2023.9.1

Released on September 6, 2023

Note

This is a hotfix release that fixes a P2P shuffling bug introduced in the 2023.9.0 release (see GH#10493).

Enhancements

• Stricter data type for dask keys (GH#10485) crusaderky

• Special handling for None in DASK_ environment variables (GH#10487) crusaderky

Bug Fixes

• Fix _partitions dtype in meta for DataFrame.set_index and DataFrame.sort_values (GH#10493) Hendrik Makait

• Handle cached_property decorators in derived_from (GH#10490) Lawrence Mitchell

Maintenance

• Bump actions/checkout from 3.6.0 to 4.0.0 (GH#10492)

• Simplify some tests that import distributed (GH#10484) crusaderky

2023.9.0

Released on September 1, 2023

Bug Fixes

• Remove support for np.int64 in keys (GH#10483) crusaderky

• Fix _partitions dtype in meta for shuffling (GH#10462) Hendrik Makait

• Don’t use exception hooks to shorten tracebacks (GH#10456) crusaderky

Documentation

• Add p2p shuffle option to DataFrame docs (GH#10477) Patrick Hoefler

Maintenance

• Skip failing tests for pandas=2.1.0 (GH#10488) Patrick Hoefler

• Update tests for pandas=2.1.0 (GH#10439) Patrick Hoefler

• Enable pytest-timeout (GH#10482) crusaderky

• Bump actions/checkout from 3.5.3 to 3.6.0 (GH#10470)

2023.8.1

Released on August 18, 2023

Enhancements

• Adding support for cgroup v2 to cpu_count (GH#10419) Johan Olsson

• Support multi-column groupby with sort=True and split_out>1 (GH#10425) Richard (Rick) Zamora

• Add DataFrame.enforce_runtime_divisions method (GH#10404) Richard (Rick) Zamora

• Enable file mode="x" with a single_file=True for Dask DataFrame to_csv (GH#10443) Genevieve Buckley

Bug Fixes

• Fix ValueError when running to_csv in append mode with single_file as True (GH#10441) Ben

Maintenance

• Add default types_mapper to from_pyarrow_table_dispatch for pandas (GH#10446) Richard (Rick) Zamora

2023.8.0

Released on August 4, 2023

Enhancements

• Fix for make_timeseries performance regression (GH#10428) Irina Truong

Documentation

• Add distributed.print to debugging docs (GH#10435) James Bourbeau

• Documenting compatibility of NumPy functions with Dask functions (GH#9941) Chiara Marmo

Maintenance

• Use SPDX in license metadata (GH#10437) John A Kirkham

• Require dask[array] in dask[dataframe] (GH#10357) John A Kirkham

• Update gpuCI RAPIDS_VER to 23.10 (GH#10427)

• Simplify compatibility code (GH#10426) Hendrik Makait

• Fix compatibility variable naming (GH#10424) Hendrik Makait

• Fix a few errors with upstream pandas and pyarrow (GH#10412) Irina Truong

2023.7.1

Released on July 20, 2023

Note

This release updates Dask DataFrame to automatically convert text data using object data types to string[pyarrow] if pandas>=2 and pyarrow>=12 are installed.

This should result in significantly reduced memory consumption and increased computation performance in many workflows that deal with text data.

You can disable this change by setting the dataframe.convert-string configuration value to False with

dask.config.set({"dataframe.convert-string": False})

Enhancements

• Convert to pyarrow strings if proper dependencies are installed (GH#10400) James Bourbeau

• Avoid repartition before shuffle for p2p (GH#10421) Patrick Hoefler

• API to generate random Dask DataFrames (GH#10392) Irina Truong

• Speed up dask.bag.Bag.random_sample (GH#10356) crusaderky

• Raise helpful ValueError for invalid time units (GH#10408) Nat Tabris

• Make repartition a no-op when divisions match (divisions provided as a list) (GH#10395) Nicolas Grandemange

Bug Fixes

• Use dataframe.convert-string in read_parquet token (GH#10411) James Bourbeau

• Category dtype is lost when concatenating MultiIndex (GH#10407) Irina Truong

• Fix FutureWarning: The provided callable... (GH#10405) Irina Truong

• Enable non-categorical hive-partition columns in read_parquet (GH#10353) Richard (Rick) Zamora

• concat ignoring DataFrame withouth columns (GH#10359) Patrick Hoefler

2023.7.0

Released on July 7, 2023

Enhancements

• Catch exceptions when attempting to load CLI entry points (GH#10380) Jacob Tomlinson

Bug Fixes

• Fix typo in _clean_ipython_traceback (GH#10385) Alexander Clausen

• Ensure that df is immutable after from_pandas (GH#10383) Patrick Hoefler

• Warn consistently for inplace in Series.rename (GH#10313) Patrick Hoefler

Documentation

• Add clarification about output shape and reshaping in rechunk documentation (GH#10377) Swayam Patil

Maintenance

• Simplify astype implementation (GH#10393) Patrick Hoefler

• Fix test_first_and_last to accommodate deprecated last (GH#10373) James Bourbeau

• Add level to create_merge_tree (GH#10391) Patrick Hoefler

• Do not derive from scipy.stats.chisquare docstring (GH#10382) Doug Davis

2023.6.1

Released on June 26, 2023

Enhancements

• Remove no longer supported clip_lower and clip_upper (GH#10371) Patrick Hoefler

• Support DataFrame.set_index(..., sort=False) (GH#10342) Miles

• Cleanup remote tracebacks (GH#10354) Irina Truong

• Add dispatching mechanisms for pyarrow.Table conversion (GH#10312) Richard (Rick) Zamora

• Choose P2P even if fusion is enabled (GH#10344) Hendrik Makait

• Validate that rechunking is possible earlier in graph generation (GH#10336) Hendrik Makait

Bug Fixes

• Fix issue with header passed to read_csv (GH#10355) GALI PREM SAGAR

• Respect dropna and observed in GroupBy.var and GroupBy.std (GH#10350) Patrick Hoefler

• Fix H5FD_lock error when writing to hdf with distributed client (GH#10309) Irina Truong

• Fix for total_mem_usage of bag.map() (GH#10341) Irina Truong

Deprecations

• Deprecate DataFrame.fillna/Series.fillna with method (GH#10349) Irina Truong

• Deprecate DataFrame.first and Series.first (GH#10352) Irina Truong

Maintenance

• Deprecate numpy.compat (GH#10370) Irina Truong

• Fix annotations and spans leaking between threads (GH#10367) Irina Truong

• Use general kwargs in pyarrow_table_dispatch functions (GH#10364) Richard (Rick) Zamora

• Remove unnecessary try/except in isna (GH#10363) Patrick Hoefler

• mypy support for numpy 1.25 (GH#10362) crusaderky

• Bump actions/checkout from 3.5.2 to 3.5.3 (GH#10348)

• Restore numba in upstream build (GH#10330) James Bourbeau

• Update nightly wheel index for pandas/numpy/scipy (GH#10346) Matthew Roeschke

• Add rechunk config values to yaml (GH#10343) Hendrik Makait

2023.6.0

Released on June 9, 2023

Enhancements

• Add missing not in predicate support to read_parquet (GH#10320) Richard (Rick) Zamora

Bug Fixes

• Fix for incorrect value_counts (GH#10323) Irina Truong

• Update empty describe top and freq values (GH#10319) James Bourbeau

Documentation

• Fix hetzner typo (GH#10332) Sarah Charlotte Johnson

Maintenance

• Test with numba and sparse on Python 3.11 (GH#10329) Thomas Grainger

• Remove numpy.find_common_type warning ignore (GH#10311) James Bourbeau

• Update gpuCI RAPIDS_VER to 23.08 (GH#10310)

2023.5.1

Released on May 26, 2023

Note

This release drops support for Python 3.8. As of this release Dask supports Python 3.9, 3.10, and 3.11. See this community issue for more details.

Enhancements

• Drop Python 3.8 support (GH#10295) Thomas Grainger

• Change Dask Bag partitioning scheme to improve cluster saturation (GH#10294) Jacob Tomlinson

• Generalize dd.to_datetime for GPU-backed collections, introduce get_meta_library utility (GH#9881) Charles Blackmon-Luca

• Add na_action to DataFrame.map (GH#10305) Patrick Hoefler

• Raise TypeError in DataFrame.nsmallest and DataFrame.nlargest when columns is not given (GH#10301) Patrick Hoefler

• Improve sizeof for pd.MultiIndex (GH#10230) Patrick Hoefler

• Support duplicated columns in a bunch of DataFrame methods (GH#10261) Patrick Hoefler

• Add numeric_only support to DataFrame.idxmin and DataFrame.idxmax (GH#10253) Patrick Hoefler

• Implement numeric_only support for DataFrame.quantile (GH#10259) Patrick Hoefler

• Add support for numeric_only=False in DataFrame.std (GH#10251) Patrick Hoefler

• Implement numeric_only=False for GroupBy.cumprod and GroupBy.cumsum (GH#10262) Patrick Hoefler

• Implement numeric_only for skew and kurtosis (GH#10258) Patrick Hoefler

• mask and where should accept a callable (GH#10289) Irina Truong

• Fix conversion from Categorical to pa.dictionary in read_parquet (GH#10285) Patrick Hoefler

Bug Fixes

• Spurious config on nested annotations (GH#10318) crusaderky

• Fix rechunking behavior for dimensions with known and unknown chunk sizes (GH#10157) Hendrik Makait

• Enable drop to support mismatched partitions (GH#10300) James Bourbeau

• Fix divisions construction for to_timestamp (GH#10304) Patrick Hoefler

• pandas ExtensionDtype raising in Series reduction operations (GH#10149) Patrick Hoefler

• Fix regression in da.random interface (GH#10247) Eray Aslan

• da.coarsen doesn’t trim an empty chunk in meta (GH#10281) Irina Truong

• Fix dtype inference for engine="pyarrow" in read_csv (GH#10280) Patrick Hoefler

Documentation

• Add meta_from_array to API docs (GH#10306) Ruth Comer

• Update Coiled links (GH#10296) Sarah Charlotte Johnson

• Add docs for demo day (GH#10288) Matthew Rocklin

Maintenance

• Explicitly install anaconda-client from conda-forge when uploading conda nightlies (GH#10316) Charles Blackmon-Luca

• Configure isort to add from __future__ import annotations (GH#10314) Thomas Grainger

• Avoid pandas Series.__getitem__ deprecation in tests (GH#10308) James Bourbeau

• Ignore numpy.find_common_type warning from pandas (GH#10307) James Bourbeau

• Add test to check that DataFrame.__setitem__ does not modify df inplace (GH#10223) Patrick Hoefler

• Clean up default value of dropna in value_counts (GH#10299) Patrick Hoefler

• Add pytest-cov to test extra (GH#10271) James Bourbeau

2023.5.0

Released on May 12, 2023

Enhancements

• Implement numeric_only=False for GroupBy.corr and GroupBy.cov (GH#10264) Patrick Hoefler

• Add support for numeric_only=False in DataFrame.var (GH#10250) Patrick Hoefler

• Add numeric_only support to DataFrame.mode (GH#10257) Patrick Hoefler

• Add DataFrame.map to dask.DataFrame API (GH#10246) Patrick Hoefler

• Adjust for DataFrame.applymap deprecation and all NA concat behaviour change (GH#10245) Patrick Hoefler

• Enable numeric_only=False for DataFrame.count (GH#10234) Patrick Hoefler

• Disallow array input in mask/where (GH#10163) Irina Truong

• Support numeric_only=True in GroupBy.corr and GroupBy.cov (GH#10227) Patrick Hoefler

• Add numeric_only support to GroupBy.median (GH#10236) Patrick Hoefler

• Support mimesis=9 in dask.datasets (GH#10241) James Bourbeau

• Add numeric_only support to min, max and prod (GH#10219) Patrick Hoefler

• Add numeric_only=True support for GroupBy.cumsum and GroupBy.cumprod (GH#10224) Patrick Hoefler

• Add helper to unpack numeric_only keyword (GH#10228) Patrick Hoefler

Bug Fixes

• Fix clone + from_array failure (GH#10211) crusaderky

• Fix dataframe reductions for ea dtypes (GH#10150) Patrick Hoefler

• Avoid scalar conversion deprecation warning in numpy=1.25 (GH#10248) James Bourbeau

• Make sure transform output has the same index as input (GH#10184) Irina Truong

• Fix corr and cov on a single-row partition (GH#9756) Irina Truong

• Fix test_groupby_numeric_only_supported and test_groupby_aggregate_categorical_observed upstream errors (GH#10243) Irina Truong

Documentation

• Clean up futures docs (GH#10266) Matthew Rocklin

• Add Index API reference (GH#10263) hotpotato

Maintenance

• Warn when meta is passed to apply (GH#10256) Patrick Hoefler

• Remove imageio version restriction in CI (GH#10260) Patrick Hoefler

• Remove unused DataFrame variance methods (GH#10252) Patrick Hoefler

• Un-xfail test_categories with pyarrow strings and pyarrow>=12 (GH#10244) Irina Truong

• Bump gpuCI PYTHON_VER 3.8->3.9 (GH#10233) Charles Blackmon-Luca

2023.4.1

Released on April 28, 2023

Enhancements

• Implement numeric_only support for DataFrame.sum (GH#10194) Patrick Hoefler

• Add support for numeric_only=True in GroupBy operations (GH#10222) Patrick Hoefler

• Avoid deep copy in DataFrame.__setitem__ for pandas 1.4 and up (GH#10221) Patrick Hoefler

• Avoid calling Series.apply with _meta_nonempty (GH#10212) Patrick Hoefler

• Unpin sqlalchemy and fix compatibility issues (GH#10140) Patrick Hoefler

Bug Fixes

• Partially revert default client discovery (GH#10225) Florian Jetter

• Support arrow dtypes in Index meta creation (GH#10170) Patrick Hoefler

• Repartitioning raises with extension dtype when truncating floats (GH#10169) Patrick Hoefler

• Adjust empty Index from fastparquet to object dtype (GH#10179) Patrick Hoefler

Documentation

• Update Kubernetes docs (GH#10232) Jacob Tomlinson

• Add DataFrame.reduction to API docs (GH#10229) James Bourbeau

• Add DataFrame.persist to docs and fix links (GH#10231) Patrick Hoefler

• Add documentation for GroupBy.transform (GH#10185) Irina Truong

• Fix formatting in random number generation docs (GH#10189) Eray Aslan

Maintenance

• Pin imageio to <2.28 (GH#10216) Patrick Hoefler

• Add note about importlib_metadata backport (GH#10207) James Bourbeau

• Add xarray back to Python 3.11 CI builds (GH#10200) James Bourbeau

• Add mindeps build with all optional dependencies (GH#10161) Charles Blackmon-Luca

• Provide proper like value for array_safe in percentiles_summary (GH#10156) Charles Blackmon-Luca

• Avoid re-opening hdf file multiple times in read_hdf (GH#10205) Thomas Grainger

• Add merge tests on nullable columns (GH#10071) Charles Blackmon-Luca

• Fix coverage configuration (GH#10203) Thomas Grainger

• Remove is_period_dtype and is_sparse_dtype (GH#10197) Patrick Hoefler

• Bump actions/checkout from 3.5.0 to 3.5.2 (GH#10201)

• Avoid deprecated is_categorical_dtype from pandas (GH#10180) Patrick Hoefler

• Adjust for deprecated is_interval_dtype and is_datetime64tz_dtype (GH#10188) Patrick Hoefler

2023.4.0

Released on April 14, 2023

Enhancements

• Override old default values in update_defaults (GH#10159) Gabe Joseph

• Add a CLI command to list and get a value from dask config (GH#9936) Irina Truong

• Handle string-based engine argument to read_json (GH#9947) Richard (Rick) Zamora

• Avoid deprecated GroupBy.dtypes (GH#10111) Irina Truong

Bug Fixes

• Revert grouper-related changes (GH#10182) Irina Truong

• GroupBy.cov raising for non-numeric grouping column (GH#10171) Patrick Hoefler

• Updates for Index supporting numpy numeric dtypes (GH#10154) Irina Truong

• Preserve dtype for partitioning columns when read with pyarrow (GH#10115) Patrick Hoefler

• Fix annotations for to_hdf (GH#10123) Hendrik Makait

• Handle None column name when checking if columns are all numeric (GH#10128) Lawrence Mitchell

• Fix valid_divisions when passed a tuple (GH#10126) Brian Phillips

• Maintain annotations in DataFrame.categorize (GH#10120) Hendrik Makait

• Fix handling of missing min/max parquet statistics during filtering (GH#10042) Richard (Rick) Zamora

Deprecations

• Deprecate use_nullable_dtypes= and add dtype_backend= (GH#10076) Irina Truong

• Deprecate convert_dtype in Series.apply (GH#10133) Irina Truong

Documentation

• Document Generator based random number generation (GH#10134) Eray Aslan

Maintenance

• Update dataframe.convert_string to dataframe.convert-string (GH#10191) Irina Truong

• Add python-cityhash to CI environments (GH#10190) Charles Blackmon-Luca

• Temporarily pin scikit-image to fix Windows CI (GH#10186) Patrick Hoefler

• Handle pandas deprecation warnings for to_pydatetime and apply (GH#10168) Patrick Hoefler

• Drop bokeh<3 restriction (GH#10177) James Bourbeau

• Fix failing tests under copy-on-write (GH#10173) Patrick Hoefler

• Allow pyarrow CI to fail (GH#10176) James Bourbeau

• Switch to Generator for random number generation in dask.array (GH#10003) Eray Aslan

• Bump peter-evans/create-pull-request from 4 to 5 (GH#10166)

• Fix flaky modf operation in test_arithmetic (GH#10162) Irina Truong

• Temporarily remove xarray from CI with pandas 2.0 (GH#10153) James Bourbeau

• Fix update_graph counting logic in test_default_scheduler_on_worker (GH#10145) James Bourbeau

• Fix documentation build with pandas 2.0 (GH#10138) James Bourbeau

• Remove dask/gpu from gpuCI update reviewers (GH#10135) Charles Blackmon-Luca

• Update gpuCI RAPIDS_VER to 23.06 (GH#10129)

• Bump actions/stale from 6 to 8 (GH#10121)

• Use declarative setuptools (GH#10102) Thomas Grainger

• Relax assert_eq checks on Scalar-like objects (GH#10125) Matthew Rocklin

• Upgrade readthedocs config to ubuntu 22.04 and Python 3.11 (GH#10124) Thomas Grainger

• Bump actions/checkout from 3.4.0 to 3.5.0 (GH#10122)

• Fix test_null_partition_pyarrow in pyarrow CI build (GH#10116) Irina Truong

• Drop distributed pack (GH#9988) Florian Jetter

• Make dask.compatibility private (GH#10114) Jacob Tomlinson

2023.3.2

Released on March 24, 2023

Enhancements

• Deprecate observed=False for groupby with categoricals (GH#10095) Irina Truong

• Deprecate axis= for some groupby operations (GH#10094) James Bourbeau

• The axis keyword in DataFrame.rolling/Series.rolling is deprecated (GH#10110) Irina Truong

• DataFrame._data deprecation in pandas (GH#10081) Irina Truong

• Use importlib_metadata backport to avoid CLI UserWarning (GH#10070) Thomas Grainger

• Port option parsing logic from dask.dataframe.read_parquet to to_parquet (GH#9981) Anton Loukianov

Bug Fixes

• Avoid using dd.shuffle in groupby-apply (GH#10043) Richard (Rick) Zamora

• Enable null hive partitions with pyarrow parquet engine (GH#10007) Richard (Rick) Zamora

• Support unknown shapes in *_like functions (GH#10064) Doug Davis

Documentation

• Add to_backend methods to API docs (GH#10093) Lawrence Mitchell

• Remove broken gpuCI link in developer docs (GH#10065) Charles Blackmon-Luca

Maintenance

• Configure readthedocs sphinx warnings as errors (GH#10104) Thomas Grainger

• Un-xfail test_division_or_partition with pyarrow strings active (GH#10108) Irina Truong

• Un-xfail test_different_columns_are_allowed with pyarrow strings active (GH#10109) Irina Truong

• Restore Entrypoints compatibility (GH#10113) Jacob Tomlinson

• Un-xfail test_to_dataframe_optimize_graph with pyarrow strings active (GH#10087) Irina Truong

• Only run test_development_guidelines_matches_ci on editable install (GH#10106) Charles Blackmon-Luca

• Un-xfail test_dataframe_cull_key_dependencies_materialized with pyarrow strings active (GH#10088) Irina Truong

• Install mimesis in CI environments (GH#10105) Charles Blackmon-Luca

• Fix for no module named ipykernel (GH#10101) Irina Truong

• Fix docs builds by installing ipykernel (GH#10103) Thomas Grainger

• Allow pyarrow build to continue on failures (GH#10097) James Bourbeau

• Bump actions/checkout from 3.3.0 to 3.4.0 (GH#10096)

• Fix test_set_index_on_empty with pyarrow strings active (GH#10054) Irina Truong

• Un-xfail pyarrow pickling tests (GH#10082) James Bourbeau

• CI environment file cleanup (GH#10078) James Bourbeau

• Un-xfail more pyarrow tests (GH#10066) Irina Truong

• Temporarily skip pyarrow_compat tests with p`andas 2.0 (GH#10063) James Bourbeau

• Fix test_melt with pyarrow strings active (GH#10052) Irina Truong

• Fix test_str_accessor with pyarrow strings active (GH#10048) James Bourbeau

• Fix test_better_errors_object_reductions with pyarrow strings active (GH#10051) James Bourbeau

• Fix test_loc_with_non_boolean_series with pyarrow strings active (GH#10046) James Bourbeau

• Fix test_values with pyarrow strings active (GH#10050) James Bourbeau

• Temporarily xfail test_upstream_packages_installed (GH#10047) James Bourbeau

2023.3.1

Released on March 10, 2023

Enhancements

• Support pyarrow strings in MultiIndex (GH#10040) Irina Truong

• Improved support for pyarrow strings (GH#10000) Irina Truong

• Fix flaky RuntimeWarning during array reductions (GH#10030) James Bourbeau

• Extend complete extras (GH#10023) James Bourbeau

• Raise an error with dataframe.convert-string=True and pandas<2.0 (GH#10033) Irina Truong

• Rename shuffle/rechunk config option/kwarg to method (GH#10013) James Bourbeau

• Add initial support for converting pandas extension dtypes to arrays (GH#10018) James Bourbeau

• Remove randomgen support (GH#9987) Eray Aslan

Bug Fixes

• Skip rechunk when rechunking to the same chunks with unknown sizes (GH#10027) Hendrik Makait

• Custom utility to convert parquet filters to pyarrow expression (GH#9885) Richard (Rick) Zamora

• Consider numpy scalars and 0d arrays as scalars when padding (GH#9653) Justus Magin

• Fix parquet overwrite behavior after an adaptive read_parquet operation (GH#10002) Richard (Rick) Zamora

Documentation

• Add and update docs for Data Transfer section (GH#10022) Miles

Maintenance

• Remove stale hive-partitioning code from pyarrow parquet engine (GH#10039) Richard (Rick) Zamora

• Increase minimum supported pyarrow to 7.0 (GH#10024) James Bourbeau

• Revert “Prepare drop packunpack (GH#9994) (GH#10037) Florian Jetter

• Have codecov wait for more builds before reporting (GH#10031) James Bourbeau

• Prepare drop packunpack (GH#9994) Florian Jetter

• Add CI job with pyarrow strings turned on (GH#10017) James Bourbeau

• Fix test_groupby_dropna_with_agg for pandas 2.0 (GH#10001) Irina Truong

• Fix test_pickle_roundtrip for pandas 2.0 (GH#10011) James Bourbeau

2023.3.0

Released on March 1, 2023

Bug Fixes

• Bag must not pick p2p as shuffle default (GH#10005) Florian Jetter

Documentation

• Minor follow-up to P2P by default (GH#10008) James Bourbeau

Maintenance

• Add minimum version to optional jinja2 dependency (GH#9999) Charles Blackmon-Luca

2023.2.1

Released on February 24, 2023

Note

This release changes the default DataFrame shuffle algorithm to p2p to improve stability and performance. Learn more here and please provide any feedback on this discussion.

If you encounter issues with this new algorithm, please see the documentation for more information, and how to switch back to the old mode.

Enhancements

• Enable P2P shuffling by default (GH#9991) Florian Jetter

• P2P rechunking (GH#9939) Hendrik Makait

• Efficient dataframe.convert-string support for read_parquet (GH#9979) Irina Truong

• Allow p2p shuffle kwarg for DataFrame merges (GH#9900) Florian Jetter

• Change split_row_groups default to “infer” (GH#9637) Richard (Rick) Zamora

• Add option for converting string data to use pyarrow strings (GH#9926) James Bourbeau

• Add support for multi-column sort_values (GH#8263) Charles Blackmon-Luca

• Generator based random-number generation in``dask.array`` (GH#9038) Eray Aslan

• Support numeric_only for simple groupby aggregations for pandas 2.0 compatibility (GH#9889) Irina Truong

Bug Fixes

• Fix profilers plot not being aligned to context manager enter time (GH#9739) David Hoese

• Relax dask.dataframe assert_eq type checks (GH#9989) Matthew Rocklin

• Restore describe compatibility for pandas 2.0 (GH#9982) James Bourbeau

Documentation

• Improving deploying Dask docs (GH#9912) Sarah Charlotte Johnson

• More docs for DataFrame.partitions (GH#9976) Tom Augspurger

• Update docs with more information on default Delayed scheduler (GH#9903) Guillaume Eynard-Bontemps

• Deployment Considerations documentation (GH#9933) Gabe Joseph

Maintenance

• Temporarily rerun flaky tests (GH#9983) James Bourbeau

• Update parsing of FULL_RAPIDS_VER/FULL_UCX_PY_VER (GH#9990) Charles Blackmon-Luca

• Increase minimum supported versions to pandas=1.3 and numpy=1.21 (GH#9950) James Bourbeau

• Fix std to work with numeric_only for pandas 2.0 (GH#9960) Irina Truong

• Temporarily xfail test_roundtrip_partitioned_pyarrow_dataset (GH#9977) James Bourbeau

• Fix copy on write failure in test_idxmaxmin (GH#9944) Patrick Hoefler

• Bump pre-commit versions (GH#9955) crusaderky

• Fix test_groupby_unaligned_index for pandas 2.0 (GH#9963) Irina Truong

• Un-xfail test_set_index_overlap_2 for pandas 2.0 (GH#9959) James Bourbeau

• Fix test_merge_by_index_patterns for pandas 2.0 (GH#9930) Irina Truong

• Bump jacobtomlinson/gha-find-replace from 2 to 3 (GH#9953) James Bourbeau

• Fix test_rolling_agg_aggregate for pandas 2.0 compatibility (GH#9948) Irina Truong

• Bump black to 23.1.0 (GH#9956) crusaderky

• Run GPU tests on python 3.8 & 3.10 (GH#9940) Charles Blackmon-Luca

• Fix test_to_timestamp for pandas 2.0 (GH#9932) Irina Truong

• Fix an error with groupby value_counts for pandas 2.0 compatibility (GH#9928) Irina Truong

• Config converter: replace all dashes with underscores (GH#9945) Jacob Tomlinson

• CI: use nightly wheel to install pyarrow in upstream test build (GH#9873) Joris Van den Bossche

2023.2.0

Released on February 10, 2023

Enhancements

• Update numeric_only default in quantile for pandas 2.0 (GH#9854) Irina Truong

• Make repartition a no-op when divisions match (GH#9924) James Bourbeau

• Update datetime_is_numeric behavior in describe for pandas 2.0 (GH#9868) Irina Truong

• Update value_counts to return correct name in pandas 2.0 (GH#9919) Irina Truong

• Support new axis=None behavior in pandas 2.0 for certain reductions (GH#9867) James Bourbeau

• Filter out all-nan RuntimeWarning at the chunk level for nanmin and nanmax (GH#9916) Julia Signell

• Fix numeric meta_nonempty index creation for pandas 2.0 (GH#9908) James Bourbeau

• Fix DataFrame.info() tests for pandas 2.0 (GH#9909) James Bourbeau

Bug Fixes

• Fix GroupBy.value_counts handling for multiple groupby columns (GH#9905) Charles Blackmon-Luca

Documentation

• Fix some outdated information/typos in development guide (GH#9893) Patrick Hoefler

• Add note about keep=False in drop_duplicates docstring (GH#9887) Jayesh Manani

• Add meta details to dask Array (GH#9886) Jayesh Manani

• Clarify task stream showing more rows than threads (GH#9906) Gabe Joseph

Maintenance

• Fix test_numeric_column_names for pandas 2.0 (GH#9937) Irina Truong

• Fix dask/dataframe/tests/test_utils_dataframe.py tests for pandas 2.0 (GH#9788) James Bourbeau

• Replace index.is_numeric with is_any_real_numeric_dtype for pandas 2.0 compatibility (GH#9918) Irina Truong

• Avoid pd.core import in dask utils (GH#9907) Matthew Roeschke

• Use label for upstream build on pull requests (GH#9910) James Bourbeau

• Broaden exception catching for sqlalchemy.exc.RemovedIn20Warning (GH#9904) James Bourbeau

• Temporarily restrict sqlalchemy < 2 in CI (GH#9897) James Bourbeau

• Update isort version to 5.12.0 (GH#9895) Lawrence Mitchell

• Remove unused skiprows variable in read_csv (GH#9892) Patrick Hoefler

2023.1.1

Released on January 27, 2023

Enhancements

• Add to_backend method to Array and _Frame (GH#9758) Richard (Rick) Zamora

• Small fix for timestamp index divisions in pandas 2.0 (GH#9872) Irina Truong

• Add numeric_only to DataFrame.cov and DataFrame.corr (GH#9787) James Bourbeau

• Fixes related to group_keys default change in pandas 2.0 (GH#9855) Irina Truong

• infer_datetime_format compatibility for pandas 2.0 (GH#9783) James Bourbeau

Bug Fixes

• Fix serialization bug in BroadcastJoinLayer (GH#9871) Richard (Rick) Zamora

• Satisfy broadcast argument in DataFrame.merge (GH#9852) Richard (Rick) Zamora

• Fix pyarrow parquet columns statistics computation (GH#9772) aywandji

Documentation

• Fix “duplicate explicit target name” docs warning (GH#9863) Chiara Marmo

• Fix code formatting issue in “Defining a new collection backend” docs (GH#9864) Chiara Marmo

• Update dashboard documentation for memory plot (GH#9768) Jayesh Manani

• Add docs section about no-worker tasks (GH#9839) Florian Jetter

Maintenance

• Additional updates for detecting a distributed scheduler (GH#9890) James Bourbeau

• Update gpuCI RAPIDS_VER to 23.04 (GH#9876)

• Reverse precedence between collection and distributed default (GH#9869) Florian Jetter

• Update xarray-contrib/issue-from-pytest-log to version 1.2.6 (GH#9865) James Bourbeau

• Dont require dask config shuffle default (GH#9826) Florian Jetter

• Un-xfail datetime64 Parquet roundtripping tests for new fastparquet (GH#9811) James Bourbeau

• Add option to manually run upstream CI build (GH#9853) James Bourbeau

• Use custom timeout in CI builds (GH#9844) James Bourbeau

• Remove kwargs from make_blockwise_graph (GH#9838) Florian Jetter

• Ignore warnings on persist call in test_setitem_extended_API_2d_mask (GH#9843) Charles Blackmon-Luca

• Fix running S3 tests locally (GH#9833) James Bourbeau

2023.1.0

Released on January 13, 2023

Enhancements

• Use distributed default clients even if no config is set (GH#9808) Florian Jetter

• Implement ma.where and ma.nonzero (GH#9760) Erik Holmgren

• Update zarr store creation functions (GH#9790) Ryan Abernathey

• iteritems compatibility for pandas 2.0 (GH#9785) James Bourbeau

• Accurate sizeof for pandas string[python] dtype (GH#9781) crusaderky

• Deflate sizeof() of duplicate references to pandas object types (GH#9776) crusaderky

• GroupBy.__getitem__ compatibility for pandas 2.0 (GH#9779) James Bourbeau

• append compatibility for pandas 2.0 (GH#9750) James Bourbeau

• get_dummies compatibility for pandas 2.0 (GH#9752) James Bourbeau

• is_monotonic compatibility for pandas 2.0 (GH#9751) James Bourbeau

• numpy=1.24 compatability (GH#9777) James Bourbeau

Documentation

• Remove duplicated encoding kwarg in docstring for to_json (GH#9796) Sultan Orazbayev

• Mention SubprocessCluster in LocalCluster documentation (GH#9784) Hendrik Makait

• Move Prometheus docs to dask/distributed (GH#9761) crusaderky

Maintenance

• Temporarily ignore RuntimeWarning in test_setitem_extended_API_2d_mask (GH#9828) James Bourbeau

• Fix flaky test_threaded.py::test_interrupt (GH#9827) Hendrik Makait

• Update xarray-contrib/issue-from-pytest-log in upstream report (GH#9822) James Bourbeau

• pip install dask on gpuCI builds (GH#9816) Charles Blackmon-Luca

• Bump actions/checkout from 3.2.0 to 3.3.0 (GH#9815)

• Resolve sqlalchemy import failures in mindeps testing (GH#9809) Charles Blackmon-Luca

• Ignore sqlalchemy.exc.RemovedIn20Warning (GH#9801) Thomas Grainger

• xfail datetime64 Parquet roundtripping tests for pandas 2.0 (GH#9786) James Bourbeau

• Remove sqlachemy 1.3 compatibility (GH#9695) McToel

• Reduce size of expected DoK sparse matrix (GH#9775) Elliott Sales de Andrade

• Remove executable flag from dask/dataframe/io/orc/utils.py (GH#9774) Elliott Sales de Andrade

2022.12.1

Released on December 16, 2022

Enhancements

• Support dtype_backend="pandas|pyarrow" configuration (GH#9719) James Bourbeau

• Support cupy.ndarray to cudf.DataFrame dispatching in dask.dataframe (GH#9579) Richard (Rick) Zamora

• Make filesystem-backend configurable in read_parquet (GH#9699) Richard (Rick) Zamora

• Serialize all pyarrow extension arrays efficiently (GH#9740) James Bourbeau

Bug Fixes

• Fix bug when repartitioning with tz-aware datetime index (GH#9741) James Bourbeau

• Partial functions in aggs may have arguments (GH#9724) Irina Truong

• Add support for simple operation with pyarrow-backed extension dtypes (GH#9717) James Bourbeau

• Rename columns correctly in case of SeriesGroupby (GH#9716) Lawrence Mitchell

Documentation

• Fix url link typo in collection backend doc (GH#9748) Shawn

• Update Prometheus docs (GH#9696) Hendrik Makait

Maintenance

• Add zarr to Python 3.11 CI environment (GH#9771) James Bourbeau

• Add support for Python 3.11 (GH#9708) Thomas Grainger

• Bump actions/checkout from 3.1.0 to 3.2.0 (GH#9753)

• Avoid np.bool8 deprecation warning (GH#9737) James Bourbeau

• Make sure dev packages aren’t overwritten in upstream CI build (GH#9731) James Bourbeau

• Avoid adding data.h5 and mydask.html files during tests (GH#9726) Thomas Grainger

2022.12.0

Released on December 2, 2022

Enhancements

• Remove statistics-based set_index logic from read_parquet (GH#9661) Richard (Rick) Zamora

• Add support for use_nullable_dtypes to dd.read_parquet (GH#9617) Ian Rose

• Fix map_overlap in order to accept pandas arguments (GH#9571) Fabien Aulaire

• Fix pandas 1.5+ FutureWarning in .str.split(..., expand=True) (GH#9704) Jacob Hayes

• Enable column projection for groupby slicing (GH#9667) Richard (Rick) Zamora

• Support duplicate column cum-functions (GH#9685) Ben

• Improve error message for failed backend dispatch call (GH#9677) Richard (Rick) Zamora

Bug Fixes

• Revise meta creation in arrow parquet engine (GH#9672) Richard (Rick) Zamora

• Fix da.fft.fft for array-like inputs (GH#9688) James Bourbeau

• Fix groupby -aggregation when grouping on an index by name (GH#9646) Richard (Rick) Zamora

Maintenance

• Avoid PytestReturnNotNoneWarning in test_inheriting_class (GH#9707) Thomas Grainger

• Fix flaky test_dataframe_aggregations_multilevel (GH#9701) Richard (Rick) Zamora

• Bump mypy version (GH#9697) crusaderky

• Disable dashboard in test_map_partitions_df_input (GH#9687) James Bourbeau

• Use latest xarray-contrib/issue-from-pytest-log in upstream build (GH#9682) James Bourbeau

• xfail ttest_1samp for upstream scipy (GH#9670) James Bourbeau

• Update gpuCI RAPIDS_VER to 23.02 (GH#9678)

2022.11.1

Released on November 18, 2022

Enhancements

• Restrict bokeh=3 support (GH#9673) Gabe Joseph

• Updates for fastparquet evolution (GH#9650) Martin Durant

Maintenance

• Update ga-yaml-parser step in gpuCI updating workflow (GH#9675) Charles Blackmon-Luca

• Revert importlib.metadata workaround (GH#9658) James Bourbeau

• Fix mindeps-distributed CI build to handle numpy/pandas not being installed (GH#9668) James Bourbeau

2022.11.0

Released on November 15, 2022

Enhancements

• Generalize from_dict implementation to allow usage from other backends (GH#9628) GALI PREM SAGAR

Bug Fixes

• Avoid pandas constructors in dask.dataframe.core (GH#9570) Richard (Rick) Zamora

• Fix sort_values with Timestamp data (GH#9642) James Bourbeau

• Generalize array checking and remove pd.Index call in _get_partitions (GH#9634) Benjamin Zaitlen

• Fix read_csv behavior for header=0 and names (GH#9614) Richard (Rick) Zamora

Documentation

• Update dashboard docs for queuing (GH#9660) Gabe Joseph

• Remove import dask as d from docstrings (GH#9644) Matthew Rocklin

• Fix link to partitions docs in read_parquet docstring (GH#9636) qheuristics

• Add API doc links to array/bag/dataframe sections (GH#9630) Matthew Rocklin

Maintenance

• Use conda-incubator/setup-miniconda@v2.2.0 (GH#9662) John A Kirkham

• Allow bokeh=3 (GH#9659) James Bourbeau

• Run upstream build with Python 3.10 (GH#9655) James Bourbeau

• Pin pyyaml version in mindeps testing (GH#9640) Charles Blackmon-Luca

• Add pre-commit to catch breakpoint() (GH#9638) James Bourbeau

• Bump xarray-contrib/issue-from-pytest-log from 1.1 to 1.2 (GH#9635)

• Remove blosc references (GH#9625) Naty Clementi

• Upgrade mypy and drop unused comments (GH#9616) Hendrik Makait

• Harden test_repartition_npartitions (GH#9585) Richard (Rick) Zamora

2022.10.2

Released on October 31, 2022

This was a hotfix and has no changes in this repository. The necessary fix was in dask/distributed, but we decided to bump this version number for consistency.

2022.10.1

Released on October 28, 2022

Enhancements

• Enable named aggregation syntax (GH#9563) ChrisJar

• Add extension dtype support to set_index (GH#9566) James Bourbeau

• Redesigning the array HTML repr for clarity (GH#9519) Shingo OKAWA

Bug Fixes

• Fix merge with emtpy left DataFrame (GH#9578) Ian Rose

Documentation

• Add note about limiting thread oversubscription by default (GH#9592) James Bourbeau

• Use sphinx-click for dask CLI (GH#9589) James Bourbeau

• Fix Semaphore API docs (GH#9584) James Bourbeau

• Render meta description in map_overlap docstring (GH#9568) James Bourbeau

Maintenance

• Require Click 7.0+ in Dask (GH#9595) John A Kirkham

• Temporarily restrict bokeh<3 (GH#9607) James Bourbeau

• Resolve importlib-related failures in upstream CI (GH#9604) Charles Blackmon-Luca

• Improve upstream CI report (GH#9603) James Bourbeau

• Fix upstream CI report (GH#9602) James Bourbeau

• Remove setuptools host dep, add CLI entrypoint (GH#9600) Charles Blackmon-Luca

• More Backend dispatch class type annotations (GH#9573) Ian Rose

2022.10.0

Released on October 14, 2022

New Features

• Backend library dispatching for IO in Dask-Array and Dask-DataFrame (GH#9475) Richard (Rick) Zamora

• Add new CLI that is extensible (GH#9283) Doug Davis

Enhancements

• Groupby median (GH#9516) Ian Rose

• Fix array copy not being a no-op (GH#9555) David Hoese

• Add support for string timedelta in map_overlap (GH#9559) Nicolas Grandemange

• Shuffle-based groupby for single functions (GH#9504) Ian Rose

• Make datetime.datetime tokenize idempotantly (GH#9532) Martin Durant

• Support tokenizing datetime.time (GH#9528) Tim Paine

Bug Fixes

• Avoid race condition in lazy dispatch registration (GH#9545) James Bourbeau

• Do not allow setitem to np.nan for int dtype (GH#9531) Doug Davis

• Stable demo column projection (GH#9538) Ian Rose

• Ensure pickle-able binops in delayed (GH#9540) Ian Rose

• Fix project CSV columns when selecting (GH#9534) Martin Durant

Documentation

• Update Parquet best practice (GH#9537) Matthew Rocklin

Maintenance

• Restrict tiledb-py version to avoid CI failures (GH#9569) James Bourbeau

• Bump actions/github-script from 3 to 6 (GH#9564)

• Bump actions/stale from 4 to 6 (GH#9551)

• Bump peter-evans/create-pull-request from 3 to 4 (GH#9550)

• Bump actions/checkout from 2 to 3.1.0 (GH#9552)

• Bump codecov/codecov-action from 1 to 3 (GH#9549)

• Bump the-coding-turtle/ga-yaml-parser from 0.1.1 to 0.1.2 (GH#9553)

• Move dependabot configuration file (GH#9547) James Bourbeau

• Add dependabot for GitHub actions (GH#9542) James Bourbeau

• Run mypy on Windows and Linux (GH#9530) crusaderky

• Update gpuCI RAPIDS_VER to 22.12 (GH#9524)

2022.9.2

Released on September 30, 2022

Enhancements

• Remove factorization logic from array auto chunking (GH#9507) James Bourbeau

Documentation

• Add docs on running Dask in a standalone Python script (GH#9513) James Bourbeau

• Clarify custom-graph multiprocessing example (GH#9511) nouman

Maintenance

• Groupby sort upstream compatibility (GH#9486) Ian Rose

2022.9.1

Released on September 16, 2022

New Features

• Add DataFrame and Series median methods (GH#9483) James Bourbeau

Enhancements

• Shuffle groupby default (GH#9453) Ian Rose

• Filter by list (GH#9419) Greg Hayes

• Added distributed.utils.key_split functionality to dask.utils.key_split (GH#9464) Luke Conibear

Bug Fixes

• Fix overlap so that set_index doesn’t drop rows (GH#9423) Julia Signell

• Fix assigning pandas Series to column when ddf.columns.min() raises (GH#9485) Erik Welch

• Fix metadata comparison stack_partitions (GH#9481) James Bourbeau

• Provide default for split_out (GH#9493) Lawrence Mitchell

Deprecations

• Allow split_out to be None, which then defaults to 1 in groupby().aggregate() (GH#9491) Ian Rose

Documentation

• Fixing enforce_metadata documentation, not checking for dtypes (GH#9474) Nicolas Grandemange

• Fix it's –> its typo (GH#9484) Nat Tabris

Maintenance

• Workaround for parquet writing failure using some datetime series but not others (GH#9500) Ian Rose

• Filter out numeric_only warnings from pandas (GH#9496) James Bourbeau

• Avoid set_index(..., inplace=True) where not necessary (GH#9472) James Bourbeau

• Avoid passing groupby key list of length one (GH#9495) James Bourbeau

• Update test_groupby_dropna_cudf based on cudf support for group_keys (GH#9482) James Bourbeau

• Remove dd.from_bcolz (GH#9479) James Bourbeau

• Added flake8-bugbear to pre-commit hooks (GH#9457) Luke Conibear

• Bind loop variables in function definitions (B023) (GH#9461) Luke Conibear

• Added assert for comparisons (B015) (GH#9459) Luke Conibear

• Set top-level default shell in CI workflows (GH#9469) James Bourbeau

• Removed unused loop control variables (B007) (GH#9458) Luke Conibear

• Replaced getattr calls for constant attributes (B009) (GH#9460) Luke Conibear

• Pin libprotobuf to allow nightly pyarrow in the upstream CI build (GH#9465) Joris Van den Bossche

• Replaced mutable data structures for default arguments (B006) (GH#9462) Luke Conibear

• Changed flake8 mirror and updated version (GH#9456) Luke Conibear

2022.9.0

Released on September 2, 2022

Enhancements

• Enable automatic column projection for groupby aggregations (GH#9442) Richard (Rick) Zamora

• Accept superclasses in NEP-13/17 dispatching (GH#6710) Gabe Joseph

Bug Fixes

• Rename by columns internally for cumulative operations on the same by columns (GH#9430) Pavithra Eswaramoorthy

• Fix get_group with categoricals (GH#9436) Pavithra Eswaramoorthy

• Fix caching-related MaterializedLayer.cull performance regression (GH#9413) Richard (Rick) Zamora

Documentation

• Add maintainer documentation page (GH#9309) James Bourbeau

Maintenance

• Revert skipped fastparquet test (GH#9439) Pavithra Eswaramoorthy

• tmpfile does not end files with period on empty extension (GH#9429) Hendrik Makait

• Skip failing fastparquet test with latest release (GH#9432) James Bourbeau

2022.8.1

Released on August 19, 2022

New Features

• Implement ma.*_like functions (GH#9378) Ruth Comer

Enhancements

• Fuse compatible annotations (GH#9402) Ian Rose

• Shuffle-based groupby aggregation for high-cardinality groups (GH#9302) Richard (Rick) Zamora

• Unpack namedtuple (GH#9361) Hendrik Makait

Bug Fixes

• Fix SeriesGroupBy cumulative functions with axis=1 (GH#9377) Pavithra Eswaramoorthy

• Sparse array reductions (GH#9342) Ian Rose

• Fix make_meta while using categorical column with index (GH#9348) Pavithra Eswaramoorthy

• Don’t allow incompatible keywords in DataFrame.dropna (GH#9366) Naty Clementi

• Make set_index handle entirely empty dataframes (GH#8896) Julia Signell

• Improve dataclass handling in unpack_collections (GH#9345) Hendrik Makait

• Fix bag sampling when there are some smaller partitions (GH#9349) Ian Rose

• Add support for empty partitions to da.min/da.max functions (GH#9268) geraninam

Documentation

• Clarify that bind() etc. regenerate the keys (GH#9385) crusaderky

• Consolidate dashboard diagnostics documentation (GH#9357) Sarah Charlotte Johnson

• Remove outdated meta information Pavithra Eswaramoorthy

Maintenance

• Use entry_points utility in sizeof (GH#9390) James Bourbeau

• Add entry_points compatibility utility (GH#9388) Jacob Tomlinson

• Upload environment file artifact for each CI build (GH#9372) James Bourbeau

• Remove werkzeug pin in CI (GH#9371) James Bourbeau

• Fix type annotations for dd.from_pandas and dd.from_delayed (GH#9362) Jordan Yap

2022.8.0

Released on August 5, 2022

Enhancements

• Ensure make_meta doesn’t hold ref to data (GH#9354) Jim Crist-Harif

• Revise divisions logic in from_pandas (GH#9221) Richard (Rick) Zamora

• Warn if user sets index with existing index (GH#9341) Julia Signell

• Add keepdims keyword for da.average (GH#9332) Ruth Comer

• Change repr methods to avoid Layer materialization (GH#9289) Richard (Rick) Zamora

Bug Fixes

• Make sure order kwarg will not crash the astype method (GH#9317) Genevieve Buckley

• Fix bug for cumsum on cupy chunked dask arrays (GH#9320) Genevieve Buckley

• Match input and output structure in _sample_reduce (GH#9272) Pavithra Eswaramoorthy

• Include meta in array serialization (GH#9240) Frédéric BRIOL

• Fix Index.memory_usage (GH#9290) James Bourbeau

• Fix division calculation in dask.dataframe.io.from_dask_array (GH#9282) Jordan Yap

Documentation

• Fow to use kwargs with custom task graphs (GH#9322) Genevieve Buckley

• Add note to da.from_array about how the order is not preserved (GH#9346) Julia Signell

• Add I/O info for async functions (GH#9326) Logan Norman

• Tidy up docs snippet for futures IO functions (GH#9340) Julia Signell

• Use consistent variable names for pandas df and Dask ddf in dataframe-groupby.rst (GH#9304) ivojuroro

• Switch js-yaml for yaml.js in config converter (GH#9306) Jacob Tomlinson

Maintenance

• Update da.linalg.solve for SciPy 1.9.0 compatibility (GH#9350) Pavithra Eswaramoorthy

• Update test_getitem_avoids_large_chunks_missing (GH#9347) Pavithra Eswaramoorthy

• Fix docs title formatting for “Extend sizeof” Doug Davis

• Import loop_in_thread fixture in tests (GH#9337) James Bourbeau

• Temporarily xfail test_solve_sym_pos (GH#9336) Pavithra Eswaramoorthy

• Fix small typo in 10 minutes to Dask page (GH#9329) Shaghayegh

• Temporarily pin werkzeug in CI to avoid test suite hanging (GH#9325) James Bourbeau

• Add tests for cupy.angle() (GH#9312) Peter Andreas Entschev

• Update gpuCI RAPIDS_VER to 22.10 (GH#9314)

• Add pandas[test] to test extra (GH#9110) Ben Beasley

• Add bokeh and scipy to upstream CI build (GH#9265) James Bourbeau

2022.7.1

Released on July 22, 2022

Enhancements

• Return Dask array if all axes are squeezed (GH#9250) Pavithra Eswaramoorthy

• Make cycle reported by toposort shorter (GH#9068) Erik Welch

• Unknown chunk slicing - raise informative error (GH#9285) Naty Clementi

Bug Fixes

• Fix bug in HighLevelGraph.cull (GH#9267) Richard (Rick) Zamora

• Sort categories (GH#9264) Pavithra Eswaramoorthy

• Use max (instead of sum) for calculating warnsize (GH#9235) Pavithra Eswaramoorthy

• Fix bug when filtering on partitioned column with pyarrow (GH#9252) Richard (Rick) Zamora

Documentation

• Updated repartition documentation to add note about partition_size (GH#9288) Dylan Stewart

• Don’t include docs in Array methods, just refer to module docs (GH#9244) Julia Signell

• Remove outdated reference to scheduler and worker dashboards (GH#9278) Pavithra Eswaramoorthy

• Fix a few typos (GH#9270) Tim Gates

• Adds an custom aggregate example using numpy methods (GH#9260) geraninam

Maintenance

• Add type annotations to dd.from_pandas and dd.from_delayed (GH#9237) Michael Milton

• Update calculate_divisions docstring (GH#9275) Tom Augspurger

• Update test_plot_multiple for upcoming bokeh release (GH#9261) James Bourbeau

• Add typing to common array properties (GH#9255) Illviljan

2022.7.0

Released on July 8, 2022

Enhancements

• Support pathlib.PurePath in normalize_token (GH#9229) Angus Hollands

• Add AttributeNotImplementedError for properties so IPython glob search works (GH#9231) Erik Welch

• map_overlap: multiple dataframe handling (GH#9145) Fabien Aulaire

• Read entrypoints in dask.sizeof (GH#7688) Angus Hollands

Bug Fixes

• Fix TypeError: 'Serialize' object is not subscriptable when writing parquet dataset with Client(processes=False) (GH#9015) Lucas Miguel Ponce

• Correct dtypes when concat with an empty dataframe (GH#9193) Pavithra Eswaramoorthy

Documentation

• Highlight note about persist (GH#9234) Pavithra Eswaramoorthy

• Update release-procedure to include more detail and helpful commands (GH#9215) Julia Signell

• Better SEO for Futures and Dask vs. Spark pages (GH#9217) Sarah Charlotte Johnson

Maintenance

• Use math.prod instead of np.prod on lists, tuples, and iters (GH#9232) crusaderky

• Only import IPython if type checking (GH#9230) Florian Jetter

• Tougher mypy checks (GH#9206) crusaderky

2022.6.1

Released on June 24, 2022

Enhancements

• Dask in pyodide (GH#9053) Ian Rose

• Create dask.utils.show_versions (GH#9144) Sultan Orazbayev

• Better error message for unsupported numpy operations on dask.dataframe objects. (GH#9201) Julia Signell

• Add allow_rechunk kwarg to dask.array.overlap function (GH#7776) Genevieve Buckley

• Add minutes and hours to dask.utils.format_time (GH#9116) Matthew Rocklin

• More retries when writing parquet to remote filesystem (GH#9175) Ian Rose

Bug Fixes

• Timedelta deterministic hashing (GH#9213) Fabien Aulaire

• Enum deterministic hashing (GH#9212) Fabien Aulaire

• shuffle_group(): avoid converting to arrays (GH#9157) Mads R. B. Kristensen

Deprecations

• Deprecate extra format_time utility (GH#9184) James Bourbeau

Documentation

• Better SEO for 10 Minutes to Dask (GH#9182) Sarah Charlotte Johnson

• Better SEO for Delayed and Best Practices (GH#9194) Sarah Charlotte Johnson

• Include known inconsistency in DataFrame str.split accessor docstring (GH#9177) Richard Pelgrim

• Add inconsistencies keyword to derived_from (GH#9192) Richard Pelgrim

• Add missing append in delayed best practices example (GH#9202) Ben

• Fix indentation in Best Practices (GH#9196) Sarah Charlotte Johnson

• Add link to Genevieve Buckley’s blog on chunk sizes (GH#9199) Pavithra Eswaramoorthy

• Update to_csv docstring (GH#9094) Sarah Charlotte Johnson

Maintenance

• Update versioneer: change from using SafeConfigParser to ConfigParser (GH#9205) Thomas A Caswell

• Remove ipython hack in CI(GH#9200) crusaderky

2022.6.0

Released on June 10, 2022

Enhancements

• Add feature to show names of layer dependencies in HLG JupyterLab repr (GH#9081) Angelos Omirolis

• Add arrow schema extraction dispatch (GH#9169) GALI PREM SAGAR

• Add sort_results argument to assert_eq (GH#9130) Pavithra Eswaramoorthy

• Add weeks to parse_timedelta (GH#9168) Matthew Rocklin

• Warn that cloudpickle is not always deterministic (GH#9148) Pavithra Eswaramoorthy

• Switch parquet default engine (GH#9140) Jim Crist-Harif

• Use deterministic hashing with _iLocIndexer / _LocIndexer (GH#9108) Fabien Aulaire

• Enfore consistent schema in to_parquet pyarrow (GH#9131) Jim Crist-Harif

Bug Fixes

• Fix pyarrow.StringArray pickle (GH#9170) Jim Crist-Harif

• Fix parallel metadata collection in pyarrow engine (GH#9165) Richard (Rick) Zamora

• Improve pyarrow partitioning logic (GH#9147) James Bourbeau

• pyarrow 8.0 partitioning fix (GH#9143) James Bourbeau

Documentation

• Better SEO for Installing Dask and Dask DataFrame Best Practices (GH#9178) Sarah Charlotte Johnson

• Update logos page in docs (GH#9167) Sarah Charlotte Johnson

• Add example using pandas Series to map_partition doctring (GH#9161) Alex-JG3

• Update docs theme for rebranding (GH#9160) Sarah Charlotte Johnson

• Better SEO for docs on Dask DataFrames (GH#9128) Sarah Charlotte Johnson

Maintenance

• Remove ensure_file from recommended practice for downstream libraries (GH#9171) Matthew Rocklin

• Test round-tripping DataFrame parquet I/O including pyspark (GH#9156) Ian Rose

• Try disabling HDF5 locking (GH#9154) Ian Rose

• Link best practices to DataFrame-parquet (GH#9150) Tom Augspurger

• Fix typo in map_partitions func parameter description (GH#9149) Christopher Akiki

• Un-xfail test_groupby_grouper_dispatch (GH#9139) GALI PREM SAGAR

• Temporarily import cleanup fixture from distributed (GH#9138) James Bourbeau

• Simplify partitioning logic in pyarrow parquet engine (GH#9041) Richard (Rick) Zamora

2022.05.2

Released on May 26, 2022

Enhancements

• Add a dispatch for non-pandas Grouper objects and use it in GroupBy (GH#9074) brandon-b-miller

• Error if read_parquet & to_parquet files intersect (GH#9124) Jim Crist-Harif

• Visualize task graphs using ipycytoscape (GH#9091) Ian Rose

Documentation

• Fix various typos (GH#9126) Ryan Russell

Maintenance

• Fix flaky test_filter_nonpartition_columns (GH#9127) Pavithra Eswaramoorthy

• Update gpuCI RAPIDS_VER to 22.08 (GH#9120)

• Include conftest.py` in sdists (GH#9115) Ben Beasley

2022.05.1

Released on May 24, 2022

New Features

• Add DataFrame.from_dict classmethod (GH#9017) Matthew Powers

• Add from_map function to Dask DataFrame (GH#8911) Richard (Rick) Zamora

Enhancements

• Improve to_parquet error for appended divisions overlap (GH#9102) Jim Crist-Harif

• Enabled user-defined process-initializer functions (GH#9087) ParticularMiner

• Mention align_dataframes=False option in map_partitions error (GH#9075) Gabe Joseph

• Add kwarg enforce_ndim to dask.array.map_blocks() (GH#8865) ParticularMiner

• Implement Series.GroupBy.fillna / DataFrame.GroupBy.fillna methods (GH#8869) Pavithra Eswaramoorthy

• Allow fillna with Dask DataFrame (GH#8950) Pavithra Eswaramoorthy

• Update error message for assignment with 1-d dask array (GH#9036) Pavithra Eswaramoorthy

• Collection Protocol (GH#8674) Doug Davis

• Patch around pandas ArrowStringArray pickling (GH#9024) Jim Crist-Harif

• Band-aid for compute_as_if_collection (GH#8998) Ian Rose

• Add p2p shuffle option (GH#8836) Matthew Rocklin

Bug Fixes

• Fixup column projection with no columns (GH#9106) Jim Crist-Harif

• Blockwise cull NumPy dtype (GH#9100) Ian Rose

• Fix column-projection bug in from_map (GH#9078) Richard (Rick) Zamora

• Prevent nulls in index for non-numeric dtypes (GH#8963) Jorge López

• Fix is_monotonic methods for more than 8 partitions (GH#9019) Julia Signell

• Handle enumerate and generator inputs to from_map (GH#9066) Richard (Rick) Zamora

• Revert is_dask_collection; back to previous implementation (GH#9062) Doug Davis

• Fix Blockwise.clone does not handle iterable literal arguments correctly (GH#8979) JSKenyon

• Array setitem hardmask (GH#9027) David Hassell

• Fix overlapping divisions error on append (GH#8997) Ian Rose

Deprecations

• Add pre-deprecation warnings for read_parquet kwargs chunksize and aggregate_files (GH#9052) Richard (Rick) Zamora

Documentation

• Document map_partitions handling of args vs kwargs, usage of partition_info (GH#9084) Charles Blackmon-Luca

• Update custom collection documentation (leverage new collection protocol) (GH#9097) Doug Davis

• Better SEO for docs on creating and storing Dask DataFrames (GH#9098) Sarah Charlotte Johnson

• Clarify chunking in imread docstring (GH#9082) Genevieve Buckley

• Rearrange docs TOC (GH#9001) Matthew Rocklin

• Corrected map_blocks() docstring for kwarg enforce_ndim (GH#9071) ParticularMiner

• Update DataFrame SQL docs references to other libraries (GH#9077) Charles Blackmon-Luca

• Update page on creating and storing Dask DataFrames (GH#9025) Sarah Charlotte Johnson

Maintenance

• Include NUMPY_LICENSE.txt in license files (GH#9113) Ben Beasley

• Increase retries when installing nightly pandas (GH#9103) James Bourbeau

• Force nightly pyarrow in the upstream build (GH#9095) Joris Van den Bossche

• Improve object handling & testing of ensure_unicode (GH#9059) John A Kirkham

• Force nightly pyarrow in the upstream build (GH#8993) Joris Van den Bossche

• Additional check on is_dask_collection (GH#9054) Doug Davis

• Update ensure_bytes (GH#9050) John A Kirkham

• Add end of file pre-commit hook (GH#9045) James Bourbeau

• Add codespell pre-commit hook (GH#9040) James Bourbeau

• Remove the HDFS tests (GH#9039) Jim Crist-Harif

• Fix flaky test_reductions_2D (GH#9037) Jim Crist-Harif

• Prevent codecov from notifying of failure too soon (GH#9031) Jim Crist-Harif

• Only test on Python 3.9 on macos (GH#9029) Jim Crist-Harif

• Update to_timedelta default unit (GH#9010) Pavithra Eswaramoorthy

2022.05.0

Released on May 2, 2022

Highlights

This is a bugfix release for this issue.

Documentation

• Add highlights section to 2022.04.2 release notes (GH#9012) James Bourbeau

2022.04.2

Released on April 29, 2022

Highlights

This release includes several deprecations/breaking API changes to dask.dataframe.read_parquet and dask.dataframe.to_parquet:

• to_parquet no longer writes _metadata files by default. If you want to write a _metadata file, you can pass in write_metadata_file=True.

• read_parquet now defaults to split_row_groups=False, which results in one Dask dataframe partition per parquet file when reading in a parquet dataset. If you’re working with large parquet files you may need to set split_row_groups=True to reduce your partition size.

• read_parquet no longer calculates divisions by default. If you require read_parquet to return dataframes with known divisions, please set calculate_divisions=True.

• read_parquet has deprecated the gather_statistics keyword argument. Please use the calculate_divisions keyword argument instead.

• read_parquet has deprecated the require_extensions keyword argument. Please use the parquet_file_extension keyword argument instead.

New Features

• Add removeprefix and removesuffix as StringMethods (GH#8912) Jorge López

Enhancements

• Call fs.invalidate_cache in to_parquet (GH#8994) Jim Crist-Harif

• Change to_parquet default to write_metadata_file=None (GH#8988) Jim Crist-Harif

• Let arg reductions pass keepdims (GH#8926) Julia Signell

• Change split_row_groups default to False in read_parquet (GH#8981) Richard (Rick) Zamora

• Improve NotImplementedError message for da.reshape (GH#8987) Jim Crist-Harif

• Simplify to_parquet compute path (GH#8982) Jim Crist-Harif

• Raise an error if you try to use vindex with a Dask object (GH#8945) Julia Signell

• Avoid pre_buffer=True when a precache method is specified (GH#8957) Richard (Rick) Zamora

• from_dask_array uses blockwise instead of merging graphs (GH#8889) Bryan Weber

• Use pre_buffer=True for “pyarrow” Parquet engine (GH#8952) Richard (Rick) Zamora

Bug Fixes

• Handle dtype=None correctly in da.full (GH#8954) Tom White

• Fix dask-sql bug caused by blockwise fusion (GH#8989) Richard (Rick) Zamora

• to_parquet errors for non-string column names (GH#8990) Jim Crist-Harif

• Make sure da.roll works even if shape is 0 (GH#8925) Julia Signell

• Fix recursion error issue with set_index (GH#8967) Paul Hobson

• Stringify BlockwiseDepDict mapping values when produces_keys=True (GH#8972) Richard (Rick) Zamora

• Use DataFram`eIOLayer in DataFrame.from_delayed (GH#8852) Richard (Rick) Zamora

• Check that values for the in predicate in read_parquet are correct (GH#8846) Bryan Weber

• Fix bug for reduction of zero dimensional arrays (GH#8930) Tom White

• Specify dtype when deciding division using np.linspace in read_sql_query (GH#8940) Cheun Hong

Deprecations

• Deprecate gather_statistics from read_parquet (GH#8992) Richard (Rick) Zamora

• Change require_extension to top-level parquet_file_extension read_parquet kwarg (GH#8935) Richard (Rick) Zamora

Documentation

• Update write_metadata_file discussion in documentation (GH#8995) Richard (Rick) Zamora

• Update DataFrame.merge docstring (GH#8966) Pavithra Eswaramoorthy

• Added description for parameter align_arrays in array.blockwise() (GH#8977) ParticularMiner

• ecommend not to use map_block(drop_axis=...) on chunked axes of an array (GH#8921) ParticularMiner

• Add copy button to code snippets in docs (GH#8956) James Bourbeau

Maintenance

• Pandas 1.5.0 compatibility (GH#8961) Ian Rose

• Add pytest-timeout to distributed envs on CI (GH#8986) Julia Signell

• Improve read_parquet docstring formatting (GH#8971) Bryan Weber

• Remove pytest.warns(None) (GH#8924) Pavithra Eswaramoorthy

• Document Python 3.10 as supported (GH#8976) Eray Aslan

• parse_timedelta option to enforce explicit unit (GH#8969) crusaderky

• mypy compatibility (GH#8854) Paul Hobson

• Add a docs page for Dask & Parquet (GH#8899) Jim Crist-Harif

• Adds configuration to ignore revs in blame (GH#8933) Bryan Weber

2022.04.1

Released on April 15, 2022

New Features

• Add missing NumPy ufuncs: abs, left_shift, right_shift, positive. (GH#8920) Tom White

Enhancements

• Avoid collecting parquet metadata in pyarrow when write_metadata_file=False (GH#8906) Richard (Rick) Zamora

• Better error for failed wildcard path in dd.read_csv() (fixes #8878) (GH#8908) Roger Filmyer

• Return da.Array rather than dd.Series for non-ufunc elementwise functions on dd.Series (GH#8558) Julia Signell

• Let get_dummies use meta computation in map_partitions (GH#8898) Julia Signell

• Masked scalars input to da.from_array (GH#8895) David Hassell

• Raise ValueError in merge_asof for duplicate kwargs (GH#8861) Bryan Weber

Bug Fixes

• Make is_monotonic work when some partitions are empty (GH#8897) Julia Signell

• Fix custom getter in da.from_array when inline_array=False (GH#8903) Ian Rose

• Correctly handle dict-specification for rechunk. (GH#8859) Richard

• Fix merge_asof: drop index column if left_on == right_on (GH#8874) Gil Forsyth

Deprecations

• Warn users that engine='auto' will change in future (GH#8907) Jim Crist-Harif

• Remove pyarrow-legacy engine from parquet API (GH#8835) Richard (Rick) Zamora

Documentation

• Add note on missing parameter out for dask.array.dot (GH#8913) Francesco Andreuzzi

• Update DataFrame.query docstring (GH#8890) Pavithra Eswaramoorthy

Maintenance

• Don’t test da.prod on large integer data (GH#8893) Jim Crist-Harif

• Add network marks to tests that fail without an internet connection (GH#8881) Paul Hobson

• Fix gpuCI GHA version (GH#8891) Charles Blackmon-Luca

• xfail/skip some flaky distributed tests (GH#8887) Jim Crist-Harif

• Remove unused (deprecated) code from ArrowDatasetEngine (GH#8885) Richard (Rick) Zamora

• Add mild typing to common utils functions, part 2 (GH#8867) crusaderky

• Documentation of Limitation of sample() (GH#8858) Nadiem Sissouno

2022.04.0

Released on April 1, 2022

Note

This is the first release with support for Python 3.10

New Features

• Add Python 3.10 support (GH#8566) James Bourbeau

Enhancements

• Add check on dtype.itemsize in order to produce a useful error (GH#8860) Davide Gavio

• Add mild typing to common utils functions (GH#8848) Matthew Rocklin

• Add sanity checks to divisions setter (GH#8806) Jim Crist-Harif

• Use Blockwise and map_partitions for more tasks (GH#8831) Bryan Weber

Bug Fixes

• Fix dataframe.merge_asof to preserve right_on column (GH#8857) Sarah Charlotte Johnson

• Fix “Buffer dtype mismatch” for pandas >= 1.3 on 32bit (GH#8851) Ben Greiner

• Fix slicing fusion by altering SubgraphCallable getter (GH#8827) Ian Rose

Deprecations

• Remove support for PyPy (GH#8863) James Bourbeau

• Drop setuptools at runtime (GH#8855) crusaderky

• Remove dataframe.tseries.resample.getnanos (GH#8834) Sarah Charlotte Johnson

Documentation

• Organize diagnostic and performance docs (GH#8871) Naty Clementi

• Add image to explain drop_axis option of map_blocks (GH#8868) ParticularMiner

Maintenance

• Update gpuCI RAPIDS_VER to 22.06 (GH#8828)

• Restore test_parquet in http (GH#8850) Bryan Weber

• Simplify gpuCI updating workflow (GH#8849) Charles Blackmon-Luca

2022.03.0

Released on March 18, 2022

New Features

• Bag: add implementation for reservoir sampling (GH#7636) Daniel Mesejo-León

• Add ma.count to Dask array (GH#8785) David Hassell

• Change to_parquet default to compression="snappy" (GH#8814) Jim Crist-Harif

• Add weights parameter to dask.array.reduction (GH#8805) David Hassell

• Add ddf.compute_current_divisions to get divisions on a sorted index or column (GH#8517) Julia Signell

Enhancements

• Pass __name__ and __doc__ through on DelayedLeaf (GH#8820) Leo Gao

• Raise exception for not implemented merge how option (GH#8818) Naty Clementi

• Move Bag.map_partitions to Blockwise (GH#8646) Richard (Rick) Zamora

• Improve error messages for malformed config files (GH#8801) Jim Crist-Harif

• Revise column-projection optimization to capture common dask-sql patterns (GH#8692) Richard (Rick) Zamora

• Useful error for empty divisions (GH#8789) Pavithra Eswaramoorthy

• Scipy 1.8.0 compat: copy private classes into dask/array/stats.py (GH#8694) Julia Signell

• Raise warning when using multiple types of schedulers where one is distributed (GH#8700) Pedro Silva

Bug Fixes

• Fix bug in applying != filter in read_parquet (GH#8824) Richard (Rick) Zamora

• Fix set_index when directly passed a dask Index (GH#8680) Paul Hobson

• Quick fix for unbounded memory usage in tensordot (GH#7980) Genevieve Buckley

• If hdf file is empty, don’t fail on meta creation (GH#8809) Julia Signell

• Update clone_key("x") to retain prefix (GH#8792) crusaderky

• Fix “physical” column bug in pyarrow-based read_parquet (GH#8775) Richard (Rick) Zamora

• Fix groupby.shift bug caused by unsorted partitions after shuffle (GH#8782) kori73

• Fix serialization bug (GH#8786) Richard (Rick) Zamora

Deprecations

• Bump diagnostics bokeh dependency to 2.4.2 (GH#8791) Charles Blackmon-Luca

• Deprecate bcolz support (GH#8754) Pavithra Eswaramoorthy

• Finish making map_overlap default boundary kwarg 'none' (GH#8743) Genevieve Buckley

Documentation

• Custom collection example docs fix (GH#8807) Doug Davis

• Add Series.str, Series.dt, and Series.cat accessors to docs (GH#8757) Sarah Charlotte Johnson

• Fix docstring for ddf.compute_current_divisions (GH#8793) Julia Signell

• Dashboard docs on /status page (GH#8648) Naty Clementi

• Clarify divisions kwarg in repartition docstring (GH#8781) Sarah Charlotte Johnson

• Update Docker images to use ghcr.io (GH#8774) Jacob Tomlinson

Maintenance

• Reduce gpuci pytest parallelism (GH#8826) GALI PREM SAGAR

• absolufy-imports - No relative imports - PEP8 (GH#8796) Julia Signell

• Tidy up assert_eq calls in array tests (GH#8812) Julia Signell

• Avoid pytest.warns(None) (GH#8718) LSturtew

• Fix test_describe_empty to work without global -Werror (GH#8291) Michał Górny

• Temporarily xfail graphviz tests on windows (GH#8794) Jim Crist-Harif

• Use packaging.parse for md5 compatibility (GH#8763) James Bourbeau

• Make tokenize work in a FIPS 140-2 environment (GH#8762) Jim Crist-Harif

• Label issues and PRs on open with ‘needs triage’ (GH#8761) Julia Signell

• Add some extra test coverage (GH#8302) lrjball

• Specify action version and change from pull_request_target to pull_request (GH#8767) Julia Signell

• Make scheduler kwarg pass though to sub functions in da.assert_eq (GH#8755) Julia Signell

2022.02.1

Released on February 25, 2022

New Features

• Add aggregate functions first and last to dask.dataframe.pivot_table (GH#8649) Knut Nordanger

• Add std() support for datetime64 dtype for pandas-like objects (GH#8523) Ben Glossner

• Add materialized task counts to HighLevelGraph and Layer html reprs (GH#8589) kori73

Enhancements

• Do not allow iterating a DataFrameGroupBy (GH#8696) Bryan Weber

• Fix missing newline after info() call on empty DataFrame (GH#8727) Naty Clementi

• Add groupby.compute as a not implemented method (GH#8734) Dranaxel

• Improve multi dataframe join performance (GH#8740) Holden Karau

• Include bool type for Index (GH#8732) Naty Clementi

• Allow ArrowDatasetEngine subclass to override pandas->arrow conversion also for partitioned write (GH#8741) Joris Van den Bossche

• Increase performance of k-diagonal extraction in da.diag() and da.diagonal() (GH#8689) ParticularMiner

• Change linspace creation to match numpy when num equal to 0 (GH#8676) Peter

• Tokenize dataclasses (GH#8557) Gabe Joseph

• Update tokenize to treat dict and kwargs differently (GH#8655) James Bourbeau

Bug Fixes

• Fix bug in dask.array.roll() for roll-shifts that match the size of the input array (GH#8723) ParticularMiner

• Fix for normalize_function dataclass methods (GH#8527) Sarah Charlotte Johnson

• Fix rechunking with zero-size-chunks (GH#8703) ParticularMiner

• Move creation of sqlalchemy connection for picklability (GH#8745) Julia Signell

Deprecations

• Drop Python 3.7 (GH#8572) James Bourbeau

• Deprecate iteritems (GH#8660) James Bourbeau

• Deprecate dataframe.tseries.resample.getnanos (GH#8752) Sarah Charlotte Johnson

• Add deprecation warning for pyarrow-legacy engine (GH#8758) Richard (Rick) Zamora

Documentation

• Update link typos in changelog (GH#8717) James Bourbeau

• Clarify dask.visualize docstring (GH#8710) Dranaxel

• Update Docker example to use current best practices (GH#8731) Jacob Tomlinson

• Update docs to include distributed.Client.preload (GH#8679) Bryan Weber

• Document monthly social meeting (GH#8595) Thomas Grainger

• Add docs for Gen2 access with RBAC/ACL i.e. security principal (GH#8748) Martin Thøgersen

• Use Dask configuration extension from dask-sphinx-theme (GH#8751) Benjamin Zaitlen

Maintenance

• Unpin coverage in CI (GH#8690) James Bourbeau

• Add manual trigger for running test suite (GH#8716) James Bourbeau

• Xfail scheduler_HLG_unpack_import; flaky test (GH#8724) Mike McCarty

• Temporarily remove scipy upstream CI build (GH#8725) James Bourbeau

• Bump pre-release version to be greater than stable releases (GH#8728) Charles Blackmon-Luca

• Move custom sort function logic to internal sort_values (GH#8571) Charles Blackmon-Luca

• Pin cloudpickle and scipy in docs requirements (GH#8737) Julia Signell

• Make the labeler not delete labels, and look for the docs at the right spot (GH#8746) Julia Signell

• Fix docs build warnings (GH#8432) Kristopher Overholt

• Update test status badge (GH#8747) James Bourbeau

• Fix parquet test_pandas_timestamp_overflow_pyarrow test (GH#8733) Joris Van den Bossche

• Only run PR builds on changes to relevant files (GH#8756) Charles Blackmon-Luca

2022.02.0

Released on February 11, 2022

Note

This is the last release with support for Python 3.7

New Features

• Add region to to_zarr when using existing array (GH#8590) Chris Roat

• Add engine_kwargs support to dask.dataframe.to_sql (GH#8609) Amir Kadivar

• Add include_path_column arg to read_json (GH#8603) Bryan Weber

• Add expand_dims to Dask array (GH#8687) Tom White

Enhancements

• Add scheduler option to assert_eq utilities (GH#8610) Xinrong Meng

• Fix eye inconsistency with NumPy for dtype=None (GH#8685) Tom White

• Fix concatenate inconsistency with NumPy for axis=None (GH#8686) Tom White

• Type annotations, part 1 (GH#8295) crusaderky

• Really allow any iterable to be passed as a meta (GH#8629) Julia Signell

• Use map_partitions (Blockwise) in to_parquet (GH#8487) Richard (Rick) Zamora

Bug Fixes

• Result of reducing an array should not depend on its chunk-structure (GH#8637) ParticularMiner

• Pass place-holder metadata to map_partitions in ACA code path (GH#8643) Richard (Rick) Zamora

Deprecations

• Deprecate is_monotonic (GH#8653) James Bourbeau

• Remove some deprecations (GH#8605) James Bourbeau

Documentation

• Add Domino Data Lab to Hosted / managed Dask clusters (GH#8675) Ray Bell

• Fix inter-linking and remove deprecated function (GH#8715) Julia Signell

• Fix imbalanced backticks. (GH#8693) Matthias Bussonnier

• Add documentation for high level graph visualization (GH#8483) Genevieve Buckley

• Update documentation of ProgressBar out parameter (GH#8604) Pedro Silva

• Improve documentation of dask.config.set (GH#8705) crusaderky

• Revert mention to mypy among type checkers (GH#8699) crusaderky

Maintenance

• Update warning handling in get_dummies tests (GH#8651) James Bourbeau

• Add a github changelog template (GH#8714) Julia Signell

• Update year in LICENSE.txt (GH#8665) David Hoese

• Update pre-commit version (GH#8691) James Bourbeau

• Include scipy in upstream CI build (GH#8681) James Bourbeau

• Temporarily pin scipy < 1.8.0 in CI (GH#8683) James Bourbeau

• Pin scipy to less than 1.8.0 in GPU CI (GH#8698) Julia Signell

• Avoid pytest.warns(None) in test_multi.py (GH#8678) James Bourbeau

• Update GHA concurrent job cancellation (GH#8652) James Bourbeau

• Make test__get_paths robust to site.PREFIXES being set (GH#8644) James Bourbeau

• Bump gpuCI PYTHON_VER to 3.9 (GH#8642) Charles Blackmon-Luca

2022.01.1

Released on January 28, 2022

New Features

• Add dask.dataframe.series.view() (GH#8533) Pavithra Eswaramoorthy

Enhancements

• Update tz for fastparquet + pandas 1.4.0 (GH#8626) Martin Durant

• Cleaning up misc tests for pandas compat (GH#8623) Julia Signell

• Moving to SQLAlchemy >= 1.4 (GH#8158) McToel

• Pandas compat: Filter sparse warnings (GH#8621) Julia Signell

• Fail if meta is not a pandas object (GH#8563) Julia Signell

• Use fsspec.parquet module for better remote-storage read_parquet performance (GH#8339) Richard (Rick) Zamora

• Move DataFrame ACA aggregations to HLG (GH#8468) Richard (Rick) Zamora

• Add optional information about originating function call in DataFrameIOLayer (GH#8453) Richard (Rick) Zamora

• Blockwise array creation redux (GH#7417) Ian Rose

• Refactor config default search path retrieval (GH#8573) James Bourbeau

• Add optimize_graph flag to Bag.to_dataframe function (GH#8486) Maxim Lippeveld

• Make sure that delayed output operations still return lists of paths (GH#8498) Julia Signell

• Pandas compat: Fix to_frame name to not pass None (GH#8554) Julia Signell

• Pandas compat: Fix axis=None warning (GH#8555) Julia Signell

• Expand Dask YAML config search directories (GH#8531) abergou

Bug Fixes

• Fix groupby.cumsum with series grouped by index (GH#8588) Julia Signell

• Fix derived_from for pandas methods (GH#8612) Thomas J. Fan

• Enforce boolean ascending for sort_values (GH#8440) Charles Blackmon-Luca

• Fix parsing of __setitem__ indices (GH#8601) David Hassell

• Avoid divide by zero in slicing (GH#8597) Doug Davis

Deprecations

• Downgrade meta error in (GH#8563) to warning (GH#8628) Julia Signell

• Pandas compat: Deprecate append when pandas >= 1.4.0 (GH#8617) Julia Signell

Documentation

• Replace outdated columns argument with meta in DataFrame constructor (GH#8614) kori73

• Refactor deploying docs (GH#8602) Jacob Tomlinson

Maintenance

• Pin coverage in CI (GH#8631) James Bourbeau

• Move cached_cumsum imports to be from dask.utils (GH#8606) James Bourbeau

• Update gpuCI RAPIDS_VER to 22.04 (GH#8600)

• Update cocstring for from_delayed function (GH#8576) Kirito1397

• Handle plot_width / plot_height deprecations (GH#8544) Bryan Van de Ven

• Remove unnecessary pyyaml importorskip (GH#8562) James Bourbeau

• Specify scheduler in DataFrame assert_eq (GH#8559) Gabe Joseph

2022.01.0

Released on January 14, 2022

New Features

• Add groupby.shift method (GH#8522) kori73

• Add DataFrame.nunique (GH#8479) Sarah Charlotte Johnson

• Add da.ndim to match np.ndim (GH#8502) Julia Signell

Enhancements

• Only show percentile interpolation= keyword warning if NumPy version >= 1.22 (GH#8564) Julia Signell

• Raise PerformanceWarning when limit and "array.slicing.split-large-chunks" are None (GH#8511) Julia Signell

• Define normalize_seq function at import time (GH#8521) Illviljan

• Ensure that divisions are alway tuples (GH#8393) Charles Blackmon-Luca

• Allow a callable scheduler for bag.groupby (GH#8492) Julia Signell

• Save Zarr arrays with dask-on-ray scheduler (GH#8472) TnTo

• Make byte blocks more even in read_bytes (GH#8459) Martin Durant

• Improved the efficiency of matmul() by completely removing concatenation (GH#8423) ParticularMiner

• Limit max chunk size when reshaping dask arrays (GH#8124) Genevieve Buckley

• Changes for fastparquet superthrift (GH#8470) Martin Durant

Bug Fixes

• Fix boolean indices in array assignment (GH#8538) David Hassell

• Detect default dtype on array-likes (GH#8501) aeisenbarth

• Fix optimize_blockwise bug for duplicate dependency names (GH#8542) Richard (Rick) Zamora

• Update warnings for DataFrame.GroupBy.apply and transform (GH#8507) Sarah Charlotte Johnson

• Track HLG layer name in Delayed (GH#8452) Gabe Joseph

• Fix single item nanmin and nanmax reductions (GH#8484) Julia Signell

• Make read_csv with comment kwarg work even if there is a comment in the header (GH#8433) Julia Signell

Deprecations

• Replace interpolation with method and method with internal_method (GH#8525) Julia Signell

• Remove daily stock demo utility (GH#8477) James Bourbeau

Documentation

• Add a join example in docs that be run with copy/paste (GH#8520) kori73

• Mention dashboard link in config (GH#8510) Ray Bell

• Fix changelog section hyperlinks (GH#8534) Aneesh Nema

• Hyphenate “single-machine scheduler” for consistency (GH#8519) Deepyaman Datta

• Normalize whitespace in doctests in slicing.py (GH#8512) Maren Westermann

• Best practices storage line typo (GH#8529) Michael Delgado

• Update figures (GH#8401) Sarah Charlotte Johnson

• Remove pyarrow-only reference from split_row_groups in read_parquet docstring (GH#8490) Naty Clementi

Maintenance

• Remove obsolete LocalFileSystem tests that fail for fsspec>=2022.1.0 (GH#8565) Richard (Rick) Zamora

• Tweak: “RuntimeWarning: invalid value encountered in reciprocal” (GH#8561) crusaderky

• Fix skipna=None for DataFrame.sem (GH#8556) Julia Signell

• Fix PANDAS_GT_140 (GH#8552) Julia Signell

• Collections with HLG must always implement __dask_layers__ (GH#8548) crusaderky

• Work around race condition in import llvmlite (GH#8550) crusaderky

• Set a minimum version for pyyaml (GH#8545) Gaurav Sheni

• Adding nodefaults to environments to fix tiledb + mac issue (GH#8505) Julia Signell

• Set ceiling for setuptools (GH#8509) Julia Signell

• Add workflow / recipe to generate Dask nightlies (GH#8469) Charles Blackmon-Luca

• Bump gpuCI CUDA_VER to 11.5 (GH#8489) Charles Blackmon-Luca

2021.12.0

Released on December 10, 2021

New Features

• Add Series and Index is_monotonic* methods (GH#8304) Daniel Mesejo-León

Enhancements

• Blockwise map_partitions with partition_info (GH#8310) Gabe Joseph

• Better error message for length of array with unknown chunk sizes (GH#8436) Doug Davis

• Use by instead of index internally on the Groupby class (GH#8441) Julia Signell

• Allow custom sort functions for sort_values (GH#8345) Charles Blackmon-Luca

• Add warning to read_parquet when statistics and partitions are misaligned (GH#8416) Richard (Rick) Zamora

• Support where argument in ufuncs (GH#8253) mihir

• Make visualize more consistent with compute (GH#8328) JSKenyon

Bug Fixes

• Fix map_blocks not using own arguments in name generation (GH#8462) David Hoese

• Fix for index error with reading empty parquet file (GH#8410) Sarah Charlotte Johnson

• Fix nullable-dtype error when writing partitioned parquet data (GH#8400) Richard (Rick) Zamora

• Fix CSV header bug (GH#8413) Richard (Rick) Zamora

• Fix empty chunk causes exception in nanmin/nanmax (GH#8375) Boaz Mohar

Deprecations

• Deprecate token keyword argument to map_blocks (GH#8464) James Bourbeau

• Deprecation warning for default value of boundary kwarg in map_overlap (GH#8397) Genevieve Buckley

Documentation

• Clarify block_info documentation (GH#8425) Genevieve Buckley

• Output from alt text sprint (GH#8456) Sarah Charlotte Johnson

• Update talks and presentations (GH#8370) Naty Clementi

• Update Anaconda link in “Paid support” section of docs (GH#8427) Martin Durant

• Fixed broken dask-gateway link in ecosystem.rst (GH#8424) ofirr

• Fix CuPy doctest error (GH#8412) Genevieve Buckley

Maintenance

• Bump Bokeh min version to 2.1.1 (GH#8431) Bryan Van de Ven

• Fix following fsspec=2021.11.1 release (GH#8428) Martin Durant

• Add dask/ml.py to pytest exclude list (GH#8414) Genevieve Buckley

• Update gpuCI RAPIDS_VER to 22.02 (GH#8394)

• Unpin graphviz and improve package management in environment-3.7 (GH#8411) Julia Signell

2021.11.2

Released on November 19, 2021

• Only run gpuCI bump script daily (GH#8404) Charles Blackmon-Luca

• Actually ignore index when asked in assert_eq (GH#8396) Gabe Joseph

• Ensure single-partition join divisions is tuple (GH#8389) Charles Blackmon-Luca

• Try to make divisions behavior clearer (GH#8379) Julia Signell

• Fix typo in set_index partition_size parameter description (GH#8384) FredericOdermatt

• Use blockwise in single_partition_join (GH#8341) Gabe Joseph

• Use more explicit keyword arguments (GH#8354) Boaz Mohar

• Fix .loc of DataFrame with nullable boolean dtype (GH#8368) Marco Rossi

• Parameterize shuffle implementation in tests (GH#8250) Ian Rose

• Remove some doc build warnings (GH#8369) Boaz Mohar

• Include properties in array API docs (GH#8356) Julia Signell

• Fix Zarr for upstream (GH#8367) Julia Signell

• Pin graphviz to avoid issue with windows and Python 3.7 (GH#8365) Julia Signell

• Import graphviz.Diagraph from top of module, not from dot (GH#8363) Julia Signell

2021.11.1

Released on November 8, 2021

Patch release to update distributed dependency to version 2021.11.1.

2021.11.0

Released on November 5, 2021

• Fx required_extension behavior in read_parquet (GH#8351) Richard (Rick) Zamora

• Add align_dataframes to map_partitions to broadcast a dataframe passed as an arg (GH#6628) Julia Signell

• Better handling for arrays/series of keys in dask.dataframe.loc (GH#8254) Julia Signell

• Point users to Discourse (GH#8332) Ian Rose

• Add name_function option to to_parquet (GH#7682) Matthew Powers

• Get rid of environment-latest.yml and update to Python 3.9 (GH#8275) Julia Signell

• Require newer s3fs in CI (GH#8336) James Bourbeau

• Groupby Rolling (GH#8176) Julia Signell

• Add more ordering diagnostics to dask.visualize (GH#7992) Erik Welch

• Use HighLevelGraph optimizations for delayed (GH#8316) Ian Rose

• demo_tuples produces malformed HighLevelGraph (GH#8325) crusaderky

• Dask calendar should show events in local time (GH#8312) Genevieve Buckley

• Fix flaky test_interrupt (GH#8314) crusaderky

• Deprecate AxisError (GH#8305) crusaderky

• Fix name of cuDF in extension documentation. (GH#8311) Vyas Ramasubramani

• Add single eq operator (=) to parquet filters (GH#8300) Ayush Dattagupta

• Improve support for Spark output in read_parquet (GH#8274) Richard (Rick) Zamora

• Add dask.ml module (GH#6384) Matthew Rocklin

• CI fixups (GH#8298) James Bourbeau

• Make slice errors match NumPy (GH#8248) Julia Signell

• Fix API docs misrendering with new sphinx theme (GH#8296) Julia Signell

• Replace block property with blockview for array-like operations on blocks (GH#8242) Davis Bennett

• Deprecate file_path and make it possible to save from within a notebook (GH#8283) Julia Signell

2021.10.0

Released on October 22, 2021

• da.store to create well-formed HighLevelGraph (GH#8261) crusaderky

• CI: force nightly pyarrow in the upstream build (GH#8281) Joris Van den Bossche

• Remove chest (GH#8279) James Bourbeau

• Skip doctests if optional dependencies are not installed (GH#8258) Genevieve Buckley

• Update tmpdir and tmpfile context manager docstrings (GH#8270) Daniel Mesejo-León

• Unregister callbacks in doctests (GH#8276) James Bourbeau

• Fix typo in docs (GH#8277) JoranDox

• Stale label GitHub action (GH#8244) Genevieve Buckley

• Client-shutdown method appears twice (GH#8273) German Shiklov

• Add pre-commit to test requirements (GH#8257) Genevieve Buckley

• Refactor read_metadata in fastparquet engine (GH#8092) Richard (Rick) Zamora

• Support Path objects in from_zarr (GH#8266) Samuel Gaist

• Make nested redirects work (GH#8272) Julia Signell

• Set memory_usage to True if verbose is True in info (GH#8222) Kinshuk Dua

• Remove individual API doc pages from sphinx toctree (GH#8238) James Bourbeau

• Ignore whitespace in gufunc signature (GH#8267) James Bourbeau

• Add workflow to update gpuCI (GH#8215) Charles Blackmon-Luca

• DataFrame.head shouldn’t warn when there’s one partition (GH#8091) Pankaj Patil

• Ignore arrow doctests if pyarrow not installed (GH#8256) Genevieve Buckley

• Fix debugging.html redirect (GH#8251) James Bourbeau

• Fix null sorting for single partition dataframes (GH#8225) Charles Blackmon-Luca

• Fix setup.html redirect (GH#8249) Florian Jetter

• Run pyupgrade in CI (GH#8246) crusaderky

• Fix label typo in upstream CI build (GH#8237) James Bourbeau

• Add support for “dependent” columns in DataFrame.assign (GH#8086) Suriya Senthilkumar

• add NumPy array of Dask keys to Array (GH#7922) Davis Bennett

• Remove unnecessary dask.multiprocessing import in docs (GH#8240) Ray Bell

• Adjust retrieving _max_workers from Executor (GH#8228) John A Kirkham

• Update function signatures in delayed best practices docs (GH#8231) Vũ Trung Đức

• Docs reoganization (GH#7984) Julia Signell

• Fix df.quantile on all missing data (GH#8129) Julia Signell

• Add tokenize.ensure-deterministic config option (GH#7413) Hristo Georgiev

• Use inclusive rather than closed with pandas>=1.4.0 and pd.date_range (GH#8213) Julia Signell

• Add dask-gateway, Coiled, and Saturn-Cloud to list of Dask setup tools (GH#7814) Kristopher Overholt

• Ensure existing futures get passed as deps when serializing HighLevelGraph layers (GH#8199) Jim Crist-Harif

• Make sure that the divisions of the single partition merge is left (GH#8162) Julia Signell

• Refactor read_metadata in pyarrow parquet engines (GH#8072) Richard (Rick) Zamora

• Support negative drop_axis in map_blocks and map_overlap (GH#8192) Gregory R. Lee

• Fix upstream tests (GH#8205) Julia Signell

• Add support for scalar item assignment by Series (GH#8195) Charles Blackmon-Luca

• Add some basic examples to doc strings on dask.bag all, any, count methods (GH#7630) Nathan Danielsen

• Don’t have upstream report depend on commit message (GH#8202) James Bourbeau

• Ensure upstream CI cron job runs (GH#8200) James Bourbeau

• Use pytest.param to properly label param-specific GPU tests (GH#8197) Charles Blackmon-Luca

• Add test_set_index to tests ran on gpuCI (GH#8198) Charles Blackmon-Luca

• Suppress tmpfile OSError (GH#8191) James Bourbeau

• Use s.isna instead of pd.isna(s) in set_partitions_pre (fix cudf CI) (GH#8193) Charles Blackmon-Luca

• Open an issue for test-upstream failures (GH#8067) Wallace Reis

• Fix to_parquet bug in call to pyarrow.parquet.read_metadata (GH#8186) Richard (Rick) Zamora

• Add handling for null values in sort_values (GH#8167) Charles Blackmon-Luca

• Bump RAPIDS_VER for gpuCI (GH#8184) Charles Blackmon-Luca

• Dispatch walks MRO for lazily registered handlers (GH#8185) Jim Crist-Harif

• Configure SSHCluster instructions (GH#8181) Ray Bell

• Preserve HighLevelGraphs in DataFrame.from_delayed (GH#8174) Gabe Joseph

• Deprecate inplace argument for Dask series renaming (GH#8136) Marcel Coetzee

• Fix rolling for compatibility with pandas > 1.3.0 (GH#8150) Julia Signell

• Raise error when setitem on unknown chunks (GH#8166) Julia Signell

• Include divisions when doing Index.to_series (GH#8165) Julia Signell

2021.09.1

Released on September 21, 2021

• Fix groupby for future pandas (GH#8151) Julia Signell

• Remove warning filters in tests that are no longer needed (GH#8155) Julia Signell

• Add link to diagnostic visualize function in local diagnostic docs (GH#8157) David Hoese

• Add datetime_is_numeric to dataframe.describe (GH#7719) Julia Signell

• Remove references to pd.Int64Index in anticipation of deprecation (GH#8144) Julia Signell

• Use loc if needed for series __get_item__ (GH#7953) Julia Signell

• Specifically ignore warnings on mean for empty slices (GH#8125) Julia Signell

• Skip groupby nunique test for pandas >= 1.3.3 (GH#8142) Julia Signell

• Implement ascending arg for sort_values (GH#8130) Charles Blackmon-Luca

• Replace operator.getitem (GH#8015) Naty Clementi

• Deprecate zero_broadcast_dimensions and homogeneous_deepmap (GH#8134) SnkSynthesis

• Add error if drop_index is negative (GH#8064) neel iyer

• Allow scheduler to be an Executor (GH#8112) John A Kirkham

• Handle asarray/asanyarray cases where like is a dask.Array (GH#8128) Peter Andreas Entschev

• Fix index_col duplication if index_col is type str (GH#7661) McToel

• Add dtype and order to asarray and asanyarray definitions (GH#8106) Julia Signell

• Deprecate dask.dataframe.Series.__contains__ (GH#7914) Julia Signell

• Fix edge case with like-arrays in _wrapped_qr (GH#8122) Peter Andreas Entschev

• Deprecate boundary_slice kwarg: kind for pandas compat (GH#8037) Julia Signell

2021.09.0

Released on September 3, 2021

• Fewer open files (GH#7303) Julia Signell

• Add FileNotFound to expected http errors (GH#8109) Martin Durant

• Add DataFrame.sort_values to API docs (GH#8107) Benjamin Zaitlen

• Change to dask.order: be more eager at times (GH#7929) Erik Welch

• Add pytest color to CI (GH#8090) James Bourbeau

• FIX: make_people works with processes scheduler (GH#8103) Dahn

• Adds deep param to Dataframe copy method and restrict it to False (GH#8068) João Paulo Lacerda

• Fix typo in configuration docs (GH#8104) Robert Hales

• Update formatting in DataFrame.query docstring (GH#8100) James Bourbeau

• Un-xfail sparse tests for 0.13.0 release (GH#8102) James Bourbeau

• Add axes property to DataFrame and Series (GH#8069) Jordan Jensen

• Add CuPy support in da.unique (values only) (GH#8021) Peter Andreas Entschev

• Unit tests for sparse.zeros_like (xfailed) (GH#8093) crusaderky

• Add explicit like kwarg support to array creation functions (GH#8054) Peter Andreas Entschev

• Separate Array and DataFrame mindeps builds (GH#8079) James Bourbeau

• Fork out percentile_dispatch to dask.array (GH#8083) GALI PREM SAGAR

• Ensure filepath exists in to_parquet (GH#8057) James Bourbeau

• Update scheduler plugin usage in test_scheduler_highlevel_graph_unpack_import (GH#8080) James Bourbeau

• Add DataFrame.shuffle to API docs (GH#8076) Martin Fleischmann

• Order requirements alphabetically (GH#8073) John A Kirkham

2021.08.1

Released on August 20, 2021

• Add ignore_metadata_file option to read_parquet (pyarrow-dataset and fastparquet support only) (GH#8034) Richard (Rick) Zamora

• Add reference to pytest-xdist in dev docs (GH#8066) Julia Signell

• Include tz in meta from to_datetime (GH#8000) Julia Signell

• CI Infra Docs (GH#7985) Benjamin Zaitlen

• Include invalid DataFrame key in assert_eq check (GH#8061) James Bourbeau

• Use __class__ when creating DataFrames (GH#8053) Mads R. B. Kristensen

• Use development version of distributed in gpuCI build (GH#7976) James Bourbeau

• Ignore whitespace when gufunc signature (GH#8049) James Bourbeau

• Move pandas import and percentile dispatch refactor (GH#8055) GALI PREM SAGAR

• Add colors to represent high level layer types (GH#7974) Freyam Mehta

• Upstream instance fix (GH#8060) Jacob Tomlinson

• Add dask.widgets and migrate HTML reprs to jinja2 (GH#8019) Jacob Tomlinson

• Remove wrap_func_like_safe, not required with NumPy >= 1.17 (GH#8052) Peter Andreas Entschev

• Fix threaded scheduler memory backpressure regression (GH#8040) David Hoese

• Add percentile dispatch (GH#8029) GALI PREM SAGAR

• Use a publicly documented attribute obj in groupby rather than private _selected_obj (GH#8038) GALI PREM SAGAR

• Specify module to import rechunk from (GH#8039) Illviljan

• Use dict to store data for {nan,}arg{min,max} in certain cases (GH#8014) Peter Andreas Entschev

• Fix blocksize description formatting in read_pandas (GH#8047) Louis Maddox

• Fix “point” -> “pointers” typo in docs (GH#8043) David Chudzicki

2021.08.0

Released on August 13, 2021

• Fix to_orc delayed compute behavior (GH#8035) Richard (Rick) Zamora

• Don’t convert to low-level task graph in compute_as_if_collection (GH#7969) James Bourbeau

• Fix multifile read for hdf (GH#8033) Julia Signell

• Resolve warning in distributed tests (GH#8025) James Bourbeau

• Update to_orc collection name (GH#8024) James Bourbeau

• Resolve skipfooter problem (GH#7855) Ross

• Raise NotImplementedError for non-indexable arg passed to to_datetime (GH#7989) Doug Davis

• Ensure we error on warnings from distributed (GH#8002) James Bourbeau

• Added dict format in to_bag accessories of DataFrame (GH#7932) gurunath

• Delayed docs indirect dependencies (GH#8016) aa1371

• Add tooltips to graphviz high-level graphs (GH#7973) Freyam Mehta

• Close 2021 User Survey (GH#8007) Julia Signell

• Reorganize CuPy tests into multiple files (GH#8013) Peter Andreas Entschev

• Refactor and Expand Dask-Dataframe ORC API (GH#7756) Richard (Rick) Zamora

• Don’t enforce columns if enforce=False (GH#7916) Julia Signell

• Fix map_overlap trimming behavior when drop_axis is not None (GH#7894) Gregory R. Lee

• Mark gpuCI CuPy test as flaky (GH#7994) Peter Andreas Entschev

• Avoid using Delayed in to_csv and to_parquet (GH#7968) Matthew Rocklin

• Removed redundant check_dtypes (GH#7952) gurunath

• Use pytest.warns instead of raises for checking parquet engine deprecation (GH#7993) Joris Van den Bossche

• Bump RAPIDS_VER in gpuCI to 21.10 (GH#7991) Charles Blackmon-Luca

• Add back pyarrow-legacy test coverage for pyarrow>=5 (GH#7988) Richard (Rick) Zamora

• Allow pyarrow>=5 in to_parquet and read_parquet (GH#7967) Richard (Rick) Zamora

• Skip CuPy tests requiring NEP-35 when NumPy < 1.20 is available (GH#7982) Peter Andreas Entschev

• Add tail and head to SeriesGroupby (GH#7935) Daniel Mesejo-León

• Update Zoom link for monthly meeting (GH#7979) James Bourbeau

• Add gpuCI build script (GH#7966) Charles Blackmon-Luca

• Deprecate daily_stock utility (GH#7949) James Bourbeau

• Add distributed.nanny to configuration reference docs (GH#7955) James Bourbeau

• Require NumPy 1.18+ & Pandas 1.0+ (GH#7939) John A Kirkham

2021.07.2

Released on July 30, 2021

Note

This is the last release with support for NumPy 1.17 and pandas 0.25. Beginning with the next release, NumPy 1.18 and pandas 1.0 will be the minimum supported versions.

• Add dask.array SVG to the HTML Repr (GH#7886) Freyam Mehta

• Avoid use of Delayed in to_parquet (GH#7958) Matthew Rocklin

• Temporarily pin pyarrow<5 in CI (GH#7960) James Bourbeau

• Add deprecation warning for top-level ucx and rmm config values (GH#7956) James Bourbeau

• Remove skips from doctests (4 of 6) (GH#7865) Zhengnan Zhao

• Remove skips from doctests (5 of 6) (GH#7864) Zhengnan Zhao

• Adds missing prepend/append functionality to da.diff (GH#7946) Peter Andreas Entschev

• Change graphviz font family to sans (GH#7931) Freyam Mehta

• Fix read-csv name - when path is different, use different name for task (GH#7942) Julia Signell

• Update configuration reference for ucx and rmm changes (GH#7943) James Bourbeau

• Add meta support to __setitem__ (GH#7940) Peter Andreas Entschev

• NEP-35 support for slice_with_int_dask_array (GH#7927) Peter Andreas Entschev

• Unpin fastparquet in CI (GH#7928) James Bourbeau

• Remove skips from doctests (3 of 6) (GH#7872) Zhengnan Zhao

2021.07.1

Released on July 23, 2021

• Make array assert_eq check dtype (GH#7903) Julia Signell

• Remove skips from doctests (6 of 6) (GH#7863) Zhengnan Zhao

• Remove experimental feature warning from actors docs (GH#7925) Matthew Rocklin

• Remove skips from doctests (2 of 6) (GH#7873) Zhengnan Zhao

• Separate out Array and Bag API (GH#7917) Julia Signell

• Implement lazy Array.__iter__ (GH#7905) Julia Signell

• Clean up places where we inadvertently iterate over arrays (GH#7913) Julia Signell

• Add numeric_only kwarg to DataFrame reductions (GH#7831) Julia Signell

• Add pytest marker for GPU tests (GH#7876) Charles Blackmon-Luca

• Add support for histogram2d in dask.array (GH#7827) Doug Davis

• Remove skips from doctests (1 of 6) (GH#7874) Zhengnan Zhao

• Add node size scaling to the Graphviz output for the high level graphs (GH#7869) Freyam Mehta

• Update old Bokeh links (GH#7915) Bryan Van de Ven

• Temporarily pin fastparquet in CI (GH#7907) James Bourbeau

• Add dask.array import to progress bar docs (GH#7910) Fabian Gebhart

• Use separate files for each DataFrame API function and method (GH#7890) Julia Signell

• Fix pyarrow-dataset ordering bug (GH#7902) Richard (Rick) Zamora

• Generalize unique aggregate (GH#7892) GALI PREM SAGAR

• Raise NotImplementedError when using pd.Grouper (GH#7857) Ruben van de Geer

• Add aggregate_files argument to enable multi-file partitions in read_parquet (GH#7557) Richard (Rick) Zamora

• Un-xfail test_daily_stock (GH#7895) James Bourbeau

• Update access configuration docs (GH#7837) Naty Clementi

• Use packaging for version comparisons (GH#7820) Elliott Sales de Andrade

• Handle infinite loops in merge_asof (GH#7842) gerrymanoim

2021.07.0

Released on July 9, 2021

• Include fastparquet in upstream CI build (GH#7884) James Bourbeau

• Blockwise: handle non-string constant dependencies (GH#7849) Mads R. B. Kristensen

• fastparquet now supports new time types, including ns precision (GH#7880) Martin Durant

• Avoid ParquetDataset API when appending in ArrowDatasetEngine (GH#7544) Richard (Rick) Zamora

• Add retry logic to test_shuffle_priority (GH#7879) Richard (Rick) Zamora

• Use strict channel priority in CI (GH#7878) James Bourbeau

• Support nested dask.distributed imports (GH#7866) Matthew Rocklin

• Should check module name only, not the entire directory filepath (GH#7856) Genevieve Buckley

• Updates due to https://github.com/dask/fastparquet/pull/623 (GH#7875) Martin Durant

• da.eye fix for chunks=-1 (GH#7854) Naty Clementi

• Temporarily xfail test_daily_stock (GH#7858) James Bourbeau

• Set priority annotations in SimpleShuffleLayer (GH#7846) Richard (Rick) Zamora

• Blockwise: stringify constant key inputs (GH#7838) Mads R. B. Kristensen

• Allow mixing dask and numpy arrays in @guvectorize (GH#6863) Julia Signell

• Don’t sample dict result of a shuffle group when calculating its size (GH#7834) Florian Jetter

• Fix scipy tests (GH#7841) Julia Signell

• Deterministically tokenize datetime.date (GH#7836) James Bourbeau

• Add sample_rows to read_csv-like (GH#7825) Martin Durant

• Fix typo in config.deserialize docstring (GH#7830) Geoffrey Lentner

• Remove warning filter in test_dataframe_picklable (GH#7822) James Bourbeau

• Improvements to histogramdd (for handling inputs that are sequences-of-arrays). (GH#7634) Doug Davis

• Make PY_VERSION private (GH#7824) James Bourbeau

2021.06.2

Released on June 22, 2021

• layers.py compare parts_out with set(self.parts_out) (GH#7787) Genevieve Buckley

• Make check_meta understand pandas dtypes better (GH#7813) Julia Signell

• Remove “Educational Resources” doc page (GH#7818) James Bourbeau

2021.06.1

Released on June 18, 2021

• Replace funding page with ‘Supported By’ section on dask.org (GH#7817) James Bourbeau

• Add initial deprecation utilities (GH#7810) James Bourbeau

• Enforce dtype conservation in ufuncs that explicitly use dtype= (GH#7808) Doug Davis

• Add Coiled to list of paid support organizations (GH#7811) Kristopher Overholt

• Small tweaks to the HTML repr for Layer & HighLevelGraph (GH#7812) Genevieve Buckley

• Add dark mode support to HLG HTML repr (GH#7809) Jacob Tomlinson

• Remove compatibility entries for old distributed (GH#7801) Elliott Sales de Andrade

• Implementation of HTML repr for HighLevelGraph layers (GH#7763) Genevieve Buckley

• Update default blockwise token to avoid DataFrame column name clash (GH#6546) James Bourbeau

• Use dispatch concat for merge_asof (GH#7806) Julia Signell

• Fix upstream freq tests (GH#7795) Julia Signell

• Use more context managers from the standard library (GH#7796) James Bourbeau

• Simplify skips in parquet tests (GH#7802) Elliott Sales de Andrade

• Remove check for outdated bokeh (GH#7804) Elliott Sales de Andrade

• More test coverage uploads (GH#7799) James Bourbeau

• Remove ImportError catching from dask/__init__.py (GH#7797) James Bourbeau

• Allow DataFrame.join() to take a list of DataFrames to merge with (GH#7578) Krishan Bhasin

• Fix maximum recursion depth exception in dask.array.linspace (GH#7667) Daniel Mesejo-León

• Fix docs links (GH#7794) Julia Signell

• Initial da.select() implementation and test (GH#7760) Gabriel Miretti

• Layers must implement get_output_keys method (GH#7790) Genevieve Buckley

• Don’t include or expect freq in divisions (GH#7785) Julia Signell

• A HighLevelGraph abstract layer for map_overlap (GH#7595) Genevieve Buckley

• Always include kwarg name in drop (GH#7784) Julia Signell

• Only rechunk for median if needed (GH#7782) Julia Signell

• Add add_(prefix|suffix) to DataFrame and Series (GH#7745) tsuga

• Move read_hdf to Blockwise (GH#7625) Richard (Rick) Zamora

• Make Layer.get_output_keys officially an abstract method (GH#7775) Genevieve Buckley

• Non-dask-arrays and broadcasting in ravel_multi_index (GH#7594) Gabe Joseph

• Fix for paths ending with “/” in parquet overwrite (GH#7773) Martin Durant

• Fixing calling .visualize() with filename=None (GH#7740) Freyam Mehta

• Generate unique names for SubgraphCallable (GH#7637) Bruce Merry

• Pin fsspec to 2021.5.0 in CI (GH#7771) James Bourbeau

• Evaluate graph lazily if meta is provided in from_delayed (GH#7769) Florian Jetter

• Add meta support for DatetimeTZDtype (GH#7627) gerrymanoim

• Add dispatch label to automatic PR labeler (GH#7701) James Bourbeau

• Fix HDFS tests (GH#7752) Julia Signell

2021.06.0

Released on June 4, 2021

• Remove abstract tokens from graph keys in rewrite_blockwise (GH#7721) Richard (Rick) Zamora

• Ensure correct column order in csv project_columns (GH#7761) Richard (Rick) Zamora

• Renamed inner loop variables to avoid duplication (GH#7741) Boaz Mohar

• Do not return delayed object from to_zarr (GH#7738) Chris Roat

• Array: correct number of outputs in apply_gufunc (GH#7669) Gabe Joseph

• Rewrite da.fromfunction with da.blockwise (GH#7704) John A Kirkham

• Rename make_meta_util to make_meta (GH#7743) GALI PREM SAGAR

• Repartition before shuffle if the requested partitions are less than input partitions (GH#7715) Vibhu Jawa

• Blockwise: handle constant key inputs (GH#7734) Mads R. B. Kristensen

• Added raise to apply_gufunc (GH#7744) Boaz Mohar

• Show failing tests summary in CI (GH#7735) Genevieve Buckley

• sizeof sets in Python 3.9 (GH#7739) Mads R. B. Kristensen

• Warn if using pandas datetimelike string in dataframe.__getitem__ (GH#7749) Julia Signell

• Highlight the client.dashboard_link (GH#7747) Genevieve Buckley

• Easier link for subscribing to the Google calendar (GH#7733) Genevieve Buckley

• Automatically show graph visualization in Jupyter notebooks (GH#7716) Genevieve Buckley

• Add autofunction for unify_chunks in API docs (GH#7730) James Bourbeau

2021.05.1

Released on May 28, 2021

• Pandas compatibility (GH#7712) Julia Signell

• Fix optimize_dataframe_getitem bug (GH#7698) Richard (Rick) Zamora

• Update make_meta import in docs (GH#7713) Benjamin Zaitlen

• Implement da.searchsorted (GH#7696) Tom White

• Fix format string in error message (GH#7706) Jiaming Yuan

• Fix read_sql_table returning wrong result for single column loads (GH#7572) c-thiel

• Add slack join link in support.rst (GH#7679) Naty Clementi

• Remove unused alphabet variable (GH#7700) James Bourbeau

• Fix meta creation incase of object (GH#7586) GALI PREM SAGAR

• Add dispatch for union_categoricals (GH#7699) GALI PREM SAGAR

• Consolidate array Dispatch objects (GH#7505) James Bourbeau

• Move DataFrame dispatch.registers to their own file (GH#7503) Julia Signell

• Fix delayed with dataclasses where init=False (GH#7656) Julia Signell

• Allow a column to be named divisions (GH#7605) Julia Signell

• Stack nd array with unknown chunks (GH#7562) Chris Roat

• Promote the 2021 Dask User Survey (GH#7694) Genevieve Buckley

• Fix typo in DataFrame.set_index() (GH#7691) James Lamb

• Cleanup array API reference links (GH#7684) David Hoese

• Accept axis tuple for flip to be consistent with NumPy (GH#7675) Andrew Champion

• Bump pre-commit hook versions (GH#7676) James Bourbeau

• Cleanup to_zarr docstring (GH#7683) David Hoese

• Fix the docstring of read_orc (GH#7678) Justus Magin

• Doc ipyparallel & mpi4py concurrent.futures (GH#7665) John A Kirkham

• Update tests to support CuPy 9 (GH#7671) Peter Andreas Entschev

• Fix some HighLevelGraph documentation inaccuracies (GH#7662) Mads R. B. Kristensen

• Fix spelling in Series getitem error message (GH#7659) Maisie Marshall

2021.05.0

Released on May 14, 2021

• Remove deprecated kind kwarg to comply with pandas 1.3.0 (GH#7653) Julia Signell

• Fix bug in DataFrame column projection (GH#7645) Richard (Rick) Zamora

• Merge global annotations when packing (GH#7565) Mads R. B. Kristensen

• Avoid inplace= in pandas set_categories (GH#7633) James Bourbeau

• Change the active-fusion default to False for Dask-Dataframe (GH#7620) Richard (Rick) Zamora

• Array: remove extraneous code from RandomState (GH#7487) Gabe Joseph

• Implement str.concat when others=None (GH#7623) Daniel Mesejo-León

• Fix dask.dataframe in sandboxed environments (GH#7601) Noah D. Brenowitz

• Support for cupyx.scipy.linalg (GH#7563) Benjamin Zaitlen

• Move timeseries and daily-stock to Blockwise (GH#7615) Richard (Rick) Zamora

• Fix bugs in broadcast join (GH#7617) Richard (Rick) Zamora

• Use Blockwise for DataFrame IO (parquet, csv, and orc) (GH#7415) Richard (Rick) Zamora

• Adding chunk & type information to Dask HighLevelGraph s (GH#7309) Genevieve Buckley

• Add pyarrow sphinx intersphinx_mapping (GH#7612) Ray Bell

• Remove skip on test freq (GH#7608) Julia Signell

• Defaults in read_parquet parameters (GH#7567) Ray Bell

• Remove ignore_abc_warning (GH#7606) Julia Signell

• Harden DataFrame merge between column-selection and index (GH#7575) Richard (Rick) Zamora

• Get rid of ignore_abc decorator (GH#7604) Julia Signell

• Remove kwarg validation for bokeh (GH#7597) Julia Signell

• Add loky example (GH#7590) Naty Clementi

• Delayed: nout when arguments become tasks (GH#7593) Gabe Joseph

• Update distributed version in mindep CI build (GH#7602) James Bourbeau

• Support all or no overlap between partition columns and real columns (GH#7541) Richard (Rick) Zamora

2021.04.1

Released on April 23, 2021

• Handle Blockwise HLG pack/unpack for concatenate=True (GH#7455) Richard (Rick) Zamora

• map_partitions: use tokenized info as name of the SubgraphCallable (GH#7524) Mads R. B. Kristensen

• Using tmp_path and tmpdir to avoid temporary files and directories hanging in the repo (GH#7592) Naty Clementi

• Contributing to docs (development guide) (GH#7591) Naty Clementi

• Add more packages to Python 3.9 CI build (GH#7588) James Bourbeau

• Array: Fix NEP-18 dispatching in finalize (GH#7508) Gabe Joseph

• Misc fixes for numpydoc (GH#7569) Matthias Bussonnier

• Avoid pandas level= keyword deprecation (GH#7577) James Bourbeau

• Map e.g. .repartition(freq="M") to .repartition(freq="MS") (GH#7504) Ruben van de Geer

• Remove hash seeding in parallel CI runs (GH#7128) Elliott Sales de Andrade

• Add defaults in parameters in to_parquet (GH#7564) Ray Bell

• Simplify transpose axes cleanup (GH#7561) Julia Signell

• Make ValueError in len(index_names) > 1 explicit it’s using fastparquet (GH#7556) Ray Bell

• Fix dict-column appending for pyarrow parquet engines (GH#7527) Richard (Rick) Zamora

• Add a documentation auto label (GH#7560) Doug Davis

• Add dask.delayed.Delayed to docs so it can be referenced by other sphinx docs (GH#7559) Doug Davis

• Fix upstream idxmaxmin for uneven split_every (GH#7538) Julia Signell

• Make normalize_token for pandas Series/DataFrame future proof (no direct block access) (GH#7318) Joris Van den Bossche

• Redesigned __setitem__ implementation (GH#7393) David Hassell

• histogram, histogramdd improvements (docs; return consistencies) (GH#7520) Doug Davis

• Force nightly pyarrow in the upstream build (GH#7530) Joris Van den Bossche

• Fix Configuration Reference (GH#7533) Benjamin Zaitlen

• Use .to_parquet on dask.dataframe in doc string (GH#7528) Ray Bell

• Avoid double msgpack serialization of HLGs (GH#7525) Mads R. B. Kristensen

• Encourage usage of yaml.safe_load() in configuration doc (GH#7529) Hristo Georgiev

• Fix reshape bug. Add relevant test. Fixes #7171. (GH#7523) JSKenyon

• Support custom_metadata= argument in to_parquet (GH#7359) Richard (Rick) Zamora

• Clean some documentation warnings (GH#7518) Daniel Mesejo-León

• Getting rid of more docs warnings (GH#7426) Julia Signell

• Added product (alias of prod) (GH#7517) Freyam Mehta

• Fix upstream __array_ufunc__ tests (GH#7494) Julia Signell

• Escape from map_overlap to map_blocks if depth is zero (GH#7481) Genevieve Buckley

• Add check_type to array assert_eq (GH#7491) Julia Signell

2021.04.0

Released on April 2, 2021

• Adding support for multidimensional histograms with dask.array.histogramdd (GH#7387) Doug Davis

• Update docs on number of threads and workers in default LocalCluster (GH#7497) cameron16

• Add labels automatically when certain files are touched in a PR (GH#7506) Julia Signell

• Extract ignore_order from kwargs (GH#7500) GALI PREM SAGAR

• Only provide installation instructions when distributed is missing (GH#7498) Matthew Rocklin

• Start adding isort (GH#7370) Julia Signell

• Add ignore_order parameter in dd.concat (GH#7473) Daniel Mesejo-León

• Use powers-of-two when displaying RAM (GH#7484) crusaderky

• Added License Classifier (GH#7485) Tom Augspurger

• Replace conda with mamba (GH#7227) crusaderky

• Fix typo in array docs (GH#7478) James Lamb

• Use concurrent.futures in local scheduler (GH#6322) John A Kirkham

2021.03.1

Released on March 26, 2021

• Add a dispatch for is_categorical_dtype to handle non-pandas objects (GH#7469) brandon-b-miller

• Use multiprocessing.Pool in test_read_text (GH#7472) John A Kirkham

• Add missing meta kwarg to gufunc class (GH#7423) Peter Andreas Entschev

• Example for memory-mapped Dask array (GH#7380) Dieter Weber

• Fix NumPy upstream failures xfail pandas and fastparquet failures (GH#7441) Julia Signell

• Fix bug in repartition with freq (GH#7357) Ruben van de Geer

• Fix __array_function__ dispatching for tril/triu (GH#7457) Peter Andreas Entschev

• Use concurrent.futures.Executors in a few tests (GH#7429) John A Kirkham

• Require NumPy >=1.16 (GH#7383) crusaderky

• Minor sort_values housekeeping (GH#7462) Ryan Williams

• Ensure natural sort order in parquet part paths (GH#7249) Ryan Williams

• Remove global env mutation upon running test_config.py (GH#7464) Hristo Georgiev

• Update NumPy intersphinx URL (GH#7460) Gabe Joseph

• Add rot90 (GH#7440) Trevor Manz

• Update docs for required package for endpoint (GH#7454) Nick Vazquez

• Master -> main in slice_array docstring (GH#7453) Gabe Joseph

• Expand dask.utils.is_arraylike docstring (GH#7445) Doug Davis

• Simplify BlockwiseIODeps importing (GH#7420) Richard (Rick) Zamora

• Update layer annotation packing method (GH#7430) James Bourbeau

• Drop duplicate test in test_describe_empty (GH#7431) John A Kirkham

• Add Series.dot method to dataframe module (GH#7236) Madhu94

• Added df kurtosis-method and testing (GH#7273) Jan Borchmann

• Avoid quadratic-time performance for HLG culling (GH#7403) Bruce Merry

• Temporarily skip problematic sparse test (GH#7421) James Bourbeau

• Update some CI workflow names (GH#7422) James Bourbeau

• Fix HDFS test (GH#7418) Julia Signell

• Make changelog subtitles match the hierarchy (GH#7419) Julia Signell

• Add support for normalize in value_counts (GH#7342) Julia Signell

• Avoid unnecessary imports for HLG Layer unpacking and materialization (GH#7381) Richard (Rick) Zamora

• Bincount fix slicing (GH#7391) Genevieve Buckley

• Add sliding_window_view (GH#7234) Deepak Cherian

• Fix typo in docs/source/develop.rst (GH#7414) Hristo Georgiev

• Switch documentation builds for PRs to readthedocs (GH#7397) James Bourbeau

• Adds sort_values to dask.DataFrame (GH#7286) gerrymanoim

• Pin sqlalchemy<1.4.0 in CI (GH#7405) James Bourbeau

• Comment fixes (GH#7215) Ryan Williams

• Dead code removal / fixes (GH#7388) Ryan Williams

• Use single thread for pa.Table.from_pandas calls (GH#7347) Richard (Rick) Zamora

• Replace 'container' with 'image' (GH#7389) James Lamb

• DOC hyperlink repartition (GH#7394) Ray Bell

• Pass delimiter to fsspec in bag.read_text (GH#7349) Martin Durant

• Update read_hdf default mode to "r" (GH#7039) rs9w33

• Embed literals in SubgraphCallable when packing Blockwise (GH#7353) Mads R. B. Kristensen

• Update test_hdf.py to not reuse file handlers (GH#7044) rs9w33

• Require additional dependencies: cloudpickle, partd, fsspec, toolz (GH#7345) Julia Signell

• Prepare Blockwise + IO infrastructure (GH#7281) Richard (Rick) Zamora

• Remove duplicated imports from test_slicing.py (GH#7365) Hristo Georgiev

• Add test deps for pip development (GH#7360) Julia Signell

• Support int slicing for non-NumPy arrays (GH#7364) Peter Andreas Entschev

• Automatically cancel previous CI builds (GH#7348) James Bourbeau

• dask.array.asarray should handle case where xarray class is in top-level namespace (GH#7335) Tom White

• HighLevelGraph length without materializing layers (GH#7274) Gabe Joseph

• Drop support for Python 3.6 (GH#7006) James Bourbeau

• Fix fsspec usage in create_metadata_file (GH#7295) Richard (Rick) Zamora

• Change default branch from master to main (GH#7198) Julia Signell

• Add Xarray to CI software environment (GH#7338) James Bourbeau

• Update repartition argument name in error text (GH#7336) Eoin Shanaghy

• Run upstream tests based on commit message (GH#7329) James Bourbeau

• Use pytest.register_assert_rewrite on util modules (GH#7278) Bruce Merry

• Add example on using specific chunk sizes in from_array() (GH#7330) James Lamb

• Move NumPy skip into test (GH#7247) Julia Signell

2021.03.0

Released on March 5, 2021

Note

This is the first release with support for Python 3.9 and the last release with support for Python 3.6

• Bump minimum version of distributed (GH#7328) James Bourbeau

• Fix percentiles_summary with dask_cudf (GH#7325) Peter Andreas Entschev

• Temporarily revert recent Array.__setitem__ updates (GH#7326) James Bourbeau

• Blockwise.clone (GH#7312) crusaderky

• NEP-35 duck array update (GH#7321) James Bourbeau

• Don’t allow setting .name for array (GH#7222) Julia Signell

• Use nearest interpolation for creating percentiles of integer input (GH#7305) Kyle Barron

• Test exp with CuPy arrays (GH#7322) John A Kirkham

• Check that computed chunks have right size and dtype (GH#7277) Bruce Merry

• pytest.mark.flaky (GH#7319) crusaderky

• Contributing docs: add note to pull the latest git tags before pip installing Dask (GH#7308) Genevieve Buckley

• Support for Python 3.9 (GH#7289) crusaderky

• Add broadcast-based merge implementation (GH#7143) Richard (Rick) Zamora

• Add split_every to graph_manipulation (GH#7282) crusaderky

• Typo in optimize docs (GH#7306) Julius Busecke

• dask.graph_manipulation support for xarray.Dataset (GH#7276) crusaderky

• Add plot width and height support for Bokeh 2.3.0 (GH#7297) James Bourbeau

• Add NumPy functions tri, triu_indices, triu_indices_from, tril_indices, tril_indices_from (GH#6997) Illviljan

• Remove “cleanup” task in DataFrame on-disk shuffle (GH#7260) Sinclair Target

• Use development version of distributed in CI (GH#7279) James Bourbeau

• Moving high level graph pack/unpack Dask (GH#7179) Mads R. B. Kristensen

• Improve performance of merge_percentiles (GH#7172) Ashwin Srinath

• DOC: add dask-sql and fugue (GH#7129) Ray Bell

• Example for working with categoricals and parquet (GH#7085) McToel

• Adds tree reduction to bincount (GH#7183) Thomas J. Fan

• Improve documentation of name in from_array (GH#7264) Bruce Merry

• Fix cumsum for empty partitions (GH#7230) Julia Signell

• Add map_blocks example to dask array creation docs (GH#7221) Julia Signell

• Fix performance issue in dask.graph_manipulation.wait_on() (GH#7258) crusaderky

• Replace coveralls with codecov.io (GH#7246) crusaderky

• Pin to a particular black rev in pre-commit (GH#7256) Julia Signell

• Minor typo in documentation: array-chunks.rst (GH#7254) Magnus Nord

• Fix bugs in Blockwise and ShuffleLayer (GH#7213) Richard (Rick) Zamora

• Fix parquet filtering bug for "pyarrow-dataset" with pyarrow-3.0.0 (GH#7200) Richard (Rick) Zamora

• graph_manipulation without NumPy (GH#7243) crusaderky

• Support for NEP-35 (GH#6738) Peter Andreas Entschev

• Avoid running unit tests during doctest CI build (GH#7240) James Bourbeau

• Run doctests on CI (GH#7238) Julia Signell

• Cleanup code quality on set arithmetics (GH#7196) crusaderky

• Add dask.array.delete (GH#7125) Julia Signell

• Unpin graphviz now that new conda-forge recipe is built (GH#7235) Julia Signell

• Don’t use NumPy 1.20 from conda-forge on Mac (GH#7211) crusaderky

• map_overlap: Don’t rechunk axes without overlap (GH#7233) Deepak Cherian

• Pin graphviz to avoid issue with latest conda-forge build (GH#7232) Julia Signell

• Use html_css_files in docs for custom CSS (GH#7220) James Bourbeau

• Graph manipulation: clone, bind, checkpoint, wait_on (GH#7109) crusaderky

• Fix handling of filter expressions in parquet pyarrow-dataset engine (GH#7186) Joris Van den Bossche

• Extend __setitem__ to more closely match numpy (GH#7033) David Hassell

• Clean up Python 2 syntax (GH#7195) crusaderky

• Fix regression in Delayed._length (GH#7194) crusaderky

• __dask_layers__() tests and tweaks (GH#7177) crusaderky

• Properly convert HighLevelGraph in multiprocessing scheduler (GH#7191) Jim Crist-Harif

• Don’t fail fast in CI (GH#7188) James Bourbeau

2021.02.0

Released on February 5, 2021

• Add percentile support for NEP-35 (GH#7162) Peter Andreas Entschev

• Added support for Float64 in column assignment (GH#7173) Nils Braun

• Coarsen rechunking error (GH#7127) Davis Bennett

• Fix upstream CI tests (GH#6896) Julia Signell

• Revise HighLevelGraph Mapping API (GH#7160) crusaderky

• Update low-level graph spec to use any hashable for keys (GH#7163) James Bourbeau

• Generically rebuild a collection with different keys (GH#7142) crusaderky

• Make easier to link issues in PRs (GH#7130) Ray Bell

• Add dask.array.append (GH#7146) D-Stacks

• Allow dask.array.ravel to accept array_like argument (GH#7138) D-Stacks

• Fixes link in array design doc (GH#7152) Thomas J. Fan

• Fix example of using blockwise for an outer product (GH#7119) Bruce Merry

• Deprecate HighlevelGraph.dicts in favor of .layers (GH#7145) Amit Kumar

• Align FastParquetEngine with pyarrow engines (GH#7091) Richard (Rick) Zamora

• Merge annotations (GH#7102) Ian Rose

• Simplify contents of parts list in read_parquet (GH#7066) Richard (Rick) Zamora

• check_meta(): use __class__ when checking DataFrame types (GH#7099) Mads R. B. Kristensen

• Cache several properties (GH#7104) Illviljan

• Fix parquet getitem optimization (GH#7106) Richard (Rick) Zamora

• Add cytoolz back to CI environment (GH#7103) James Bourbeau

2021.01.1

Released on January 22, 2021

• Partially fix cumprod (GH#7089) Julia Signell

• Test pandas 1.1.x / 1.2.0 releases and pandas nightly (GH#6996) Joris Van den Bossche

• Use assign to avoid SettingWithCopyWarning (GH#7092) Julia Signell

• 'mode' argument passed to bokeh.output_file() (GH#7034) (GH#7075) patquem

• Skip empty partitions when doing groupby.value_counts (GH#7073) Julia Signell

• Add error messages to assert_eq() (GH#7083) James Lamb

• Make cached properties read-only (GH#7077) Illviljan

2021.01.0

Released on January 15, 2021

• map_partitions with review comments (GH#6776) Kumar Bharath Prabhu

• Make sure that population is a real list (GH#7027) Julia Signell

• Propagate storage_options in read_csv (GH#7074) Richard (Rick) Zamora

• Remove all BlockwiseIO code (GH#7067) Richard (Rick) Zamora

• Fix CI (GH#7069) James Bourbeau

• Add option to control rechunking in reshape (GH#6753) Tom Augspurger

• Fix linalg.lstsq for complex inputs (GH#7056) Johnnie Gray

• Add compression='infer' default to read_csv (GH#6960) Richard (Rick) Zamora

• Revert parameter changes in svd_compressed #7003 (GH#7004) Eric Czech

• Skip failing s3 test (GH#7064) Martin Durant

• Revert BlockwiseIO (GH#7048) Richard (Rick) Zamora

• Add some cross-references to DataFrame.to_bag() and Series.to_bag() (GH#7049) Rob Malouf

• Rewrite matmul as blockwise without contraction/concatenate (GH#7000) Rafal Wojdyla

• Use functools.cached_property in da.shape (GH#7023) Illviljan

• Use meta value in series non_empty (GH#6976) Julia Signell

• Revert “Temporarly pin sphinx version to 3.3.1 (GH#7002)” (GH#7014) Rafal Wojdyla

• Revert python-graphviz pinning (GH#7037) Julia Signell

• Accidentally committed print statement (GH#7038) Julia Signell

• Pass dropna and observed in agg (GH#6992) Julia Signell

• Add index to meta after .str.split with expand (GH#7026) Ruben van de Geer

• CI: test pyarrow 2.0 and nightly (GH#7030) Joris Van den Bossche

• Temporarily pin python-graphviz in CI (GH#7031) James Bourbeau

• Underline section in numpydoc (GH#7013) Matthias Bussonnier

• Keep normal optimizations when adding custom optimizations (GH#7016) Matthew Rocklin

• Temporarily pin sphinx version to 3.3.1 (GH#7002) Rafal Wojdyla

• DOC: Misc formatting (GH#6998) Matthias Bussonnier

• Add inline_array option to from_array (GH#6773) Tom Augspurger

• Revert “Initial pass at blockwise array creation routines (GH#6931)” (:pr:`6995) James Bourbeau

• Set npartitions in set_index (GH#6978) Julia Signell

• Upstream config serialization and inheritance (GH#6987) Jacob Tomlinson

• Bump the minimum time in test_minimum_time (GH#6988) Martin Durant

• Fix pandas dtype inference for read_parquet (GH#6985) Richard (Rick) Zamora

• Avoid data loss in set_index with sorted=True (GH#6980) Richard (Rick) Zamora

• Bugfix in read_parquet for handling un-named indices with index=False (GH#6969) Richard (Rick) Zamora

• Use __class__ when comparing meta data (GH#6981) Mads R. B. Kristensen

• Comparing string versions won’t always work (GH#6979) Rafal Wojdyla

• Fix GH#6925 (GH#6982) sdementen

• Initial pass at blockwise array creation routines (GH#6931) Ian Rose

• Simplify has_parallel_type() (GH#6927) Mads R. B. Kristensen

• Handle annotation unpacking in BlockwiseIO (GH#6934) Simon Perkins

• Avoid deprecated yield_fixture in test_sql.py (GH#6968) Richard (Rick) Zamora

• Remove bad graph logic in BlockwiseIO (GH#6933) Richard (Rick) Zamora

• Get config item if variable is None (GH#6862) Jacob Tomlinson

• Update from_pandas docstring (GH#6957) Richard (Rick) Zamora

• Prevent fuse_roots from clobbering annotations (GH#6955) Simon Perkins

2020.12.0

Released on December 10, 2020

Highlights

• Switched to CalVer for versioning scheme.

• Introduced new APIs for HighLevelGraph to enable sending high-level representations of task graphs to the distributed scheduler.

• Introduced new HighLevelGraph layer objects including BasicLayer, Blockwise, BlockwiseIO, ShuffleLayer, and more.

• Added support for applying custom Layer-level annotations like priority, retries, etc. with the dask.annotations context manager.

• Updated minimum supported version of pandas to 0.25.0 and NumPy to 1.15.1.

• Support for the pyarrow.dataset API to read_parquet.

• Several fixes to Dask Array’s SVD.

All changes

• Make observed kwarg optional (GH#6952) Julia Signell

• Min supported pandas 0.25.0 numpy 1.15.1 (GH#6895) Julia Signell

• Make order of categoricals unambiguous (GH#6949) Julia Signell

• Improve “pyarrow-dataset” statistics performance for read_parquet (GH#6918) Richard (Rick) Zamora

• Add observed keyword to groupby (GH#6854) Julia Signell

• Make sure include_path_column works when there are multiple partitions per file (GH#6911) Julia Signell

• Fix: array.overlap and array.map_overlap block sizes are incorrect when depth is an unsigned bit type (GH#6909) GFleishman

• Fix syntax error in HLG docs example (GH#6946) Mark

• Return a Bag from sample (GH#6941) Shang Wang

• Add ravel_multi_index (GH#6939) Illviljan

• Enable parquet metadata collection in parallel (GH#6921) Richard (Rick) Zamora

• Avoid using _file in progressbar if it is None (GH#6938) Mark Harfouche

• Add Zarr to upstream CI build (GH#6932) James Bourbeau

• Introduce BlockwiseIO layer (GH#6878) Richard (Rick) Zamora

• Transmit Layer Annotations to Scheduler (GH#6889) Simon Perkins

• Update opportunistic caching page to remove experimental warning (GH#6926) Timost

• Allow pyarrow >2.0.0 (GH#6772) Richard (Rick) Zamora

• Support pyarrow.dataset API for read_parquet (GH#6534) Richard (Rick) Zamora

• Add more informative error message to da.coarsen when coarsening factors do not divide shape (GH#6908) Davis Bennett

• Only run the cron CI on dask/dask not forks (GH#6905) Jacob Tomlinson

• Add annotations to ShuffleLayers (GH#6913) Matthew Rocklin

• Temporarily xfail test_from_s3 (GH#6915) James Bourbeau

• Added dataframe skew method (GH#6881) Jan Borchmann

• Fix dtype in array meta (GH#6893) Julia Signell

• Missing name arg in helm install ... (GH#6903) Ruben van de Geer

• Fix: exception when reading an item with filters (GH#6901) Martin Durant

• Add support for cupyx sparse to dask.array.dot (GH#6846) Akira Naruse

• Pin array mindeps up a bit to get the tests to pass [test-mindeps] (GH#6894) Julia Signell

• Update/remove pandas and numpy in mindeps (GH#6888) Julia Signell

• Fix ArrowEngine bug in use of clear_known_categories (GH#6887) Richard (Rick) Zamora

• Fix documentation about task scheduler (GH#6879) Zhengnan Zhao

• Add human relative time formatting utility (GH#6883) Jacob Tomlinson

• Possible fix for 6864 set_index issue (GH#6866) Richard (Rick) Zamora

• BasicLayer: remove dependency arguments (GH#6859) Mads R. B. Kristensen

• Serialization of Blockwise (GH#6848) Mads R. B. Kristensen

• Address columns=[] bug (GH#6871) Richard (Rick) Zamora

• Avoid duplicate parquet schema communication (GH#6841) Richard (Rick) Zamora

• Add create_metadata_file utility for existing parquet datasets (GH#6851) Richard (Rick) Zamora

• Improve ordering for workloads with a common terminus (GH#6779) Tom Augspurger

• Stringify utilities (GH#6852) Mads R. B. Kristensen

• Add keyword overwrite=True to to_parquet to remove dangling files when overwriting a pyarrow Dataset. (GH#6825) Greg Hayes

• Removed map_tasks() and map_basic_layers() (GH#6853) Mads R. B. Kristensen

• Introduce QR iteration to svd_compressed (GH#6813) RogerMoens

• __dask_distributed_pack__() now takes a client argument (GH#6850) Mads R. B. Kristensen

• Use map_partitions instead of delayed in set_index (GH#6837) Mads R. B. Kristensen

• Add doc hit for as_completed().update(futures) (GH#6817) manuels

• Bump GHA setup-miniconda version (GH#6847) Jacob Tomlinson

• Remove nans when setting sorted index (GH#6829) Rockwell Weiner

• Fix transpose of u in SVD (GH#6799) RogerMoens

• Migrate to GitHub Actions (GH#6794) Jacob Tomlinson

• Fix sphinx currentmodule usage (GH#6839) James Bourbeau

• Fix minimum dependencies CI builds (GH#6838) James Bourbeau

• Avoid graph materialization during Blockwise culling (GH#6815) Richard (Rick) Zamora

• Fixed typo (GH#6834) Devanshu Desai

• Use HighLevelGraph.merge in collections_to_dsk (GH#6836) Mads R. B. Kristensen

• Respect dtype in svd compression_matrix #2849 (GH#6802) RogerMoens

• Add blocksize to task name (GH#6818) Julia Signell

• Check for all-NaN partitions (GH#6821) Rockwell Weiner

• Change “institutional” SQL doc section to point to main SQL doc (GH#6823) Martin Durant

• Fix: DataFrame.join doesn’t accept Series as other (GH#6809) David Katz

• Remove to_delayed operations from to_parquet (GH#6801) Richard (Rick) Zamora

• Layer annotation docstrings improvements (GH#6806) Simon Perkins

• Avro reader (GH#6780) Martin Durant

• Rechunk array if smallest chunk size is smaller than depth (GH#6708) Julia Signell

• Add Layer Annotations (GH#6767) Simon Perkins

• Add “view code” links to documentation (GH#6793) manuels

• Add optional IO-subgraph to Blockwise Layers (GH#6715) Richard (Rick) Zamora

• Add high level graph pack/unpack for distributed (GH#6786) Mads R. B. Kristensen

• Add missing methods of the Dataframe API (GH#6789) Stephannie Jimenez Gacha

• Add doc on managing environments (GH#6778) Martin Durant

• HLG: get_all_external_keys() (GH#6774) Mads R. B. Kristensen

• Avoid rechunking in reshape with chunksize=1 (GH#6748) Tom Augspurger

• Try to make categoricals work on join (GH#6205) Julia Signell

• Fix some minor typos and trailing whitespaces in array-slice.rst (GH#6771) Magnus Nord

• Bugfix for parquet metadata writes of empty dataframe partitions (pyarrow) (GH#6741) Callum Noble

• Document meta kwarg in map_blocks and map_overlap. (GH#6763) Peter Andreas Entschev

• Begin experimenting with parallel prefix scan for cumsum and cumprod (GH#6675) Erik Welch

• Clarify differences in boolean indexing between dask and numpy arrays (GH#6764) Illviljan

• Efficient serialization of shuffle layers (GH#6760) James Bourbeau

• Config array optimize to skip fusion and return a HLG (GH#6751) Mads R. B. Kristensen

• Temporarily use pyarrow<2 in CI (GH#6759) James Bourbeau

• Fix meta for min/max reductions (GH#6736) Peter Andreas Entschev

• Add 2D possibility to da.linalg.lstsq - mirroring numpy (GH#6749) Pascal Bourgault

• CI: Fixed bug causing flaky test failure in pivot (GH#6752) Tom Augspurger

• Serialization of layers (GH#6693) Mads R. B. Kristensen

• Add attrs property to Series/Dataframe (GH#6742) Illviljan

• Removed Mutable Default Argument (GH#6747) Mads R. B. Kristensen

• Adjust parquet ArrowEngine to allow more easy subclass for writing (GH#6505) Joris Van den Bossche

• Add ShuffleStage HLG Layer (GH#6650) Richard (Rick) Zamora

• Handle literal in meta_from_array (GH#6731) Peter Andreas Entschev

• Do balanced rechunking even if chunks are the same (GH#6735) Chris Roat

• Fix docstring DataFrame.set_index (GH#6739) Gil Forsyth

• Ensure HighLevelGraph layers always contain Layer instances (GH#6716) James Bourbeau

• Map on HighLevelGraph Layers (GH#6689) Mads R. B. Kristensen

• Update overlap *_like function calls and CuPy tests (GH#6728) Peter Andreas Entschev

• Fixes for svd with __array_function__ (GH#6727) Peter Andreas Entschev

• Added doctest extension for documentation (GH#6397) Jim Circadian

• Minor fix to #5628 using @pentschev’s suggestion (GH#6724) John A Kirkham

• Change type of Dask array when meta type changes (GH#5628) Matthew Rocklin

• Add az (GH#6719) Ray Bell

• HLG: get_dependencies() of single keys (GH#6699) Mads R. B. Kristensen

• Revert “Revert “Use HighLevelGraph layers everywhere in collections (GH#6510)” (GH#6697)” (GH#6707) Tom Augspurger

• Allow *_like array creation functions to respect input array type (GH#6680) Genevieve Buckley

• Update dask-sphinx-theme version (GH#6700) Gil Forsyth

2.30.0 / 2020-10-06

Array

• Allow rechunk to evenly split into N chunks (GH#6420) Scott Sievert

2.29.0 / 2020-10-02

Array

• _repr_html_: color sides darker instead of drawing all the lines (GH#6683) Julia Signell

• Removes warning from nanstd and nanvar (GH#6667) Thomas J. Fan

• Get shape of output from original array - map_overlap (GH#6682) Julia Signell

• Replace np.searchsorted with bisect in indexing (GH#6669) Joachim B Haga

Bag

• Make sure subprocesses have a consistent hash for bag groupby (GH#6660) Itamar Turner-Trauring

Core

• Revert “Use HighLevelGraph layers everywhere in collections (GH#6510)” (GH#6697) Tom Augspurger

• Use pandas.testing (GH#6687) John A Kirkham

• Improve 128-bit floating-point skip in tests (GH#6676) Elliott Sales de Andrade

DataFrame

• Allow setting dataframe items using a bool dataframe (GH#6608) Julia Signell

Documentation

• Fix typo (GH#6692) garanews

• Fix a few typos (GH#6678) Pav A

2.28.0 / 2020-09-25

Array

• Partially reverted changes to Array indexing that produces large changes. This restores the behavior from Dask 2.25.0 and earlier, with a warning when large chunks are produced. A configuration option is provided to avoid creating the large chunks, see Efficiency. (GH#6665) Tom Augspurger

• Add meta to to_dask_array (GH#6651) Kyle Nicholson

• Fix GH#6631 and GH#6611 (GH#6632) Rafal Wojdyla

• Infer object in array reductions (GH#6629) Daniel Saxton

• Adding v_based flag for svd_flip (GH#6658) Eric Czech

• Fix flakey array mean (GH#6656) Sam Grayson

Core

• Removed dsk equality check from SubgraphCallable.__eq__ (GH#6666) Mads R. B. Kristensen

• Use HighLevelGraph layers everywhere in collections (GH#6510) Mads R. B. Kristensen

• Adds hash dunder method to SubgraphCallable for caching purposes (GH#6424) Andrew Fulton

• Stop writing commented out config files by default (GH#6647) Matthew Rocklin

DataFrame

• Add support for collect list aggregation via agg API (GH#6655) Madhur Tandon

• Slightly better error message (GH#6657) Julia Signell

2.27.0 / 2020-09-18

Array

• Preserve dtype in svd (GH#6643) Eric Czech

Core

• store(): create a single HLG layer (GH#6601) Mads R. B. Kristensen

• Add pre-commit CI build (GH#6645) James Bourbeau

• Update .pre-commit-config to latest black. (GH#6641) Julia Signell

• Update super usage to remove Python 2 compatibility (GH#6630) Poruri Sai Rahul

• Remove u string prefixes (GH#6633) Poruri Sai Rahul

DataFrame

• Improve error message for to_sql (GH#6638) Julia Signell

• Use empty list as categories (GH#6626) Julia Signell

Documentation

• Add autofunction to array api docs for more ufuncs (GH#6644) James Bourbeau

• Add a number of missing ufuncs to dask.array docs (GH#6642) Ralf Gommers

• Add HelmCluster docs (GH#6290) Jacob Tomlinson

2.26.0 / 2020-09-11

Array

• Backend-aware dtype inference for single-chunk svd (GH#6623) Eric Czech

• Make array.reduction docstring match for dtype (GH#6624) Martin Durant

• Set lower bound on compression level for svd_compressed using rows and cols (GH#6622) Eric Czech

• Improve SVD consistency and small array handling (GH#6616) Eric Czech

• Add svd_flip #6599 (GH#6613) Eric Czech

• Handle sequences containing dask Arrays (GH#6595) Gabe Joseph

• Avoid large chunks from getitem with lists (GH#6514) Tom Augspurger

• Eagerly slice numpy arrays in from_array (GH#6605) Deepak Cherian

• Restore ability to pickle dask arrays (GH#6594) Noah D. Brenowitz

• Add SVD support for short-and-fat arrays (GH#6591) Eric Czech

• Add simple chunk type registry and defer as appropriate to upcast types (GH#6393) Jon Thielen

• Align coarsen chunks by default (GH#6580) Deepak Cherian

• Fixup reshape on unknown dimensions and other testing fixes (GH#6578) Ryan Williams

Core

• Add validation and fixes for HighLevelGraph dependencies (GH#6588) Mads R. B. Kristensen

• Fix linting issue (GH#6598) Tom Augspurger

• Skip bokeh version 2.0.0 (GH#6572) John A Kirkham

DataFrame

• Added bytes/row calculation when using meta (GH#6585) McToel

• Handle min_count in Series.sum / prod (GH#6618) Daniel Saxton

• Update DataFrame.set_index docstring (GH#6549) Timost

• Always compute 0 and 1 quantiles during quantile calculations (GH#6564) Erik Welch

• Fix wrong path when reading empty csv file (GH#6573) Abdulelah Bin Mahfoodh

Documentation

• Doc: Troubleshooting dashboard 404 (GH#6215) Kilian Lieret

• Fixup extraConfig example (GH#6625) Tom Augspurger

• Update supported Python versions (GH#6609) Julia Signell

• Document dask/daskhub helm chart (GH#6560) Tom Augspurger

2.25.0 / 2020-08-28

Core

• Compare key hashes in subs() (GH#6559) Mads R. B. Kristensen

• Rerun with latest black release (GH#6568) James Bourbeau

• License update (GH#6554) Tom Augspurger

DataFrame

• Add gs read_parquet example (GH#6548) Ray Bell

Documentation

• Remove version from documentation page names (GH#6558) James Bourbeau

• Update kubernetes-helm.rst (GH#6523) David Sheldon

• Stop 2020 survey (GH#6547) Tom Augspurger

2.24.0 / 2020-08-22

Array

• Fix setting random seed in tests. (GH#6518) Elliott Sales de Andrade

• Support meta in apply gufunc (GH#6521) joshreback

• Replace cupy.sparse with cupyx.scipy.sparse (GH#6530) John A Kirkham

Dataframe

• Bump up tolerance for rolling tests (GH#6502) Julia Signell

• Implement DatFrame.__len__ (GH#6515) Tom Augspurger

• Infer arrow schema in to_parquet (for ArrowEngine`) (GH#6490) Richard (Rick) Zamora

• Fix parquet test when no pyarrow (GH#6524) Martin Durant

• Remove problematic filter arguments in ArrowEngine (GH#6527) Richard (Rick) Zamora

• Avoid schema validation by default in ArrowEngine (GH#6536) Richard (Rick) Zamora

Core

• Use unpack_collections in make_blockwise_graph (GH#6517) Thomas J. Fan

• Move key_split() from optimization.py to utils.py (GH#6529) Mads R. B. Kristensen

• Make tests run on moto server (GH#6528) Martin Durant

2.23.0 / 2020-08-14

Array

• Reduce np.zeros, ones, and full array size with broadcasting (GH#6491) Matthias Bussonnier

• Add missing meta= for trim in map_overlap (GH#6494) Peter Andreas Entschev

Bag

• Bag repartition partition size (GH#6371) joshreback

Core

• Scalar.__dask_layers__() to return self._name instead of self.key (GH#6507) Mads R. B. Kristensen

• Update dependencies correctly in fuse_root optimization (GH#6508) Mads R. B. Kristensen

DataFrame

• Adds items to dataframe (GH#6503) Thomas J. Fan

• Include compression in write_table call (GH#6499) Julia Signell

• Fixed warning in nonempty_series (GH#6485) Tom Augspurger

• Intelligently determine partitions based on type of first arg (GH#6479) Matthew Rocklin

• Fix pyarrow mkdirs (GH#6475) Julia Signell

• Fix duplicate parquet output in to_parquet (GH#6451) michaelnarodovitch

Documentation

• Fix documentation da.histogram (GH#6439) Roberto Panai

• Add agg nunique example (GH#6404) Ray Bell

• Fixed a few typos in the SQL docs (GH#6489) Mike McCarty

• Docs for SQLing (GH#6453) Martin Durant

2.22.0 / 2020-07-31

Array

• Compatibility for NumPy dtype deprecation (GH#6430) Tom Augspurger

Core

• Implement sizeof for some bytes-like objects (GH#6457) John A Kirkham

• HTTP error for new fsspec (GH#6446) Martin Durant

• When RecursionError is raised, return uuid from tokenize function (GH#6437) Julia Signell

• Install deps of upstream-dev packages (GH#6431) Tom Augspurger

• Use updated link in setup.cfg (GH#6426) Zhengnan Zhao

DataFrame

• Add single quotes around column names if strings (GH#6471) Gil Forsyth

• Refactor ArrowEngine for better read_parquet performance (GH#6346) Richard (Rick) Zamora

• Add tolist dispatch (GH#6444) GALI PREM SAGAR

• Compatibility with pandas 1.1.0rc0 (GH#6429) Tom Augspurger

• Multi value pivot table (GH#6428) joshreback

• Duplicate argument definitions in to_csv docstring (GH#6411) Jun Han (Johnson) Ooi

Documentation

• Add utility to docs to convert YAML config to env vars and back (GH#6472) Jacob Tomlinson

• Fix parameter server rendering (GH#6466) Scott Sievert

• Fixes broken links (GH#6403) Jim Circadian

• Complete parameter server implementation in docs (GH#6449) Scott Sievert

• Fix typo (GH#6436) Jack Xiaosong Xu

2.21.0 / 2020-07-17

Array

• Correct error message in array.routines.gradient() (GH#6417) johnomotani

• Fix blockwise concatenate for array with some dimension=1 (GH#6342) Matthias Bussonnier

Bag

• Fix bag.take example (GH#6418) Roberto Panai

Core

• Groups values in optimization pass should only be graph and keys – not an optimization + keys (GH#6409) Benjamin Zaitlen

• Call custom optimizations once, with kwargs provided (GH#6382) Clark Zinzow

• Include pickle5 for testing on Python 3.7 (GH#6379) John A Kirkham

DataFrame

• Correct typo in error message (GH#6422) Tom McTiernan

• Use pytest.warns to check for UserWarning (GH#6378) Richard (Rick) Zamora

• Parse bytes_per_chunk keyword from string (GH#6370) Matthew Rocklin

Documentation

• Numpydoc formatting (GH#6421) Matthias Bussonnier

• Unpin numpydoc following 1.1 release (GH#6407) Gil Forsyth

• Numpydoc formatting (GH#6402) Matthias Bussonnier

• Add instructions for using conda when installing code for development (GH#6399) Ray Bell

• Update visualize docstrings (GH#6383) Zhengnan Zhao

2.20.0 / 2020-07-02

Array

• Register sizeof for numpy zero-strided arrays (GH#6343) Matthias Bussonnier

• Use concatenate_lookup in concatenate (GH#6339) John A Kirkham

• Fix rechunking of arrays with some zero-length dimensions (GH#6335) Matthias Bussonnier

DataFrame

• Dispatch iloc` calls to getitem (GH#6355) Gil Forsyth

• Handle unnamed pandas RangeIndex in fastparquet engine (GH#6350) Richard (Rick) Zamora

• Preserve index when writing partitioned parquet datasets with pyarrow (GH#6282) Richard (Rick) Zamora

• Use ignore_index for pandas’ group_split_dispatch (GH#6251) Richard (Rick) Zamora

Documentation

• Add doc describing argument (GH#6318) asmith26

2.19.0 / 2020-06-19

Array

• Cast chunk sizes to python int dtype (GH#6326) Gil Forsyth

• Add shape=None to *_like() array creation functions (GH#6064) Anderson Banihirwe

Core

• Update expected error msg for protocol difference in fsspec (GH#6331) Gil Forsyth

• Fix for floats < 1 in parse_bytes (GH#6311) Gil Forsyth

• Fix exception causes all over the codebase (GH#6308) Ram Rachum

• Fix duplicated tests (GH#6303) James Lamb

• Remove unused testing function (GH#6304) James Lamb

DataFrame

• Add high-level CSV Subgraph (GH#6262) Gil Forsyth

• Fix ValueError when merging an index-only 1-partition dataframe (GH#6309) Krishan Bhasin

• Make index.map clear divisions. (GH#6285) Julia Signell

Documentation

• Add link to 2020 survey (GH#6328) Tom Augspurger

• Update bag.rst (GH#6317) Ben Shaver

2.18.1 / 2020-06-09

Array

• Don’t try to set name on full (GH#6299) Julia Signell

• Histogram: support lazy values for range/bins (another way) (GH#6252) Gabe Joseph

Core

• Fix exception causes in utils.py (GH#6302) Ram Rachum

• Improve performance of HighLevelGraph construction (GH#6293) Julia Signell

Documentation

• Now readthedocs builds unrelased features’ docstrings (GH#6295) Antonio Ercole De Luca

• Add asyncssh intersphinx mappings (GH#6298) Jacob Tomlinson

2.18.0 / 2020-06-05

Array

• Cast slicing index to dask array if same shape as original (GH#6273) Julia Signell

• Fix stack error message (GH#6268) Stephanie Gott

• full & full_like: error on non-scalar fill_value (GH#6129) Huite

• Support for multiple arrays in map_overlap (GH#6165) Eric Czech

• Pad resample divisions so that edges are counted (GH#6255) Julia Signell

Bag

• Random sampling of k elements from a dask bag #4799 (GH#6239) Antonio Ercole De Luca

DataFrame

• Add dropna, sort, and ascending to sort_values (GH#5880) Julia Signell

• Generalize from_dask_array (GH#6263) GALI PREM SAGAR

• Add derived docstring for SeriesGroupby.nunique (GH#6284) Julia Signell

• Remove NotImplementedError in resample with rule (GH#6274) Abdulelah Bin Mahfoodh

• Add dd.to_sql (GH#6038) Ryan Williams

Documentation

• Update remote data section (GH#6258) Ray Bell

2.17.2 / 2020-05-28

Core

• Re-add the complete extra (GH#6257) Jim Crist-Harif

DataFrame

• Raise error if resample isn’t going to give right answer (GH#6244) Julia Signell

2.17.1 / 2020-05-28

Array

• Empty array rechunk (GH#6233) Andrew Fulton

Core

• Make pyyaml required (GH#6250) Jim Crist-Harif

• Fix install commands from ImportError (GH#6238) Gaurav Sheni

• Remove issue template (GH#6249) Jacob Tomlinson

DataFrame

• Pass ignore_index to dd_shuffle from DataFrame.shuffle (GH#6247) Richard (Rick) Zamora

• Cope with missing HDF keys (GH#6204) Martin Durant

• Generalize describe & quantile apis (GH#5137) GALI PREM SAGAR

2.17.0 / 2020-05-26

Array

• Small improvements to da.pad (GH#6213) Mark Boer

• Return tuple if multiple outputs in dask.array.apply_gufunc, add test to check for tuple (GH#6207) Kai Mühlbauer

• Support stack with unknown chunksizes (GH#6195) swapna

Bag

• Random Choice on Bags (GH#6208) Antonio Ercole De Luca

Core

• Raise warning delayed.visualise() (GH#6216) Amol Umbarkar

• Ensure other pickle arguments work (GH#6229) John A Kirkham

• Overhaul fuse() config (GH#6198) crusaderky

• Update dask.order.order to consider “next” nodes using both FIFO and LIFO (GH#5872) Erik Welch

DataFrame

• Use 0 as fill_value for more agg methods (GH#6245) Julia Signell

• Generalize rearrange_by_column_tasks and add DataFrame.shuffle (GH#6066) Richard (Rick) Zamora

• Xfail test_rolling_numba_engine for newer numba and older pandas (GH#6236) James Bourbeau

• Generalize fix_overlap (GH#6240) GALI PREM SAGAR

• Fix DataFrame.shape with no columns (GH#6237) noreentry

• Avoid shuffle when setting a presorted index with overlapping divisions (GH#6226) Krishan Bhasin

• Adjust the Parquet engine classes to allow more easily subclassing (GH#6211) Marius van Niekerk

• Fix dd.merge_asof with left_on='col' & right_index=True (GH#6192) noreentry

• Disable warning for concat (GH#6210) Tung Dang

• Move AUTO_BLOCKSIZE out of read_csv signature (GH#6214) Jim Crist-Harif

• .loc indexing with callable (GH#6185) Endre Mark Borza

• Avoid apply in _compute_sum_of_squares for groupby std agg (GH#6186) Richard (Rick) Zamora

• Minor correction to test_parquet (GH#6190) Brian Larsen

• Adhering to the passed pat for delimeter join and fix error message (GH#6194) GALI PREM SAGAR

• Skip test_to_parquet_with_get if no parquet libs available (GH#6188) Scott Sanderson

Documentation

• Added documentation for distributed.Event class (GH#6231) Nils Braun

• Doc write to remote (GH#6124) Ray Bell

2.16.0 / 2020-05-08

Array

• Fix array general-reduction name (GH#6176) Nick Evans

• Replace dim with shape in unravel_index (GH#6155) Julia Signell

• Moment: handle all elements being masked (GH#5339) Gabe Joseph

Core

• Remove Redundant string concatenations in dask code-base (GH#6137) GALI PREM SAGAR

• Upstream compat (GH#6159) Tom Augspurger

• Ensure sizeof of dict and sequences returns an integer (GH#6179) James Bourbeau

• Estimate python collection sizes with random sampling (GH#6154) Florian Jetter

• Update test upstream (GH#6146) Tom Augspurger

• Skip test for mindeps build (GH#6144) Tom Augspurger

• Switch default multiprocessing context to “spawn” (GH#4003) Itamar Turner-Trauring

• Update manifest to include dask-schema (GH#6140) Benjamin Zaitlen

DataFrame

• Harden inconsistent-schema handling in pyarrow-based read_parquet (GH#6160) Richard (Rick) Zamora

• Add compute kwargs to methods that write data to disk (GH#6056) Krishan Bhasin

• Fix issue where unique returns an index like result from backends (GH#6153) GALI PREM SAGAR

• Fix internal error in map_partitions with collections (GH#6103) Tom Augspurger

Documentation

• Add phase of computation to index TOC (GH#6157) Benjamin Zaitlen

• Remove unused imports in scheduling script (GH#6138) James Lamb

• Fix indent (GH#6147) Martin Durant

• Add Tom’s log config example (GH#6143) Martin Durant

2.15.0 / 2020-04-24

Array

• Update dask.array.from_array to warn when passed a Dask collection (GH#6122) James Bourbeau

• Un-numpy like behaviour in dask.array.pad (GH#6042) Mark Boer

• Add support for repeats=0 in da.repeat (GH#6080) James Bourbeau

Core

• Fix yaml layout for schema (GH#6132) Benjamin Zaitlen

• Configuration Reference (GH#6069) Benjamin Zaitlen

• Add configuration option to turn off task fusion (GH#6087) Matthew Rocklin

• Skip pyarrow on windows (GH#6094) Tom Augspurger

• Set limit to maximum length of fused key (GH#6057) Lucas Rademaker

• Add test against #6062 (GH#6072) Martin Durant

• Bump checkout action to v2 (GH#6065) James Bourbeau

DataFrame

• Generalize categorical calls to support cudf Categorical (GH#6113) GALI PREM SAGAR

• Avoid reading _metadata on every worker (GH#6017) Richard (Rick) Zamora

• Use group_split_dispatch and ignore_index in apply_concat_apply (GH#6119) Richard (Rick) Zamora

• Handle new (dtype) pandas metadata with pyarrow (GH#6090) Richard (Rick) Zamora

• Skip test_partition_on_cats_pyarrow if pyarrow is not installed (GH#6112) James Bourbeau

• Update DataFrame len to handle columns with the same name (GH#6111) James Bourbeau

• ArrowEngine bug fixes and test coverage (GH#6047) Richard (Rick) Zamora

• Added mode (GH#5958) Adam Lewis

Documentation

• Update “helm install” for helm 3 usage (GH#6130) JulianWgs

• Extend preload documentation (GH#6077) Matthew Rocklin

• Fixed small typo in DataFrame map_partitions() docstring (GH#6115) Eugene Huang

• Fix typo: “double” should be times, not plus (GH#6091) David Chudzicki

• Fix first line of array.random.* docs (GH#6063) Martin Durant

• Add section about Semaphore in distributed (GH#6053) Florian Jetter

2.14.0 / 2020-04-03

Array

• Added np.iscomplexobj implementation (GH#6045) Tom Augspurger

Core

• Update test_rearrange_disk_cleanup_with_exception to pass without cloudpickle installed (GH#6052) James Bourbeau

• Fixed flaky test-rearrange (GH#5977) Tom Augspurger

DataFrame

• Use _meta_nonempty for dtype casting in stack_partitions (GH#6061) mlondschien

• Fix bugs in _metadata creation and filtering in parquet ArrowEngine (GH#6023) Richard (Rick) Zamora

Documentation

• DOC: Add name caveats (GH#6040) Tom Augspurger

2.13.0 / 2020-03-25

Array

• Support dtype and other keyword arguments in da.random (GH#6030) Matthew Rocklin

• Register support for cupy sparse hstack/vstack (GH#5735) Corey J. Nolet

• Force self.name to str in dask.array (GH#6002) Chuanzhu Xu

Bag

• Set rename_fused_keys to None by default in bag.optimize (GH#6000) Lucas Rademaker

Core

• Copy dict in to_graphviz to prevent overwriting (GH#5996) JulianWgs

• Stricter pandas xfail (GH#6024) Tom Augspurger

• Fix CI failures (GH#6013) James Bourbeau

• Update toolz to 0.8.2 and use tlz (GH#5997) Ryan Grout

• Move Windows CI builds to GitHub Actions (GH#5862) James Bourbeau

DataFrame

• Improve path-related exceptions in read_hdf (GH#6032) psimaj

• Fix dtype handling in dd.concat (GH#6006) mlondschien

• Handle cudf’s leftsemi and leftanti joins (GH#6025) Richard J Zamora

• Remove unused npartitions variable in dd.from_pandas (GH#6019) Daniel Saxton

• Added shuffle to DataFrame.random_split (GH#5980) petiop

Documentation

• Fix indentation in scheduler-overview docs (GH#6022) Matthew Rocklin

• Update task graphs in optimize docs (GH#5928) Julia Signell

• Optionally get rid of intermediary boxes in visualize, and add more labels (GH#5976) Julia Signell

2.12.0 / 2020-03-06

Array

• Improve reuse of temporaries with numpy (GH#5933) Bruce Merry

• Make map_blocks with block_info produce a Blockwise (GH#5896) Bruce Merry

• Optimize make_blockwise_graph (GH#5940) Bruce Merry

• Fix axes ordering in da.tensordot (GH#5975) Gil Forsyth

• Adds empty mode to array.pad (GH#5931) Thomas J. Fan

Core

• Remove toolz.memoize dependency in dask.utils (GH#5978) Ryan Grout

• Close pool leaking subprocess (GH#5979) Tom Augspurger

• Pin numpydoc to 0.8.0 (fix double autoescape) (GH#5961) Gil Forsyth

• Register deterministic tokenization for range objects (GH#5947) James Bourbeau

• Unpin msgpack in CI (GH#5930) JAmes Bourbeau

• Ensure dot results are placed in unique files. (GH#5937) Elliott Sales de Andrade

• Add remaining optional dependencies to Travis 3.8 CI build environment (GH#5920) James Bourbeau

DataFrame

• Skip parquet getitem optimization for some keys (GH#5917) Tom Augspurger

• Add ignore_index argument to rearrange_by_column code path (GH#5973) Richard J Zamora

• Add DataFrame and Series memory_usage_per_partition methods (GH#5971) James Bourbeau

• xfail test_describe when using Pandas 0.24.2 (GH#5948) James Bourbeau

• Implement dask.dataframe.to_numeric (GH#5929) Julia Signell

• Add new error message content when columns are in a different order (GH#5927) Julia Signell

• Use shallow copy for assign operations when possible (GH#5740) Richard J Zamora

Documentation

• Changed above to below in dask.array.triu docs (GH#5984) Henrik Andersson

• Array slicing: fix typo in slice_with_int_dask_array error message (GH#5981) Gabe Joseph

• Grammar and formatting updates to docstrings (GH#5963) James Lamb

• Update develop doc with conda option (GH#5939) Ray Bell

• Update title of DataFrame extension docs (GH#5954) James Bourbeau

• Fixed typos in documentation (GH#5962) James Lamb

• Add original class or module as a kwarg on _bind_* methods (GH#5946) Julia Signell

• Add collect list example (GH#5938) Ray Bell

• Update optimization doc for python 3 (GH#5926) Julia Signell

2.11.0 / 2020-02-19

Array

• Cache result of Array.shape (GH#5916) Bruce Merry

• Improve accuracy of estimate_graph_size for rechunk (GH#5907) Bruce Merry

• Skip rechunk steps that do not alter chunking (GH#5909) Bruce Merry

• Support dtype and other kwargs in coarsen (GH#5903) Matthew Rocklin

• Push chunk override from map_blocks into blockwise (GH#5895) Bruce Merry

• Avoid using rewrite_blockwise for a singleton (GH#5890) Bruce Merry

• Optimize slices_from_chunks (GH#5891) Bruce Merry

• Avoid unnecessary __getitem__ in block() when chunks have correct dimensionality (GH#5884) Thomas Robitaille

Bag

• Add include_path option for dask.bag.read_text (GH#5836) Yifan Gu

• Fixes ValueError in delayed execution of bagged NumPy array (GH#5828) Surya Avala

Core

• CI: Pin msgpack (GH#5923) Tom Augspurger

• Rename test_inner to test_outer (GH#5922) Shiva Raisinghani

• quote should quote dicts too (GH#5905) Bruce Merry

• Register a normalizer for literal (GH#5898) Bruce Merry

• Improve layer name synthesis for non-HLGs (GH#5888) Bruce Merry

• Replace flake8 pre-commit-hook with upstream (GH#5892) Julia Signell

• Call pip as a module to avoid warnings (GH#5861) Cyril Shcherbin

• Close ThreadPool at exit (GH#5852) Tom Augspurger

• Remove dask.dataframe import in tokenization code (GH#5855) James Bourbeau

DataFrame

• Require pandas>=0.23 (GH#5883) Tom Augspurger

• Remove lambda from dataframe aggregation (GH#5901) Matthew Rocklin

• Fix exception chaining in dataframe/__init__.py (GH#5882) Ram Rachum

• Add support for reductions on empty dataframes (GH#5804) Shiva Raisinghani

• Expose sort= argument for groupby (GH#5801) Richard J Zamora

• Add df.empty property (GH#5711) rockwellw

• Use parquet read speed-ups from fastparquet.api.paths_to_cats. (GH#5821) Igor Gotlibovych

Documentation

• Deprecate doc_wraps (GH#5912) Tom Augspurger

• Update array internal design docs for HighLevelGraph era (GH#5889) Bruce Merry

• Move over dashboard connection docs (GH#5877) Matthew Rocklin

• Move prometheus docs from distributed.dask.org (GH#5876) Matthew Rocklin

• Removing duplicated DO block at the end (GH#5878) K.-Michael Aye

• map_blocks see also (GH#5874) Tom Augspurger

• More derived from (GH#5871) Julia Signell

• Fix typo (GH#5866) Yetunde Dada

• Fix typo in cloud.rst (GH#5860) Andrew Thomas

• Add note pointing to code of conduct and diversity statement (GH#5844) Matthew Rocklin

2.10.1 / 2020-01-30

• Fix Pandas 1.0 version comparison (GH#5851) Tom Augspurger

• Fix typo in distributed diagnostics documentation (GH#5841) Gerrit Holl

2.10.0 / 2020-01-28

• Support for pandas 1.0’s new BooleanDtype and StringDtype (GH#5815) Tom Augspurger

• Compatibility with pandas 1.0’s API breaking changes and deprecations (GH#5792) Tom Augspurger

• Fixed non-deterministic tokenization of some extension-array backed pandas objects (GH#5813) Tom Augspurger

• Fixed handling of dataclass class objects in collections (GH#5812) Matteo De Wint

• Fixed resampling with tz-aware dates when one of the endpoints fell in a non-existent time (GH#5807) dfonnegra

• Delay initial Zarr dataset creation until the computation occurs (GH#5797) Chris Roat

• Use parquet dataset statistics in more cases with the pyarrow engine (GH#5799) Richard J Zamora

• Fixed exception in groupby.std() when some of the keys were large integers (GH#5737) H. Thomson Comer

2.9.2 / 2020-01-16

Array

• Unify chunks in broadcast_arrays (GH#5765) Matthew Rocklin

Core

• xfail CSV encoding tests (GH#5791) Tom Augspurger

• Update order to handle empty dask graph (GH#5789) James Bourbeau

• Redo dask.order.order (GH#5646) Erik Welch

DataFrame

• Add transparent compression for on-disk shuffle with partd (GH#5786) Christian Wesp

• Fix repr for empty dataframes (GH#5781) Shiva Raisinghani

• Pandas 1.0.0RC0 compat (GH#5784) Tom Augspurger

• Remove buggy assertions (GH#5783) Tom Augspurger

• Pandas 1.0 compat (GH#5782) Tom Augspurger

• Fix bug in pyarrow-based read_parquet on partitioned datasets (GH#5777) Richard J Zamora

• Compat for pandas 1.0 (GH#5779) Tom Augspurger

• Fix groupby/mean error with with categorical index (GH#5776) Richard J Zamora

• Support empty partitions when performing cumulative aggregation (GH#5730) Matthew Rocklin

• set_index accepts single-item unnested list (GH#5760) Wes Roach

• Fixed partitioning in set index for ordered Categorical (GH#5715) Tom Augspurger

Documentation

• Note additional use case for normalize_token.register (GH#5766) Thomas A Caswell

• Update bag repartition docstring (GH#5772) Timost

• Small typos (GH#5771) Maarten Breddels

• Fix typo in Task Expectations docs (GH#5767) James Bourbeau

• Add docs section on task expectations to graph page (GH#5764) Devin Petersohn

2.9.1 / 2019-12-27

Array

• Support Array.view with dtype=None (GH#5736) Anderson Banihirwe

• Add dask.array.nanmedian (GH#5684) Deepak Cherian

Core

• xfail test_temporary_directory on Python 3.8 (GH#5734) James Bourbeau

• Add support for Python 3.8 (GH#5603) James Bourbeau

• Use id to dedupe constants in rewrite_blockwise (GH#5696) Jim Crist

DataFrame

• Raise error when converting a dask dataframe scalar to a boolean (GH#5743) James Bourbeau

• Ensure dataframe groupby-variance is greater than zero (GH#5728) Matthew Rocklin

• Fix DataFrame.__iter__ (GH#5719) Tom Augspurger

• Support Parquet filters in disjunctive normal form, like PyArrow (GH#5656) Matteo De Wint

• Auto-detect categorical columns in ArrowEngine-based read_parquet (GH#5690) Richard J Zamora

• Skip parquet getitem optimization tests if no engine found (GH#5697) James Bourbeau

• Fix independent optimization of parquet-getitem (GH#5613) Tom Augspurger

Documentation

• Update helm config doc (GH#5750) Ray Bell

• Link to examples.dask.org in several places (GH#5733) Tom Augspurger

• Add missing ” in performance report example (GH#5724) James Bourbeau

• Resolve several documentation build warnings (GH#5685) James Bourbeau

• add info on performance_report (GH#5713) Benjamin Zaitlen

• Add more docs disclaimers (GH#5710) Julia Signell

• Fix simple typo: wihout -> without (GH#5708) Tim Gates

• Update numpydoc dependency (GH#5694) James Bourbeau

2.9.0 / 2019-12-06

Array

• Fix da.std to work with NumPy arrays (GH#5681) James Bourbeau

Core

• Register sizeof functions for Numba and RMM (GH#5668) John A Kirkham

• Update meeting time (GH#5682) Tom Augspurger

DataFrame

• Modify dd.DataFrame.drop to use shallow copy (GH#5675) Richard J Zamora

• Fix bug in _get_md_row_groups (GH#5673) Richard J Zamora

• Close sqlalchemy engine after querying DB (GH#5629) Krishan Bhasin

• Allow dd.map_partitions to not enforce meta (GH#5660) Matthew Rocklin

• Generalize concat_unindexed_dataframes to support cudf-backend (GH#5659) Richard J Zamora

• Add dataframe resample methods (GH#5636) Benjamin Zaitlen

• Compute length of dataframe as length of first column (GH#5635) Matthew Rocklin

Documentation

• Doc fixup (GH#5665) James Bourbeau

• Update doc build instructions (GH#5640) James Bourbeau

• Fix ADL link (GH#5639) Ray Bell

• Add documentation build (GH#5617) James Bourbeau

2.8.1 / 2019-11-22

Array

• Use auto rechunking in da.rechunk if no value given (GH#5605) Matthew Rocklin

Core

• Add simple action to activate GH actions (GH#5619) James Bourbeau

DataFrame

• Fix “file_path_0” bug in aggregate_row_groups (GH#5627) Richard J Zamora

• Add chunksize argument to read_parquet (GH#5607) Richard J Zamora

• Change test_repartition_npartitions to support arch64 architecture (GH#5620) ossdev07

• Categories lost after groupby + agg (GH#5423) Oliver Hofkens

• Fixed relative path issue with parquet metadata file (GH#5608) Nuno Gomes Silva

• Enable gpu-backed covariance/correlation in dataframes (GH#5597) Richard J Zamora

Documentation

• Fix institutional faq and unknown doc warnings (GH#5616) James Bourbeau

• Add doc for some utils (GH#5609) Tom Augspurger

• Removes html_extra_path (GH#5614) James Bourbeau

• Fixed See Also referencence (GH#5612) Tom Augspurger

2.8.0 / 2019-11-14

Array

• Implement complete dask.array.tile function (GH#5574) Bouwe Andela

• Add median along an axis with automatic rechunking (GH#5575) Matthew Rocklin

• Allow da.asarray to chunk inputs (GH#5586) Matthew Rocklin

Bag

• Use key_split in Bag name (GH#5571) Matthew Rocklin

Core

• Switch Doctests to Py3.7 (GH#5573) Ryan Nazareth

• Relax get_colors test to adapt to new Bokeh release (GH#5576) Matthew Rocklin

• Add dask.blockwise.fuse_roots optimization (GH#5451) Matthew Rocklin

• Add sizeof implementation for small dicts (GH#5578) Matthew Rocklin

• Update fsspec, gcsfs, s3fs (GH#5588) Tom Augspurger

DataFrame

• Add dropna argument to groupby (GH#5579) Richard J Zamora

• Revert “Remove import of dask_cudf, which is now a part of cudf (GH#5568)” (GH#5590) Matthew Rocklin

Documentation

• Add best practice for dask.compute function (GH#5583) Matthew Rocklin

• Create FUNDING.yml (GH#5587) Gina Helfrich

• Add screencast for coordination primitives (GH#5593) Matthew Rocklin

• Move funding to .github repo (GH#5589) Tom Augspurger

• Update calendar link (GH#5569) Tom Augspurger

2.7.0 / 2019-11-08

This release drops support for Python 3.5

Array

• Reuse code for assert_eq util method (GH#5496) Vijayant

• Update da.array to always return a dask array (GH#5510) James Bourbeau

• Skip transpose on trivial inputs (GH#5523) Ryan Abernathey

• Avoid NumPy scalar string representation in tokenize (GH#5527) James Bourbeau

• Remove unnecessary tiledb shape constraint (GH#5545) Norman Barker

• Removes bytes from sparse array HTML repr (GH#5556) James Bourbeau

Core

• Drop Python 3.5 (GH#5528) James Bourbeau

• Update the use of fixtures in distributed tests (GH#5497) Matthew Rocklin

• Changed deprecated bokeh-port to dashboard-address (GH#5507) darindf

• Avoid updating with identical dicts in ensure_dict (GH#5501) James Bourbeau

• Test Upstream (GH#5516) Tom Augspurger

• Accelerate reverse_dict (GH#5479) Ryan Grout

• Update test_imports.sh (GH#5534) James Bourbeau

• Support cgroups limits on cpu count in multiprocess and threaded schedulers (GH#5499) Albert DeFusco

• Update minimum pyarrow version on CI (GH#5562) James Bourbeau

• Make cloudpickle optional (GH#5511) crusaderky

DataFrame

• Add an example of index_col usage (GH#3072) Bruno Bonfils

• Explicitly use iloc for row indexing (GH#5500) Krishan Bhasin

• Accept dask arrays on columns assignemnt (GH#5224) Henrique Ribeiro-

• Implement unique and value_counts for SeriesGroupBy (GH#5358) Scott Sievert

• Add sizeof definition for pyarrow tables and columns (GH#5522) Richard J Zamora

• Enable row-group task partitioning in pyarrow-based read_parquet (GH#5508) Richard J Zamora

• Removes npartitions=’auto’ from dd.merge docstring (GH#5531) James Bourbeau

• Apply enforce error message shows non-overlapping columns. (GH#5530) Tom Augspurger

• Optimize meta_nonempty for repetitive dtypes (GH#5553) Petio Petrov

• Remove import of dask_cudf, which is now a part of cudf (GH#5568) Mads R. B. Kristensen

Documentation

• Make capitalization more consistent in FAQ docs (GH#5512) Matthew Rocklin

• Add CONTRIBUTING.md (GH#5513) Jacob Tomlinson

• Document optional dependencies (GH#5456) Prithvi MK

• Update helm chart docs to reflect new chart repo (GH#5539) Jacob Tomlinson

• Add Resampler to API docs (GH#5551) James Bourbeau

• Fix typo in read_sql_table (GH#5554) Eric Dill

• Add adaptive deployments screencast [skip ci] (GH#5566) Matthew Rocklin

2.6.0 / 2019-10-15

Core

• Call ensure_dict on graphs before entering toolz.merge (GH#5486) Matthew Rocklin

• Consolidating hash dispatch functions (GH#5476) Richard J Zamora

DataFrame

• Support Python 3.5 in Parquet code (GH#5491) Benjamin Zaitlen

• Avoid identity check in warn_dtype_mismatch (GH#5489) Tom Augspurger

• Enable unused groupby tests (GH#3480) Jörg Dietrich

• Remove old parquet and bcolz dataframe optimizations (GH#5484) Matthew Rocklin

• Add getitem optimization for read_parquet (GH#5453) Tom Augspurger

• Use _constructor_sliced method to determine Series type (GH#5480) Richard J Zamora

• Fix map(series) for unsorted base series index (GH#5459) Justin Waugh

• Fix KeyError with Groupby label (GH#5467) Ryan Nazareth

Documentation

• Use Zoom meeting instead of appear.in (GH#5494) Matthew Rocklin

• Added curated list of resources (GH#5460) Javad

• Update SSH docs to include SSHCluster (GH#5482) Matthew Rocklin

• Update “Why Dask?” page (GH#5473) Matthew Rocklin

• Fix typos in docstrings (GH#5469) garanews

2.5.2 / 2019-10-04

Array

• Correct chunk size logic for asymmetric overlaps (GH#5449) Ben Jeffery

• Make da.unify_chunks public API (GH#5443) Matthew Rocklin

DataFrame

• Fix dask.dataframe.fillna handling of Scalar object (GH#5463) Zhenqing Li

Documentation

• Remove boxes in Spark comparison page (GH#5445) Matthew Rocklin

• Add latest presentations (GH#5446) Javad

• Update cloud documentation (GH#5444) Matthew Rocklin

2.5.0 / 2019-09-27

Core

• Add sentinel no_default to get_dependencies task (GH#5420) James Bourbeau

• Update fsspec version (GH#5415) Matthew Rocklin

• Remove PY2 checks (GH#5400) Jim Crist

DataFrame

• Add option to not check meta in dd.from_delayed (GH#5436) Christopher J. Wright

• Fix test_timeseries_nulls_in_schema failures with pyarrow master (GH#5421) Richard J Zamora

• Reduce read_metadata output size in pyarrow/parquet (GH#5391) Richard J Zamora

• Test numeric edge case for repartition with npartitions. (GH#5433) amerkel2

• Unxfail pandas-datareader test (GH#5430) Tom Augspurger

• Add DataFrame.pop implementation (GH#5422) Matthew Rocklin

• Enable merge/set_index for cudf-based dataframes with cupy values (GH#5322) Richard J Zamora

• drop_duplicates support for positional subset parameter (GH#5410) Wes Roach

Documentation

• Add screencasts to array, bag, dataframe, delayed, futures and setup (GH#5429) (GH#5424) Matthew Rocklin

• Fix delimeter parsing documentation (GH#5428) Mahmut Bulut

• Update overview image (GH#5404) James Bourbeau

2.4.0 / 2019-09-13

Array

• Adds explicit h5py.File mode (GH#5390) James Bourbeau

• Provides method to compute unknown array chunks sizes (GH#5312) Scott Sievert

• Ignore runtime warning in Array compute_meta (GH#5356) estebanag

• Add _meta to Array.__dask_postpersist__ (GH#5353) Benoit Bovy

• Fixup da.asarray and da.asanyarray for datetime64 dtype and xarray objects (GH#5334) Stephan Hoyer

• Add shape implementation (GH#5293) Tom Augspurger

• Add chunktype to array text repr (GH#5289) James Bourbeau

• Array.random.choice: handle array-like non-arrays (GH#5283) Gabe Joseph

Core

• Remove deprecated code (GH#5401) Jim Crist

• Fix funcname when vectorized func has no __name__ (GH#5399) James Bourbeau

• Truncate funcname to avoid long key names (GH#5383) Matthew Rocklin

• Add support for numpy.vectorize in funcname (GH#5396) James Bourbeau

• Fixed HDFS upstream test (GH#5395) Tom Augspurger

• Support numbers and None in parse_bytes/timedelta (GH#5384) Matthew Rocklin

• Fix tokenizing of subindexes on memmapped numpy arrays (GH#5351) Henry Pinkard

• Upstream fixups (GH#5300) Tom Augspurger

DataFrame

• Allow pandas to cast type of statistics (GH#5402) Richard J Zamora

• Preserve index dtype after applying dd.pivot_table (GH#5385) therhaag

• Implement explode for Series and DataFrame (GH#5381) Arpit Solanki

• set_index on categorical fails with less categories than partitions (GH#5354) Oliver Hofkens

• Support output to a single CSV file (GH#5304) Hongjiu Zhang

• Add groupby().transform() (GH#5327) Oliver Hofkens

• Adding filter kwarg to pyarrow dataset call (GH#5348) Richard J Zamora

• Implement and check compression defaults for parquet (GH#5335) Sarah Bird

• Pass sqlalchemy params to delayed objects (GH#5332) Arpit Solanki

• Fixing schema handling in arrow-parquet (GH#5307) Richard J Zamora

• Add support for DF and Series groupby().idxmin/max() (GH#5273) Oliver Hofkens

• Add correlation calculation and add test (GH#5296) Benjamin Zaitlen

Documentation

• Numpy docstring standard has moved (GH#5405) Wes Roach

• Reference correct NumPy array name (GH#5403) Wes Roach

• Minor edits to Array chunk documentation (GH#5372) Scott Sievert

• Add methods to API docs (GH#5387) Tom Augspurger

• Add namespacing to configuration example (GH#5374) Matthew Rocklin

• Add get_task_stream and profile to the diagnostics page (GH#5375) Matthew Rocklin

• Add best practice to load data with Dask (GH#5369) Matthew Rocklin

• Update institutional-faq.rst (GH#5345) DomHudson

• Add threads and processes note to the best practices (GH#5340) Matthew Rocklin

• Update cuDF links (GH#5328) James Bourbeau

• Fixed small typo with parentheses placement (GH#5311) Eugene Huang

• Update link in reshape docstring (GH#5297) James Bourbeau

2.3.0 / 2019-08-16

Array

• Raise exception when from_array is given a dask array (GH#5280) David Hoese

• Avoid adjusting gufunc’s meta dtype twice (GH#5274) Peter Andreas Entschev

• Add meta= keyword to map_blocks and add test with sparse (GH#5269) Matthew Rocklin

• Add rollaxis and moveaxis (GH#4822) Tobias de Jong

• Always increment old chunk index (GH#5256) James Bourbeau

• Shuffle dask array (GH#3901) Tom Augspurger

• Fix ordering when indexing a dask array with a bool dask array (GH#5151) James Bourbeau

Bag

• Add workaround for memory leaks in bag generators (GH#5208) Marco Neumann

Core

• Set strict xfail option (GH#5220) James Bourbeau

• test-upstream (GH#5267) Tom Augspurger

• Fixed HDFS CI failure (GH#5234) Tom Augspurger

• Error nicely if no file size inferred (GH#5231) Jim Crist

• A few changes to config.set (GH#5226) Jim Crist

• Fixup black string normalization (GH#5227) Jim Crist

• Pin NumPy in windows tests (GH#5228) Jim Crist

• Ensure parquet tests are skipped if fastparquet and pyarrow not installed (GH#5217) James Bourbeau

• Add fsspec to readthedocs (GH#5207) Matthew Rocklin

• Bump NumPy and Pandas to 1.17 and 0.25 in CI test (GH#5179) John A Kirkham

DataFrame

• Fix DataFrame.query docstring (incorrect numexpr API) (GH#5271) Doug Davis

• Parquet metadata-handling improvements (GH#5218) Richard J Zamora

• Improve messaging around sorted parquet columns for index (GH#5265) Martin Durant

• Add rearrange_by_divisions and set_index support for cudf (GH#5205) Richard J Zamora

• Fix groupby.std() with integer colum names (GH#5096) Nicolas Hug

• Add Series.__iter__ (GH#5071) Blane

• Generalize hash_pandas_object to work for non-pandas backends (GH#5184) GALI PREM SAGAR

• Add rolling cov (GH#5154) Ivars Geidans

• Add columns argument in drop function (GH#5223) Henrique Ribeiro

Documentation

• Update institutional FAQ doc (GH#5277) Matthew Rocklin

• Add draft of institutional FAQ (GH#5214) Matthew Rocklin

• Make boxes for dask-spark page (GH#5249) Martin Durant

• Add motivation for shuffle docs (GH#5213) Matthew Rocklin

• Fix links and API entries for best-practices (GH#5246) Martin Durant

• Remove “bytes” (internal data ingestion) doc page (GH#5242) Martin Durant

• Redirect from our local distributed page to distributed.dask.org (GH#5248) Matthew Rocklin

• Cleanup API page (GH#5247) Matthew Rocklin

• Remove excess endlines from install docs (GH#5243) Matthew Rocklin

• Remove item list in phases of computation doc (GH#5245) Martin Durant

• Remove custom graphs from the TOC sidebar (GH#5241) Matthew Rocklin

• Remove experimental status of custom collections (GH#5236) James Bourbeau

• Adds table of contents to Why Dask? (GH#5244) James Bourbeau

• Moves bag overview to top-level bag page (GH#5240) James Bourbeau

• Remove use-cases in favor of stories.dask.org (GH#5238) Matthew Rocklin

• Removes redundant TOC information in index.rst (GH#5235) James Bourbeau

• Elevate dashboard in distributed diagnostics documentation (GH#5239) Martin Durant

• Updates “add” layer in HLG docs example (GH#5237) James Bourbeau

• Update GUFunc documentation (GH#5232) Matthew Rocklin

2.2.0 / 2019-08-01

Array

• Use da.from_array(…, asarray=False) if input follows NEP-18 (GH#5074) Matthew Rocklin

• Add missing attributes to from_array documentation (GH#5108) Peter Andreas Entschev

• Fix meta computation for some reduction functions (GH#5035) Peter Andreas Entschev

• Raise informative error in to_zarr if unknown chunks (GH#5148) James Bourbeau

• Remove invalid pad tests (GH#5122) Tom Augspurger

• Ignore NumPy warnings in compute_meta (GH#5103) Peter Andreas Entschev

• Fix kurtosis calc for single dimension input array (GH#5177) @andrethrill

• Support Numpy 1.17 in tests (GH#5192) Matthew Rocklin

Bag

• Supply pool to bag test to resolve intermittent failure (GH#5172) Tom Augspurger

Core

• Base dask on fsspec (GH#5064) (GH#5121) Martin Durant

• Various upstream compatibility fixes (GH#5056) Tom Augspurger

• Make distributed tests optional again. (GH#5128) Elliott Sales de Andrade

• Fix HDFS in dask (GH#5130) Martin Durant

• Ignore some more invalid value warnings. (GH#5140) Elliott Sales de Andrade

DataFrame

• Fix pd.MultiIndex size estimate (GH#5066) Brett Naul

• Generalizing has_known_categories (GH#5090) GALI PREM SAGAR

• Refactor Parquet engine (GH#4995) Richard J Zamora

• Add divide method to series and dataframe (GH#5094) msbrown47

• fix flaky partd test (GH#5111) Tom Augspurger

• Adjust is_dataframe_like to adjust for value_counts change (GH#5143) Tom Augspurger

• Generalize rolling windows to support non-Pandas dataframes (GH#5149) Nick Becker

• Avoid unnecessary aggregation in pivot_table (GH#5173) Daniel Saxton

• Add column names to apply_and_enforce error message (GH#5180) Matthew Rocklin

• Add schema keyword argument to to_parquet (GH#5150) Sarah Bird

• Remove recursion error in accessors (GH#5182) Jim Crist

• Allow fastparquet to handle gather_statistics=False for file lists (GH#5157) Richard J Zamora

Documentation

• Adds NumFOCUS badge to the README (GH#5086) James Bourbeau

• Update developer docs [ci skip] (GH#5093) Jim Crist

• Document DataFrame.set_index computataion behavior Natalya Rapstine

• Use pip install . instead of calling setup.py (GH#5139) Matthias Bussonier

• Close user survey (GH#5147) Tom Augspurger

• Fix Google Calendar meeting link (GH#5155) Loïc Estève

• Add docker image customization example (GH#5171) James Bourbeau

• Update remote-data-services after fsspec (GH#5170) Martin Durant

• Fix typo in spark.rst (GH#5164) Xavier Holt

• Update setup/python docs for async/await API (GH#5163) Matthew Rocklin

• Update Local Storage HPC documentation (GH#5165) Matthew Rocklin