|
| 1 | +--- |
| 2 | +title: 'CuPy-Xarray: Xarray on GPUs!' |
| 3 | +date: '2024-01-17' |
| 4 | +authors: |
| 5 | + - name: Negin Sobhani |
| 6 | + github: negin513 |
| 7 | + - name: Deepak Cherian |
| 8 | + github: dcherian |
| 9 | + - name: Max Jones |
| 10 | + github: maxrjones |
| 11 | +summary: 'CuPy-Xarray is a Python library that leverages CuPy, a GPU array library, and Xarray, a library for multi-dimensional labeled array computations, to enable fast and efficient data processing on GPUs.' |
| 12 | +--- |
| 13 | + |
| 14 | +## TLDR |
| 15 | + |
| 16 | +The [CuPy-Xarray](https://github.com/xarray-contrib/cupy-xarray) project makes mixing GPU acceleration with Xarray workflows very convenient! Explore the [new documentation](https://cupy-xarray.readthedocs.io/) and tutorials to explore how CuPy-Xarray enables GPU accelerations on large multidimensional datasets. 🎉 🥳 🚀 |
| 17 | + |
| 18 | +## Background |
| 19 | + |
| 20 | +### What is CuPy-Xarray? |
| 21 | + |
| 22 | +[CuPy](https://cupy.dev) is a GPU-accelerated library for numerical computations. CuPy provides a NumPy-like array object -- a duck array -- that follows various standard array protocols and executes computations on CUDA-capable devices. Xarray can [wrap duck array](https://docs.xarray.dev/en/stable/user-guide/duckarrays.html) objects (i.e. NumPy-like arrays) that follow specific protocols. |
| 23 | + |
| 24 | +Thus Xarray can handle CuPy arrays, and `cupy-xarray` provides a number of useful methods under the `xarray_object.cupy` namespace, allowing seamless transition between CPU and GPU computations in your data pipeline. |
| 25 | + |
| 26 | +### Why is this important? |
| 27 | + |
| 28 | +GPU acceleration is becoming increasingly important in scientific research, data analysis, and AI/ML techniques due to its ability to perform massively parallel computations. GPUs can greatly accelerate the processing of array datasets, allowing for faster analysis and modeling of large datasets. By leveraging the power of GPUs with tools such as CuPy and CuPy-Xarray, Xarray users can gain significant performance improvements and unlock new opportunities for scientific discovery. |
| 29 | + |
| 30 | +## New Documentation and Tutorials |
| 31 | + |
| 32 | +We have recently created detailed documentation with examples to help users get started with CuPy-Xarray. Check it out at [this link](https://cupy-xarray.readthedocs.io/). |
| 33 | + |
| 34 | +The new documentation offers the following topics: |
| 35 | + |
| 36 | +1. **[Basics of CuPy](https://cupy-xarray.readthedocs.io/source/cupy-basics.html)** : An introduction to CuPy, basics of GPU computing, and data transfer between host and device. |
| 37 | +2. **[Introduction to CuPy-Xarray](https://cupy-xarray.readthedocs.io/source/introduction.html)** |
| 38 | +3. **[Basic Computations with CuPy-Xarray](https://cupy-xarray.readthedocs.io/source/basic-computations.html)** |
| 39 | +4. **[High-level Computation with CuPy-Xarray](https://cupy-xarray.readthedocs.io/source/basic-computations.html#)** : Applying high-level functions like `groupby`, `resample`, `rolling`, and `apply_ufunc` to xarray objects. |
| 40 | +5. **[Custom Kernels with `apply_ufunc`](https://cupy-xarray.readthedocs.io/source/apply-ufunc.html)** : Custom CUDA kernels for `apply_ufunc` and how to use `apply_ufunc` with `groupby` and `resample`. |
| 41 | +6. **[A real world example](https://cupy-xarray.readthedocs.io/source/real-example-1.html)** : This section introduces how to use CuPy-Xarray to accelerate a real world earth system model analysis workflow. In this demo, we used the NASA Earth Exchange Global Daily Downscaled Projections (NEX-GDDP-CMIP6) to demonstrate how to use CuPy-Xarray to speed-up computations on climate data variables. |
| 42 | + |
| 43 | +If you have any questions, encounter issues, or want to contribute, the [community forum](https://discourse.pangeo.io) is a great place to start. |
| 44 | + |
| 45 | +## Upstream Work |
| 46 | + |
| 47 | +We also worked to improve upstream support for the primitives that Xarray needs. For example this [pull request](https://github.com/cupy/cupy/pull/7575) enabled the use of Xarray's `.rolling` methods. An open [pull request](https://github.com/cupy/cupy/pull/7811), when merged, will make it more clear when Xarray objects are wrapping CuPy arrays. |
| 48 | + |
| 49 | +## Summary |
| 50 | + |
| 51 | +CuPy-Xarray is a Python library helps you use CuPy, a GPU array library, and Xarray, a library for multi-dimensional labeled array computations, to enable fast and friendly data processing on GPUs. With the new documentation and tutorials, users can quickly adapt to this integration and optimize their data science workflows.🚀 |
| 52 | + |
| 53 | +## Acknowledgments |
| 54 | + |
| 55 | +A special thanks to the Xarray, CuPy, and Pangeo communities for making this integration possible. Collaborations like these are a testament to the power of open-source and community-driven development. 💪 |
| 56 | +This work was partly funded by NSF Earthcube award ["Jupyter Meets the Earth" (1928374)](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1928374); and NASA's Open Source Tools, Frameworks, and Libraries award "Enhancing analysis of NASA data with the open-source Python Xarray Library" (80NSSC22K0345). |
| 57 | + |
| 58 | +## Appendix I: Installation Instructions |
| 59 | + |
| 60 | +From anaconda: |
| 61 | + |
| 62 | +```shell |
| 63 | +conda install cupy-xarray -c conda-forge |
| 64 | +``` |
| 65 | + |
| 66 | +From PyPI: |
| 67 | + |
| 68 | +```shell |
| 69 | +python -m pip install cupy-xarray |
| 70 | +``` |
| 71 | + |
| 72 | +## Appendix II: Additional Resources |
| 73 | + |
| 74 | +[CuPy User Guide](https://docs.cupy.dev/en/stable/user_guide/index.html) |
| 75 | +[Xarray User Guide](https://docs.xarray.dev/en/stable/user-guide/index.html) |
| 76 | +[Cupy-Xarray Github](https://github.com/xarray-contrib/cupy-xarray.git) |
| 77 | +[NCAR GPU Workshop](https://github.com/NCAR/GPU_workshop) |
0 commit comments