Skip to content

Arrow

Apache Arrow is a cross-language development platform for in-memory data. It uses a standardized language-independent columnar memory format for flat and hierarchical data, organised for efficient analytic operations. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.

CUDA

Arrow is also available with CUDA. 

module load gcc arrow/X.Y.Z cuda
where X.Y.Z represents the desired version.

Python bindings

The module contains bindings for multiple Python versions. To discover which are the compatible Python versions, run: 

module spider arrow/X.Y.Z
where X.Y.Z represents the desired version.

Or search for pyarrow directly, by running: 

module spider pyarrow

PyArrow

The Arrow Python bindings (also named PyArrow) have first-class integration with NumPy, Pandas, and built-in Python objects. They are based on the C++ implementation of Arrow.

  1. Load the required modules. 

    module load gcc arrow/X.Y.Z python/3.11
    where X.Y.Z represents the desired version.

  2. Import PyArrow. 

    python -c "import pyarrow"
    If the command displays nothing, the import was successful.

For more information, see the Arrow Python documentation.

Fulfilling other Python package dependencies

Other Python packages depend on PyArrow in order to be installed. With the arrow module loaded, your package dependency for pyarrow will be satisfied. 

pip list | grep pyarrow
Output: 
pyarrow    17.0.0

If pip list shows an entry, then pyarrow is available and seen by pip. Otherwise, if there is no entry, pyarrow is not available.

Apache Parquet format

The Parquet file format is available.

To import the Parquet module, execute the previous steps for pyarrow, then run: 

python -c "import pyarrow.parquet"
If the command displays nothing, the import was successful.

R bindings

The Arrow package exposes an interface to the Arrow C++ library to access many of its features in R. This includes support for analysing large, multi-file datasets (open_dataset()), working with individual Parquet files (read_parquet(), write_parquet()) and Feather files (read_feather(), write_feather()), as well as lower-level access to the Arrow memory and messages.

Installation

  1. Load the required modules. 

    module load StdEnv/2020 gcc/9.3.0 arrow/8 r/4.1 boost/1.72.0

  2. Specify the local installation directory. 

    mkdir -p ~/.local/R/$EBVERSIONR/
export R_LIBS=~/.local/R/$EBVERSIONR/

  3. Export the required variables to ensure you are using the system installation. 

    export PKG_CONFIG_PATH=$EBROOTARROW/lib/pkgconfig
export INCLUDE_DIR=$EBROOTARROW/include
export LIB_DIR=$EBROOTARROW/lib

  4. Install the bindings. 

    R -e 'install.packages("arrow", repos="https://cloud.r-project.org/")'

Usage

After the bindings are installed, they have to be loaded.

  1. Load the required modules. 

    module load StdEnv/2020 gcc/9.3.0 arrow/8 r/4.1

  2. Load the library. 

    R -e "library(arrow)"
    Output: 
    > library("arrow")
Attaching package: ‘arrow’

For more information, see the Arrow R documentation.

Troubleshooting

This is a normal error generated by this dummy wheel.

See the Dummy wheel page.

ModuleNotFoundError: No module named 'pyarrow'

When importing pyarrow, one may get the following error: 

python -c "import pyarrow"
Output: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pyarrow'
This usually means one of the following cases: 1. A module for Arrow was not loaded. 2. A Python module was not loaded.

Module Arrow not loaded

Find a compatible arrow module and load it. See PyArrow.

Python module not loaded

When omitting to load a Python module, and activating a virtual environment, the Python bindings will not be available, hence resulting in pyarrow not being seen.

Solution:

  1. Deactivate the Python virtual environment. 

    test $VIRTUAL_ENV && deactivate
    !!! note If you had a virtual environment activated, it is important to deactivate it first, then load the module, before reactivating the virtual environment.

  2. Load the module. 

    module load arrow/x.y.z python/x.y.z

  3. Check that it is visible by pip: 

    pip list | grep pyarrow
    Output: 
    pyarrow            23.0.1
    and is accessible for your currently loaded Python module: 
    python -c 'import pyarrow'
    If no errors are raised, then everything is OK!