Getting Started

IOWarp

IOWarp is a powerful cutting-edge data management platform designed to streamline and accelerate scientific workflows, especially those intensified by the rise of artificial intelligence (AI). It optimizes data flow throughout the research process, ensuring efficient handling of diverse data types, formats, and the exponential growth of data. IOWarp aims to significantly reduce data access times, accelerate the pace of scientific discovery, and foster a collaborative research environment.

Dependencies

Currently, IOWarp is Linux-only. We are working on extending portability.

Spack is the easiest way to install IOWarp as of right now.

We also have a set of docker containers.

Installation

Clone the IOWarp Install Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

User

Spack (Linux-Only)

Install Spack

cd ${HOME}
git clone https://github.com/spack/spack.git
cd spack
git checkout tags/v0.22.2
echo ". ${PWD}/share/spack/setup-env.sh" >> ~/.bashrc
source ~/.bashrc

Add IOWarp to Spack

spack repo add iowarp-install/iowarp-spack

Install IOWarp

spack install iowarp

Vcpkg (Linux-Only)

This installer is experimental and will be improving as we add Windows support. Spack is the most well-maintained installer at this time. However, these are the general steps for vcpkg.

Install vcpkg + add our ports

git clone  --branch 2025.03.19 --single-branch https://github.com/microsoft/vcpkg
cp -r iowarp-install/ports/* ./vcpkg/ports/
cd vcpkg
./bootstrap-vcpkg.sh
export PATH="${PWD}:${PATH}"

Install IOWarp

./vcpkg install content-transfer-engine

Manual Podman (Any)

Clone the IOWarp Install Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
podman build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
podman tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
podman build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
podman tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
podman build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
podman tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
podman build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
podman tag iowarp-user iowarp/iowarp-user:latest

Create Podman Compose File

mkdir ~/iowarp-podman
cd ~/iowarp-podman
nano podman-compose.yml

Add the following:

services:
  iowarp:
    image: iowarp/iowarp-user:latest
    container_name: iowarp
    shm_size: 8g
    mem_limit: 8g
    stdin_open: true
    tty: true
    network_mode: host

Run the container

podman compose up -d  # For recent podman
podman-compose up -d  # For older podman

Access the Container

podman exec -it iowarp bash

Docker (Any)

Clone the IOWarp Install Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

Pull

Pull prebuilt containers

docker pull iowarp/iowarp-user:latest

Manual

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
docker build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
docker tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
docker build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
docker tag iowarp-user iowarp/iowarp-user:latest

Run the container

cd ~/iowarp-install/docker/user
docker compose up -d  # Only for recent dockers
docker-compose up -d  # Only for older dockers

For reference, the user docker-compose.yml contains:

services:
  iowarp:
    image: iowarp/iowarp-user:latest
    container_name: iowarp
    shm_size: 8g
    mem_limit: 8g
    stdin_open: true
    tty: true
    network_mode: host

Access the Container

docker exec -it iowarp bash

Example test

The following will run an example iowarp unit test:

cd ~/iowarp-install/docker
docker build --no-cache --build-arg SHM_SIZE=8g -t iowarp-test . -f iowarp-test.Dockerfile
docker run -it --shm-size=8g iowarp-test /bin/bash
spack load iowarp
jarvis ppl run

Core Developer

This is for people that will be editing either cte-hermes-shm, iowarp-runtime, or content-transfer-engine directly. If you are developing something relying on these repos, install iowarp like a regular user instead. E.g., developing the CAE, PPI, or some other runtime plugin.

Project Environment

Decide on where you want to place your iowarp repos. This guide will assume you have set a variable named IOWARP.

export IOWARP="${HOME}"

Set IOWARP to where you want your packages.

NOTE: This variable needs to be reset for every new terminal. On Linux, you could add the above command to your ~/.bashrc to automate.

Clone the IOWarp Spack Repo

cd ${IOWARP}
git clone https://github.com/iowarp/iowarp-install.git

Clone the repos

cd ${IOWARP}
git clone https://github.com/iowarp/cte-hermes-shm.git
git clone https://github.com/iowarp/iowarp-runtime
git clone https://github.com/iowarp/content-transfer-engine.git

For Pull Requests

You may find the github command line useful for making PRs.

gh auth login
gh repo set-default
gh pr create --title "Your pull request title" --body "Your pull request description"

Install Dependencies

Spack (Linux-Only)

Install Spack

cd ${HOME}
git clone https://github.com/spack/spack.git
cd spack
git checkout tags/v0.22.2
echo ". ${PWD}/share/spack/setup-env.sh" >> ~/.bashrc
source ~/.bashrc

Add iowarp to spack

spack repo add ${IOWARP}/iowarp-install/iowarp-spack

Install Dependencies

To install iowarp dependencies:

spack install iowarp +nocompile

For developer installations, we will use a tool named scspkg. scspkg is installed automatically by iowarp. It is a tool for managing modulefiles, which set various environment variables that streamline the installation of our tools.

Dockerfiles (Any)

Pull

Pull prebuilt containers

docker pull iowarp/iowarp-user:latest

Manual

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest

Build Dependency Container

We will rebuild this container so that jarvis introspects your machine:

# iowarp-deps
docker build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
docker tag iowarp-deps iowarp/iowarp-deps:latest

Launch Docker Compose

mkdir -p ${IOWARP}/docker
cp -r ${IOWARP}/iowarp-install/docker/development ${IOWARP}/docker
cd ${IOWARP}/docker/development
# Copy-paste the above into the file
docker compose up -d  # Only for recent dockers
docker-compose up -d  # Only for older dockers

Feel free to edit ${IOWARP}/docker/development/docker-compose.yml. It assumes the environment variable ${IOWARP} is set.

NOTE: shm_size and mem_limit are set to 8GB. Feel free to increase or decrease that value.

Access the Container

docker exec -it iowarp bash

Load Basic Packages

spack load iowarp

Make sure to understand your environment. If you get errors like package X was not found, it is most likely you forgot to do either spack load X or module load X.

Configure SCSPKG

Read this guide to configure scspkg.

Ensure packages are up-to-date

Make sure to try and update all your packages:

cd ~/iowarp-install
git pull
cd ${IOWARP}/cte-hermes-shm
git pull
cd ${IOWARP}/iowarp-runtime
git pull
cd ${IOWARP}/content-transfer-engine
git pull

Things failing to build after an update

If you find that things are failing to build for you after an update, try destroying all build directories.

cd ${IOWARP}/cte-hermes-shm
rm -rf build
cd ${IOWARP}/iowarp-runtime
rm -rf build
cd ${IOWARP}/content-transfer-engine
rm -rf build

Hermes-SHM

cd ${IOWARP}/cte-hermes-shm
scspkg create hermes_shm
mkdir build
cd build
cmake ../ \
-DCMAKE_INSTALL_PREFIX=$(scspkg pkg root hermes_shm) \
-DHSHM_ENABLE_COMPRESS=ON \
-DHSHM_ENABLE_ENCRYPT=ON \
-DHSHM_RPC_THALLIUM=ON \
-DHSHM_USE_ELF=ON
make -j32 install

If you will be working on hermes-shm, create your own personal fork instead of cloning directly. The following will allow you to keep your fork updated with changes to the global repo.

cd ${IOWARP}/cte-hermes-shm
git remote add iowarp https://github.com/iowarp/cte-hermes-shm.git
git pull iowarp

IOWarp Runtime: Chimaera

cd ${IOWARP}/iowarp-runtime
scspkg create iowarp_runtime
module unload iowarp_runtime
module load hermes_shm
mkdir build
cd build
cmake ../ \
-DCMAKE_INSTALL_PREFIX=$(scspkg pkg root iowarp_runtime)
make -j32 install

NOTE: If you will be working on iowarp-runtime, create your own personal fork instead of cloning directly. The following will allow you to keep your fork updated with changes to the global repo.

cd ${IOWARP}/iowarp-runtime
git remote add iowarp https://github.com/iowarp/iowarp-runtime
git pull iowarp

Content-Transfer-Engine: Hermes

cd ${IOWARP}/content-transfer-engine
scspkg create cte
module unload cte
module load hermes_shm iowarp_runtime
mkdir build
cd build
cmake ../ \
-DCMAKE_INSTALL_PREFIX=$(scspkg pkg root cte)
make -j32 install

NOTE: If you will be working on the cte, create your own personal fork instead of cloning directly. The following will allow you to keep your fork updated with changes to the global repo.

cd ${IOWARP}/content-transfer-engine
git remote add iowarp https://github.com/iowarp/content-transfer-engine.git
git pull iowarp

Another note on environments

By the end of this installation, you will have the following spack:

spack load iowarp

And the following modules:

module load hermes_shm iowarp_runtime cte

Understanding how your environment works is important. When compiling you must make sure that only the necessary modules are loaded. For example, the iowarp_runtime install only requires spack load iowarp and module load hermes_shm.

If you have module load iowarp_runtime as well, there is a chance the compiler will find header files in the module's directory, instead of the build directory. This can lead to compilation errors since the module directory will likely be outdated compared to the code you just wrote.