IOWarp

IOWarp is a task-based I/O system which enables users to create custom I/O pipelines. IOWarp is built on top of the Content Transfer Engine (CTE) and Hermes-SHM.

Installation

User

Installation Method

Spack (Linux-Only)

Clone the IOWarp Spack Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git
spack repo add iowarp-install/iowarp-spack

Install IOWarp

spack install iowarp
spack load iowarp

Docker (Any)

Clone the IOWarp Install Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

Installation Method

Pull

Pull prebuilt containers

docker pull iowarp/iowarp-user:latest

Manual

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
docker build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
docker tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
docker build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
docker tag iowarp-user iowarp/iowarp-user:latest

Run the container

cd ~/iowarp-install/docker/user
docker compose up -d  # Only for recent dockers
docker-compose up -d  # Only for older dockers

For reference, the user docker-compose.yml contains:

services:
  iowarp:
    image: iowarp/iowarp-user:latest
    container_name: iowarp
    shm_size: 8g
    mem_limit: 8g
    stdin_open: true
    tty: true
    network_mode: host

Access the Container

docker exec -it iowarp bash

Example test

The following will run an example iowarp unit test:

cd ~/iowarp-install/docker
docker build --no-cache --build-arg SHM_SIZE=8g -t iowarp-test . -f iowarp-test.Dockerfile
docker run -it --shm-size=8g iowarp-test /bin/bash
spack load iowarp
jarvis ppl run

Podman (Any)

Clone the IOWarp Install Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
podman build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
podman tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
podman build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
podman tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
podman build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
podman tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
podman build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
podman tag iowarp-user iowarp/iowarp-user:latest

Create Podman Compose File

mkdir ~/iowarp-podman
cd ~/iowarp-podman
nano podman-compose.yml

Add the following:

services:
  iowarp:
    image: iowarp/iowarp-user:latest
    container_name: iowarp
    shm_size: 8g
    mem_limit: 8g
    stdin_open: true
    tty: true
    network_mode: host

Run the container

podman compose up -d  # For recent podman
podman-compose up -d  # For older podman

Access the Container

podman exec -it iowarp bash

Vcpkg (Any)

Clone the IOWarp Vcpkg Repo

cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git

Install IOWarp

cd iowarp-install/vcpkg
./bootstrap-vcpkg.sh
./vcpkg install iowarp

Core Developer

Clone the IOWarp Install Repo

export IOWARP=${HOME}/iowarp
mkdir -p ${IOWARP}
git clone https://github.com/iowarp/iowarp-install.git

Install WrpGit

wrpgit is a command line utility built in Python for manually installing iowarp packages. Manual installation is necessary for when developing certain packages.

wrpgit should remain in the iowarp-install. Do not install this file to another directory using something like pip.

cd ${IOWARP}/iowarp-install
pip install -r requirements.txt
chmod +x wrpgit

Initialize WrpGit

The following initializes wrpgit YAML configuration:

./wrpgit init

It will be stored in $IOWARP/iowarp-install/wrpgit.yaml. This file looks roughly as follows:

username: iowarp 
protocol: ssh
modules:
- name: cte-hermes-shm
  build: True
  fork: False
  type: cmake
  preset: debug
- name: iowarp-runtime
  build: True
  fork: False
  type: cmake
  preset: debug
  depends_on: cte-hermes-shm

• username: your github username
• protocol: the protocol to use for clone. Typically ssh.
• modules: the set of manual builds to create. You can add or remove packages from this list.

Fork IOWarp Repos

If you are working on iowarp, but are not considered full-time staff, we will have you fork our code instead of being a branch maintainer. You should be told the specific repos you are expected to work on in this case. Make the following changes to the wrpgit.yaml:

username: [MY GITHUB USERNAME]
protocol: ssh
token: [MY TOKEN]
modules:
- name: cte-hermes-shm
  build: True
  fork: True  # MAKE THIS TRUE
  type: cmake
  preset: debug
- name: iowarp-runtime
  build: True
  fork: True  # MAKE THIS TRUE
  type: cmake
  preset: debug
  depends_on: cte-hermes-shm

To set token:, create a personal access token. At minimum, the permissions should be:

repo: checked
admin:org: checked

And then run:

./wrpgit fork

Any package with fork: True will be forked from iowarp into the github account [MY GITHUB USERNAME]. It is ok if you already forked on your own -- the command will not create a duplicate fork.

Clone IOWarp Repos

./wrpgit clone

This will clone all packages in the modules list. E.g., name: iowarp-runtime is assumed to be the real name of the github repo. When fork: False, it will clone from iowarp. When fork: True, it will clone from your personal account. It will also add iowarp as a new git remote.

Install Dependencies

Spack (Linux-Only)

Install IOWarp

spack repo add iowarp-install/iowarp-spack
spack install iowarp
spack load iowarp

Configure SCSPKG

Read this guide to configure scspkg.

Dockerfiles (Any)

Pull

Pull prebuilt containers

docker pull iowarp/iowarp-user:latest

Manual

Create the containers

cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest

Launch Docker Compose

mkdir -p ${IOWARP}/docker
cp -r ${IOWARP}/iowarp-install/docker/development ${IOWARP}/docker
cd ${IOWARP}/docker/development
# Copy-paste the above into the file
docker compose up -d  # Only for recent dockers
docker-compose up -d  # Only for older dockers

Feel free to edit ${IOWARP}/docker/development/docker-compose.yml. It assumes the environment variable ${IOWARP} is set.

NOTE: shm_size and mem_limit are set to 8GB. Feel free to increase or decrease that value.

Access the Container

docker exec -it iowarp bash

Load Basic Packages

spack load iowarp

Make sure to understand your environment. If you get errors like package X was not found, it is most likely you forgot to do either spack load X or module load X.

Setup Repo Environments

./wrpgit setup

This will construct modulefiles and environments for each of the packages. You should rerun this if you spack packages or any dependencies of iowarp.

Build + Install Repos

./wrpgit build

This will build and install all repos with build: True.

Clean Build Directories

./wrpgit clean

This will destroy all build directories. You should rerun this if you spack packages or any dependencies of iowarp.

Recreate Modules

./wrpgit recreate

Modules are created using scspkg. These are used to store the "installed" files (e.g., cmake install). However, these can become corrupted too. This will destroy scspkgs and then rebuild them.

For Pull Requests

You may find the github command line useful for making PRs.

gh auth login
gh repo set-default
gh pr create --title "Your pull request title" --body "Your pull request description"