IOWarp
IOWarp is a task-based I/O system which enables users to create custom I/O pipelines. IOWarp is built on top of the Content Transfer Engine (CTE) and Hermes-SHM.
Installation
- User
- Core Developer
Installation Method
- Spack (Linux-Only)
- Docker (Any)
- Podman (Any)
- Vcpkg (Any)
Clone the IOWarp Spack Repo
cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git
spack repo add iowarp-install/iowarp-spack
Install IOWarp
spack install iowarp
spack load iowarp
Clone the IOWarp Install Repo
cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git
Installation Method
- Pull
- Manual
Pull prebuilt containers
docker pull iowarp/iowarp-user:latest
Create the containers
cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
docker build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
docker tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
docker build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
docker tag iowarp-user iowarp/iowarp-user:latest
Run the container
cd ~/iowarp-install/docker/user
docker compose up -d # Only for recent dockers
docker-compose up -d # Only for older dockers
For reference, the user docker-compose.yml contains:
services:
iowarp:
image: iowarp/iowarp-user:latest
container_name: iowarp
shm_size: 8g
mem_limit: 8g
stdin_open: true
tty: true
network_mode: host
Access the Container
docker exec -it iowarp bash
Example test
The following will run an example iowarp unit test:
cd ~/iowarp-install/docker
docker build --no-cache --build-arg SHM_SIZE=8g -t iowarp-test . -f iowarp-test.Dockerfile
docker run -it --shm-size=8g iowarp-test /bin/bash
spack load iowarp
jarvis ppl run
Clone the IOWarp Install Repo
cd ${HOME}
git clone https://github.com/iowarp/iowarp-install.git
Create the containers
cd ~/iowarp-install/docker
# iowarp-base
podman build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
podman tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
podman build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
podman tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
# iowarp-deps
podman build --no-cache -t iowarp-deps . -f iowarp-deps.Dockerfile
podman tag iowarp-deps iowarp/iowarp-deps:latest
#iowarp-user
podman build --no-cache -t iowarp-user . -f iowarp-user.Dockerfile
podman tag iowarp-user iowarp/iowarp-user:latest
Create Podman Compose File
mkdir ~/iowarp-podman
cd ~/iowarp-podman
nano podman-compose.yml
Add the following:
services:
iowarp:
image: iowarp/iowarp-user:latest
container_name: iowarp
shm_size: 8g
mem_limit: 8g
stdin_open: true
tty: true
network_mode: host
Run the container
podman compose up -d # For recent podman
podman-compose up -d # For older podman
Access the Container
podman exec -it iowarp bash
Clone the IOWarp Install Repo
export IOWARP=${HOME}/iowarp
mkdir -p ${IOWARP}
git clone https://github.com/iowarp/iowarp-install.git
Install WrpGit
wrpgit is a command line utility built in Python for manually installing iowarp packages. Manual installation is necessary for when developing certain packages.
wrpgit should remain in the iowarp-install. Do not install this file to another directory using something like pip.
cd ${IOWARP}/iowarp-install
pip install -r requirements.txt
chmod +x wrpgit
Initialize WrpGit
The following initializes wrpgit YAML configuration:
./wrpgit init
It will be stored in $IOWARP/iowarp-install/wrpgit.yaml
.
This file looks roughly as follows:
username: iowarp
protocol: ssh
modules:
- name: cte-hermes-shm
build: True
fork: False
type: cmake
preset: debug
- name: iowarp-runtime
build: True
fork: False
type: cmake
preset: debug
depends_on: cte-hermes-shm
username
: your github usernameprotocol
: the protocol to use for clone. Typically ssh.modules
: the set of manual builds to create. You can add or remove packages from this list.
Fork IOWarp Repos
If you are working on iowarp, but are not considered full-time staff, we will have you fork our code instead of being a branch maintainer. You should be told the specific repos you are expected to work on in this case. Make the following changes to the wrpgit.yaml:
username: [MY GITHUB USERNAME]
protocol: ssh
token: [MY TOKEN]
modules:
- name: cte-hermes-shm
build: True
fork: True # MAKE THIS TRUE
type: cmake
preset: debug
- name: iowarp-runtime
build: True
fork: True # MAKE THIS TRUE
type: cmake
preset: debug
depends_on: cte-hermes-shm
To set token:
, create a personal access token.
At minimum, the permissions should be:
repo: checked
admin:org: checked
And then run:
./wrpgit fork
Any package with fork: True
will be forked from iowarp into the github account
[MY GITHUB USERNAME]. It is ok if you already forked on your own -- the command will
not create a duplicate fork.
Clone IOWarp Repos
./wrpgit clone
This will clone all packages in the modules list. E.g., name: iowarp-runtime
is assumed
to be the real name of the github repo. When fork: False, it will clone from iowarp. When fork: True,
it will clone from your personal account. It will also add iowarp as a new git remote.
Install Dependencies
- Spack (Linux-Only)
- Dockerfiles (Any)
Install IOWarp
spack repo add iowarp-install/iowarp-spack
spack install iowarp
spack load iowarp
Configure SCSPKG
Read this guide to configure scspkg.
- Pull
- Manual
Pull prebuilt containers
docker pull iowarp/iowarp-user:latest
Create the containers
cd ~/iowarp-install/docker
# iowarp-base
docker build --no-cache -t iowarp-base . -f iowarp-base.Dockerfile
docker tag iowarp-base iowarp/iowarp-base:latest
# iowarp-deps-spack
docker build --no-cache -t iowarp-deps-spack . -f iowarp-deps-spack.Dockerfile
docker tag iowarp-deps-spack iowarp/iowarp-deps-spack:latest
Launch Docker Compose
mkdir -p ${IOWARP}/docker
cp -r ${IOWARP}/iowarp-install/docker/development ${IOWARP}/docker
cd ${IOWARP}/docker/development
# Copy-paste the above into the file
docker compose up -d # Only for recent dockers
docker-compose up -d # Only for older dockers
Feel free to edit ${IOWARP}/docker/development/docker-compose.yml
.
It assumes the environment variable ${IOWARP}
is set.
NOTE: shm_size
and mem_limit
are set to 8GB. Feel free to increase or decrease
that value.
Access the Container
docker exec -it iowarp bash
Load Basic Packages
spack load iowarp
Make sure to understand your environment. If you get errors
like package X was not found
, it is most likely you forgot
to do either spack load X
or module load X
.
Setup Repo Environments
./wrpgit setup
This will construct modulefiles and environments for each of the packages. You should rerun this if you spack packages or any dependencies of iowarp.
Build + Install Repos
./wrpgit build
This will build and install all repos with build: True
.
Clean Build Directories
./wrpgit clean
This will destroy all build directories. You should rerun this if you spack packages or any dependencies of iowarp.
Recreate Modules
./wrpgit recreate
Modules are created using scspkg. These are used to store the "installed" files (e.g., cmake install). However, these can become corrupted too. This will destroy scspkgs and then rebuild them.
For Pull Requests
You may find the github command line useful for making PRs.
gh auth login
gh repo set-default
gh pr create --title "Your pull request title" --body "Your pull request description"