Parallel Monte Carlo runs

The example script below shows how to run SWIM in parallel on the PIK cluster or on your local CPUs, using the example of a Monte Carlo random parameter setting. It makes use of the project cloning functionality of SWIMpy and SLURM sbatch job submission that is executed in the background.

import pandas as pd
import numpy as np

import swimpy

# number of runs
nruns = 10

# parameter and ranges (extend as needed)
parameter = {'ecal': (0.8, 1.2),
             'sccor': (0.1, 10),
             'smrate': (0.1, 0.6)}

# load the project instance
p = swimpy.Project()

# create list of basin_parameter arguments
args = [{k: np.random.uniform(*b) for k, b in parameter.items()} for _ in range(nruns)]

# run and return set of runs
runs = p.cluster.run_parallel(args=args, files=['station_daily_discharge'])

/Users/wortmann/Desktop/source/swimpy/swimpy/utils.py:352: UserWarning: Using multiprocessing on 4 CPUs.
  warnings.warn(msg)

# unpack Blankenstein station_daily_discharge and plot
q = pd.DataFrame({r.id: r.station_daily_discharge['BLANKENSTEIN'] for r in runs})
q.plot()

<matplotlib.axes._subplots.AxesSubplot at 0x10709ecd0>

The default SLURM settings may be extended like a normal dictionary. E.g.:

p.cluster_slurmargs
p.cluster_slurmargs.update({'qos': 'medium', 'time': '0:0:01'})
p.cluster_slurmargs

{'account': 'swim', 'qos': 'medium', 'time': '0:0:01'}

Download as Notebook