Installing and running LPJmL C Version 4.0.002

Installation for Unix-like systems

This text describes how to install and run LPJmL on your system. The code has been tested for AIX, Linux, Mac OS X, and Windows with cygwin and Microsoft C compiler (sequential version without MPI). Cygwin is a Linux-like environment for Windows (http://www.cygwin.com) On Mac OS the xcode suite must be installed in order to have the gcc compiler.

Clone it from the Github repository:

git clone https://github.com/PIK-LPJmL/LPJmL
Go to LPjmL directory:

cd LPJmL
Directory tree after extracting the tar files:

--lpjml
|
+-config        : OS- and compiler-specific Makefiles
|
+-bin           : Directory for executables and shell scripts
|
+-man           : Manual pages root directory
| |
| +-man1        : Manual pages for programs and scripts
| |
| +-man3        : Manual pages for functions
| |
| +-man5        : Manual pages for file formats
|
+--html         : Documentation and man pages in HTML format
|
+--par          : Parameter files for PFTs and soils
|
+--lib          : library files (created by make)
|
+--include      : include files
|
+--src          : source tree
   |
   +--numeric   : source for numerical routines
   |
   +--tools     : source for tools
   |
   +--pnet      : source for library for distributed networks
   |
   +--climate   : climate code 
   |
   +--lpj       : lpj functions
   |
   +--soil      : soil code
   |
   +--grass     : grass PFT code
   |
   +--tree      : tree PFT code
   |
   +--crop      : crop PFT code
   |
   +--landuse   : land use code
   |
   +--spitfire  : spitfire fire code
   |
   +--reservoir : reservoir code
   |
   +--socket    : socket communication library
   |
   +--image     : Coupler to IMAGE model
   |
   +--netcdf    : NetCDF input/output functions
   |
   +--utils     : utility programs
Configure for your operating system

On the HLRS2015 cluster at PIK you have to set the lpjml module for compilation and linking:
module load lpjml
% ./configure.sh
Configuring LPJmL 4.0.002...
Operating system is Linux
Intel MPI found
Create executables with make all
Put . $PWD/bin/lpj_paths.sh in your $HOME/.profile
The configure script creates scripts lpj_paths.sh/lpj_paths.csh for the bash/csh shell that sets all necessary environment variables. They are set by invoking
. bin/lpj_paths.sh
from the LPJmL root directory. It is recommended to put this statement into your ~./.profile. If configure script exits with message "Unsupported operating system", Makefile.$osname is created from Makefile.gcc and probably has to be modified for your operating system/compiler. If the configure script finds a MPI environment a parallel version of lpjml is built. This is done by checking for mpicc/mpiicc in the search path for commands. The configure script creates a copy of the following OS-specific makefiles from directory config:

MakefileDescription
Makefile.aix IBM AIX settings (xlc compiler)
Makefile.aix_mpi IBM AIX and MPI environment
Makefile.gcc GNU C-compiler settings
Makefile.intel Intel C-compiler settings
Makefile.intel_mpi Intel C-compiler and Intel MPI settings
Makefile.cluster2015 Intel C-compiler and Intel MPI on cluster2015 at PIK
Makefile.mpich GNU C-Compiler and MPI Chameleon settings
Makefile.win32 Windows settings (used by configure.bat)

Compilation flags

Compilation of lpjml is customized by definition of macros in the LPJFLAGS section of Makefile.inc:
LPJFLAGS= -Dflag1 ...
FlagDescription
DAILY_ESTABLISHMENT enable daily establishment
DEBUG diagnostic output is generated for debugging purposes
DOUBLE_HARVEST adding correct sequencing of harvest events
IMAGE Coupler to IMAGE model enabled via socket library
LINEAR_DECAY use linearized functions for litter decay
MICRO_HEATING Enable microbial heating
NO_FAIL_BALANCE lpjml does not terminate on balance errors
SAFE code is compiled with additional checks
STORECLIMATE Store climate input data in memory for spin up phase
USE_JSON enable JSON format for LPJmL configuration files
USE_MPI compile parallel version of LPJmL
USE_NETCDF enable NetCDF input/output
USE_NETCDF4 enable NetCDF version 4 input/output
USE_RAND48 use drand48() random number generator
USE_UDUNITS enable unit conversion in NetCDF files
WITH_FPE floating point exceptions are enabled for debugging purposes
Create executables:

make    
One executable in directory bin is created:

lpjml - LPJmL simulation code

Utility programs are compiled by

make utils
The following programs are created in the bin directory:

ProgramDescription
adddrain Adds river basin to coordinate file
asc2clm Convert CRU ASCII files to clm data files for LPJmL
backtrace Create backtrace from core file
bin2cdf Convert binary output files into NetCDF files
catclm Concatenate climate data files
cdf2bin Convert NetCDF files into raw binary data
cdf2clm convert NetCDF files into CLM files
cdf2coord extract CLM grid file from NetCDF file
cdf2soil convert NetCDF file into binary file
cft2clm convert binary land use data files to clm data files for LPJmL
clm2cdf convert CLM files into NetCDF files
cru2clm convert raw binary data files to clm data files for LPJmL
cutclm Cuts climate data files
grid2clm convert grid data file to clm data files for LPJmL
lpjcat Concatenates restart files from distributed LPJmL simulations.
lpjcheck Checks syntax of LPJmL configuration files
lpj Dynamic global vegetation model with river routing
lpjfiles Print list of input/output files of LPJmL
lpjml Dynamic global vegetation model with managed landuse and river routing
lpjprint Print contents of restart file of LPJmL model runs
lpjrun Run parallel LPJmL program interactively.
printclm Print contents of clm files for LPJmL
printheader Print header of clm files for LPJmL
txt2clm Convert CRU ASCII files to clm data files for LPJmL
txt2grid (1) Convert text files to clm grid data files for LPJmL
cat2bsq Concatenates output files from distributed LPJmL simulations
An installation directory can be defined during configuration:
./configure.sh -prefix /data/biosx/lpjml-4.0.002
Then the binaries can be copied in the installation directory by invoking

make install
The necessary parameter and include files will be copied, too. Create output and restart directory:

make test
On the compute cluster at PIK it is better to create symbolic links of the input, output, and restart directory to the parallel GPFS filesystem /scratch. This improves performance in particular for the parallel code.

% mkdir -p /p/tmp/$USER/lpj/restart
% mkdir -p /p/tmp/$USER/lpj/output
% ln -sf /p/tmp/$USER/lpj/restart restart
% ln -sf /p/tmp/$USER/lpj/output output

Man pages

Manual pages for the man command are located in the $LPJROOT/man directory. The MANPAPTH variable is set by the lpj_paths.sh script. HTML versions of the manual pages are located here. Invoking

apropos lpj
yields list of man pages related to LPJmL.

Running the program

Sequential version can be started by invoking:

./bin/lpjml 
By default the configuration file lpjml.conf is read. The contents of this file can be viewed here. If you run lpjml outside the root directory of LPJmL, the following environment variable should be set (done by lpj_paths.sh):

export LPJROOT=$HOME/lpjml
Then all includes are found.

Environment variables

The following environment variables are used by LPJmL:

Environment variableDescription
LPJCONFIG default LPJmL configuration filename
LPJPREP defines preprocessor command for LPJmL configuration file, default is cpp -P
LPJROOT defines the root directory for LPJmL. This directory is added to the include directories of the preprocessor. Is usually set by lpj_paths.sh.
LPJIMAGE sets host where IMAGE model is running
LPJWAITIMAGE sets time to wait for connection to IMAGE model
LPJINPATH Path append to the input filenames. Only done for filenames without absolute path.
LPJRESTARTPATH Path append to the restart filenames. Only done for filenames without absolute path.
LPJOPTIONS preprocessor runtime options for LPJmL
LPJOUTPATH Path appended to the output filenames. Only done for filenames without absolute path.
LPOUTPUT Default method for generating output files. Valid values are write, mpi2, gather, and socket. Method mpi2 and gather are used by the MPI version only.

Runtime options of LPJmL

The following runtime options are defined for lpjml:

OptionDescription
-Iincludepath Adds include path for LPJmL configuration file
-Dmacro[=value] Defines macro
-h print usage of lpjml
-l print license
-v print compiler used and LPJmL flags set
-param print LPJmL parameter for soils and PFTs
-pp cmd set preprocessor program to cmd. Default is cpp -P.
-fpe enable floating point exceptions
-image host set host where IMAGE model is running. Default is 'localhost'. Option is only availabel for the IMAGE version.
-wait time set time to wait for connection to IMAGE model measured in sec. Default is 10 sec.
-output method Method for generating output files. Valid values for are write, mpi2, gather, and socket. Methods mpi2 and gather are only available for the MPI version
-inpath dir input directory path
-outpath dir output directory path
-restartpath dir restart directory path
The first two options will be sent to the preprocessor. For the parallel version LoadLeveler files for Linux and AIX are provided. A job can be submitted by invoking the lpjsubmit script:

lpjsubmit ntasks LPJmL_args...
Depending on your MPI version installed the program can be started interactively:

% mpirun -np 256 $LPJROOT/bin/lpjml
******************************************************************************
****        _     ____     _           _       _  _          ___          ****
****       | |   |  _ \   | |_ __ ___ | |     | || |        / _ \         ****
****       | |   | |_) |  | | '_ ` _ \| |     | || |_      | | | |        ****
****       | |___|  __/ |_| | | | | | | |___  |__   _|  _  | |_| |        ****
****       |_____|_|   \___/|_| |_| |_|_____|    |_|   (_)  \___/         ****
****                                                                      ****
****                lpjml C Version 4.0.002 (Nov  6 2018)                 ****
****    Dynamic global vegetation model with natural and managed land     ****
****                     Compiled for Linux with MPI                      ****
****       (C) Potsdam Institute for Climate Impact Research (PIK),       ****
****                          see COPYRIGHT file                          ****
****              Authors, and contributors see AUTHORS file              ****
**** This version of LPJmL is licensed under GNU AGPL Version 3 or later  ****
****        See LICENSE file or go to http://www.gnu.org/licenses/        ****
****                 or invoke lpjml -l to print license                  ****
****          Contact: https://github.com/PIK-LPJmL/LPJmL /lpjml          ****
****                                                                      ****
******************************************************************************

Running for user user on cs-e14c05b07 at Tue Nov  6 20:14:43 2018
Reading configuration from 'lpjml.js' with options '-DFROM_RESTART'.
==============================================================================
Simulation "LPJmL Run" running on 256 tasks
Simulation with random precipitation, fire, river routing, permafrost,
new phenology, land use, limited irrigation, intercropping,
fixed sowing date after 1970, dam reservoirs, water use.
Working directory: /home/user/git/lpjml
Starting from restart file '/p/tmp/user/lpjml/restart/restart_1840_nv_stdfire.lpj'.
Input files:
Variable   Fmt  Filename
---------- ---- --------------------------------------------------------------
soil       meta /p/projects/lpjml/input/historical/input_VERSION2/soil.descr
coord      clm  /p/projects/lpjml/input/historical/input_VERSION2/grid.bin
temp       clm  /p/projects/lpjml/input/historical/CRUDATA_TS3_23/cru_ts3.23.1901.2014.tmp.dat.clm
prec       clm  /p/projects/lpjml/input/historical/CRUDATA_TS3_23/gpcc_v7_cruts3_23_precip_1901_2013.clm
lwnet      clm  /p/projects/lpjml/input/historical/input_VERSION2/lwnet_erainterim_1901-2011.clm
swdown     clm  /p/projects/lpjml/input/historical/input_VERSION2/swdown_erainterim_1901-2011.clm
co2        txt  /p/projects/lpjml/input/historical/input_VERSION2/co2_1841-2012.dat
countries  clm  /p/projects/lpjml/input/historical/input_VERSION2/cow_mg_2006_full.bin
landuse    clm  /p/projects/lpjml/input/historical/input_VERSION2/cft1700_2005_irrigation_systems_64bands.bin
elevation  clm  /p/projects/lpjml/input/historical/input_VERSION2/elevation.bin
reservoir  clm  /p/projects/lpjml/input/historical/input_VERSION2/reservoir_info_grand5.bin
wetdays    clm  /p/projects/lpjml/input/historical/CRUDATA_TS3_23/gpcc_v7_cruts3_23_wet_1901_2013.clm
drainage   clm  /p/projects/lpjml/input/historical/input_VERSION2/drainagestn.bin
lakes      meta /p/projects/lpjml/input/historical/input_VERSION2/glwd_lakes_and_rivers.descr
neighbour  clm  /p/projects/lpjml/input/historical/input_VERSION2/neighb_irrig_stn.bin
wateruse   clm  /p/projects/lpjml/input/historical/input_VERSION2/wateruse_1900_2000.bin
---------- ---- --------------------------------------------------------------
Writing restart file '/p/tmp/user/lpjml/restart/restart_1900_crop_stdfire.lpj' after year 1900.
Random seed: 2
Number of output files:       45
Output written in year:       1901
Byte order in output files:   little endian
Output method:                gathered
Variable         Fmt Unit         Type   Filename
---------------- --- ------------ ------ -------------------------------------
            grid raw       degree  short /p/tmp/user/lpjml/output/grid.bin
             fpc raw            -  float /p/tmp/user/lpjml/output/fpc.bin
            mnpp raw  gC/m2/month  float /p/tmp/user/lpjml/output/mnpp.bin
            mgpp raw  gC/m2/month  float /p/tmp/user/lpjml/output/mgpp.bin
             mrh raw  gC/m2/month  float /p/tmp/user/lpjml/output/mrh.bin
          mfapar raw            -  float /p/tmp/user/lpjml/output/mfapar.bin
         mtransp raw     mm/month  float /p/tmp/user/lpjml/output/mtransp.bin
         mrunoff raw     mm/month  float /p/tmp/user/lpjml/output/mrunoff.bin
           mevap raw     mm/month  float /p/tmp/user/lpjml/output/mevap.bin
         minterc raw     mm/month  float /p/tmp/user/lpjml/output/minterc.bin
           mswc1 raw            -  float /p/tmp/user/lpjml/output/mswc1.bin
           mswc2 raw            -  float /p/tmp/user/lpjml/output/mswc2.bin
           firec raw     gC/m2/yr  float /p/tmp/user/lpjml/output/firec.bin
           firef raw           yr  float /p/tmp/user/lpjml/output/firef.bin
            vegc raw        gC/m2  float /p/tmp/user/lpjml/output/vegc.bin
           soilc raw        gC/m2  float /p/tmp/user/lpjml/output/soilc.bin
            litc raw        gC/m2  float /p/tmp/user/lpjml/output/litc.bin
      flux_estab raw     gC/m2/yr  float /p/tmp/user/lpjml/output/flux_estab.bin
      mphen_tmin raw            -  float /p/tmp/user/lpjml/output/mphen_tmin.bin
      mphen_tmax raw            -  float /p/tmp/user/lpjml/output/mphen_tmax.bin
     mphen_light raw            -  float /p/tmp/user/lpjml/output/mphen_light.bin
     mphen_water raw            -  float /p/tmp/user/lpjml/output/mphen_water.bin
          mfirec raw  gC/m2/month  float /p/tmp/user/lpjml/output/mfirec.bin
      mdischarge raw        hm3/d  float /p/tmp/user/lpjml/output/mdischarge.bin
    mwateramount raw          hm3  float /p/tmp/user/lpjml/output/mwateramount.bin
         harvest raw     gC/m2/yr  float /p/tmp/user/lpjml/output/flux_harvest.bin
           sdate raw            -  short /p/tmp/user/lpjml/output/sdate.bin
     pft_harvest raw     gC/m2/yr  float /p/tmp/user/lpjml/output/pft_harvest.pft.bin
         cftfrac raw            -  float /p/tmp/user/lpjml/output/cftfrac.bin
     seasonality raw            -  short /p/tmp/user/lpjml/output/seasonality.bin
            mpet raw     mm/month  float /p/tmp/user/lpjml/output/mpet.bin
         malbedo raw            -  float /p/tmp/user/lpjml/output/malbedo.bin
   maxthaw_depth raw           mm  float /p/tmp/user/lpjml/output/maxthaw_depth.bin
      msoiltemp1 raw      Celsius  float /p/tmp/user/lpjml/output/msoiltemp1.bin
      msoiltemp2 raw      Celsius  float /p/tmp/user/lpjml/output/msoiltemp2.bin
      msoiltemp3 raw      Celsius  float /p/tmp/user/lpjml/output/msoiltemp3.bin
     soilc_layer raw        gC/m2  float /p/tmp/user/lpjml/output/soilc_layer.bin
             agb raw        gC/m2  float /p/tmp/user/lpjml/output/agb.bin
        agb_TREE raw        gC/m2  float /p/tmp/user/lpjml/output/agb_tree.bin
  mreturn_flow_b raw     mm/month  float /p/tmp/user/lpjml/output/mreturn_flow_b.bin
       mtransp_b raw     mm/month  float /p/tmp/user/lpjml/output/mtransp_b.bin
         mevap_b raw     mm/month  float /p/tmp/user/lpjml/output/mevap_b.bin
       minterc_b raw     mm/month  float /p/tmp/user/lpjml/output/mintec_b.bin
 aconv_loss_evap raw        mm/yr  float /p/tmp/user/lpjml/output/aconv_loss_evap.bin
aconv_loss_drain raw        mm/yr  float /p/tmp/user/lpjml/output/aconv_loss_drain.bin
---------------- --- ------------ ------ -------------------------------------
Spinup years:                   390
Cycle length during spinup:      30
First year:                    1901
Last year:                     2011
Number of grid cells:         67420
==============================================================================
Simulation begins...

                  Carbon flux (GtC)                                Water (km3)
       --------------------------------------- ---------------------------------------------
Spinup NEP     estab   fire    harvest total   transp     evap    interc  wd      discharge
------ ------- ------- ------- ------- ------- ---------- ------- ------- ------- ----------
  1511   2.088   0.004   3.620   0.000  -1.528    47957.3  9976.7  6895.2     0.0    51348.1
  1512  -1.062   0.036   3.629   0.095  -4.750    45776.6 11529.4  6555.8    20.8    50481.0
  1513   2.715   0.035   3.613   0.458  -1.321    47354.0 11195.4  6634.2    22.1    50551.0
  1514   4.357   0.035   3.229   0.739   0.423    48562.6 10554.8  6682.2    23.1    50039.0
...
  2009  13.477   0.162   3.074   8.476   2.090    47712.6  9357.4  8168.6  2496.7    52281.5
  2010  14.199   0.162   3.069   8.765   2.528    48559.7 10116.4  8488.2  2356.5    54515.1
  2011  15.201   0.162   3.063   8.667   3.633    48735.4  9536.9  8527.0  2577.8    56086.4
Simulation ended.
lpjml successfully terminated, 67420 grid cells processed.
Wall clock time:        1776 sec, 5.3e-05 sec/cell/year.
Total wall clock time:  1778 sec (00:29:38).

Error codes

If lpjml fails an error message is displayed in the following format

ERRORxxx: message
where xxx is the error code returned. The following error codes are defined:

Error code Description Error type
1 Error reading configuration External
2 Error initializing input data External
3 Error initializing grid External
4 Invalid carbon balance Internal
5 Invalid water balance Internal
6 Negative discharge Internal
7 Negative fire probability Internal
8 Negative soil moisture Internal
9 Error allocating memory External
10 Negative stand fraction Internal
11 Stand fraction sum error Internal
12 List is empty in dellistitem Internal
13 Index out of range in dellistitem Internal
15 Invalid year in getco2() External
16 Crop fraction >1 Internal
17 No natural stand for deforest Internal
18 Wrong cultivation type Internal
19 Floating point error Internal
21 PFT list is not empty in setaside Internal
22 Negative establishment rate Internal
23 Output channel is broken in socket connection External
24 Sending data to IMAGE model failed External
25 Opening connection to IMAGE model failed External
26 Not enough setaside stand created Internal
27 Forest left after deforestation Internal
28 Outflow reservoir error Internal
29 Permafrost error Internal
30 Global waterbalance error Internal
31 Cannot store climate data External
32 NO FMS coupler External
33 Cannot initialize soil temperature External
34 Invalid radiation model Internal
External errors are caused by invalid or missing input files while internal errors are caused by problems inside the LPJmL code. The latter will cause a core dump and have to be fixed by program changes. Some errors will only be generated if -DSAFE is set in the compile options of Makefile.inc. The options set at compile time can be obtained by invoking
% lpjml -v
lpjml C Version 4.0.002 (Nov  7 2018)
Operating system: Linux, little endian
Compiler:         Intel C version 15.00, 64 bit
Compile flags:    "-DUSE_RAND48 -DUSE_MPI -DSAFE -DWITH_FPE -DSTORECLIMATE -DUSE_NETCDF -DUSE_UDUNITS -DUSE_JSON "
It is recommended to compile the code without optimization and inlining making the inspection of the core file easier. Configuring in $LPJROOT with
./configure.sh -debug
make clean
make
will do the job. If no core file is generated set the user limit for core files:
% ulimit -c
0
% ulimit -c unlimited
Then the core can be analysed with the gdb debugger:
gdb $LPJROOT/bin/lpjml core
or by invoking the backtrace script:
backtrace

Building with Visual C++ 2005

How to get the software

Microsoft free version of the Visual Studio allows you to download the program, including a debugger. You can get it at https://www.visualstudio.com/downloads and choose Visual Studio Community. The link might only work for the newest version 2017, since that is available now. You should still be able to download the 2015 version at https://www.microsoft.com/de-DE/download/details.aspx?id=48146 or https://www.visualstudio.com/vs/older-downloads/. If you are working with windows systems anyway, this is a nice and handy developer environment. The main difference to alternatives (eclipse, text editors, etc.) is that VS uses no makefiles. Instead, you'll have to configure the project by hand. However, you still may use the makefiles supplied with the code in windows systems, if you have VS C++ installed.

SVN management

The plugin VisualSVN (download here) allows for smoothly integrating the SVN functionality of Tortoise in Visual Studio and really makes things nice there. However, it seems that PIK computers are member of an Active Directory domain, so buying a license is needed... :-(

Setting up a VS project

lick near the icon in the upper left to start a new project a new project on the downward arrow and choose 'start project from existing code', which starts a project wizard:
In case you are not able to create a project (error message: "Exception from HRESULT: 0x80041FE2"), go to the Windows control panel → uninstall a program → VS2015 → 'right click': change → MODIFY → drop down menu: Programming Languages → Visual C++ → check the box: 'Windows XP Support for C++' → Update That should solve the problem and you can now create a project from existing code. After the project has been created, you need to exclude all .c files from the project that are not essential to the code. This can be done in the Solution Explorer (upper left hand corner). Finally, you need to specify the project properties. For this, right-click on the project in the Solution Explorer.

Done. Now right-click on the project in the Solution Explorer and select rebuild or build.
Last change: 7 November 2018 by Werner von Bloh, PIK Potsdam