Version 0.12 released
Highlights of this release are support for weights in all aggregate functions, choice() with probabilities which can be different per individual (e.g. depending on gender), the ability to load globals mid-simulation, the possibility to do simulations without any .h5 output file, global constants and a lot of minor improvements and fixes.
Download this release from the download section.
Syntax changes
defining functions without any argument without using parentheses is now a warning. Closes issue 162. In a future release, this will be an error. The goal of this change is both to make models more explicit and consistent (a function definition always has parentheses) and to make LIAM2 internal code simpler in the long run.
Given that literally all models ever written need to be updated and doing so by hand would take a lot of time, the LIAM2 model upgrader can be used to automatically upgrade model files to the new syntax (the original version of your model(s) will be saved in a .bak file).
If you only have a few model files and you use the Windows bundle, you could open each of your model in turn and use the LIAM2: upgrade model command in the Macro menu of Notepad++.
If you do not use the bundle, or have many models/files to upgrade, you should rather use the command line:
liam2 upgrade <pattern_for_your_models>
For example:
liam2 upgrade examples/*.yml liam2 upgrade */*.yml
Note that if you are using the Windows bundle the executable is not liam2 but liam2/main.exe
New features
added support for weights in all aggregate functions (closes issue 226):
- avg(income, filter=WORKING, weights=weight)
implemented choice with a different probability per individual (closes issue 211):
- p0: if(gender, 0.1, 0.3) - p5: if(gender, 0.2, 0.4) - p10: if(gender, 0.7, 0.3) - intchoice: choice([0, 5, 10], [p0, p5, p10]) # the same using a global array (choices must be the first dimension) # this particular case will become easier/nicer in a future release - global_choice: choice(ARRAY2D.pvalues[0], ARRAY2D[:, gender * 1])
implemented simulation without any user-visible .h5 output file (closes issue 220). One can use an output section without declaring any “file”, or with file: ‘’:
output: file: ''In that case, LIAM2 will create an hidden minimal output file and delete it automatically at the end of the simulation. For people who do not use the .h5 output file, this can substantially improve disk usage and slightly improve performance when using large datasets. The minimal output file will contain only the fields used in lag expression going back in time more than one period (because those are not kept in memory).
implemented load() function which can load both arrays and tables in the middle of a simulation.
- array: load('param/mig.csv', type=float) - table: load('param/othertable.csv', fields=[('PERIOD', int), ('INTFIELD', int), ('FLOATFIELD', float)])the above array and table variable can then be used, within the function, exactly like if they were respectively a global array or a global table. The only difference is that they are local to the function and thus are discarded when the function terminates. This can be used as a way to transfer groupby arrays from one entity to the other, which is otherwise not possible at the moment.
implemented align(link=) to use proportions in combination with the Chenard algorithm (closes issue 216).
implemented a way of declaring global constants (based on pull request 206 by Mahdi Ben Jelloul). For example:
globals: MY_BOOL_CONSTANT: True MY_FLOAT_CONSTANT: 3.1415 MY_INT_CONSTANT: 42 MY_STRING_CONSTANT: "hello"chart functions gained xmin, xmax, ymin, ymax optional arguments to provide manual bounds for axes. By default, they are automatically inferred from the data as before (closes issue 209).
implemented totals argument to groupby to specify whether or not totals should be computed. Defaults to True like before.
added support for a new “msg” argument to all assertion functions. It specifies a custom message to append to the normal message displayed when the assertion fails. This is most useful when the condition of the assertion is complex. The message can be any expression or tuple of expressions, and it is only evaluated if the assertion fails (closes issue 208).
- assertTrue(all(age >= 0), msg="we have persons with negative age!")
will display:
AssertionError: all((age >= 100)) is not True: we have persons with negative age!
instead of just:
AssertionError: all((age >= 100)) is not True
Using dump(), csv(dump()) or breakpoint() as the msg argument can be useful.
- assertTrue(all(age < 150), msg=("we have abnormally old persons", dump(filter=age >= 150)) - assertTrue(all(total_income >= 0), breakpoint())implemented assertRaises to check for expected errors.
implemented access to the error function (erf) function if the scipy package is installed (which is NOT the case in the Windows bundle). Thanks to Mahdi Ben Jelloul (pull request 234).
Miscellaneous improvements
improve installation/getting started instructions and include installation instructions in the documentation (instead of only providing them in the source archive). It now includes a section concerning installation on Mac OS X. Closes issue 192. Thanks to Paul Williamson for his help concerning installation on Macs.
use “Read the Docs” theme for the documentation as it is nicer.
better error messages when trying to use data that is not ordered by period (display row and period) or which contains duplicated ids for a period (show row in addition to period and id).
better error message when a field default value is not of the field type.
When installing LIAM2 via python setup.py install, a liam2 script will be created in the Python installation Scripts directory (which is in the system PATH in most cases), so that one can use liam2 on the command line without specifying its path. For example:
liam2 run model.yml
the view command can be called without any file argument. This will launch the embedded ViTables without opening any file. In other words, one can now use liam2 view or python main.py view without another extra argument (closes issue 194).
made charts (matplotlib) work even without PyQt installed (i.e. fallback to the Tk backend).
avoid evaluating assertions arguments when using assertions: skip. Previously, only the final test was skipped.
improved check on the sum of probabilities in sidewalk alignment (pull request 197). Thanks to Mahdi Ben Jelloul.
misc improvements to the code, test models and the documentation, some of which done by Mahdi Ben Jelloul.
messages when an assertion concerning float values fails now contain all available decimals, instead of rounding at the 12th decimal.
Fixes
fixed running the bundled LIAM2 when another Python distribution is installed in the PATH of the system (closes issue 222).
fixed csv(dump()) rounding float values at the 12th decimal instead of using all available precision when the missing argument is used (closes issue 252).
fixed bug which made it impossible to override an existing field or global definition when importing another model. It was using the original/imported field definition and ignoring the overridden definition (closes issue 264).
fixed fields declared as initialdata: False to load data from the input file anyway if a field with the same name existed in the corresponding table (closes issue 227). Additionally, those fields used the type from the input table instead of the one declared (if different).
fixed output: False fields acting like temporary globals (i.e. being wiped at the end of each period) instead of like a field (closes issue 230). This had the indirect consequence of having both a field and a temporary variable with the same name, which confused dump().
fixed importing models using relative paths in some cases. Also makes the display of the imported model path nicer in that case (pull request 200). Thanks to Mahdi Ben Jelloul.
fixed skip_shows: True in simulation file being ignored.
fixed --skiptimings=False being ignored if timings: True was specified in the simulation file.
fixed subsetting an array created by indexing a global with a field when the result is an array.
- array: global[field1, :, field2] - first_item: array[0]
fixed show() and csv() on arrays created using some combinations of groupby and global arrays.
fixed using .transpose() without argument on a LabeledArray (a global or the result of a groupby).
fixed running LIAM2 in a debugger in some cases.
fixed some random number generator functions being referenced twice in the documentation index.
fixed alignment when take or leave filters are a single constant for all individuals (set as a Python bool).
fixed remove() failing if an array temporary variable is defined in the same function (closes issue 222).
fixed expressions with both a “non-simple” operation (everything except + - * / and where) and a function call failing to evaluate (closes issue 186).
Version 0.11 released
Highlights of this release are support for the “sidewalk” alignment method, support for default values for fields and some basic help in the interactive console.
Download this release from the download section.
New features
- added support for the “sidewalk” method in align_abs() in addition to the existing (and default) “bysorting” method. Thanks to Alexis Eidelman and Mahdi Ben Jelloul (pull request 189).
- added support for defining a default value for fields. This default value will be assigned to the corresponding fields in new() and if the fields are not present in the initial data (see the fields declaration section for details). Thanks to Alexis Eidelman and Mahdi Ben Jelloul (pull request 190).
- added “functions” command to the interactive console. It displays the list of available functions.
- added “help [function]” console command. It displays the signature of the function (ie the list of possible arguments and their default value if any). For random generator functions it also displays a detailed description (from numpy).
- added a ‘limit’ argument to dump() to limit the number of rows of the output.
Miscellaneous improvements
- improved the logging message for align with take or leave filters: the take and leave filters are combined with the “normal” filter before computing the take and leave numbers displayed. It does not change the results in any way, only the numbers logged to the console.
- made the demonstration models use 3d alignment files in some cases instead of separate alignment files for men and women.
- better error message when a user-defined function is called with the wrong number of arguments.
- better error message on failed assertEqual if shapes are different.
- added check for duplicate ids when loading datasets.
Miscellaneous improvements for developers
- implemented Simulation.from_str to create a simulation object from a string (using Python).
Fixes
- fixed various problems with the demonstration models.
- fixed grouping by a constant.
Version 0.10.5 released
This is a bug fix only release.
Download this release from the download section.
Fixes
- fixed using one2many in combination with lags of more than 1 period (or any past period in the interactive console).
Version 0.10.4 released
This is a bug fix only release.
Download this release from the download section.
Fixes
- fixed a few issues with the demonstration models.
- fixed 0.10.3 change log to mention Christophe Benz.
Version 0.10.3 released
Highlights of this release are improved demonstration models, better Linux support, better automated tests and the usual assortment of fixes and miscellaneous improvements.
Download this release from the download section.
New features
- added seed() function to (re)set the pseudo-random generator seed mid-simulation.
Miscellaneous improvements
- updated demo models:
- improved the dataset (which is now generated by a LIAM2 model: generate.yml).
- reorganise the models to group presented features more thematically.
- added more comments.
- added example of progressive .csv files via mode=’a’.
- added installation instructions and generally improve the out of the box experience on Linux. Thanks to Mahdi Ben Jelloul (pull request 181).
- enabled running the unit tests automatically on each change to the GitHub repository by using Travis-CI (which runs tests on build servers). Thanks to Mahdi Ben Jelloul and Christophe Benz (pull request 182).
- added an explicit error message when using “empty” processes (ie a “-” without any other text).
- moved website to its own repository (http://github.com/liam2/website).
- renamed “src” folder to “liam2” to follow standard practices. Thanks to Mahdi Ben Jelloul (pull request 181).
- improved error message of assertEqual if the array shapes are different.
- added explicit test for logit_score.
Fixes
- the list of available periods in the console is sorted.
- fixed time travel in the console at the end of a simulation.
- fixed writing technical error (traceback) in error.log.
- fixed the explore command on .h5 files.
Version 0.10.2 released
This release fixes a few more bugs I remembered about just after I finished the 0.10.1 release.
Download this release from the download section.
Version 0.10.1 released
This is a minor mostly bug fix release.
Download this release from the download section.
New features
- allowed to override all simulation options except “init” and “processes” using the command-line. Run “main run -h” for details (closes issue 173).
Fixes
- fixed the error message when using an unknown variable to be more explicit like it was in versions before 0.9 (closes issue 167).
- the output of the demonstration models is no longer included in the bundles (closes issue 169).
- made simulations which terminate in an error exit with the correct error code even without –debug (closes issue 174).
Presentation at the IMA conference in Luxembourg
I will be giving a presentation titled “LIAM2 - overview and recently added features” at the Fifth World Congress of the International Microsimulation Association, 2-4 September 2015, Luxembourg.
Abstract
This presentation will be divided into two parts. First a quick overview of what is LIAM2, then a description of all the new features since the LIAM2 course at the IMA conference in 2013 and more briefly those since our initial presentation in 2011. Each of those features will be briefly explained with examples of how they were used in real models.
Overview
LIAM2 is a free, open source, user-friendly modelling and simulation framework. It is made as generic as possible so that it can be used to develop almost any type of discrete-time dynamic microsimulation model with cross-sectional dynamic ageing (i.e. all individuals are simulated at the same time for one period, then for the next period, etc.). LIAM2 is clearly aiming to free “modellers” from having to develop or care about having state-of-the-art methods for data-handling or expression evaluation and yet be able to handle relatively large datasets at a reasonable speed. For example, a model like MIDAS in Belgium simulated over 60 years with 2.2 million individuals initially could be developed in a user-friendly environment and is run in less than 4 hours. To date, LIAM2 has been adopted by modellers in at least 7 countries.
New features since 2011
- model imports and model variants
- many new functions including:
- charting functions
- aggregate functions: gini, percentile, ...
- new alignment options (using absolute values or on a linked entity using Chenard’s algorithm)
- debugging functions (assertions, ...)
- data viewer (HDF5 viewer)
- syntax changes
- officially open source and hosted on GitHub
New features since 2013
- user defined functions
- while loops
- improved handling of external data
- new matching algorithms
- more random number generators
Autumn school on modelling using LIAM2 at LISER (Luxembourg)
In the context of the InGRID network, the Luxembourg Institute of Socio-Economic Research (LISER, formerly CEPS/INSTEAD) is organizing a workshop on “Elaborating a discrete-time dynamic microsimulation model with LIAM2, an open source development tool”.
The workshop is free of charge and will take place from 16 to 18 November 2015 in Luxembourg. Travel costs and accommodation are reimbursed up to certain amounts. See http://inclusivegrowth.be/events/call24/call24 for all the details.
Version 0.10 released
Highlights of this release are the implementation of while loops and calling user-defined functions, potentially with arguments. Please also see the complete release notes below for a potential migration issue concerning code defined outside of functions/procedures.
Download this release from the download section.
New features
- implemented support for calling user-defined functions (previously called procedures) from within user code, optionally passing arguments to them and retrieving their return value. As a consequence of this, every mention of “procedures” in the documentation has been replaced by “functions”, including the name of the “procedures” logging level which is now deprecated. Closes issue 152.
- implemented while loops (closes issue 151).
- added support for not storing some fields in the output file (closes issue 58).
- added support for using the “inf” (infinite) constant.
Miscellaneous improvements
improved our error handling code to display the (part of the) line where the error occurred in more cases and to not strip the traceback (error.log) of some important information in the cases where the line was already displayed.
configured the bundle editor (Notepad++) to display some warning and error lines in red in the console log. Additionally, when the error message contains a filename and/or a line number (this is currently too rare unfortunately), it is now click-able (to jump directly at the file/line).
defining a process outside of a function has been deprecated because it is ambiguous. For example, this code will now trigger a warning and will be an error in a future version:
entities: person: fields: agegroup: int processes: agegroup: 10 * trunc(age / 10) simulation: processes: - person: [agegroup]It should be replaced by (or possibly moved into another existing function):
entities: person: fields: agegroup: int processes: compute_agegroup: - agegroup: 10 * trunc(age / 10) simulation: processes: - person: [compute_agegroup]If this construct was used to have a temporary field (ie the field was not declared in the fields section) accessible from several functions, like
entities: person: processes: tempfield: 0 func1: tempfield: count() func2: otherfield: tempfield + 1One should now declare that field with output: False instead.
entities: person: fields: tempfield: {type: int, initialdata: False, output: False} processes: func1: tempfield: count() func2: otherfield: tempfield + 1This closes issue 124.
made bcolz optional (which is only useful for interpolating the dataset during import). Thanks to Mahdi Ben Jelloul (pull request 161).
allow simulations with no processes section (but an init section).
reworked a few sections of the documentation.
trying to set a value to an unknown variable in new/clone produces a warning instead of being silently ignored (or even crashing with –debug)
