The clear documentation of simulations is important for their communication, replication, and comprehension. The ODD protocol (Grimm et al. 2006, 2010; Polhill et al. 2008; Polhill 2010) is a
standard layout for describing individual- and agent-based simulation models (ABMs), especially for journal articles, conference papers, and other academic literature. It consists of seven elements which can be grouped into three blocks: Overview, Design concepts, Details. The purpose of ODD is to facilitate writing and reading of model descriptions, to better enable replication of model-based research, and to establish a set of design concepts that should be taken into account while developing an ABM. It does this in a relatively lightweight way, avoiding over-formal approaches whilst ensuring that the essentials of a simulation are explicitly described in a flexible yet appropriate manner.
Thanks Volker. Indeed ODD is good for us ;-).
However, I gone through 1000s of ABM publications (see my talk on the Catalog App) and often ODD is used in spirit but still a very limited model description is given due to the focus on qualitative narrative instead of using mathematical equations, psuedo code or flow charts. In fact, when I teach ODD in my ABM course student’s first application is to ignore a lot of the technical details of the model and focus on the narrative.
I wonder whether in the new ODD protocol the use of math equations, pseudo code and flow charts need to be recommended explicitly since some users seem to focus on the qualitative narrative.
Hi Marco,
Thanks for your feedback. I fully agree. In our new article about ODD, which is about to be submitted to JASSS in November, we wrote the section below. There will also be section, and guidance, on writing “summary ODDs”, which focusses on the narrative, while ODD itself is, and always was meant to be, a technical document, not a story.
" 4.6 Reducing ambiguity by linking ODD to code
To many simulation scientists, a model’s computer code is its most authoritative and unambiguous description. Therefore, providing clear links between ODD and the computer code can make a model seem more transparent and its description less ambiguous. If readers can easily find and read the code that implements any part of an ODD, they are more likely to thoroughly understand and even replicate the model. Most programming languages are relatively similar and seem understandable over the few lines typically needed to code ABM algorithms or submodels, so even readers unfamiliar with a language can try to understand an algorithm from its code and prove whether it matches the ODD narrative. Here, we recommend ways to provide such links. Such links obviously require making a model’s code (appropriately licensed and copyrighted) available with its ODD. Most journals do not require making code available but doing so is widely considered good scientific practice in line with current trends towards open science.
There are well-known limitations of publishing model code. Teams that have invested years in the development of complex models certainly want to benefit from this investment before others do; however, code for key algorithms and data structures could be provided instead of the full program. Furthermore, a model’s code itself is not always sufficient for understanding exactly how it executes; information may also be required about the compiler or interpreter, code libraries, any numerical solution methods (Seppelt and Richter 2005), and even the hardware and operating system. Therefore, we recommend that at least revision numbers of external software/library, architecture (e.g. x86, 32/64 bits) and operating system version always be provided. Further, computer code can be misinterpreted, and important software details can be specified in places other than the code statements; especially, the popular NetLogo platform (Wilensky 1999) provides many extremely useful primitives that must be fully understood (almost always possible from their documentation) to understand a model code.
One way to link ODD descriptions to computer code is simply through careful naming conventions. Using the same names for variables, parameters, and submodels in both ODD and code makes it easier to find the code implementing specific parts of a model. Similarly, code comments can be used to identify where specific ODD elements, algorithms, or even numbered equations, are programmed.
Links between ODD and code can also be more comprehensive and specific. Becher et al (2014; https://doi.org/10.1111/1365-2664.12222) provided both the ODD description and the computer code for their complex model of honey bee colonies, BEEHAVE, in a single file that includes hyperlinks from the ODD to the corresponding code. Another potential approach is to add notes or a table to the ODD to identify the code locations (file, procedure or function name, or even line number) where each ODD element, submodel, or algorithm is implemented."
Cheers,
Volker
Hi Volker,
Thanks for this nice contribution. Happy to know that a new update will be soon published, ODD is definitively a “must” now!
I like very much the idea of combining source-code documentation with the ODD protocol. In his paper published in 2010, Chris Topping also used this approach, calling it ODdox (see https://www.sciencedirect.com/science/article/pii/S0304380009006346).
I’d like also to mention the paper of Ahmed Laatabi and his colleagues: ODD+2D: An ODD Based Protocol for Mapping Data to Empirical ABMs (see http://jasss.soc.surrey.ac.uk/21/2/9.html).
Best regards,
Clp
Dear Christophe,
In our new paper we cited ODD+2D as an extension of ODD which has a special focus on linking a model to its data base.
I know the ODDox paper and agree that is a good idea, but it should be noted that it is incomplete, due to the complexity of the described model, ALMASS. Some central features of it, in particular the scheduling and farming practice, are not fully documented but available as the source code only. I know because we used ALMASS ourselves, with the help of its developer, Chris Topping: Langhammer, M., Grimm, V., PĂĽtz, S., & Topping, C. J. (2017). A modelling approach to evaluating the effectiveness of ecological focus areas: the case of the European brown hare. Land Use Policy , 61 , 63-79.
Cheers,
Volker
Dear Volker,
I am really looking forward to the next update to the ODD protocol! Your work is extremely important for pulling the community together.
First, I want to echo Marco’s comments regarding flowcharts. I’ve found this to be a great way to quickly communicate the flow of control - both to ABM experts and novices alike. Particularly for submodels in the ODD protocol, I like to use flowcharts.
But I also have a question for you. Have you encountered the practice of “Literate Programming?”
I completely agree with you that ABM code is not self-descriptive - and on its own, it is not sufficient for replication. In fact, other disciplines have also encountered the challenge of code that is uninterpretable. One solution to this challenge is called Literate Programming - and it involves a specific kind of annotation that is interleaved with your code. If you’ve seen RMarkdown, this could be considered one implementation of Literate Programming - and I think it is a practical improvement for my own R code.
Do you think it would work to use ODD to “mark up” ABM code in the manner of Literate Programming?
Hi Ian,
I know of Knuth, of course, but have not heard about Literate Programming, thanks a lot for the hint! This is certainly something to think about for the future. I will have a look into RMarkdown and perhaps can include your hint in our current ODD manuscript.
Thanks again, and best regards,
Volker