Quantitative Framework

MDP(Markov Decision Process)


1. Description

Markov chain combined with decision tree produces Markov decision process (MDP), which is a tool for multi-stage decision making with finite states and options. Transition probabilities among states of different stages and utilities (or costs) at all stages are the elements for decision making and the goal is to find an option (or combination of options at different stages) to maximize the expected utility of entire process.

2. Evaluation

2.1 Principle
  • MDP is a rigorous mathematical tool for multi-stage decision making with Markov dependence between the states in different stages.
  • It requires estimates of the transition probabilities associated with each option.
  • MDP may be applicable to all medical decision problems due to its dynamic nature.

2.2 Feature
  • MDP allows multiple criteria and multiple options.
  • Multiple criteria may make the structure of MDP very complicated.
  • The application of MDP relies on the availability of transition probabilities between states within the model.
  • MDP allows the benefit and risk status to change with time, but it may be challenging to obtain the transition probabilities data.
2.3 Visualisation

The main visualisations for MDP are:
  • Network diagram

2.4 Assesability and accessibility
  • Stakeholders may include their value preferences at each stage in the model.
  • Understanding of the model may require extensive mathematical knowledge.
  • MDP may be suitable for post-market decision based on long-term data.
  • MDP may also be suitable to model the possible outcomes of patient's decision to undergo a certain therapy considering all the possibilities and consequences.
  • It is often very difficult to clearly define health states which are central to the application of MDP.
  • MDP is often modelled using software with programming language:
Matlab (http://www.mathworks.co.uk/products/matlab)
R (http://www.r-project.org).
  • A specialist software for modelling simpler MDP models is:
Treeage (http://www.treeage.com)

3. References

[1]Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making 1993 Oct;13(4):322-38.
[2]Thompson JP, Noyes K, Dorsey ER, Schwid SR, Holloway RG. Quantitative risk-benefit analysis of natalizumab. Neurology 2008 Jul 29;71(5):357-64.