This document outlines the inner-workings of the simulation that is used to select the source/post pairs that each participant is shown, and that decides the changes to the credibility and followers of participants as they play through a study.
A simulation is performed for each participant that loads the study, to determine what they are shown during the study. These simulations are performed in three phases.
The initial follower number and credibility of each source is sampled from the default skew-normal distributions specified in the configuration spreadsheet. These values are sampled independently for each participant that loads the study.
All the source/post pairs that will be shown to each participant is pre-generated by running a simulation. This simulation is run for every participant that loads the study, before they start the study. Therefore, the interactions of participants with posts have no impact on the source/post pairs that they will be shown.
The participant’s followers and credibility are updated as they react to posts. The changes to their followers and credibility is dependent upon the reactions they choose, and may be configured for each post.
Throughout the game, the credibility and followers of the sources of posts will change. The change to the credibility and followers of the sources is the same change that is applied to participants after they share a post. If share reactions are disabled, then the credibility and followers of sources will not change. The changes to source’s followers and credibility is applied before the source/post pair is shown to the participant. This is done to increase immersion by showing that the credibility and followers of the sources also change. However, it also means that if one source is selected to show many true or many false posts, their credibility may rise to 100 or fall to zero over the course of a game.
The source/post pair selection dictates what posts participants will be shown, when, and from which source. All the source/post pairs are pre-generated when a participant starts the study. The participant’s interactions with the posts have no impact on the sources and posts that will be shown to them.
There are four source/post selection methods that can be used for your studies. The simulation of each source/post selection method is detailed in the sections below. For all the methods other than Pre-Defined, the sources will be chosen before the posts. This is described in the Source Selection section.
The game can be configured such that some sources can be associated with many posts, however each post will only ever be shown once to a participant.
The source to be shown is always chosen first for all source/post selection methods other than Pre-Defined. Sources are chosen randomly, and their choice is weighted by their maximum number of posts. Therefore, if a source has a maximum of 10 posts, it will be 10x as likely to be chosen as a source that has a maximum of 1 post. Sources that have an unlimited number of posts will be weighted with the mean max-post number of all sources that have a maximum. Once the maximum number of posts has been reached for a given source, it will not be chosen again.
When using the Overall-Ratio source/post selection method, posts will be chosen to target an overall percentage of true and false posts to be shown to the user. For example, if you use an overall true-post percentage of 60%, then the simulation will have a 60% chance of selecting a true post, and a 40% chance of selecting a false post. This is achieved by probabilistically selecting whether to show a true or a false post before the post itself is selected. If it is chosen that a true post should be displayed, it will then select a true post randomly from the pool of all true posts that have not yet been shown. Similarly, if it is chosen that a false post should be displayed, it will select a false post randomly from the pool of all false posts that have not yet been shown.
However, if there are no remaining true posts, or no remaining false posts, then the selection method will have to choose from whatever posts are remaining. For example, if your study has a length of 10 and includes 20 false posts but only 5 true posts, it is not possible that the game could ever show more than 50% true posts to the participant. After it has shown 5 true posts to the participant, it will only be able to show false posts from then on.
Also note that due to the probabilistic sampling, it is possible that the actual ratio of true and false posts deviates from the target ratio, especially with small studies (e.g., with only 10 posts shown, even with a 60%:40% true:false ratio, one could easily end up with only 3 true posts and 7 false posts, by chance).
When using the Source-Ratios source/post selection method, each source must be given their own true-post percentage. The post selection then acts in the exact same way as when using the Overall-Ratio selection method, except that the true-post percentage is source-specific.
When using the Credibility source/post selection method, the true-post percentage to use for selecting posts is determined by a configurable linear relationship from the credibility of each source. Once the true-post percentage is calculated from the credibility of the source, it is then used in the same way as the Overall-Ratio selection method to choose the posts.
Care must be taken when using the credibility source/post selection method, as a source’s credibility can increase to 100 or decrease to 0 over the course of a game if a source is used to show many true or false posts. This may be accelerated under Credibility selection due to a feedback cycle where a high credibility causes a source to show more true posts, which increases credibility. Adjusting the relationship between credibility and true-post percentage to be flatter can help alleviate this issue. Additionally, if many sources are used, then this will probably not cause an issue.
The Pre-Defined source/post selection method allows specification of an exact set of source/post pairs to be shown to participants. Every participant is always shown the exact same set of source/post pairs that is pre-defined in the configuration spreadsheet. The order of the source/post pairs may also be randomised for each participant if enabled in the study configuration.
Back to Top