[vector]

algorithm packages accept only one argument

[vector]

algorithm packages accept only one argument

[vector]

algorithm packages accept only one argument

[string]

Specifies the learning policy to be used. This determines how the model updates action values based on observed or simulated choices. It can be either "off" or "on".

• Off-Policy (Q-learning): This is the most common approach for modeling reinforcement learning in Two-Alternative Forced Choice (TAFC) tasks. In this mode, the model's goal is to learn the underlying value of each option by observing the human participant's behavior. It achieves this by consistently updating the value of the option that the human actually chose. The focus is on understanding the value representation that likely drove the participant's decisions.

• Off-Policy (SARSA): In this mode, the target policy and the behavior policy are identical. The model first computes the selection probability for each option based on their current values. Critically, it then uses these probabilities to sample its own action. The value update is then performed on the action that the model itself selected. This approach focuses more on directly mimicking the stochastic choice patterns of the agent, rather than just learning the underlying values from a fixed sequence of actions.

default: policy = "off"

[string]

Estimation method. Can be either "MLE" or "MAP".

• "MLE": (Default) Maximum Likelihood Estimation. This method finds the parameter values that maximize the log-likelihood of the data. A higher log-likelihood indicates that the parameters provide a better explanation for the observed human behavior. In other words, data simulated using these parameters would most closely resemble the actual human data. This method does not consider any prior information about the parameters.

• "MAP": Maximum A Posteriori Estimation. This method finds the parameter values that maximize the posterior probability. It is an iterative process based on the Expectation-Maximization (EM) framework.

  • Initialization: The process begins by assuming a uniform distribution as the prior for each parameter, making the initial log-prior zero. The first optimization is thus equivalent to MLE.

  • Iteration: After finding the best parameters for all subjects, the algorithm assesses the actual distribution of each parameter and fits a normal distribution to it. This fitted distribution becomes the new empirical prior.

  • Re-estimation: The parameters are then re-optimized to maximize the updated posterior probability.

  • Convergence: This cycle repeats until the posterior probability converges or the maximum number of iterations (specified by iteration_g) is reached.

  Using this method requires that the priors argument be specified to define the initial prior distributions.

default: estimate = "MLE"

[data.frame]

This data should include the following mandatory columns:

• sub "Subject"

• time_line "Block" "Trial"

• L_choice "L_choice"

• R_choice "R_choice"

• L_reward "L_reward"

• R_reward "R_reward"

• sub_choose "Sub_Choose"

[CharacterVector]

A vector specifying the subject ID(s) for which parameters should be fitted. The function will process only the subjects provided in this vector.

To fit all subjects, you can either explicitly set the argument as id = unique(data$Subject) or leave it as the default (id = NULL). Both approaches will direct the function to fit parameters for every unique subject in the dataset.

It is strongly recommended to avoid using simple numeric sequences like id = 1:4. This practice can lead to errors if subject IDs are stored as strings (e.g., subject four is stored as "004") or are not sequentially numbered.

default: id = NULL

[integer]

Represents the total number of trials a single subject experienced in the experiment. If this parameter is kept at its default value of NULL, the program will automatically detect how many trials a subject experienced from the provided data. This information is primarily used for calculating model fit statistics such as AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion).

default: n_trials = NULL

[CharacterVector]

A character vector containing the names of all user-defined functions required for the computation. When parallel computation is enabled (i.e., nc > 1), user-defined models and their custom functions might not be automatically accessible within the parallel environment.

Therefore, if you have created your own reinforcement learning model that modifies the package's default six default functions (default functions: util_func = func_gamma, rate_func = func_eta, expl_func = func_epsilon, bias_func = func_pi, prob_func = func_tau, loss_func = func_logl ), you must explicitly provide the names of your custom functions as a vector here.

[List]

The name of fit modals

e.g. model_name = c("TD", "RSTD", "Utility")

[List]

A collection of functions applied to fit models to the data.

e.g. fit_model = list(binaryRL::TD, binaryRL::RSTD, binaryRL::Utility)

[List]

The lower bounds for model fit models

e.g. lower = list(c(0, 0), c(0, 0, 0), c(0, 0, 0))

[List]

The upper bounds for model fit models

e.g. upper = list(c(1, 10), c(1, 1, 10), c(1, 1, 10))

[List]

A list specifying the prior distributions for the model parameters. This argument is mandatory when using estimate = "MAP". There are two primary scenarios for its use:

1. Static MAP Estimation (Non-Hierarchical) This approach is used when you have a strong, pre-defined belief about the parameter priors and do not want the model to update them iteratively.

Configuration:

• Set estimate = "MAP".

• Provide a list defining your confident prior distributions.

• Keep iteration_g = 0 (the default).

Behavior:

The algorithm maximizes the posterior probability based solely on your specified priors. It will not use the EM (Expectation-Maximization) framework to learn new priors from the data.

2. Hierarchical Bayesian Estimation via EM This approach is used to let the model learn the group-level (hierarchical) prior distributions directly from the data.

Configuration:

• Set estimate = "MAP".

• Specify a weak or non-informative initial prior, such as a uniform distribution for all parameters.

• Set iteration_g to a value greater than 0.

Behavior:

With a uniform prior, the initial log-posterior equals the log-likelihood, making the first estimation step equivalent to MLE. The algorithm then initiates the EM procedure: it iteratively assesses the actual parameter distribution across all subjects and updates the group-level priors. This cycle continues until the posterior converges or iteration_g is reached.

default: priors = NULL

[double]

Convergence threshold for MAP estimation. If the change in log posterior probability between iterations is smaller than this value, the algorithm is considered to have converged and the program will stop.

default: tolerance = 0.001

[integer]

The number of iterations the optimization algorithm will perform when searching for the best-fitting parameters during the fitting phase. A higher number of iterations may increase the likelihood of finding a global optimum but also increases computation time.

default: iteration_i = 10.

[integer]

The maximum number of iterations for the Expectation-Maximization (EM) based MAP estimation. The algorithm will stop once this iteration count is reached, even if the change in the log-posterior value has not yet fallen below the tolerance threshold.

default: iteration_g = 0.

[NumericVector]

Initial values for the free parameters that the optimization algorithm will search from. These are primarily relevant when using algorithms that require an explicit starting point, such as L-BFGS-B. If not specified, the function will automatically generate initial values close to zero.

default: initial_params = NA.

[integer]

This parameter corresponds to the population size in genetic algorithms (GA). It specifies the number of initial candidate solutions that the algorithm starts with for its evolutionary search. This parameter is only required for optimization algorithms that operate on a population, such as GA or DEoptim.

default: initial_size = 50.

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

[integer]

Number of cores to use for parallel processing. Since fitting optimal parameters for each subject is an independent task, parallel computation can significantly speed up the fitting process:

• nc = 1: The fitting proceeds sequentially. Parameters for one subject are fitted completely before moving to the next subject.

• nc > 1: The fitting is performed in parallel across subjects. For example, if nc = 4, the algorithm will simultaneously fit data for four subjects. Once these are complete, it will proceed to fit the next batch of subjects (e.g., subjects 5-8), and so on, until all subjects are processed.

default: nc = 1

[string]

Choose an algorithm package from L-BFGS-B, GenSA,GA,DEoptim,PSO, Bayesian, CMA-ES.

In addition, any algorithm from the nloptr package is also supported. If your chosen nloptr algorithm requires a local search, you need to input a character vector. The first element represents the algorithm used for global search, and the second element represents the algorithm used for local search.

The current row number.

The frequency of left option appearance

The frequency of right option appearance

The number of times left option was picked

The number of times left option was picked

The value of the left option

The value of the right option

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

[integer] Controls the initial exploration phase in the epsilon-first strategy. This is the number of early trials where the subject makes purely random choices, as they haven't yet learned the options' values. For example, threshold = 20 means random choices for the first 20 trials. For epsilon-greedy or epsilon-decreasing strategies, threshold should be kept at its default value.

P(x) = \begin{cases} \text{trial} \le \text{threshold}, & x=1 \text{ (random choosing)} \\ \text{trial} > \text{threshold}, & x=0 \text{ (value-based choosing)} \end{cases}

default: threshold = 1

epsilon-first: threshold = 20, epsilon = NA, lambda = NA

[numeric] A parameter used in the epsilon-greedy exploration strategy. It defines the probability of making a completely random choice, as opposed to choosing based on the relative values of the left and right options. For example, if epsilon = 0.1, the subject has a 10 choice and a 90 relevant when threshold is at its default value (1) and lambda is not set.

P(x) = \begin{cases} \epsilon, & x=1 \text{ (random choosing)} \\ 1-\epsilon, & x=0 \text{ (value-based choosing)} \end{cases}

epsilon-greedy: threshold = 1, epsilon = 0.1, lambda = NA

[vector] A numeric value that controls the decay rate of exploration probability in the epsilon-decreasing strategy. A higher lambda value means the probability of random choice will decrease more rapidly as the number of trials increases.

P(x) = \begin{cases} \frac{1}{1+\lambda \cdot trial}, & x=1 \text{ (random choosing)} \\ \frac{\lambda \cdot trial}{1+\lambda \cdot trial}, & x=0 \text{ (value-based choosing)} \end{cases}

epsilon-decreasing threshold = 1, epsilon = NA, lambda = 0.5

[vector] Extra parameters that may be used in functions.

[vector] Extra parameters that may be used in functions.

The current row number.

The frequency of left option appearance

The frequency of right option appearance

The number of times left option was picked

The number of times left option was picked

The value of the left option

The value of the right option

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

The expected value of the stimulus in the subject's mind at this point in time.

The subjective value that the subject assigns to the objective reward.

The objective reward received by the subject after selecting a stimulus.

The number of times the same stimulus has been chosen.

[vector] Parameters used in the Learning Rate Function, rate_func representing the rate at which the subject updates the difference (prediction error) between the reward and the expected value in the subject's mind.

The structure of eta depends on the model type:

• For the Temporal Difference (TD) model, where a single learning rate is used throughout the experiment

  V_{new} = V_{old} + \eta \cdot (R - V_{old})

• For the Risk-Sensitive Temporal Difference (RDTD) model, where two different learning rates are used depending on whether the reward is lower or higher than the expected value:

  V_{new} = V_{old} + \eta_{+} \cdot (R - V_{old}), R > V_{old}

  V_{new} = V_{old} + \eta_{-} \cdot (R - V_{old}), R < V_{old}

TD: eta = 0.3

RSTD: eta = c(0.3, 0.7)

[vector] Extra parameters that may be used in functions.

[vector] Extra parameters that may be used in functions.

The current row number.

The frequency of left option appearance

The frequency of right option appearance

The number of times left option was picked

The number of times left option was picked

The value of the left option

The value of the right option

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

The expected value of the stimulus in the subject's mind at this point in time.

The subjective value that the subject assigns to the objective reward.

The objective reward received by the subject after selecting a stimulus.

The number of times the same stimulus has been chosen.

[vector] This parameter represents the exponent in utility functions, fcun_gamma, specifically:

• Stevens' Power Law: Utility is modeled as:

  U(R) = {R}^{\gamma}

• Kahneman's Prospect Theory: This exponent is applied differently based on the sign of the reward:

  U(R) = \begin{cases} R^{\gamma_{1}}, & R > 0 \\ \beta \cdot R^{\gamma_{2}}, & R < 0 \end{cases}

[vector] Extra parameters that may be used in functions.

[vector] Extra parameters that may be used in functions.

The current row number.

The frequency of left option appearance

The frequency of right option appearance

The number of times left option was picked

The number of times left option was picked

The value of the left option

The value of the right option

Whether the participant chose the left option.

Whether the participant chose the right option.

The probability that the model assigns to choosing the left option.

The probability that the model assigns to choosing the left option.

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

Are you calculating the probability for the left option or the right option?

If the choice was random, the value is 1; If the choice was based on value, the value is 0.

The expected value of the stimulus in the subject's mind at this point in time.

The subjective value that the subject assigns to the objective reward.

The objective reward received by the subject after selecting a stimulus.

The number of times the same stimulus has been chosen.

[vector] Extra parameters that may be used in functions.

[vector] Extra parameters that may be used in functions.

The current row number.

The frequency of left option appearance

The frequency of right option appearance

The number of times left option was picked

The number of times left option was picked

The value of the left option

The value of the right option

[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

Are you calculating the probability for the left option or the right option?

[vector] Parameter used in the Upper-Confidence-Bound (UCB) action selection formula. func_pi controls the degree of exploration by scaling the uncertainty bonus given to less-explored options. A larger value of pi (denoted as c in Sutton and Barto(1998) ) increases the influence of this bonus, leading to more exploration of actions with uncertain estimated values. Conversely, a smaller pi results in less exploration.

A_t = \arg \max_{a} \left[ V_t(a) + \pi \sqrt{\frac{\ln(t)}{N_t(a)}} \right]

default: pi = NA

[vector] Extra parameters that may be used in functions.

[vector] Extra parameters that may be used in functions.

[numeric]

The current row number.

[numeric]

The frequency of left option appearance

[numeric]

The frequency of right option appearance

[numeric]

The number of times left option was picked

[numeric]

The number of times left option was picked

[numeric]

The value of the left option with bias (if pi != 0)

[numeric]

The value of the right option with bias (if pi != 0)

[character]

Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = "Extra_Var1"

[character]

Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = "Extra_Var2"

[character]

Are you calculating the probability for the left option or the right option?

LR = "L"; LR = "R"

[numeric]

If the choice was random, the value is 1; If the choice was based on value, the value is 0.

[vector]

Parameters used in the Soft-Max Function. prob_func representing the sensitivity of the subject to the value difference when making decisions. It determines the probability of selecting the left option versus the right option based on their values. A larger value of tau indicates greater sensitivity to the value difference between the options. In other words, even a small difference in value will make the subject more likely to choose the higher-value option.

P_L = \frac{1}{1+e^{-(V_L-V_R) \cdot \tau}}; P_R = \frac{1}{1+e^{-(V_R-V_L) \cdot \tau}}

e.g., tau = c(0.5)

[numeric]

A numeric value between 0 and 1, representing the lapse rate.

You can interpret this parameter as the probability of the agent "slipping" or making a random choice, irrespective of the learned action values. This accounts for moments of inattention or motor errors. In this sense, it represents the minimum probability with which any given option will be selected. It is a free parameter that acknowledges that individuals do not always make decisions with full concentration throughout an experiment.

From a modeling perspective, the lapse rate is crucial for preventing the log-likelihood calculation from returning -Inf. This issue arises when the model assigns a probability of zero to an action that the participant actually chose (log(0) is undefined). By ensuring every option has a non-zero minimum probability, the lapse parameter makes the fitting process more stable and robust against noise in the data.

P_{final} = (1 - lapse) \cdot P_{softmax} + \frac{lapse}{N_{choices}}

default: lapse = 0.02

This ensures each option has a minimum selection probability of 1 percent in TAFC tasks.

[vector]

Extra parameters that may be used in functions.

[vector]

Extra parameters that may be used in functions.

[character]

Specifies the learning policy to be used. This determines how the model updates action values based on observed or simulated choices. It can be either "off" or "on".

• Off-Policy (Q-learning): This is the most common approach for modeling reinforcement learning in Two-Alternative Forced Choice (TAFC) tasks. In this mode, the model's goal is to learn the underlying value of each option by observing the human participant's behavior. It achieves this by consistently updating the value of the option that the human actually chose. The focus is on understanding the value representation that likely drove the participant's decisions.

• Off-Policy (SARSA): In this mode, the target policy and the behavior policy are identical. The model first computes the selection probability for each option based on their current values. Critically, it then uses these probabilities to sample its own action. The value update is then performed on the action that the model itself selected. This approach focuses more on directly mimicking the stochastic choice patterns of the agent, rather than just learning the underlying values from a fixed sequence of actions.

default: policy = "off"

[character]

Estimation method. Can be either "MLE" or "MAP".

• "MLE": (Default) Maximum Likelihood Estimation. This method finds the parameter values that maximize the log-likelihood of the data. A higher log-likelihood indicates that the parameters provide a better explanation for the observed human behavior. In other words, data simulated using these parameters would most closely resemble the actual human data. This method does not consider any prior information about the parameters.

• "MAP": Maximum A Posteriori Estimation. This method finds the parameter values that maximize the posterior probability. It is an iterative process based on the Expectation-Maximization (EM) framework.

  • Initialization: The process begins by assuming a uniform distribution as the prior for each parameter, making the initial log-prior zero. The first optimization is thus equivalent to MLE.

  • Iteration: After finding the best parameters for all subjects, the algorithm assesses the actual distribution of each parameter and fits a normal distribution to it. This fitted distribution becomes the new empirical prior.

  • Re-estimation: The parameters are then re-optimized to maximize the updated posterior probability.

  • Convergence: This cycle repeats until the posterior probability converges or the maximum number of iterations (specified by iteration_g) is reached.

  Using this method requires that the priors argument be specified to define the initial prior distributions.

default: estimate = "MLE"

[data.frame]

This data should include the following mandatory columns:

• "sub"

• "time_line" (e.g., "Block", "Trial")

• "L_choice"

• "R_choice"

• "L_reward"

• "R_reward"

• "sub_choose"

[character]

Specifies the ID of the subject whose optimal parameters will be fitted. This parameter accepts either string or numeric values. The provided ID must correspond to an existing subject identifier within the raw dataset provided to the function.

[integer]

The total number of trials in your experiment.

[integer]

The number of free parameters in your model.

[function]

The objective function that the optimization algorithm package accepts. This function must strictly take only one argument, fit_p (a vector of model parameters). Its output must be a single numeric value representing the loss function to be minimized. For more detailed requirements and examples, please refer to the relevant documentation ( TD, RSTD, Utility ).

[vector]

Lower bounds of free parameters

[vector]

Upper bounds of free parameters

[list] A list specifying the prior distributions for the model parameters. This argument is mandatory when using estimate = "MAP". There are two primary scenarios for its use:

1. Static MAP Estimation (Non-Hierarchical) This approach is used when you have a strong, pre-defined belief about the parameter priors and do not want the model to update them iteratively.

Configuration:

• Set estimate = "MAP".

• Provide a list defining your confident prior distributions.

• Keep iteration_g = 0 (the default).

Behavior:

The algorithm maximizes the posterior probability based solely on your specified priors. It will not use the EM (Expectation-Maximization) framework to learn new priors from the data.

2. Hierarchical Bayesian Estimation via EM This approach is used to let the model learn the group-level (hierarchical) prior distributions directly from the data.

Configuration:

• Set estimate = "MAP".

• Specify a weak or non-informative initial prior, such as a uniform distribution for all parameters.

• Set iteration_g to a value greater than 0.

Behavior:

With a uniform prior, the initial log-posterior equals the log-likelihood, making the first estimation step equivalent to MLE. The algorithm then initiates the EM procedure: it iteratively assesses the actual parameter distribution across all subjects and updates the group-level priors. This cycle continues until the posterior converges or iteration_g is reached.

default: priors = NULL

[vector]

Initial values for the free parameters that the optimization algorithm will search from. These are primarily relevant when using algorithms that require an explicit starting point, such as L-BFGS-B. If not specified, the function will automatically generate initial values close to zero.

default: initial_params = NA.

[integer]

This parameter corresponds to the population size in genetic algorithms (GA). It specifies the number of initial candidate solutions that the algorithm starts with for its evolutionary search. This parameter is only required for optimization algorithms that operate on a population, such as 'GA' or 'DEoptim'.

default: initial_size = 50.

[integer]

The number of iterations the optimization algorithm will perform when searching for the best-fitting parameters during the fitting phase. A higher number of iterations may increase the likelihood of finding a global optimum but also increases computation time.

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

[character]

Choose an algorithm package from L-BFGS-B, GenSA,GA,DEoptim,PSO, Bayesian, CMA-ES.

In addition, any algorithm from the nloptr package is also supported. If your chosen nloptr algorithm requires a local search, you need to input a character vector. The first element represents the algorithm used for global search, and the second element represents the algorithm used for local search.

[string]

Specifies the learning policy to be used. This determines how the model updates action values based on observed or simulated choices. It can be either "off" or "on".

• Off-Policy (Q-learning): This is the most common approach for modeling reinforcement learning in Two-Alternative Forced Choice (TAFC) tasks. In this mode, the model's goal is to learn the underlying value of each option by observing the human participant's behavior. It achieves this by consistently updating the value of the option that the human actually chose. The focus is on understanding the value representation that likely drove the participant's decisions.

• Off-Policy (SARSA): In this mode, the target policy and the behavior policy are identical. The model first computes the selection probability for each option based on their current values. Critically, it then uses these probabilities to sample its own action. The value update is then performed on the action that the model itself selected. This approach focuses more on directly mimicking the stochastic choice patterns of the agent, rather than just learning the underlying values from a fixed sequence of actions.

default: policy = "off"

[data.frame]

This data should include the following mandatory columns:

• sub "Subject"

• time_line "Block" "Trial"

• L_choice "L_choice"

• R_choice "R_choice"

• L_reward "L_reward"

• R_reward "R_reward"

• sub_choose "Sub_Choose"

[CharacterVector]

Specifies which subject's data to use. In parameter and model recovery analyses, the specific subject ID is often irrelevant. Although the experimental trial order might have some randomness for each subject, the sequence of reward feedback is typically pseudo-random.

The default value for this argument is NULL. When id = NULL, the program automatically detects existing subject IDs within the dataset. It then randomly selects one subject as a sample, and the parameter and model recovery procedures are performed based on this selected subject's data.

default: id = NULL

[integer]

Represents the total number of trials a single subject experienced in the experiment. If this parameter is kept at its default value of NULL, the program will automatically detect how many trials a subject experienced from the provided data. This information is primarily used for calculating model fit statistics such as AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion).

default: n_trials = NULL

[CharacterVector]

A character vector containing the names of all user-defined functions required for the computation. When parallel computation is enabled (i.e., nc > 1), user-defined models and their custom functions might not be automatically accessible within the parallel environment.

Therefore, if you have created your own reinforcement learning model that modifies the package's default six default functions (default functions: util_func = func_gamma, rate_func = func_eta, expl_func = func_epsilon, bias_func = func_pi, prob_func = func_tau, loss_func = func_logl ), you must explicitly provide the names of your custom functions as a vector here.

[List]

The names of fit modals

e.g. model_names = c("TD", "RSTD", "Utility")

[List]

A collection of functions used to generate simulated data.

[List]

The lower bounds for simulate models

e.g. simulate_lower = list(c(0, 0), c(0, 0, 0), c(0, 0, 0))

[List] The upper bounds for simulate models

e.g. simulate_upper = list(c(1, 1), c(1, 1, 1), c(1, 1, 1))

[List]

A collection of functions applied to fit models to the data.

e.g. fit_models = list(binaryRL::TD, binaryRL::RSTD, binaryRL::Utility)

[List]

The lower bounds for model fit models

e.g. fit_lower = list(c(0, 0), c(0, 0, 0), c(0, 0, 0))

[List]

The upper bounds for model fit models

e.g. fit_upper = list(c(1, 1), c(1, 1, 1), c(1, 1, 10))

[integer]

This parameter determines how many simulated datasets are created for subsequent model and parameter recovery analyses.

default: iteration_s = 10

[integer]

The number of iterations the optimization algorithm will perform when searching for the best-fitting parameters during the fitting phase. A higher number of iterations may increase the likelihood of finding a global optimum but also increases computation time.

default: iteration_f = 10

[NumericVector]

Initial values for the free parameters that the optimization algorithm will search from. These are primarily relevant when using algorithms that require an explicit starting point, such as L-BFGS-B. If not specified, the function will automatically generate initial values close to zero.

default: initial_params = NA.

[integer]

This parameter corresponds to the population size in genetic algorithms (GA). It specifies the number of initial candidate solutions that the algorithm starts with for its evolutionary search. This parameter is only required for optimization algorithms that operate on a population, such as GA or DEoptim.

default: initial_size = 50.

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

[integer]

Number of cores to use for parallel processing. Since fitting optimal parameters for each subject is an independent task, parallel computation can significantly speed up the fitting process:

• nc = 1: The fitting proceeds sequentially. Parameters for one subject are fitted completely before moving to the next subject.

• nc > 1: The fitting is performed in parallel across subjects. For example, if nc = 4, the algorithm will simultaneously fit data for four subjects. Once these are complete, it will proceed to fit the next batch of subjects (e.g., subjects 5-8), and so on, until all subjects are processed.

default: nc = 1

[string]

Choose an algorithm package from L-BFGS-B, GenSA,GA,DEoptim,PSO, Bayesian, CMA-ES.

In addition, any algorithm from the nloptr package is also supported. If your chosen nloptr algorithm requires a local search, you need to input a character vector. The first element represents the algorithm used for global search, and the second element represents the algorithm used for local search.

[list]

A list generated by function simulate_list()

[vector]

Specifies which subject's data to use. In parameter and model recovery analyses, the specific subject ID is often irrelevant. Although the experimental trial order might have some randomness for each subject, the sequence of reward feedback is typically pseudo-random.

The default value for this argument is NULL. When id = NULL, the program automatically detects existing subject IDs within the dataset. It then randomly selects one subject as a sample, and the parameter and model recovery procedures are performed based on this selected subject's data.

default: id = NULL

[integer]

The total number of trials in your experiment.

[integer]

The number of free parameters in your model.

[character]

A character vector containing the names of all user-defined functions required for the computation. When parallel computation is enabled (i.e., 'nc > 1'), user-defined models and their custom functions might not be automatically accessible within the parallel environment.

Therefore, if you have created your own reinforcement learning model that modifies the package's default four default functions (default functions: util_func = func_gamma, rate_func = func_eta, expl_func = func_epsilon bias_func = func_pi prob_func = func_tau ), you must explicitly provide the names of your custom functions as a vector here.

[character]

Specifies the learning policy to be used. This determines how the model updates action values based on observed or simulated choices. It can be either "off" or "on".

• Off-Policy (Q-learning): This is the most common approach for modeling reinforcement learning in Two-Alternative Forced Choice (TAFC) tasks. In this mode, the model's goal is to learn the underlying value of each option by observing the human participant's behavior. It achieves this by consistently updating the value of the option that the human actually chose. The focus is on understanding the value representation that likely drove the participant's decisions.

• Off-Policy (SARSA): In this mode, the target policy and the behavior policy are identical. The model first computes the selection probability for each option based on their current values. Critically, it then uses these probabilities to sample its own action. The value update is then performed on the action that the model itself selected. This approach focuses more on directly mimicking the stochastic choice patterns of the agent, rather than just learning the underlying values from a fixed sequence of actions.

default: policy = "off"

[character]

The name of your modal

[function]

fit model

[vector]

Lower bounds of free parameters

[vector]

Upper bounds of free parameters

[numeric]

Initial values for the free parameters that the optimization algorithm will search from. These are primarily relevant when using algorithms that require an explicit starting point, such as L-BFGS-B. If not specified, the function will automatically generate initial values close to zero.

default: initial_params = NA.

[integer]

This parameter corresponds to the population size in genetic algorithms (GA). It specifies the number of initial candidate solutions that the algorithm starts with for its evolutionary search. This parameter is only required for optimization algorithms that operate on a population, such as 'GA' or 'DEoptim'.

default: initial_size = 50.

[integer]

The number of iterations the optimization algorithm will perform when searching for the best-fitting parameters during the fitting phase. A higher number of iterations may increase the likelihood of finding a global optimum but also increases computation time.

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

[integer]

Number of cores to use for parallel processing. Since fitting optimal parameters for each subject is an independent task, parallel computation can significantly speed up the fitting process:

• 'nc = 1': The fitting proceeds sequentially. Parameters for one subject are fitted completely before moving to the next subject.

• 'nc > 1': The fitting is performed in parallel across subjects. For example, if 'nc = 4', the algorithm will simultaneously fit data for four subjects. Once these are complete, it will proceed to fit the next batch of subjects (e.g., subjects 5-8), and so on, until all subjects are processed.

default: nc = 1

[character] Choose an algorithm package from L-BFGS-B, GenSA,GA,DEoptim,PSO, Bayesian, CMA-ES.

In addition, any algorithm from the nloptr package is also supported. If your chosen nloptr algorithm requires a local search, you need to input a character vector. The first element represents the algorithm used for global search, and the second element represents the algorithm used for local search.

[data.frame]

This data should include the following mandatory columns:

• sub "Subject"

• time_line "Block" "Trial"

• L_choice "L_choice"

• R_choice "R_choice"

• L_reward "L_reward"

• R_reward "R_reward"

• sub_choose "Sub_Choose"

[CharacterVector]

A vector specifying the subject ID(s) for which parameters should be fitted. The function will process only the subjects provided in this vector.

To fit all subjects, you can either explicitly set the argument as id = unique(data$Subject) or leave it as the default (id = NULL). Both approaches will direct the function to fit parameters for every unique subject in the dataset.

It is strongly recommended to avoid using simple numeric sequences like id = 1:4. This practice can lead to errors if subject IDs are stored as strings (e.g., subject four is stored as "004") or are not sequentially numbered.

default: id = NULL

[data.frame]

Output data generated by the fit_p() function. Each row represents model fit results for a subject.

[Function]

A model function to be applied in evaluating the experimental effect.

[string]

A character string specifying the name of the model to extract from the result.

[string]

A prefix string used to identify parameter columns in the result data

default: param_prefix = "param_"

[integer]

Represents the total number of trials a single subject experienced in the experiment. If this parameter is kept at its default value of NULL, the program will automatically detect how many trials a subject experienced from the provided data. This information is primarily used for calculating model fit statistics such as AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion).

default: n_trials = NULL

[string]

The name of your RL model

[string]

This parameter controls the function's operational mode. It has three possible values, each typically associated with a specific function:

• "simulate": Should be used when working with rcv_d.

• "fit": Should be used when working with fit_p.

• "replay": Should be used when working with rpl_e.

In most cases, you won't need to modify this parameter directly, as suitable default values are set for different contexts.

[string]

Specifies the learning policy to be used. This determines how the model updates action values based on observed or simulated choices. It can be either "off" or "on".

• Off-Policy (Q-learning): This is the most common approach for modeling reinforcement learning in Two-Alternative Forced Choice (TAFC) tasks. In this mode, the model's goal is to learn the underlying value of each option by observing the human participant's behavior. It achieves this by consistently updating the value of the option that the human actually chose. The focus is on understanding the value representation that likely drove the participant's decisions.

• Off-Policy (SARSA): In this mode, the target policy and the behavior policy are identical. The model first computes the selection probability for each option based on their current values. Critically, it then uses these probabilities to sample its own action. The value update is then performed on the action that the model itself selected. This approach focuses more on directly mimicking the stochastic choice patterns of the agent, rather than just learning the underlying values from a fixed sequence of actions.

[data.frame]

This data should include the following mandatory columns:

• sub "Subject"

• time_line "Block" "Trial"

• L_choice "L_choice"

• R_choice "R_choice"

• L_reward "L_reward"

• R_reward "R_reward"

• sub_choose "Sub_Choose"

[string]

Which subject is going to be analyzed. The value should correspond to an entry in the "sub" column, which must contain the subject IDs.

e.g. id = 18

[integer]

The number of free parameters in your model.

[integer]

The total number of trials in your experiment.

[NumericVector]

Note: This should not be confused with the discount rate parameter (also named gamma) found in Temporal Difference (TD) models. Rescorla-Wagner model does not include a discount rate. Here, gamma is used as a free parameter to shape the utility function.

• Stevens' Power Law: Utility is modeled as:

  U(R) = {R}^{\gamma}

• Kahneman's Prospect Theory: This exponent is applied differently based on the sign of the reward:

  U(R) = \begin{cases} R^{\gamma_{1}}, & R > 0 \\ \beta \cdot R^{\gamma_{2}}, & R < 0 \end{cases}

default: gamma = 1

[NumericVector]

Parameters used in the Learning Rate Function, rate_func, representing the rate at which the subject updates the difference (prediction error) between the reward and the expected value in the subject's mind.

The structure of eta depends on the model type:

• For the Temporal Difference (TD) model, where a single learning rate is used throughout the experiment

  V_{new} = V_{old} + \eta \cdot (R - V_{old})

• For the Risk-Sensitive Temporal Difference (RDTD) model, where two different learning rates are used depending on whether the reward is lower or higher than the expected value:

  V_{new} = V_{old} + \eta_{+} \cdot (R - V_{old}), R > V_{old}

  V_{new} = V_{old} + \eta_{-} \cdot (R - V_{old}), R < V_{old}

TD: eta = 0.3

RSTD: eta = c(0.3, 0.7)

[double]

Subject's initial expected value for each stimulus's reward. If this value is not set initial_value = NA, the subject will use the reward received after the first trial as the initial value for that stimulus. In other words, the learning rate for the first trial is 100

default: initial_value = NA_real_

[integer]

Controls the initial exploration phase in the epsilon-first strategy. This is the number of early trials where the subject makes purely random choices, as they haven't yet learned the options' values. For example, threshold = 20 means random choices for the first 20 trials. For epsilon-greedy or epsilon-decreasing strategies, threshold should be kept at its default value.

P(x) = \begin{cases} \text{trial} \le \text{threshold}, & x=1 \text{ (random choosing)} \\ \text{trial} > \text{threshold}, & x=0 \text{ (value-based choosing)} \end{cases}

default: threshold = 1

epsilon-first: threshold = 20, epsilon = NA, lambda = NA

[NumericVector]

A parameter used in the epsilon-greedy exploration strategy. It defines the probability of making a completely random choice, as opposed to choosing based on the relative values of the left and right options. For example, if epsilon = 0.1, the subject has a 10 choice and a 90 relevant when threshold is at its default value (1) and lambda is not set.

P(x) = \begin{cases} \epsilon, & x=1 \text{ (random choosing)} \\ 1-\epsilon, & x=0 \text{ (value-based choosing)} \end{cases}

epsilon-greedy: threshold = 1, epsilon = 0.1, lambda = NA

[NumericVector]

A numeric value that controls the decay rate of exploration probability in the epsilon-decreasing strategy. A higher lambda value means the probability of random choice will decrease more rapidly as the number of trials increases.

P(x) = \begin{cases} \frac{1}{1+\lambda \cdot trial}, & x=1 \text{ (random choosing)} \\ \frac{\lambda \cdot trial}{1+\lambda \cdot trial}, & x=0 \text{ (value-based choosing)} \end{cases}

epsilon-decreasing: threshold = 1, epsilon = NA, lambda = 0.5

[NumericVector]

Parameter used in the Upper-Confidence-Bound (UCB) action selection formula. bias_func controls the degree of exploration by scaling the uncertainty bonus given to less-explored options. A larger value of pi (denoted as c in Sutton and Barto(2018) ) increases the influence of this bonus, leading to more exploration of actions with uncertain estimated values. Conversely, a smaller pi results in less exploration.

A_t = \arg \max_{a} \left[ V_t(a) + \pi \sqrt{\frac{\ln(t)}{N_t(a)}} \right]

default: pi = NA

[NumericVector]

Parameters used in the Soft-Max Function. prob_func representing the sensitivity of the subject to the value difference when making decisions. It determines the probability of selecting the left option versus the right option based on their values. A larger value of tau indicates greater sensitivity to the value difference between the options. In other words, even a small difference in value will make the subject more likely to choose the higher-value option.

P_L = \frac{1}{1+e^{-(V_L-V_R) \cdot \tau}}; P_R = \frac{1}{1+e^{-(V_R-V_L) \cdot \tau}}

default tau = NA

[double]

A numeric value between 0 and 1, representing the lapse rate.

You can interpret this parameter as the probability of the agent "slipping" or making a random choice, irrespective of the learned action values. This accounts for moments of inattention or motor errors. In this sense, it represents the minimum probability with which any given option will be selected. It is a free parameter that acknowledges that individuals do not always make decisions with full concentration throughout an experiment.

From a modeling perspective, the lapse rate is crucial for preventing the log-likelihood calculation from returning -Inf. This issue arises when the model assigns a probability of zero to an action that the participant actually chose (log(0) is undefined). By ensuring every option has a non-zero minimum probability, the lapse parameter makes the fitting process more stable and robust against noise in the data.

P_{final} = (1 - lapse) \cdot P_{softmax} + \frac{lapse}{N_{choices}}

default: lapse = 0.02

This ensures each option has a minimum selection probability of 1 percent in TAFC tasks.

[NumericVector]

Extra parameters that may be used in functions.

[NumericVector]

Extra parameters that may be used in functions.

[list]

A list specifying the prior distributions for the model parameters. This argument is mandatory when using estimate = "MAP".

default: priors = NULL

[Function]

Utility Function see func_gamma.

[Function]

Learning Rate Function see func_eta.

[Function]

Exploration Strategy Function see func_epsilon.

[Function]

Upper-Confidence-Bound see func_pi.

[Function]

Soft-Max Function see func_tau.

[Function]

Loss Function see func_logl.

[string]

Column name of subject ID

e.g. sub = "Subject"

[CharacterVector]

A vector specifying the name of the column that the sequence of the experiment. This argument defines how the experiment is structured, such as whether it is organized by "Block" with breaks in between, and multiple trials within each block.

default: time_line = c("Block", "Trial")

[string]

Column name of left choice.

default: L_choice = "Left_Choice"

[string]

Column name of right choice.

default: R_choice = "Right_Choice"

[string]

Column name of the reward of left choice

default: L_reward = "Left_reward"

[string]

Column name of the reward of right choice

default: R_reward = "Right_reward"

[string]

Column name of choices made by the subject.

default: sub_choose = "Choose"

[string]

Column name of choices made by the model, which you could ignore.

default: rob_choose = "Rob_Choose"

[CharacterVector]

Defaults to NULL. If left as NULL, it will directly capture all column names from the raw data.

[string]

Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model.

default: var1 = NA_character_

[string]

Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model.

default: var2 = NA_character_

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

[integer]

The number of decimal places to retain for columns related to value function

default: digits_1 = 2

[integer]

The number of decimal places to retain for columns related to select function.

default: digits_2 = 5

[string]

- "r": Use the pure R version of the code.

- "cpp": Use the Rcpp-optimized version.

default: engine = "cpp"

[data.frame]

This data should include the following mandatory columns:

• "sub"

• "time_line" (e.g., "Block", "Trial")

• "L_choice"

• "R_choice"

• "L_reward"

• "R_reward"

• "sub_choose"

[vector]

Specifies which subject's data to use. In parameter and model recovery analyses, the specific subject ID is often irrelevant. Although the experimental trial order might have some randomness for each subject, the sequence of reward feedback is typically pseudo-random.

The default value for this argument is NULL. When id = NULL, the program automatically detects existing subject IDs within the dataset. It then randomly selects one subject as a sample, and the parameter and model recovery procedures are performed based on this selected subject's data.

default: id = NULL

[function]

The objective function that the optimization algorithm package accepts. This function must strictly take only one argument, params (a vector of model parameters). Its output must be a single numeric value representing the loss function to be minimized. For more detailed requirements and examples, please refer to the relevant documentation ( TD, RSTD, Utility ).

[integer]

The number of free parameters in your model.

[integer]

The total number of trials in your experiment.

[vector]

Lower bounds of free parameters

[vector]

Upper bounds of free parameters

[integer]

This parameter determines how many simulated datasets are created for subsequent model and parameter recovery analyses.

[integer]

Random seed. This ensures that the results are reproducible and remain the same each time the function is run.

default: seed = 123

binaryRL result

others