With clustered data, such as where students are nested within schools or employees are nested within organizations, it is often of interest to estimate and compare associations among variables separately for each level....With clustered data, such as where students are nested within schools or employees are nested within organizations, it is often of interest to estimate and compare associations among variables separately for each level. While researchers routinely estimate between-cluster effects using the sample cluster means of a predictor, previous research has shown that such practice leads to biased estimates of coefficients at the between level, and recent research has recommended the use of latent cluster means with the multilevel structural equation modeling framework. However, the latent cluster mean approach may not always be the best choice as it (a) relies on the assumption that the population cluster sizes are close to infinite, (b) requires a relatively large number of clusters, and (c) is currently only implemented in specialized software such as Mplus. In this paper, we show how using empirical Bayes estimates of the cluster means can also lead to consistent estimates of between-level coefficients, and illustrate how the empirical Bayes estimate can incorporate finite population corrections when information on population cluster sizes is available. Through a series of Monte Carlo simulation studies, we show that the empirical Bayes cluster-mean approach performs similarly to the latent cluster mean approach for estimating the between-cluster coefficients in most conditions when the infinite-population assumption holds, and applying the finite population correction provides reasonable point and interval estimates when the population is finite. The performance of EBM can be further improved with restricted maximum likelihood estimation and likelihood-based confidence intervals. We also provide an R function that implements the empirical Bayes cluster-mean approach, and illustrate it using data from the classic High School and Beyond Study.
Network analysis has gained popularity as an approach to investigate psychological constructs. However, there are currently no guidelines for applied researchers when encountering missing values. In this simulation study...Network analysis has gained popularity as an approach to investigate psychological constructs. However, there are currently no guidelines for applied researchers when encountering missing values. In this simulation study, we compared the performance of a two-step EM algorithm with separated steps for missing handling and regularization, a combined direct EM algorithm, and pairwise deletion. We investigated conditions with varying network sizes, numbers of observations, missing data mechanisms, and percentages of missing values. These approaches are evaluated with regard to recovering population networks in terms of loss in the precision matrix, edge set identification and network statistics. The simulation showed adequate performance only in conditions with large samples () or small networks ( = 10). Comparing the missing data approaches, the direct EM appears to be more sensitive and superior in nearly all chosen conditions. The two-step EM yields better results when the ratio of n/p is very large - being less sensitive but more specific. Pairwise deletion failed to converge across numerous conditions and yielded inferior results overall. Overall, direct EM is recommended in most cases, as it is able to mitigate the impact of missing data quite well, while modifications to two-step EM could improve its performance.
Multivariate Behav Res
· 2024 · PMID 38160329
·
Full text
Multilevel autoregressive models are popular choices for the analysis of intensive longitudinal data in psychology. Empirical studies have found a positive correlation between autoregressive parameters of affective time...Multilevel autoregressive models are popular choices for the analysis of intensive longitudinal data in psychology. Empirical studies have found a positive correlation between autoregressive parameters of affective time series and the between-person measures of psychopathology, a phenomenon known as the . However, it has been argued that such findings may represent a statistical artifact: Although common models assume normal error distributions, empirical data (for instance, measurements of negative affect among healthy individuals) often exhibit the , that is response distributions with high , low mean, and low variability. In this paper, we investigated whether-and to what extent-the floor effect leads to erroneous conclusions by means of a simulation study. We describe three dynamic models which have meaningful substantive interpretations and can produce floor-effect data. We simulate multilevel data from these models, varying skewness independent of individuals' autoregressive parameters, while also varying the number of time points and cases. Analyzing these data with the standard multilevel AR(1) model we found that positive bias only occurs when modeling with random residual variance, whereas modeling with fixed residual variance leads to negative bias. We discuss the implications of our study for data collection and modeling choices.
Single case experimental designs are an important research design in behavioral and medical research. Although there are design standards prescribed by the What Works Clearinghouse for single case experimental designs, t...Single case experimental designs are an important research design in behavioral and medical research. Although there are design standards prescribed by the What Works Clearinghouse for single case experimental designs, these standards do not include statistically derived power computations. Recently we derived the equations for computing power for (AB) designs. However, these computations and the software code in R may not be accessible to applied researchers who are most likely to want to compute power for their studies. Therefore, we have developed an (AB) power calculator Shiny App (https://abkpowercalculator.shinyapps.io/ABkpowercalculator/) that researchers can use with no software training. These power computations assume that the researcher would be interested in fitting multilevel models with autocorrelations or conduct similar analyses. The purpose of this software contribution is to briefly explain how power is derived for balanced (AB) designs and to elaborate on how to use the Shiny App. The app works well on not just computers but mobile phones without installing the R program. We believe this can be a valuable tool for practitioners and applied researchers who want to plan their single case studies with sufficient power to detect appropriate effect sizes.
Increasingly, behavioral scientists encounter data where several individuals were measured on multiple variables over numerous occasions. Many current methods combine these data, assuming all individuals are randomly equ...Increasingly, behavioral scientists encounter data where several individuals were measured on multiple variables over numerous occasions. Many current methods combine these data, assuming all individuals are randomly equivalent. An extreme alternative assumes no one is randomly equivalent. We propose state space mixture modeling as one possible compromise. State space mixture modeling assumes that unknown groups of people exist who share the same parameters of a state space model, and simultaneously estimates both the state space parameters and group membership. The goal is to find people that are undergoing similar change processes over time. The present work demonstrates state space mixture modeling on a simulated data set, and summarizes the results from a large simulation study. The illustration shows how the analysis is conducted, whereas the simulation provides evidence of its general validity and applicability. In the simulation study, sample size had the greatest influence on parameter estimation and the dimension of the change process had the greatest impact on correctly grouping people together, likely due to the distinctiveness of their patterns of change. State space mixture modeling offers one of the best-performing methods for simultaneously drawing conclusions about individual change processes while also analyzing multiple people.
Forced-choice questionnaires involve presenting items in blocks and asking respondents to provide a full or partial ranking of the items within each block. To prevent involuntary or voluntary response distortions, blocks...Forced-choice questionnaires involve presenting items in blocks and asking respondents to provide a full or partial ranking of the items within each block. To prevent involuntary or voluntary response distortions, blocks are usually formed of items that possess similar levels of desirability. Assembling forced-choice blocks is not a trivial process, because in addition to desirability, both the direction and magnitude of relationships between items and the traits being measured (i.e., factor loadings) need to be carefully considered. Based on simulations and empirical studies using item pairs, we provide recommendations on how to construct item pairs matched by desirability. When all pairs contain items keyed in the same direction, score reliability is improved by maximizing within-block loading differences. Higher reliability is obtained when even a small number of pairs consist of unequally keyed items.
The inverse probability of treatment weighting (IPTW) approach is commonly used in propensity score analysis to infer causal effects in regression models. Due to oversized IPTW weights and errors associated with propensi...The inverse probability of treatment weighting (IPTW) approach is commonly used in propensity score analysis to infer causal effects in regression models. Due to oversized IPTW weights and errors associated with propensity score estimation, the IPTW approach can underestimate the standard error of causal effect. To remediate this, bootstrap standard errors have been recommended to replace the IPTW standard error, but the ordinary bootstrap (OB) procedure might still result in underestimation of the standard error because of its inefficient resampling scheme and untreated oversized weights. In this paper, we develop a generalized bootstrap (GB) procedure for estimating the standard error and confidence intervals of the IPTW approach. Compared with the OB procedure and other three procedures in comparison, the GB procedure has the highest precision and yields conservative standard error estimates. As a result, the GB procedure produces short confidence intervals with highest coverage rates. We demonstrate the effectiveness of the GB procedure two simulation studies and a dataset from the National Educational Longitudinal Study-1988 (NELS-88).
A multilevel-discrete time survival model may be appropriate for purely hierarchical data, but when data are non-purely hierarchical due to individual mobility across clusters, a cross-classified discrete time survival m...A multilevel-discrete time survival model may be appropriate for purely hierarchical data, but when data are non-purely hierarchical due to individual mobility across clusters, a cross-classified discrete time survival model may be necessary. The purpose of this research was to investigate the performance of a cross-classified discrete-time survival model and assess the impact of ignoring a cross-classified data structure on the model parameters of a conventional discrete-time survival model and a multilevel discrete-time survival model. A Monte Carlo simulation was used to examine the performance of three discrete-time survival models when individuals are mobile across clusters. Simulation factors included the value of the between-clusters variance, number of clusters, within-cluster sample size, Weibull scale parameter, and mobility rate. The results suggest that substantial relative parameter bias, unacceptable coverage of the 95% confidence intervals, and severely biased standard errors are possible for all model parameters when a discrete-time survival model is used that ignores the cross-classified data structure. The findings presented in this study are useful for methodologists and practitioners in educational research, public health, and other social sciences where discrete-time survival analysis is a common methodological technique for analyzing event-history data.
Many measurement designs produce domain factors with small variances and factor loadings. The current study investigates the cause, prevalence, and problematic consequences of such domain factors. We collected a meta-ana...Many measurement designs produce domain factors with small variances and factor loadings. The current study investigates the cause, prevalence, and problematic consequences of such domain factors. We collected a meta-analytic sample of empirical applications, conducted a simulation study on statistical power and estimation precision, and provide a reanalysis of an empirical example. The meta-analysis shows that about a quarter of all standardized domain factor loadings is in the range of and about a third of all domains is measured by five or fewer indicators, resulting in small factor variances. The simulation study examines the associated difficulties concerning statistical power, trait recovery, irregular estimates, and estimation precision for a range of such realistic cases. The empirical example illustrates the challenge to develop measures that produce clearly interpretable domain factors. Study planning and interpretation need to take the (expected) sum of squared factor loadings per domain factor into account. This is relevant even if influences of domain factors are desired to be small, and equally applies to different model variants. We propose several strategies for how researchers may better unlock the bifactor model's full potential and clarify its interpretation.
The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evalu...The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evaluation of the performance of three types of FC measures (FC2, FC4 and FC5 with 2, 4 and 5 response options, respectively) and compared their performance to their Likert (LK) counterparts (LK2, LK4, and LK5) on (1) psychometric properties, (2) respondent reactions, and (3) susceptibility to response styles. Results showed that, compared to LK measures with the same number of response options, the three FC scales provided better support for the hypothesized factor structure, were perceived as more faking-resistant and cognitive demanding, and were less susceptible to response styles. FC4/5 and LK4/5 demonstrated similarly good reliability, while LK2 provided more reliable scores than FC2. When compared across the three FC measures, FC4 and FC5 displayed comparable psychometric performance and respondent reactions. FC4 exhibited a moderate presence of extreme response style, while FC5 had a weak presence of both extreme and middle response styles. Based on these findings, the study recommends the use of graded FC over dichotomous FC and LK, particularly FC5 when extreme response style is a concern.
Multivariate Behav Res
· 2024 · PMID 37624870
·
Full text
Self-regulating systems change along different timescales. Within a given week, a depressed person's affect might oscillate around a low equilibrium point. However, when the timeframe is expanded to capture the year duri...Self-regulating systems change along different timescales. Within a given week, a depressed person's affect might oscillate around a low equilibrium point. However, when the timeframe is expanded to capture the year during which they onboarded antidepressant medication, their equilibrium and oscillatory patterns might reorganize around a higher affective point. To simultaneously account for the meaningful change processes that happen at different time scales in complex self-regulatory systems, we propose a single model that combines a second-order linear differential equation for short timescale regulation and a first-order linear differential equation for long timescale adaptation of equilibrium. This model allows for individual-level moderation of short-timescale model parameters. The model is tested in a simulation study which shows that, surprisingly, the short and long timescales can fully overlap and the model still converges to the reasonable estimates. Finally, an application of this model to self-regulation of emotional well-being in recent widows is presented and discussed.
In psychology, the use of portable technology and wearable devices to ease participant burden in data collection is on the rise. This creates increased interest in collecting real-time or near real-time data from individ...In psychology, the use of portable technology and wearable devices to ease participant burden in data collection is on the rise. This creates increased interest in collecting real-time or near real-time data from individuals within their natural environments. As a result, vast amounts of observational time series data are generated. Often, motivation for collecting this data hinges on understanding within-person processes that underlie psychological phenomena. Motivated by the body of Dr. Peter Molenaar's life work calling for analytical approaches that consider potential heterogeneity and non-ergodicity, the focus of this paper is on using idiographic analyses to generate population inferences for within-person processes. Meta-analysis techniques using one-stage and two-stage random effects meta-analysis as implemented in single-case experimental designs are presented. The case for preferring a two-stage approach for meta-analysis of single-subject observational time series data is made and demonstrated using an empirical example. This provides a novel implementation of the methodology as prior implementations focus on applications to short time series with experimental designs. Inspired by Dr. Molenaar's work, we describe how an approach, two-stage random effects meta-analysis (2SRE-MA), aligns with recent calls to consider idiographic approaches when making population-level inferences regarding within-person processes.
In a cluster randomized trial clusters of persons, for instance, schools or health centers, are assigned to treatments, and all persons in the same cluster get the same treatment. Although less powerful than individual r...In a cluster randomized trial clusters of persons, for instance, schools or health centers, are assigned to treatments, and all persons in the same cluster get the same treatment. Although less powerful than individual randomization, cluster randomization is a good alternative if individual randomization is impossible or leads to severe treatment contamination (carry-over). Focusing on cluster randomized trials with a pretest and post-test of a quantitative outcome, this paper shows the equivalence of four methods of analysis: a three-level mixed (multilevel) regression for repeated measures with as levels cluster, person, and time, and allowing for unstructured between-cluster and within-cluster covariance matrices; a two-level mixed regression with as levels cluster and person, using change from baseline as outcome; a two-level mixed regression with as levels cluster and time, using cluster means as data; a one-level analysis of cluster means of change from baseline. Subsequently, similar equivalences are shown between a constrained mixed model and methods using the pretest as covariate. All methods are also compared on a cluster randomized trial on mental health in children. From these equivalences follows a simple method to calculate the sample size for a cluster randomized trial with baseline measurement, which is demonstrated step-by-step.
Park JJ, Fisher ZF, Chow SM
… +1 more, Molenaar PCM
Multivariate Behav Res
· 2024 · PMID 37590440
·
Full text
Rapid developments over the last several decades have brought increased focus and attention to the role of time scales and heterogeneity in the modeling of human processes. To address these emerging questions, subgroupin...Rapid developments over the last several decades have brought increased focus and attention to the role of time scales and heterogeneity in the modeling of human processes. To address these emerging questions, subgrouping methods developed in the discrete-time framework-such as the vector autoregression (VAR)-have undergone widespread development to identify shared nomothetic trends from idiographic modeling results. Given the dependence of VAR-based parameters on the measurement intervals of the data, we sought to clarify the strengths and limitations of these methods in recovering subgroup dynamics under different measurement intervals. Building on the work of Molenaar and collaborators for subgrouping individual time-series by means of the subgrouped chain graphical VAR (scgVAR) and the subgrouping option in the group iterative multiple model estimation (S-GIMME), we present results from a Monte Carlo study aimed at addressing the implications of identifying subgroups using these discrete-time methods when applied to continuous-time data. Results indicate that discrete-time subgrouping methods perform well at recovering true subgroups when the measurement intervals are large enough to capture the full range of a system's dynamics, either lagged or contemporaneous effects. Further implications and limitations are discussed therein.
Gender is person-specific, and it influences and is influenced by a breadth of multidimensional psychological factors, including cognition. Directionality is important for research on gender and cognition, as debate surr...Gender is person-specific, and it influences and is influenced by a breadth of multidimensional psychological factors, including cognition. Directionality is important for research on gender and cognition, as debate surrounds, for instance, whether masculine self-concepts precede spatial skills, or whether the reverse is true. In order to provide novel insights into the individualized nature of these relations, a person-specific network approach devised by Peter Molenaar and the first author - group iterative multiple model estimation for multiple solutions (GIMME-MS) - was applied to 75-day intensive longitudinal data on gender self-concept (i.e., femininity-masculinity, instrumentality, and expressivity) and cognition (i.e., mental rotations and verbal recall) from 103 young adults. GIMME-MS estimates individualized networks that contain same-day and next-day directed relations, prioritizing relations common across participants. It is ideal for analyzing behavioral time series with unclear directionality, as it generates multiple solutions from which an optimal one is selected. GIMME-MS revealed notable heterogeneity in the presence, direction, and nature of relations from gender self-concept to cognition (∼26% of participants) and vice versa (∼21% of participants). Findings are wholly novel in revealing the person-specific nature of gender and its cognitive dynamics, yet somehow, unsurprising given the revolutionary corpus of Peter Molenaar.
Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for th...Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom.
Dziak JJ, Almirall D, Dempsey W
… +2 more, Stanger C, Nahum-Shani I
Multivariate Behav Res
· 2024 · PMID 37459401
·
Full text
Sequential Multiple-Assignment Randomized Trials (SMARTs) play an increasingly important role in psychological and behavioral health research. This experimental approach enables researchers to answer scientific questions...Sequential Multiple-Assignment Randomized Trials (SMARTs) play an increasingly important role in psychological and behavioral health research. This experimental approach enables researchers to answer scientific questions about how to sequence and match interventions to the unique, changing needs of individuals. A variety of sample size planning resources for SMART studies have been developed, enabling researchers to plan SMARTs for addressing different types of scientific questions. However, relatively limited attention has been given to planning SMARTs with binary (dichotomous) outcomes, which often require higher sample sizes relative to continuous outcomes. Existing resources for estimating sample size requirements for SMARTs with binary outcomes do not consider the potential to improve power by including a baseline measurement and/or multiple repeated outcome measurements. The current paper addresses this issue by providing sample size planning simulation procedures and approximate formulas for two-wave repeated measures binary outcomes (i.e., two measurement times for the outcome variable, before and after intervention delivery). The simulation results agree well with the formulas. We also discuss how to use simulations to calculate power for studies with more than two outcome measurement occasions. Results show that having at least one repeated measurement of the outcome can substantially improve power under certain conditions.
Bruins S, Hottenga JJ, Neale MC
… +3 more, Pool R, Boomsma DI, Dolan CV
Multivariate Behav Res
· 2024 · PMID 37439516
·
Full text
One type of genotype-environment interaction occurs when genetic effects on a phenotype are moderated by an environment; or when environmental effects on a phenotype are moderated by genes. Here we outline these types of...One type of genotype-environment interaction occurs when genetic effects on a phenotype are moderated by an environment; or when environmental effects on a phenotype are moderated by genes. Here we outline these types of genotype-environment interaction models, and propose a test of genotype-environment interaction based on the classical twin design, which includes observed genetic variables (polygenic scores: PGSs) that account for part of the genetic variance of the phenotype. We introduce environment-by-PGS interaction and the results of a simulation study to address statistical power and parameter recovery. Next, we apply the model to empirical data on anxiety and negative affect in children. The power to detect environment-by-PGS interaction depends on the heritability of the phenotype, and the strength of the PGS. The simulation results indicate that under realistic conditions of sample size, heritability and strength of the interaction, the environment-by-PGS model is a viable approach to detect genotype-environment interaction. In 7-year-old children, we defined two PGS based on the largest genetic association studies for 2 traits that are genetically correlated to childhood anxiety and negative affect, namely major depression (MDD) and intelligence (IQ). We find that common environmental influences on negative affect are amplified for children with a lower IQ-PGS.
Advances in ability to comprehensively record individuals' digital lives and in AI modeling of those data facilitate new possibilities for describing, predicting, and generating a wide variety of behavioral processes. In...Advances in ability to comprehensively record individuals' digital lives and in AI modeling of those data facilitate new possibilities for describing, predicting, and generating a wide variety of behavioral processes. In this paper, we consider these advances from a person-specific perspective, including whether the pervasive concerns about of results might be productively reframed with respect to of models, and how self-supervision and new deep neural network architectures that facilitate transfer learning can be applied in a person-specific way to the super-intensive longitudinal data arriving in the Human Screenome Project. In developing the possibilities, we suggest Molenaar add a statement to the person-specific Manifesto - "In short, the concerns about commonly leveled at the person-specific paradigm are unfounded and can be fully and completely replaced with discussion and demonstrations of ."
With the increased use of time series data in human research, ranging from ecological momentary assessments to data passively obtained, researchers can explore dynamic processes more than ever before. An important questi...With the increased use of time series data in human research, ranging from ecological momentary assessments to data passively obtained, researchers can explore dynamic processes more than ever before. An important question researchers must ask themselves is, do I think all individuals have similar processes? If not, how different, and in what ways? Dr. Peter Molenaar's work set the foundation to answer these questions by providing insight into individual-level analysis for processes that are assumed to differ across individuals in at least some aspects. Currently, such assumptions do not have a clear taxonomy regarding the degree of homogeneity in the patterns of relations among variables and the corresponding parameter values. This paper provides the language with which researchers can discuss assumptions inherent in their analyses. We define strict homogeneity as the assumption that all individuals have an identical pattern of relations as well as parameter values; pattern homogeneity assumes the same pattern of relations but parameter values can differ; weak homogeneity assumes there are some (but not all) generalizable aspects of the process; and no homogeneity explicitly assumes no population-level similarities in dynamic processes across individuals. We demonstrate these assumptions with an empirical data set of daily emotions in couples.