Bayesian Item Response Modeling in PyMC (Alva Labs)

Event Description In this panel discussion, we discuss IRT (Item Response Theory), GRM (Graded Response Model) and the advantages to using the Bayesian approach at Alva Labs. Item response theory, also known as the latent response theory, refers to a family of mathematical models that attempt to explain the relationship between latent traits (unobservable characteristic or attribute) and their manifestations (i.e. observed outcomes, responses or performance). Graded response model (or Ordered Categorical Responses Model) is a family of mathematical models for grading responses. #BayesianIRT #ItemResponseModeling #psychometrics #BayesianStatistics #LatentTraitModel #TestTheory #mcmc #ModelFit #MeasurementAccuracy #RaschModel #PersonItemMap #TestEquating #ConvergenceDiagnostics #EducationalMeasurement #IRTResearch #statisticalmodeling ## Timestamps #2PLModel #3PLModel #ICC 00:00 Introduction to PyMC Labs 02:48 Panelists introduction 06:05 Outline of the talk by Morgan 06:51 Alva Labs 08:33 Alva Labs personality test 12:14 Item Response Theory and its advantages 16:12 Question: Won't people fake answers to the personality questionnaire if they know what the company is looking for? 18:09 Question: What algorithm is used for combining data points? 19:36 Graded Response Model 20:50 Bayesian Inference 23:19 ALva Labs workflow 25:08 Is the person trait supposed to be a measure of performance and how is it quantified for training?On which data is the model trained? 27:37 Emerging challenges over the years 30:45 How PyMC helped Alva Labs improve their personality model 32:33 Understanding the problem at Alva Labs 34:34 Bayesian workflow 35:32 Simulate the data generating process 38:12 Develop the model 40:25 Evaluate alternative parameterizations 43:40 Test different inference engines 46:05 Use the mode4 with real data 47:52 The final deliverable 49:48 Results of the new model compared to the original Alva Labs model 51:12 Question: Could you comment on how much faster the sampler becomes and why do you care about memory? 53:40 Question: How is the trained model validated and how do you know the person trait is useful and how is the usefulness measured? 55:02 How do you often rerun the model to update the parameters 56:02 Thank you! Alva Labs (https://www.alvalabs.io/about) uses psychometric tests and machine learning for candidate assessments. ## About the speakers Morgan Pihl Morgan is Head of R&D at the Swedish startup Alva Labs, a recruitment software provider. He has used PyMC and NumPyro for building best-in-class psychometric assessments that have so far been completed by over 300k users. He did his MSc in psychology at Umeå University. LinkedIn: https://www.linkedin.com/in/morganpihl Tomás Capretto Tomás is a statistician and data scientist at PyMC Labs who loves solving complex problems. He is one of the main developers of Bambi, the Python library that makes Bayesian linear models accessible for all and is built on top of PyMC. He is a part-time doctorate student in Statistics. Website: https://tomicapretto.github.io Thomas Wiecki Dr. Thomas Wiecki is an author of PyMC, the leading platform for statistical data science. To help businesses solve some of their trickiest data science problems, he assembled a world class team of Bayesian modelers founded PyMC Labs -- the Bayesian consultancy. He did his PhD at Brown University studying cognitive neuroscience. Twitter: https://twitter.com/twiecki