Intelligence measures for online studies, from 16 items to a full normed battery

0 The short answer

For an online study that needs intelligence only as a covariate, a short reasoning measure with published reliability and no licence fee, such as the 16 item ICAR or the 20 item Hagen Matrices Test, is usually the defensible choice, while a normed battery is justified only when the study needs population referenced scores or a profile across several abilities. The ICAR paper reports a Cronbach alpha of 0.81 for its 16 item composite and the Hagen Matrices Test documentation reports a Kuder-Richardson reliability of 0.78, but neither reports population norms in the documents we opened. The options that do (the NIH Toolbox, Creyos, ACIS) cost more in minutes, money or examiner qualification. The decisive distinction is between a score that adjusts one analysis and a score that describes a person against a reference population.

1 Which Question Is the Intelligence Measure Answering?

The right measure depends on what the score will do in your analysis, and four jobs cover most of the reasons a team adds one to an online study. The first job is a covariate: the study is about attitudes, health behavior, decisions or language, and general cognitive ability is entered to adjust for a confound. The second is a screen or inclusion rule, where a score above or below a line decides who stays. The third is an outcome or profile, where cognition itself is the subject. The fourth is sample description, where the paper must say how the participants compare with a reference population. Each job asks a different thing of the score, and a measure that is excellent at one can be unusable at another.

Minutes are the budget that most designs ignore. Every minute a measure takes is paid for once per participant, so it scales with your sample. A 45 minute battery given to 1,000 people consumes 750 hours of participant time, and a 25 minute test consumes about 417 hours, our arithmetic from the durations. A short test is a different tool from a long one: it trades precision for the participant's time.

This page covers one decision: which measure to place in the study. It does not repeat other pages on neighboring questions. The page on how cognitive assessment platforms compare by use covers the software that delivers tasks, the page on the cost of testing other people prices recruitment and platforms, and who can administer an IQ test explains qualification levels. Screening out inattentive or duplicate responders belongs to the sibling page on online testing data quality, and writing the score up for readers belongs to reporting IQ in research. Clinical use, personnel selection and admission decisions are outside this page.

2 How This Page Compares the Options

Every row of the comparison uses the same six checks: how many items and minutes the measure takes, what construct it targets, what reliability its source reports, whether norms exist, what licence or fee applies, and what evidence exists from unsupervised online use. A figure appears only if it was in a paper, manual or vendor page that we opened, and prices carry the date we read them. Where a vendor page lists no price, the table says so and does not estimate one. Instruments we could not document from a source we opened are left out rather than described from memory.

Reliability deserves one piece of arithmetic that changes how the numbers read. A reliability coefficient converts to a standard error of measurement through the standard deviation of the score and the square root of one minus the reliability. If a score were placed on the familiar scale with a standard deviation of 15, a reliability of 0.81 would imply a standard error of about 6.5 points, a reliability of 0.78 about 7 points, and a reliability of 0.93 about 4 points. These are our own illustrations, not figures from any source, and they ignore the fact that the instruments do not report on that scale. The page on reliability and validity explains the underlying logic, and the page on the standard deviation of 15 explains the scale.

The same coefficient also limits a covariate. A measure with reliability 0.81 can correlate with a perfectly measured criterion at no more than the square root of 0.81, which is 0.90, and a measure with reliability 0.78 at no more than about 0.88, again our arithmetic. In an adjustment, that gap is the part of general ability the covariate does not remove. For an exploratory study the loss is small. For a study whose conclusion is that an effect survives control for cognitive ability, it is the size of the doubt.

Validity evidence travels less well than reliability. A measure validated in supervised students has shown a relation in that group, not in crowdsourced participants on phones, so each section flags which sources studied unsupervised online use. The page on what IQ measures and the page on the CHC model define the constructs (Gf, Gc, Gq, Gv, Gwm, Gs) used here.

3 The ICAR16 and ICAR60: Public Domain Items Built for Online Use

The International Cognitive Ability Resource is the best documented of the free options we found for unsupervised online use, and its own paper is also the most candid about the price of being free. Condon and Revelle (2014) describe it in Intelligence as a public-domain measure built because the field lacked one for large-scale, remote data collection. Their abstract names the cost openly: the lack of copyright protection poses a theoretical threat to test validity, and the size of that threat is unknown. The paper then tests it on an online sample. In the version of the paper posted on the project site, that sample is 96,958 people from 199 countries, each given 12 to 16 items from a 60 item pool through a matrix sampling design, untimed and unsupervised.

The 60 items cover four formats: letter and number series (9 items), matrix reasoning (11 items), verbal reasoning (16 items) and three dimensional rotation (24 items). The 16 item ICAR Sample Test, the ICAR16, takes four items from each format, chosen as a representative set by difficulty and factor loading. Reliability is reported for each format and for the composites.

Two features of the table matter for design. First, the 16 item composite is adequate rather than strong, and the authors themselves describe the alpha of 0.81 as adequate and the 60 item alpha of 0.93 as good. Second, the general factor saturation of the short form (omega hierarchical 0.66) is slightly higher than that of the long form (0.61), which means the short form is not a diluted version of the same signal, but a substantial share of its variance is still specific to its four formats. Matrix reasoning, the format closest to the classic Raven design, is the least reliable of the four, at 11 items.

Validity evidence in the paper is moderate and reported with its corrections. After correcting for the unreliability of self-reported scores, the ICAR16 correlated 0.59 with combined SAT scores and 0.52 with the ACT composite. In a university sample the ICAR16 was given alongside the Shipley-2, a brief commercial measure. The uncorrected correlations with the Shipley-2 composites were 0.41; corrected for restricted range they were 0.68; corrected for range and reliability they were 0.82 and 0.81. The authors note that these corrected values are comparable to the correlations the Shipley-2 itself shows with other ability measures. A researcher should read the uncorrected figure as what a student sample delivers in practice, and the corrected figures as an estimate of the relation in a broader population.

The measure has since been used outside its original site. In the Journal of Intelligence, Young and colleagues (2025) evaluated two ICAR formats (Puzzle Completion and Block Rotation) implemented in the Mobile Toolbox with 100 adults aged 18 to 82. They reported acceptable reliability, a correlation of 0.40 between Puzzle Completion and Raven's Progressive Matrices, 0.46 between Block Rotation and a mental rotation test, and non-significant practice effects. Those are formats from the ICAR family and not the ICAR16 composite, so they support the family rather than any one cut of it. A children's adaptation, the Ch-ICAR, was built for ages 11 to 14 because its authors wanted an alternative to proprietary Wechsler tests, which they say carry unfeasible administration times and financial costs.

On licence, read the project's own page. The ICAR project page at the Technical University of Dortmund, read on October 7, 2026, states that the ICAR is intended for academic use exclusively, lists a 60 item ICAR with a 12 item sample test, and points to an archive at the Leibniz Institute for Psychology. It does not name a licence type. "Public domain" in the paper and "academic use" on the project page are not identical statements, and a 12 item sample test is not the 16 item Sample Test of the 2014 paper, so confirm which version you are citing and which terms apply, especially for a commercial or non-academic study. The items are also public, which is the exposure risk the paper acknowledges. The page on the ICAR test and its limits covers interpretation for people who take it, including its norms. This page treats the ICAR as a research tool.

4 The Hagen Matrices Test: 20 Items, Free for Non-Commercial Research

The Hagen Matrices Test is a figural matrices test documented in an open access instrument database, and its documentation is unusually plain about where it should and should not be used. The entry in the GESIS collection of social science instruments, written by Heydasch and Schnaedter (2020) as an excerpt of Heydasch's 2014 dissertation material, describes a web-based test of induction and fluid reasoning in the CHC framework. It has 20 items, each a 3 by 3 incomplete matrix with a two minute limit per task, and exists in German, English and Spanish. The documentation gives the processing time as about 25 minutes, five for instructions and about 20 for the test, with a measured mean of 24.4 minutes and a standard deviation of 12.60.

The reliability figures are modest and stated without embellishment: a Kuder-Richardson reliability of 0.78 and a test-retest correlation of 0.75 over a mean interval of 78 days. The item analysis, on 1,339 participants, shows a mean difficulty of 0.37, with items ranging from 0.10 to 0.88, item-total correlations from 0.19 to 0.50, and a strong relation between position and difficulty. The first two items are very easy and seven are strongly difficult. The documentation reports convergent, divergent and criterion validity for the German version, with the highest correlations against the reasoning measures of the I-S-T 2000 R, a German intelligence test.

The documentation sets limits that many vendors would not print. It recommends the test for non-clinical adults aged 18 to 65, preferably with above average cognitive abilities, and warns that floor effects may arise, with frustration, in samples of lower cognitive ability. It was validated in samples aged 17 to 57. It should be used for group comparisons and correlative studies with sample sizes above 50, and it is not for individual diagnostics. The quality criteria were checked only for the German version, so an English or Spanish administration carries translation risk that the documentation does not resolve. The page on how IQ scores are normed explains why a score without a reference population cannot be read as an IQ.

On cost, the documentation states that the aim was a web-based, economical, valid alternative that is free of cost. The excerpt carries a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 licence, and the test itself can be requested by email from the author for non-commercial research. That is a request process, not a download, so build a few days into the schedule and keep the permission on file. Compared with the ICAR16, the Hagen test is longer, timed per item, documented as validated in one language, and recommended for samples with above average ability. In exchange it measures one construct, induction, and it documents a retest correlation of 0.75 over about eleven weeks, our conversion of 78 days.

5 Short Raven Forms: What the Nine Item Versions Do and Do Not Establish

The nine item Raven forms predict the full 60 item score closely, but predicting a total is a narrower claim than measuring intelligence, and the forms carry no norms of their own. Bilker and colleagues (2012), in Assessment, started from the 60 item Raven's Standard Progressive Matrices, which their abstract describes as a nonverbal estimate of fluid intelligence often used in clinical batteries and in research on patients with cognitive deficits. They used a Poisson predictive model to find a small subset of items that predicts the total score, and a second non-overlapping subset for repeated administrations.

Using nine items as predictors, the correlations with the full score were 0.9836 and 0.9782 in the fitting data for the two forms, and 0.9063 and 0.8978 in the validation data. The authors conclude that nine items predict the 60 item total with good accuracy, and that the two forms save 75 percent of administration time compared with a published 30 item form while achieving similar item and test level characteristics. The gap between the fitting and validation figures is the reason to quote the lower pair. A correlation of about 0.90 with the total means roughly 81 percent shared variance, our arithmetic from the correlation.

Three limits apply. First, the target of the paper is the 60 item total, so the forms inherit whatever the full test measures and whatever norms the full test has; they do not have their own. Second, the context in the abstract is clinical research, and the paper does not show how the forms behave in unsupervised crowdsourced samples. Third, the items belong to a published test, and the abstract says nothing about licence, so permission from the test's publisher is a question to settle before use. We did not verify the publisher's terms and this page does not state them. The page on Raven's 2 describes the current Raven product, and the page on fluid intelligence tests explains what matrix tasks measure and where they stop.

For a researcher who wants a matrices task online, the practical comparison is with the other short matrix instruments. The ICAR includes matrix reasoning items whose stimuli the paper describes as similar to those of the Raven, with an alpha of 0.68 on 11 items. The Hagen test is entirely matrices, with 20 items and an alpha of 0.78. A nine item Raven form is shorter than either and has the strongest published link to a long, well known test, but in the abstract we read it has no evidence from unsupervised online samples.

6 Vocabulary Measures: Wordsum and the Shipley-2

Vocabulary tests target crystallized ability (Gc), which makes them a different covariate from matrices, and their documentation shows how much item selection matters in a ten item test. The Wordsum is the ten item vocabulary test of the General Social Survey. Cor, Haertel, Krosnick and Malhotra (2012) analyzed it with item response theory and found that its items are very difficult or very easy, with the moderately difficult ones missing. Adding four moderately difficult items to make a 14 item battery, which they called Wordsumplus, outperformed the original on classical test theory indicators, reduced the standard error of ability estimates in the middle of the range, and showed higher concurrent validity, using national samples of thousands of American adults. Their recommendation is to use all 14 items.

The lesson for a researcher is that a short test can be reliable for the wrong part of the range. A ten item vocabulary test made mostly of very easy and very hard items separates the top and bottom of a sample well and the middle poorly, and most online samples live in the middle. It also measures acquired knowledge of word meaning, so it is sensitive to language background and schooling in a way that a matrix task is designed not to be. The page on verbal IQ tests covers what verbal measures capture.

The Shipley-2 is the commercial counterpart. The publisher listing at PAR, read on October 7, 2026, gives an administration time of 20 to 25 minutes, ages 7 to 89, three scales (Vocabulary for crystallized knowledge, Abstraction for fluid reasoning, and Block Patterns as a fluid alternative), Qualification Level B and a complete kit price of 293 dollars. It describes the test as a way to obtain quick ability estimates, screen for cognitive dysfunction or qualify participants for research studies, with normative data on 2,826 individuals and standard scores and percentiles. In the ICAR paper, university students completed it in 15 to 25 minutes, and Condon and Revelle cite its manual for correlations of 0.86 and 0.85 between its composites and the Full Scale IQ of the Wechsler Adult Intelligence Scale.

What the listing does not settle is delivery. A physical kit with a qualification level is built for supervised use, and the page we opened gave no terms for presenting its items in an unsupervised web survey. Before planning around it, ask the publisher whether and how its items may be delivered online. Its listing pairs published norms with a fee and a qualification level, which is a useful reminder that norms usually arrive together with both.

7 The Wonderlic and the Cognitive Reflection Test: Built for Other Jobs

The Wonderlic was built to screen job applicants and the Cognitive Reflection Test was built to study decision making, so neither is a drop-in intelligence covariate, and each needs a specific reason to be chosen. The Wonderlic Select page, read on October 7, 2026, states that its cognitive ability test consists of 50 questions answered in 12 minutes, and it describes the test as a way for employers to see whether applicants can learn from experience, solve problems and comprehend complex ideas, designed to predict job performance. The page lists no price and directs visitors to request a demo. That is 720 seconds for 50 questions, an average of 14.4 seconds each, our arithmetic, so the format rewards speed as much as depth. Its research use is not addressed on the page we opened, and a research licence is a question for the vendor.

The Wonderlic does appear in the research literature as a comparison point. Condon and Revelle report, citing the Shipley-2 manual, correlations of 0.64 and 0.60 between the two Shipley-2 composites and the Wonderlic Personnel Test, lower than the 0.86 and 0.85 the same composites show with the WAIS Full Scale IQ. A researcher who wants a quick score for a study would not obtain norms, a documented research licence and a published reliability from the vendor page alone. The page on how a Wonderlic score converts to IQ explains the conversion problem, and the page on pre-employment cognitive tests covers the hiring use that this page does not.

The Cognitive Reflection Test is a different animal. Frederick (2005), in the Journal of Economic Perspectives, introduced it as a three item test of cognitive reflection and showed that scores predicted patterns of decision making under risk and time. Toplak, West and Stanovich (2011) tested it against measures of cognitive ability, thinking dispositions and executive functioning in Memory & Cognition. They report that the CRT has a substantial correlation with cognitive ability and yet was a unique predictor of performance on heuristics-and-biases tasks, accounting for additional variance after the other individual difference measures were controlled. Their conclusion is that it measures a tendency toward miserly processing that intelligence tests do not capture.

That is a reason to include the CRT when your hypothesis concerns reflective thinking, and a reason not to use it as a stand-in for general ability. A three item test also has little room for reliability: three items give a score from 0 to 3, which cannot separate many participants. Public exposure is the other problem for a test this short, because online participants may have met it before. The page on the Cognitive Reflection Test documents that exposure for crowdsourced samples and walks through the test, so this page does not repeat it. If the aim is simply to control for ability, a matrix or series measure is the better covariate, and the CRT can sit beside it as a separate construct.

8 The NIH Toolbox and Creyos: Normed Batteries and What They Cost in Setup

The NIH Toolbox and Creyos are two of the options in this comparison that come with norms, and both ask for more setup than a free test: an examiner and a tablet for the first, a vendor quotation for the second. The NIH Toolbox Cognition Battery was described in Weintraub and colleagues (2013) in Neurology. Seven measures were designed to tap executive function, episodic memory, language, processing speed, working memory and attention. They were validated in English in 476 participants aged 3 to 85, with results on test-retest reliability, age effects and convergent and discriminant validity. The authors describe it as a brief, convenient set of measures to supplement other outcome measures in epidemiologic and longitudinal research and clinical trials, and say its computerized format and national standardization are meant to give researchers a common currency across studies.

Access is the constraint. The NIH Toolbox site, read on October 7, 2026, lists annual subscription prices for the version 3 app of 599.99 dollars for one or two devices, 1,499.99 dollars for three to six, and 2,499.99 dollars for seven to ten, for a full year of unlimited administrations. It states that administration is on an iPad and that the cognition tests require C-level qualifications or supervision by someone who holds them. Those are in-person conditions, and they explain why the Toolbox is a tool for studies where participants come to a site, not for a link sent to a crowdsourced panel. The page on who can administer an IQ test explains the qualification levels. The remote sibling is the Mobile Toolbox, in which the ICAR formats described earlier were implemented for remote self-administration, as Young and colleagues report.

Creyos, whose earlier name was Cambridge Brain Sciences, is a web platform. Its research page, read on October 7, 2026, states that the full assessment has 12 cognitive tasks and takes 35 to 45 minutes, that participants complete it online, that it draws on a normative database of more than 85,000 participants, and that its tasks have been used in more than 400 peer reviewed studies. It claims results comparable to traditional two to three hour pen and paper tests. These are the vendor's statements, and we did not verify them against the studies. The page lists no price and does not say what formats the data export takes. The page on the Creyos review examines the product in more detail.

Creyos offers twelve tasks and a vendor norm base, which is more than any free measure offers and less than a transparent technical document would. For a study that needs a multi-domain profile and cannot run a supervised session, it is a legitimate option to request a quotation for. Ask for the technical report, the norm sample description, the reliability of each task, the export format and whether participant-level data can be exported. Those questions apply to every vendor on this page, including ours.

9 Gorilla and Pavlovia: Task Libraries Are Delivery, Not Measures

Gorilla and Pavlovia deliver whatever task you build or import, so choosing them answers where the task runs and says nothing about whether the task measures intelligence. Anwyl-Irvine and colleagues (2020) presented Gorilla in Behavior Research Methods and demonstrated it with a simplified flanker task, replicating the conflict network effect in primary school children and adults, supervised and unsupervised, on participants' own computers and on computers supplied by the researcher, and over home internet and mobile connections. That shows reaction-time experiments can run reliably on the platform. It does not show that any intelligence task built on it is valid.

Bridges and colleagues (2020) compared timing across experiment software in PeerJ. Online studies did not deliver the same precision as lab systems, with slightly more variability in all measurements, though PsychoPy and Gorilla, broadly the best performers, achieved close to millisecond precision on a number of browser and system configurations. Timing precision matters for speeded and reaction time tasks and much less for untimed matrix or series items. Device matters too. Passell and colleagues (2021) examined 59,587 and 3,818 visitors to the TestMyBrain platform and found that users of mobile devices, particularly Android smartphones, showed slower measured reaction time than laptop and desktop users, after controlling for age, gender, education and an untimed vocabulary test. Their abstract does not claim that untimed reasoning tasks are immune, and neither does this page. Processing speed (Gs) tasks are the ones to watch, and the page on processing speed explains why.

Prices differ in kind. The Gorilla pricing page, read on October 7, 2026, shows no plan prices. It invites visitors to book a call and notes that prices exclude value added tax and that dollar prices are in United States dollars. The PsychoPy pricing page, read on October 7, 2026, states that PsychoPy is free, that Pavlovia credits cost 0.26 British pounds per participant for individual researchers without a licence, and that annual university and charity licences run 2,000 pounds for licence-only access, 2,200 for the standard tier and 5,200 for the plus tier. At the per-participant credit, 200 participants cost 52 pounds and 1,000 cost 260 pounds, our arithmetic and per credit consumed.

The page on the cost of testing other people adds recruitment fees, and the platform comparison page covers contracts and data terms. A library lowers the cost of delivery to almost nothing, and in exchange the validation, the scoring, the norms and the item security all become your responsibility. If you build a matrices task from scratch, you have a new, unvalidated instrument.

10 Side by Side: Items, Minutes, Reliability, Licence and Cost

Set in one table, the options sort into three groups: free reasoning tests that are short and unnormed, commercial scales with norms and fees, and platforms that sell delivery or batteries. The first table lists what each source documents. A cell that says not stated means the source we opened was silent, and silence is not evidence of absence.

Cost has two parts that the table cannot show together: the fee for the instrument and the participant time that every minute consumes. The next table puts the participant time at three sample sizes, our arithmetic from the stated durations.

The fees follow the published prices, again our arithmetic. The free tests cost nothing in licence fees at any sample size, though the Hagen test needs a permission request and the ICAR needs a terms check. Pavlovia credits at 0.26 British pounds come to 13 pounds for 50 participants, 52 for 200 and 260 for 1,000 per task run. The NIH Toolbox subscription is 599.99 dollars for the year for up to two iPads, but the examiner time for an in-person session is the real limit. Where a team bought ACIS at the consumer one time prices read on October 6, 2026, Quick at 15 dollars would total 750, 3,000 and 15,000 dollars for those three sample sizes, Optimized at 30 dollars would total 1,500, 6,000 and 30,000, and Full Scale at 50 dollars would total 2,500, 10,000 and 50,000. The ACIS research page states what a research team pays, and a buyer should read it before relying on the consumer arithmetic. Recruitment and participant payment sit on top of all of these, and the page on the cost of testing other people prices them with platform fees.

11 Which Measure for Which Design, and What a Short Measure Cannot Do

Choose by the job the score does, then by minutes, then by licence, and accept that a short measure answers a narrower question than its label suggests. For a covariate in an unsupervised study, start with a short reasoning test that has published reliability: the ICAR16 if the academic terms fit, the Hagen test if one language and an above average sample fit, and a nine item Raven form only if permission from the publisher is in hand. Add a vocabulary measure when education or language background is part of the confound, because matrices and vocabulary tap different broad abilities (Gf and Gc). If your participants come to a site and an examiner is available, the NIH Toolbox and a supervised brief scale such as the Shipley-2 become possible, at the cost of qualification and equipment.

For a multi-domain profile, a short free test is the wrong tool. The profile has to separate abilities such as fluid reasoning, working memory and processing speed, and that needs several tasks per domain. Creyos and ACIS are the two options here built for it online, and both take 35 minutes or more. For a screen with a cut score, prefer an instrument whose source describes a reference group, and read the interval around the score near the line. For sample description, report the raw score and the instrument by name and avoid IQ language unless a normed scale exists. The sibling page on reporting IQ in research covers the wording.

A short measure cannot do five things. It cannot classify an individual: the Hagen documentation says it is for group comparisons and not individual diagnostics. It cannot make a controlled-for-intelligence claim airtight, because a covariate with reliability near 0.80 leaves part of general ability uncontrolled. It cannot be compared across instruments: a raw sum on the ICAR16 and a raw sum on the Hagen test are on different scales. It cannot be trusted in a new language because evidence travels poorly. And it cannot guarantee unsupervised honesty. The ICAR paper states plainly that in unsupervised online testing there are no safeguards against external resources, including those available on the internet. Design answers to that, such as time limits, response screening and attention checks, belong to the sibling page on online testing data quality.

Adaptive designs can shorten a test by choosing items from earlier answers, and the sibling page on adaptive IQ tests explains the idea. A supervised short Wechsler form belongs to a study with a qualified examiner, and the sibling page on the WAIS-5 subtests describes its formats. Finally, preregister the measure: name the instrument, version, scoring rule and exclusions before the data arrive, so the choice cannot follow the result.

12 Where ACIS Sits, and What a Buyer Should Do

ACIS sells one of the options on this page, and for most online studies that need a covariate it is not the best fit, which is why the comparison above recommends shorter free measures first. ACIS is a self-administered online assessment of 20 subtests in six CHC domains, reported as a Full Scale IQ and six primary indices: verbal comprehension, fluid reasoning, quantitative reasoning, visual spatial, working memory and processing speed. Read on October 6, 2026 on the home page, it has three forms with one time prices and no subscription. Quick costs 15 dollars, covers 6 subtests in 3 domains and takes about 45 minutes. Optimized costs 30 dollars, covers 13 subtests in 5 domains and takes about 110 minutes. Full Scale costs 50 dollars, covers all 20 subtests in six domains and takes about 175 minutes. Breaks are allowed, a free trial needs no card, and the offer includes a 5 day quality guarantee and 30 days to complete. Prices can change.

The score report gives indices and a Full Scale IQ on the standard scale with a mean of 100 and a standard deviation of 15, scaled subtest scores from 1 to 19, percentiles and a 95 percent confidence interval. Adult norms cover ages 16 to 90. ACIS Professional is the channel for a research or organizational team to administer ACIS to its own participants, and the research page and the Professional page describe how that works and what it does not do. The evidence behind the scores is in the technical manual, and this page does not quote it. A reader who wants the evidence should read it there and judge it by the same checks used for every instrument above.

The limits are as plain as the strengths. ACIS is online and unsupervised. It is not a clinical or diagnostic instrument, and it is not for hiring, school accommodations or admission to high IQ societies. It is available in English only. At 45 minutes for its shortest form, it asks participants for more time than the Hagen test and for a fee that the ICAR and Hagen tests do not charge. It fits a study that needs a profile across several domains, scores on a standard scale with an interval, and a form a participant can complete at home within 30 days. It does not fit a design that needs a few minutes of reasoning as a covariate.

Before buying any measure, a research team should do six things:

1. Ask the vendor or author for the technical documentation and read what the reliability and norms are based on.
2. Confirm in writing that research use and online delivery are permitted.
3. Pilot the measure with a small sample, check the score distribution for floor and ceiling effects, and compute reliability in your own data.
4. Decide in advance how scores will be reported, using the guidance in the sibling page on reporting IQ in research.
5. Budget participant minutes as a cost, not an afterthought.
6. Preregister the instrument, version and exclusions.

These steps follow the logic of the Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014), the joint publication of the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education, which are open access at the standards site. Their central idea, as we read it, is that validity belongs to a proposed interpretation and use of scores, not to a test name, so the evidence has to match the use. The American Psychological Association's Guidelines for Psychological Assessment and Evaluation, approved in March 2020, point the same way. Guideline 8 asks for test selection, scoring and administration to reflect the appropriate normative comparison, and notes that the age of the norms and the continued relevance of the construct matter. Guideline 15 states that the essential criteria for evaluating technology enhanced measures are reliability, validity and fairness, and encourages review of the validation evidence when legacy tests are adapted for electronic presentation. Those guidelines are written for psychologists, but a research team choosing an instrument faces the same questions.

13 Sources Behind This Page

Every figure on this page comes from one of the sources below, opened by us. Papers are identified by DOI, and vendor and project pages carry the date we read them. ACIS facts come from the ACIS home page as read on October 6, 2026.

• Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., and Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1), 388-407. doi.org/10.3758/s13428-019-01237-x
• American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. standardsforedtesting.org
• American Psychological Association (2020). APA Guidelines for Psychological Assessment and Evaluation, approved by the APA Council of Representatives, March 2020. apa.org
• Bilker, W. B., Hansen, J. A., Brensinger, C. M., Richard, J., Gur, R. E., and Gur, R. C. (2012). Development of abbreviated nine-item forms of the Raven's standard progressive matrices test. Assessment, 19(3), 354-369. doi.org/10.1177/1073191112446655
• Bridges, D., Pitiot, A., MacAskill, M. R., and Peirce, J. W. (2020). The timing mega-study: comparing a range of experiment generators, both lab-based and online. PeerJ, 8, e9414. doi.org/10.7717/peerj.9414
• Condon, D. M., and Revelle, W. (2014). The international cognitive ability resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52-64. doi.org/10.1016/j.intell.2014.01.004. Figures were read in the version posted on the ICAR project site.
• Cor, M. K., Haertel, E., Krosnick, J. A., and Malhotra, N. (2012). Improving ability measurement in surveys by following the principles of IRT: The Wordsum vocabulary test in the General Social Survey. Social Science Research, 41(5), 1003-1016. doi.org/10.1016/j.ssresearch.2012.05.007
• Dutry, M., Vereeck, A., Duyck, W., Derous, E., Schelfhout, S., Szmalec, A., Woumans, E., Schittekatte, M., Debeer, D., and Dirix, N. (2025). Validation of the Children's International Cognitive Ability Resource (Ch-ICAR). Behavior Research Methods, 57(2), article 66. doi.org/10.3758/s13428-024-02591-1
• Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25-42. doi.org/10.1257/089533005775196732
• Heydasch, T., and Schnaedter, S. (2020). The Hagen Matrices Test (HMT). Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS), GESIS. access.gesis.org/zis/655
• Passell, E., Strong, R. W., Rutter, L. A., Kim, H., Scheuer, L., Martini, P., Grinspoon, L., and Germine, L. (2021). Cognitive test scores vary with choice of personal digital device. Behavior Research Methods, 53(6), 2544-2557. doi.org/10.3758/s13428-021-01597-3
• Toplak, M. E., West, R. F., and Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275-1289. doi.org/10.3758/s13421-011-0104-1
• Weintraub, S., Dikmen, S. S., Heaton, R. K., Tulsky, D. S., Zelazo, P. D., Bauer, P. J., and others (2013). Cognition assessment using the NIH Toolbox. Neurology, 80(11, Supplement 3), S54-S64. doi.org/10.1212/wnl.0b013e3182872ded
• Young, S. R., Kim, J., McKee, K., Doyle, D. R., Novack, M. A., Revelle, W., Gershon, R., and Dworak, E. M. (2025). Validation of International Cognitive Ability Resource (ICAR) implemented in Mobile Toolbox (MTB). Journal of Intelligence, 13(12), 154. doi.org/10.3390/jintelligence13120154
• Vendor and project pages, each read on October 7, 2026: PAR Shipley-2 listing; Wonderlic Select cognitive ability test; NIH Toolbox, Get the Toolbox; Creyos Research; Gorilla pricing; PsychoPy pricing, including Pavlovia credits; ICAR project page, Technical University of Dortmund.

14 Frequently Asked Questions