How did I create a tool for AI Literacy Assessment using AI
Have you ever wondered how much you really know about Artificial Intelligence (AI)?- the story behind the AI-IQ Assessment Tool
As an educational technology practitioner, I have always been passionate about leveraging technology to enhance learning experiences. While working on an AI and Learning project at the United Nations Institute for Training and Research, I realized that understanding artificial intelligence (AI) can be challenging for many, particularly those without a technology or computer science background. I have also noticed that people often overestimate or underestimate their knowledge of AI. I began to ponder:
What if there was a simple way for people to find out how much they know about AI?
The restlessness around this question led me to create a user-friendly online tool to help people measure their AI knowledge.
The foundation
My goal was simple: create an easy-to-use assessment tool for anyone curious about AI, regardless of technical background. The assessment should be evidence-based and aimed at non-experts in AI literacy.
The research behind the tool
My journey began with extensive research into AI literacy. I found several academic studies exploring ways to measure AI literacy.
These studies offered valuable insights, but many of them relied on self-reported knowledge. This can be tricky, as someone might overestimate or underestimate their understanding.
One study focused on measuring AI literacy in three areas: technical understanding, critical thinking, and practical use. However, it relied on individual’s self-reported knowledge, which is subjective and based on personal judgment. An example of a question in this study is, “I understand the differences between general AI and narrow AI.” Another study combined AI knowledge with broader psychological skills. Yet, it still relied on self-perceived judgments.
To address this, I looked for a method that went beyond self-assessment. It was during this time that I came across the Scale for Measuring AI Literacy among General Adult Populations (SAIL4ALL) and the objective-based assessment for non-technical individuals. Those works provided a comprehensive framework for assessing AI knowledge, and I was deeply inspired by its potential to educate and empower individuals. The questions drawn and addressed both used an objective assessment format (true/false questions). It also offers broad coverage of AI knowledge, capabilities, workings, and ethical considerations. Example Item: “AI cannot solve problems the way humans can.”
Note: This type of questionnaire still has flaws, such as surface-level assessment, guessing, ambiguity, etc. However, this approach seemed the most suitable for the assessment tool I wanted to build.
Drawing from those questionnaires, I envisioned creating a user-friendly tool that would allow anyone to assess their AI literacy easily. The goal was to provide immediate insights into their understanding of AI and offer guidance on improving their knowledge.
Getting things done (with AI)
Believe it or not, AI itself played a role in creating this AI assessment tool. I used a large language model (i.e. ChatGPT 4o) to brainstorm ideas and collaborate on the coding process. This collaboration helped refine the format and ensure the tool functioned smoothly.
In the following example, we are brainstorming the most appropriate format.
Based on the brainstorming, I decided on a quiz format that would give results in the form of levels (e.g. Foundation, Intermediate, and Advanced) based on the test score. At the end of the test, you will get a PDF with your results and some insights into your AI knowledge.
The following is an example of my “coding partner” providing me with the code according to my specifications. This process required many iterations, the code became large, and I used some of my tech background to make sense of the iterations.
AI also supported me with other related tasks, such as calculating sitting time:
The outcome
The entire development process took around 40 hours, but the satisfaction of creating a resource for everyone is truly rewarding. The tool is completely free and hosted using free hosting platforms (Github, Netlify, Softr).
Creating this tool was a labor of love. I wanted it to be accessible to everyone, regardless of their background or prior knowledge of AI. The tool is also available in Brazilian Portuguese. By taking the test, users can quickly determine their AI literacy level and receive a detailed report describing their proficiency. This report also highlights areas for improvement, guiding users on their journey to becoming more AI-literate.
Want to know your AI knowledge level? Try the AI-IQ Assessment Tool and see where you stand.
See you! 👋