It’s Scary That AI Might Give You Bad Data. It’s Terrifying That You Might Believe It.

If you ask an LLM your labor market questions, it will answer you confidently—but that confident answer is likely to include outdated, inaccurate, or misleading information. 

There’s a chance you might get an accurate, up-to-date answer. But you might also get a salary benchmark from 2024 when you asked about this year, or it gave you jobs open today when you asked for a six-month average. A smaller risk, but a more costly one, would be that the AI completely hallucinates its response. 

For organizations using AI to inform hiring budgets, training programs or investment decisions, a well-written answer can cause more harm than good, potentially concealing a mismatch between what was asked and what was measured. When your organization is relying on you for trustworthy numbers, the risk of bad information is scarier than any horror movie. 

AI isn’t the villain here. There’s no boogeyman we need to hide from. Instead we should recognize AI’s limitations in finding reliable workforce intelligence so we understand how to avoid those traps, and then use the right tools to find the data we need. 

The Ghosts Haunting the Training Data

When you use an AI large language model (LLM) or assistant, it uses its training data to formulate a response—data that has been frozen at some point in the past. That’s fine if you’re researching something well-established, the history of Halloween, but the labor market is always changing. Employers revise hiring plans, skills requirements change, and yesterday’s salary expectations may not survive today’s recruiting process. For example, the fastest growing AI job in 2026 had a total of only 174 postings for it in 2024.

To understand how well AI assistants answer questions about labor market intelligence, Lightcast tested the three leading models. Without access to any external tools (restricted only to their training data) the AIs scored an average of 38/100 on our tests. 

But you might be thinking: AI assistants can also search the web, can’t they? And sure, web search can help close the gap, but it can also retrieve old reports, undated statistics, and figures that answer a slightly different question. A fresh search does not necessarily produce fresh evidence. When given access to web search, AI scores went from 38 up to 57. 

In the haunted house of AI-generated labor market intelligence, the answers look alive…but the data behind them might be long dead.

Curse Of The Phantom Numbers

When asking for a specific figure, the AI models did even worse on our tests, providing an answer just 35% of the time. Instead, many of the responses in our test sounded authoritative but didn’t actually answer the question. 

• It may describe demand as "strong" or "growing" without quantifying it.

• When it can’t find the answer requested, it may offer different numbers as a kind of substitute, instead of admitting there’s a gap. 

• Even when it produces specific figures, they may measure something different.

In our test, we asked AI how many postings for the occupation “Data Analyst” had been posted in the US over the past three months. Each model returned a different answer: one said the data wasn’t available, another said there were 11,600 on one specific job board, and the third said there were 450,000 postings…in 2023. 

Maybe one of those answers is correct—but there’s no way all three are. In order to find out, you’d have to do the research yourself, and you’d be better off avoiding AI entirely. 

Thankfully—like sunrise in a vampire movie—there’s help on the horizon.

AI Doesn’t Have To Be Scary

The answer isn’t to stop using AI. It’s to give it access to data that is actually built to answer your workforce questions.

That’s why Lightcast built the Model Context Protocol (MCP): a way for compatible AI assistants to connect directly to Lightcast labor market intelligence. Instead of relying solely on what an AI model remembers or whatever a web search happens to surface, the assistant can query structured labor market data with consistent definitions, specific time periods, and identifiable sources.

The research we cited above, testing the AI models, comes from our new whitepaper that compares AI with our MCP to AI alone. When we added Lightcast data to AI with web search, average answer quality jumped from 57/100 to 82, and 87% of the answers improved.

When we asked AI for a specific number, it gave us one 92% of the time, compared with just 35% with web search alone.

Whether or not your AI is connected to Lightcast MCP, here are 4 important questions to ask when using AI for your workforce questions. 

1. Did it give the number you asked for? 

2. Does the answer’s time period match your question?

3. Can it tell you the assumptions, data used and methodology to arrive to its conclusion? 

4. Do you trust these sources and agree with the assumptions?

If the answer to any of these is “no,” "not really," or even “I don’t know,” now you have to do the hard part yourself, and the AI hasn’t saved you any time. 

Access to the Lightcast MCP doesn’t make AI perfect—there’s no silver bullet (or garlic, or wooden stake) that can do that—but it does give you a reliable foundation on which to base your decisions, because you know where your AI’s data comes from. 

Read the full report, When AI Meets Lightcast Data, for the findings and methodology, and explore how the Lightcast MCP connects your AI tools to trusted labor market intelligence.