Question 1

Can a well-designed prompt outperform OpenAI's o1 model?

Accepted Answer

_Human-written prompts can still beat o1's automated reasoning processes._

Yes. A well-designed prompt, meaning the human writes the instruction, can, so far, outpace OpenAI's o1. This is noteworthy because o1 is designed specifically for reasoning through an iterative, self-calling process with dedicated steps for iteration, reflection, and feedback loops. The fact that careful human-written prompts can still exceed its performance suggests that the model's reasoning capabilities, while advanced, have not yet surpassed what skilled human prompt engineering can achieve.

— [Sam Altman Says, 'We Know How To Build AGI' — Not So Fast](https://www.forbes.com/sites/lutzfinger/2025/01/07/sam-altman---agi--ai-workforce-in-2025-the-battle-of-tech-giants/) · _Forbes_

Question 2

Can we conclude that trust causes higher vaccination rates?

Accepted Answer

_Correlation found, but causation works both ways with trust._

While we have shown the impact of trust on covid vaccination, correlations do not indicate causation. Trust is a two-way street. If governments do an excellent job vaccinating their population, it might alter people's overall trust in their institutions. Appropriate, transparent responses will improve the ability of countries to deal with the effects of the COVID pandemic in a virtuous cycle, suggesting the relationship flows both ways.

— [Get Vaccinated? How Trust In Institutions Determines COVID Vaccination Rates In The EU](https://www.forbes.com/sites/lutzfinger/2021/05/06/get-vaccinated-how-trust-into-the-institutions-determines-vaccination-rates-in-the-eu/) · _Forbes_

Question 3

Did you find other factors that influenced vaccination rates besides trust?

Accepted Answer

_Other factors tested but trust stood out statistically._

There are other factors besides trust that might influence vaccination rates, including accessibility and vulnerability. We analyzed the old-age dependency ratio and health care expenditure per inhabitant but did not find statistically significant dependencies. This suggests that while these factors could theoretically matter, in our analysis trust emerged as the key differentiator when controlling for these other variables across EU member states.

— [Get Vaccinated? How Trust In Institutions Determines COVID Vaccination Rates In The EU](https://www.forbes.com/sites/lutzfinger/2021/05/06/get-vaccinated-how-trust-into-the-institutions-determines-vaccination-rates-in-the-eu/) · _Forbes_

Question 4

How are companies currently using AI tools to measure employee contributions?

Accepted Answer

_AI meeting tools are reshaping how companies evaluate employee performance._

Companies are using AI tools like Granola, Read.ai, Fireflies, and Avoma to record and transcribe meetings, tracking metrics like talk time and participation. Firms like Amazon and Meta are folding these AI-surfaced contributions into how they review people. The idea is that when the transcript becomes the record, ideas can propagate on merit rather than by who has the most power or the loudest voice. This shift means contribution gets measured by what you actually said, not by your title or office.

— [If Your Name Isn't Western, AI Could Cost You The Promotion.](https://www.forbes.com/sites/lutzfinger/2026/05/25/if-your-name-isnt-western-ai-could-cost-you-the-promotion/) · _Forbes_

Question 5

How did Varo use AI to reduce operational costs while improving customer service?

Accepted Answer

_An AI chatbot cut costs 60% while handling most inquiries._

We started by providing voice support for everybody, but it got very expensive. We moved to live person chat, then built an AI chatbot. That chatbot now contains over 55% of our inbound inquiries and allowed us to reduce operational costs by about 60%. This is part of solving the high-cost problem that made serving underserved populations uneconomical for traditional banks. The AI handles routine questions efficiently while maintaining service quality for customers who value access and responsiveness.

— [Disrupting Traditional Banking, with Colin Walsh](https://www.lutzfinger.com/transcript/2025-05-22-disrupting-traditional-banking-with-colin-walsh/) · _The Keynote on AI_

Question 6

How did you isolate trust as a factor in COVID vaccination rates?

Accepted Answer

_Using the EU to study trust's role in vaccination._

We looked towards the EU because the distribution of vaccines to Member States happened simultaneously and under the same conditions. But not all member states show the same trust, allowing us to isolate trust as a variable. We used the European Social Survey, a bi-annually collected survey of thousands of EU citizens from 29 countries, to measure trust levels. We then correlated that data with vaccination data from the European Centre for Disease Prevention and Control.

— [Get Vaccinated? How Trust In Institutions Determines COVID Vaccination Rates In The EU](https://www.forbes.com/sites/lutzfinger/2021/05/06/get-vaccinated-how-trust-into-the-institutions-determines-vaccination-rates-in-the-eu/) · _Forbes_

Question 7

How did you solve the fairness problem in peer reviews for your Cornell data products course?

Accepted Answer

_Statistical personas identify unfair reviewers in peer grading systems._

We developed a data validation process to ensure fairness in peer reviews. The process analyzes metrics like total review time, time spent on each review, and number of comments left on each project. Most variables followed a Gaussian distribution, letting us use statistical tools to identify problematic reviewers. We built student personas including Lazy Graders, Harsh and Happy Graders, and Misaligned Students. Examining just three to four projects from a single student typically gave us enough insight to determine if they fit these personas, and we would re-evaluate their feedback accordingly.

— [Hack For Fairness: Unlocking The Predictive Power Of Data Validation And Quality Assurance In Education](https://www.forbes.com/sites/lutzfinger/2023/04/03/hack-for-fairness-unlocking-the-predictive-power-of-data-validation-and-quality-assurance-in-education/) · _Forbes_

Question 8

How did you use quizzes to improve the flipped classroom model?

Accepted Answer

_Quizzes revealed where students struggled before class even started._

I asked students to do a quiz before coming to class to find out which areas they didn't understand. For example, a FiveThirtyEight video on k-means clustering Margaritas was fun, but the quiz revealed it failed to explain where the 'k' in k-means came from. This feedback let me spend more time on that point in class than I would have usually done. Content curation became the critical path, and quizzes helped me identify gaps.

— [How COVID Changed Our Classrooms? A Self-Experiment Of Online Teaching.](https://www.forbes.com/sites/lutzfinger/2020/05/14/how-covid-changed-our-classrooms-a-self-experiment-of-online-teaching/) · _Forbes_

Question 9

How do you measure quality in healthcare data?

Accepted Answer

_Quality metrics for healthcare ML._

> Knowing quality is therefore of utmost importance. Three broad measurement areas in today's healthcare: (1) Metrics about quality of care — related to treatment effectiveness and patient satisfaction. (2) Metrics about need and utilization of care — gauging health requirements of a population and how often one uses what type of service. (3) Metrics about value-based healthcare — they measure disease prevention rather than disease…

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 10

How does Think Labs AI use physics informed AI differently than general purpose models like ChatGPT?

Accepted Answer

_Grid AI must be grounded in physics, not just patterns in data._

We cannot just put SCADA measurements and smart metering information into ChatGPT or Microsoft Copilot or Gemini and ask how the grid works. It does not work like that. We lean into something called Physics Informed AI. We train our AI models using physics. It is physics driven, physics based, and physics accurate. We teach AI how the grid works and reproduce engineering analysis faithfully. The AI can turn traditional studies that take weeks to months into minutes, and go from analyzing one feeder at a time into full system analysis.

— [AI Needs Power, the Grid Needs AI, with Josh Wong](https://www.lutzfinger.com/transcript/2026-04-02-ai-needs-power-the-grid-needs-ai-with-josh-wong/) · _The Keynote on AI_

Question 11

How does your company Gooder AI address the gap between technical metrics and business value?

Accepted Answer

_*Translating predictive model performance into profit and savings estimates*_

Gooder addresses one particular place where there's a need for a new category of software: the metrics. It's a business user console for understanding potential value and making decisions about deployment. Typically, data scientists present technical metrics like area under the curve, precision, or recall, but very rarely go to stakeholders and say if you deploy this model, it has potential value of this much profit or savings. We provide visualization where both dimensions are business relevant, like the decision threshold and business metrics like savings or profit, so stakeholders can see exactly what their options are.

— [The Value Translation Gap: AI's Deployment Problem](https://www.lutzfinger.com/transcript/2024-11-28-the-value-translation-gap-ais-deployment-problem/) · _The Edge_

Question 12

How has the shift from SEO to GEO changed what websites need to do for brands?

Accepted Answer

_Websites now feed AI rather than convert human visitors directly._

Websites used to be designed to convert human visitors, but now their main purpose needs to be feeding data into AI. The three-party model of searcher, Google, and end content is disintegrating into a two-party model. Decisions are getting made before consumers hit the website, so the website is no longer the main conversion point. Instead, websites should be clean data delivery mechanisms to computers, very explicit about who you sell to, your target use case, pain points, and the entire buyer journey. If it's not explicit, the AI has no way to know and might recommend somebody else.

— [Marketing to Humans Changed, with Alex Dees](https://www.lutzfinger.com/transcript/2026-03-24-marketing-to-humans-changed-with-alex-dees/) · _The Keynote on AI_

Question 13

How have search queries changed with LLM-style interfaces compared to traditional search?

Accepted Answer

_Queries grew from one word to 23 words with conversational interfaces._

The way people search is evolving dramatically. In the past, users typed one or two words like camera. Now, when shown more natural and detailed responses, they respond in kind with much longer, conversational queries like "What's a compact camera for wildlife photography that fits in a carry-on?" Semrush backs this up: average query length is now 23 words, and sessions last over 6 minutes. Users expect answers that feel personalized and human. In our interviews, shoppers said they felt more understood and better about their purchase.

— [Study Shows LLM Conversion Rate Is 9x Better — AEO Is Coming](https://www.forbes.com/sites/lutzfinger/2025/06/19/study-shows-llm-conversion-rate-is-9x-better---aeo-is-coming/) · _Forbes_

Question 14

How much better does traffic from ChatGPT-style experiences convert compared to traditional search?

Accepted Answer

_LLM traffic converts dramatically better than traditional search results._

Traffic from ChatGPT-style experiences converts up to nine times better than traditional search, according to a study I conducted with my Cornell master's students. The reason is that large language models behave more like trusted advisors than search engines. This fundamental shift is already transforming how consumers discover and buy. If your brand isn't showing up in these conversations, you're invisible to a growing segment of consumers who rely on LLMs for recommendations.

— [Answer Engine Optimization — What Brands Need To Know](https://www.forbes.com/sites/lutzfinger/2025/06/19/answer-engine-optimization-aeo--what-brands-need-to-know/) · _Forbes_

Question 15

How much higher are conversion rates when using AI-powered search compared to traditional keyword search?

Accepted Answer

_AI search delivers up to 500% higher conversion than keyword search._

I did research comparing traditional keyword search with a more explanatory AI approach. When someone searches for a car like a Honda Odyssey, instead of just showing a list, the AI provides context about the car, options, and nearby dealerships with pricing. The conversion rate from search to buy can be five times higher in some cases. That's a 500% increase. This happens because the chatbot gives personalized recommendations based on knowing the user's full background and context, making the information land much more effectively than traditional search results.

— [Lutz Finger | From SEO to GEO: How AI is Changing How Car Buyers Find You](https://www.lutzfinger.com/transcript/2026-05-22-lutz-finger-from-seo-to-geo-how-ai-is-changing-how-car-buyers-find-you/) · _Three Key Insights_

Question 16

How much more valuable is LLM traffic compared to traditional search traffic?

Accepted Answer

_LLM traffic delivers nine times the value of traditional search._

In my research I show that the traffic from LLMs is 9x as valuable as traditional search traffic. This dramatic difference underscores why the shift to AI-powered assistants is so significant for retailers. For traditional retailers, this means the customer's path to your product no longer runs through your digital storefront. When AI becomes the curator, advisor, and shopping guide, the traditional digital mall will die. This is why retailers need to fundamentally rethink their strategies for the age of AI discovery.

— [The End Of Amazon, Walmart, Best Buy? AI-Driven Retail Unbundling](https://www.forbes.com/sites/lutzfinger/2025/06/30/the-end-of-amazon-walmart-best-buy---ai-driven-retail-unbundling/) · _Forbes_

Question 17

How should brands handle product page personalization instead of using generic chatbots?

Accepted Answer

_Personalize product pages by query, not by generic chatbot._

The fix is to weave conversation into the experience instead of pasting a clone on top. The clearest example is the product page. A product page no longer has to be average, stuffed with every keyword to please search while burying the reader in detail they never asked for. It can personalize to each shopper, based on their clicks and queries, surfacing what matters and hiding what does not. Based on my own A/B tests, this approach delivered 8.6 times better conversion.

— [Three Mistakes Brands Make In The Age Of AI-Driven E-Commerce](https://www.forbes.com/sites/lutzfinger/2026/05/23/three-mistakes-brands-make-in-the-age-of-ai-driven-e-commerce/) · _Forbes_

Question 18

How should product builders think about data bias in AI systems?

Accepted Answer

_Bias in data is inevitable; product decisions determine how much to accept._

There is never a way to have non-biased data, because data is biased. As a product person, you need to decide by how much you want to de-bias the data. Sometimes bias is actually good because you don't want averages, you want the best outcome. For example, if identifying certain patients or optimal machinery output, you don't want what the machine did in the past all the time, but just the best parts of it. You're injecting very good data versus just average data.

— [Building AI products: 5 lessons from our founders' workshop](https://www.lutzfinger.com/transcript/2024-08-01-building-ai-products-5-lessons-from-our-founders-workshop/) · _The Edge_

Question 19

What are the three broad measurement areas in today's healthcare?

Accepted Answer

_Quality, utilization, and value-based prevention are the three areas._

There are three broad measurement areas. First, metrics about quality of care, which relate to treatment effectiveness and patient satisfaction. Second, metrics about need and utilization of care, gauging health requirements of a population and how often one uses what type of service. Third, metrics about value-based healthcare, which measure disease prevention rather than disease treatment, measuring actions taken to prevent or reverse illnesses instead of just treating them.

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 20

What conversion rate difference did you find between LLM-driven traffic and traditional search traffic?

Accepted Answer

_LLM traffic converts nearly nine times better than traditional search._

In A/B tests I conducted with Cornell Master's students, we compared regular keyword search with an AI-guided, conversational search experience inside the site search of several e-commerce stores. The difference was stunning: LLM-generated traffic converted at almost 9x higher rates than traditional search. This isn't just about volume, it's about value. LLM-driven traffic behaves more like a personal recommendation than a keyword query, which explains why users convert at dramatically higher rates when they follow those links or act on suggestions.

— [Study Shows LLM Conversion Rate Is 9x Better — AEO Is Coming](https://www.forbes.com/sites/lutzfinger/2025/06/19/study-shows-llm-conversion-rate-is-9x-better---aeo-is-coming/) · _Forbes_

Question 21

What did Ben learn at LinkedIn about predicting company success with data?

Accepted Answer

_Only founder data predicts early stage success, not company metrics._

Working with the data science team at LinkedIn, we evaluated whether there are signals in the data to predict future company success. After series B, there are signals. Market signals and signals internal to LinkedIn data are predictive. Things like the quality of where you're hiring from, how many people you win compared to how many you lose to other companies. Those are very predictive. In the early stages, no company level data works. The only thing that works is data around founders specifically. This was truly revelating, showing that at the early stages, it's all about the founders.

— [The Hidden Patterns, with Ben Orthlieb](https://www.lutzfinger.com/transcript/2026-04-23-the-hidden-patterns-with-ben-orthlieb/) · _The Keynote on AI_

Question 22

What did the data show about high trust versus low trust countries?

Accepted Answer

_High trust countries vaccinate more, and the gap widens._

We plotted a historical trend of vaccine uptake delineated into high and low trust countries. The plot shows clearly that high trust countries have a higher mean vaccine uptake over low trust countries, with the gap widening over time. We regressed vaccine uptake as of April 18, 2021 with trust in EU institutions. A positive coefficient for trust in EU institutions along with a p-value of less than 0.05 clearly indicates a statistically positive relationship.

— [Get Vaccinated? How Trust In Institutions Determines COVID Vaccination Rates In The EU](https://www.forbes.com/sites/lutzfinger/2021/05/06/get-vaccinated-how-trust-into-the-institutions-determines-vaccination-rates-in-the-eu/) · _Forbes_

Question 23

What did your analysis of ShopTalk 2025 vendors reveal about generative AI adoption in e-commerce?

Accepted Answer

_Only 1% of 700 vendors offer truly integrated generative AI solutions._

I analyzed all 683 vendors at ShopTalk, the biggest e-commerce conference. While 40% claim to offer AI capabilities, only 25% (169 vendors) mention generative AI explicitly. That's surprisingly low given the hype. Even more striking, I found only seven companies, just 1% of all vendors, offering merged solutions that combine search, content personalization and chatbots using multimodal capabilities. Only one of those seven was an established player; the rest were startups. About 70% of companies focused on data and AI were startups, suggesting generative AI in e-commerce is still nascent.

— [ShopTalk 2025: Just 1% Of Vendors Use Generative AI To The Max](https://www.forbes.com/sites/lutzfinger/2025/03/21/state-of-genai--only-1---shoptalk-2025-in-numbers/) · _Forbes_

Question 24

What did your classroom study reveal about AI bias in tracking participation?

Accepted Answer

_Classroom data shows AI heard Western names five times more often._

I compared AI name recognition against my own participation ratings for students at Cornell and INSEAD. For Western names, my ratings and the AI's count had a correlation of 0.53, showing they moved together. For non-Western names, the correlation was just 0.27, statistically indistinguishable from noise. The AI also heard Western names about five times more often. For half my class, the AI's measure of contribution had almost nothing to do with how much they actually contributed.

— [If Your Name Isn't Western, AI Could Cost You The Promotion.](https://www.forbes.com/sites/lutzfinger/2026/05/25/if-your-name-isnt-western-ai-could-cost-you-the-promotion/) · _Forbes_

Question 25

What does Marpai Connect measure in value-based care, and what do you de-emphasize?

Accepted Answer

_Clinical markers and outcomes matter more than engagement metrics._

At Marpai Connect we use bio markers that are clinically proven, like HBA1C measurements for diabetes. We also use other measured outcomes such as pain reduction scores and functional health improvement scores. We include cost, which is easy for a third-party administrator to measure via claims, but cost alone is not sufficient. What we de-emphasize: engagement metrics, because they often lack evidence that they work. Our approach is working. 40% of the commercial market wants to get access to value-based care.

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 26

What engagement methods did you test and which worked best?

Accepted Answer

_Student surveys revealed multiple-choice questionnaires won for engagement._

I experimented with cold-calling via a random number generator, multiple-choice questionnaires, discussions in smaller groups, and other methods. After surveying students after each class, multiple-choice questionnaires came out on top. I created them for every 10 minutes of my course to ensure my speed was right and measure my teaching performance. I also used a Google Spreadsheet where students outlined answers in one to two sentences, then picked the most interesting ones to discuss.

— [How COVID Changed Our Classrooms? A Self-Experiment Of Online Teaching.](https://www.forbes.com/sites/lutzfinger/2020/05/14/how-covid-changed-our-classrooms-a-self-experiment-of-online-teaching/) · _Forbes_

Question 27

What has been the impact of implementing data validation in your peer review process?

Accepted Answer

_Instructors now focus time on critical reviews, elevating course quality._

Employing data validation and data quality assurance has been a game-changer for the class. As instructors, we can now concentrate on the most critical reviews, ultimately elevating the course's quality. Instead of reviewing every piece of feedback, we focus our time on re-evaluating feedback from students who fit problematic personas. Our toolset is coded in Python and can be adapted for various courses using FeedbackFruits, making this approach scalable beyond just our Designing Data Products course.

— [Hack For Fairness: Unlocking The Predictive Power Of Data Validation And Quality Assurance In Education](https://www.forbes.com/sites/lutzfinger/2023/04/03/hack-for-fairness-unlocking-the-predictive-power-of-data-validation-and-quality-assurance-in-education/) · _Forbes_

Question 28

What is LIME and how does it help with explainable AI?

Accepted Answer

_LIME removes pixels to reveal what triggers AI decisions._

LIME is a tool designed to help with explainability in AI systems, particularly for complex models like deep learning that create their own variables which might not make immediate sense for humans. LIME works by creating many different versions of the same image through removing pixels. This process helps researchers and users understand what specifically triggered a given AI response. It addresses the challenge that while decision trees, Bayesian classifiers, and linear models are easy to interpret, deep learning models cannot be explained so easily.

— [President Biden Is Man, Woman And 40 Years Old - Why We Need Algorithmic Transparency](https://www.forbes.com/sites/lutzfinger/2021/02/04/president-biden-is-man-woman-and-40-years-old---why-we-need-algorithmic-transparency/) · _Forbes_

Question 29

What is the 'Ask, Measure, Learn' framework?

Accepted Answer

_The framework from Lutz's book._

> "Ask Measure Learn" is a non-technical guide shows you how to extract significant business value from big data. It is a system that helps you ask the right questions, measure the right data, and then learn from the results. The framework was originally devised to help governments and NGOs sift through volumes of data.

— [Ask Measure Learn](https://www.lutzfinger.com/book/) · _Book — Ask, Measure, Learn_

Question 30

What is the AI Effect and how does it relate to AGI?

Accepted Answer

_AGI's definition keeps shifting as AI capabilities advance over time._

The AI Effect describes how the definition of AGI often shifts as AI systems become capable of performing tasks once thought to require general intelligence. Some colleagues joke that AGI is 'what we haven't yet built.' Altman recently downplayed the G in AGI, saying his guess is we will hit AGI sooner than most people think, and it will matter much less. Just on Sunday, he said it has become a very sloppy term. This sounds like moving the goalpost.

— [Sam Altman Says, 'We Know How To Build AGI' — Not So Fast](https://www.forbes.com/sites/lutzfinger/2025/01/07/sam-altman---agi--ai-workforce-in-2025-the-battle-of-tech-giants/) · _Forbes_

Question 31

What is the book Ask, Measure, Learn about?

Accepted Answer

_Topics covered in the O'Reilly book._

> **Publisher:** O'Reilly Media **Published:** Feb 1, 2014

— [Ask Measure Learn](https://www.lutzfinger.com/book/) · _Book — Ask, Measure, Learn_

Question 32

What is the difference between virality and contagiousness?

Accepted Answer

_Stories gain weight as they travel, not uniform infection rates._

Virality is something which is equally infectious at each stage. If I have a story which one out of 10 people like, that story doesn't change. One out of 10 people like it in the future as well. Contagiousness is different. As long as it travels, it gains weight. The more people believe in it, the more trustworthy the story becomes. Even a wrong information becomes more believed the longer it travels. At some point everyone believes it's truth no matter what because so many people believed in it. Forget about virality. The right word is contagiousness.

— [Measuring Social Media What Marketers Need To Know](https://www.lutzfinger.com/transcript/2014-08-22-measuring-social-media-what-marketers-need-to-know/) · _Social Media Examiner_

Question 33

What is value-based care and how does AI enable it?

Accepted Answer

_VBC framing and AI's role._

> Healthcare is on the brink of a seismic shift, and AI is its catalyst. Three converging trends are creating a perfect opportunity for change: (1) Foundational Models' growing ability to predict risks further into the future compared to traditional machine learning, (2) the rise of the self-insured healthcare industry motivated to address these risks, and (3) a surge in Value-Based Care (VBC) providers focused on risk reduction. These…

— [Three Trends Of Transformation: How AI & Value-Based Care (VBC) Will Drive Transformation Of Self-Insured Healthcare](https://www.forbes.com/sites/lutzfinger/2023/05/01/three-trends-of-transformation-how-ai--value-based-care-vbc-will-drive-transformation-of-self-insured-healthcare/) · _Forbes_

Question 34

What makes achieving general intelligence fundamentally different from doing many specialized tasks well?

Accepted Answer

_Generalization beyond training data defines AGI, not task breadth alone_

The key difference is the learning and generalization capability, not breadth of tasks. A system that does 95% of human jobs by copying millions of training examples is not AGI, even if economically valuable. True AGI must take a leap beyond what it was prepared for and generalize from what it knows to do new things. If you describe a novel alien civilization with bizarre customs never in training data, and the system reasons about it from multiple perspectives, that shows real inductive and abductive reasoning. AGI should figure out how to achieve unfamiliar goals in unfamiliar environments through creative imagination and trial and error, not just pattern matching against a massive catalog of prior examples.

— [Rethinking the AGI Race, with Benjamin Goertzel](https://www.lutzfinger.com/transcript/2026-05-28-rethinking-the-agi-race-with-benjamin-goertzel/) · _The Keynote on AI_

Question 35

What metrics do you use for data validation in peer review quality assurance?

Accepted Answer

_Review time and comment patterns reveal data quality issues._

Our data validation process is based on total review time, time spent on each review, the number of comments left on each project, and more. Most variables followed a Gaussian distribution, allowing us to use various statistical tools similar to those that identified cheating sumo wrestlers or overlooked baseball players. Based on predicted values from these metrics, we can identify problematic reviewing patterns. This approach mirrors data quality assurance in machine learning, where high-quality labeled data leads to better model performance.

— [Hack For Fairness: Unlocking The Predictive Power Of Data Validation And Quality Assurance In Education](https://www.forbes.com/sites/lutzfinger/2023/04/03/hack-for-fairness-unlocking-the-predictive-power-of-data-validation-and-quality-assurance-in-education/) · _Forbes_

Question 36

What outcome metrics does Marpai Connect use to evaluate care effectiveness?

Accepted Answer

_Specific metrics track adherence, improvement, and patient participation._

My team at Marpai designed outcome metrics with our value-based care partners to evaluate the effectiveness of care. These include medication adherence, functional health improvement, flu vaccination rate, and patient engagement. We also built financial alignment into the platform, where providers should only receive payment if they effectively help reduce costs and improve health on average. For example, NuvoAir committed to a total cost of care reduction of 35% based on claims analysis with their virtual-first care model.

— [Changing Healthcare - A Platform Approach Using Data & AI](https://www.forbes.com/sites/lutzfinger/2023/07/05/changing-healthcarea-platform-approach-using-data--ai/) · _Forbes_

Question 37

What pattern appears consistently in data about AI adoption and concern?

Accepted Answer

_Survey data reveals a consistent gap between concern and usage_

People express concern about AI in surveys. They worry about job displacement, privacy, hallucinations, and dependency. And then they use it anyway. This pattern shows up consistently in the data. The gap between what people say they're worried about and how they actually behave is not hypocrisy. It's simply how humans work. We routinely accept risks when the benefit feels immediate and the risk feels diffuse or abstract.

— [Chris Hemsworth Super Bowl Ad Shows Why We'll Use AI Even If We Fear It](https://www.forbes.com/sites/lutzfinger/2026/02/10/even-if-ai-could-kill-chris-hemsworth-convenience-beats-fear/) · _Forbes_

Question 38

What percentage of healthcare costs are linked to reversible or manageable diseases?

Accepted Answer

_Nearly half of healthcare spending could be avoided with better care._

At Marpai Inc., we discovered that 23% of all healthcare costs are linked to reversible diseases. An additional 18% could remain stable with better management. That's a staggering 42% of essentially avoidable costs. This finding highlights the enormous opportunity to reduce healthcare spending by focusing on prevention and better disease management rather than just treating illness.

— [Changing Healthcare - A Platform Approach Using Data & AI](https://www.forbes.com/sites/lutzfinger/2023/07/05/changing-healthcarea-platform-approach-using-data--ai/) · _Forbes_

Question 39

What should I monitor to optimize my brand for AI assistants?

Accepted Answer

_Start with questions, not metrics. Understand what customers actually ask._

Don't run and start to monitor your brand. Instead, try to understand the best set of questions to monitor. The challenge for brands is to predict what users will be looking for. Companies should be tapping into customer support tickets, community forums, social media conversations, and user interviews. In other words, you need to get the questions right first. I've built data and monitoring tools for two decades, and I've seen brands obsessively measure mentions without asking the deeper question: what exactly should we measure? Understanding authentic customer language is the foundation.

— [AI Is Changing The Rules Of Web Traffic — A GEO Playbook For Online Retailers](https://www.forbes.com/sites/lutzfinger/2025/08/20/generative-engine-optimization-geo-a-playbook-for-online-retailers/) · _Forbes_

Question 40

What unintended consequences can quality metrics create in healthcare?

Accepted Answer

_Mortality metrics can discourage doctors from treating complex cases._

As Michael Howell reminded me, the real challenges are in finding the metrics that matter and preventing unintended consequences with any given metric. Take a simple metric on mortality for surgery: patients should not die during surgery. But what if the patient's situation was complex to start with? There are cases where a patient did not receive care and died because giving him appropriate care could have resulted in a worse quality score for the doctor.

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 41

What would Clubhouse need to do to understand conversations and improve content discovery?

Accepted Answer

_Live transcription is technically possible but not currently implemented._

To find the right discussion for you, Clubhouse would need to understand all conversations, meaning it would need to live-transcribe them. Technically this is feasible. Speech to text algorithms have become very powerful. Clubhouse is currently not transcribing conversations, presumably to preserve the authenticity of the moment. However, streaming companies like Spotify or Deezer can tell you that a delay between the user's dislike and an action such as skip makes it harder for the algorithm to work correctly. Clubhouse will need data and engagement metrics to scale.

— [Clubhouse's Future Depends On Data - How To Build A TikTok Like Algorithm](https://www.forbes.com/sites/lutzfinger/2021/02/09/clubhouses-future-depends-on-datahow-to-build-a-tiktok-like-algorithm/) · _Forbes_

Question 42

Why do AI models still need human involvement even with unsupervised learning?

Accepted Answer

_Even advanced models can't escape the need for human judgment._

We need humans because there isn't always a true north for what's correct. Supervised learning works when there's a clear label, like Obama was president. That's a fact. But when you get into perception, like whether a joke is funny or which word to use about Obama's policies, it depends on context and perspective. The same happens with generated images. Art often creates things society didn't expect, so at what point is something random versus amazing? Since there's not a true label for certain things, we constantly need people to give feedback on the model and provide guardrails.

— [AI and the human factor](https://www.lutzfinger.com/transcript/2023-07-07-ai-and-the-human-factor/) · _The Edge_

Question 43

Why do LLMs convert better than search?

Accepted Answer

_Trust, personalization, and conversion lift from AI traffic._

> Some predict that by 2028, more people will discover products and information through large language models (LLMs) like ChatGPT and Gemini than through traditional search engines. But based on research I conducted with Cornell Master's students, that shift is happening much faster. LLM-driven traffic is already starting to outperform traditional search — not in volume, but in value.

— [Study Shows LLM Conversion Rate Is 9x Better — AEO Is Coming](https://www.forbes.com/sites/lutzfinger/2025/06/19/study-shows-llm-conversion-rate-is-9x-better---aeo-is-coming/) · _Forbes_

Question 44

Why does Clubhouse's future success depend on using user data?

Accepted Answer

_Finding content on Clubhouse is hard without better data use._

Clubhouse's future success depends on how it can help users find content and enable engagement through the use of user data. At the core, any social app solves two issues: to find and enable social interactions. Currently, finding interesting conversations on Clubhouse is not easy. You curate your own content by following people, similar to how Twitter used the social graph as a first approximation for your interest. A social graph is a clunky way to curate your feed. In order to scale, Clubhouse will need data and engagement metrics.

— [Clubhouse's Future Depends On Data - How To Build A TikTok Like Algorithm](https://www.forbes.com/sites/lutzfinger/2021/02/09/clubhouses-future-depends-on-datahow-to-build-a-tiktok-like-algorithm/) · _Forbes_

Question 45

Why does creating more AI-generated content fail at generative engine optimization?

Accepted Answer

_More AI content does not win at generative engine optimization._

Executives wrongly assume that because more pages once won at SEO, more AI-generated content will win at GEO. The opposite is true. AI trained on AI degrades, and model providers actively suppress pages that read like machine-generated filler. Pumping out slop is the fastest way to become invisible to the very engines you are chasing. The fix is to give models what they actually reward: novel, unique, and authentic content. That is exactly why large language models lean so heavily on sources like Reddit and LinkedIn when they train.

— [Three Mistakes Brands Make In The Age Of AI-Driven E-Commerce](https://www.forbes.com/sites/lutzfinger/2026/05/23/three-mistakes-brands-make-in-the-age-of-ai-driven-e-commerce/) · _Forbes_

Question 46

Why does traffic from AI assistants convert better than other channels?

Accepted Answer

_AI recommendations convert nine times better through personalized trust._

Studies show that traffic originating from LLMs converts up to nine times better than other channels. Why? Because recommendations feel personal, contextual, and trustworthy. One example stands out: a consumer uploaded their blood results into ChatGPT and immediately purchased €200 worth of vitamins with no browsing, no comparison, pure trust. AI-driven traffic is still small in volume, but its impact is outsized. The personalized, context-aware nature of AI recommendations creates a level of trust that traditional search channels can't match.

— [AI Is Changing The Rules Of Web Traffic — A GEO Playbook For Online Retailers](https://www.forbes.com/sites/lutzfinger/2025/08/20/generative-engine-optimization-geo-a-playbook-for-online-retailers/) · _Forbes_

Question 47

Why don't monitoring dashboards work well for tracking brand presence in LLMs?

Accepted Answer

_LLM memory makes traditional monitoring dashboards fundamentally limited._

LLMs don't behave like search engines because they remember. Google would always return the same results, even if you had already clicked those links before. Every new session was a reset with no context. That's no longer true. LLMs build context and recall prior interactions, and that memory shapes future recommendations. To fully understand how your brand is being represented, you'd have to know the personalized memory of every single user, an impossible task for any dashboard. The best measurement is just watching your actual traffic from ChatGPT, Gemini or Perplexity.

— [Answer Engine Optimization — What Brands Need To Know](https://www.forbes.com/sites/lutzfinger/2025/06/19/answer-engine-optimization-aeo--what-brands-need-to-know/) · _Forbes_

Question 48

Why is eval data a moat?

Accepted Answer

_Evaluation data as the strategic differentiator._

> All of them matter, and all of them are being fought over. But the real value may sit in the one thing OpenAI, Anthropic, and Gemini have not yet made central: your own eval data. Sure, there is the occasional thumbs-up button. But who is really using that signal?

— [The Missing Moat In AI: Your Eval Data](https://www.forbes.com/sites/lutzfinger/2026/05/26/the-missing-moat-in-ai-your-eval-data/) · _Forbes_

Question 49

Why is it important to measure appropriateness of medical interventions, not just quality?

Accepted Answer

_Fee-for-service payment can incentivize unnecessary procedures._

As Marty Makary told me, it is not sufficient to just measure the quality of medical procedures, but we also need to evaluate the appropriateness of these interventions. The current fee-for-service payment structure might encourage some doctors to perform multiple surgeries rather than one, as they are paid per procedure. Transparency in correcting misalignments is key. This is why utilization metrics look at whether someone should have gotten a given intervention in the first place.

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 50

Why is it so hard to know the quality of healthcare we receive?

Accepted Answer

_Patients lack data because they are not the customer._

We rate Uber drivers, restaurants, and Amazon delivery, but when it comes to our own health experience, we have little to no data. The reason is structural: the business model of the healthcare provider is to get money from the insurance company, not the patient. The patient is not the customer. Because of this misalignment, knowing quality is of utmost importance, yet the information is not readily available to us as patients.

— [Healthcare And Data - How To Measure Quality](https://www.forbes.com/sites/lutzfinger/2023/06/27/healthcare-and-datahow-to-measure-quality/) · _Forbes_

Question 51

Why is measuring social media so difficult despite all the tools available today?

Accepted Answer

_Tools are everywhere, but strategy comes first, not scores._

The problem is that measurement used to require specialists with big databases and tooling. Now with social media, you suddenly have all these tools that give you cloud scores and metrics instantly. People see a score and decide something based on it without planning what they actually want to achieve. If you haven't formulated your business goal first, any measurement won't work. People start with the measurement and then think about what to do with it, while the correct way is the opposite: start with a question, then come up with a measure.

— [Measuring Social Media What Marketers Need To Know](https://www.lutzfinger.com/transcript/2014-08-22-measuring-social-media-what-marketers-need-to-know/) · _Social Media Examiner_

Question 52

Why is workflow integration important for OpenAI's feedback loops?

Accepted Answer

_Integration into workflows helps measure answer quality and drives specialization._

Feedback loops work best if they are integral to an existing workflow. We will see more and more workflows become supercharged with LLMs. Once OpenAI is integrated into those workflows, it's easier to see when and whether a user accepts a change or suggestion. OpenAI will need to invest more efforts into workflows and products to be more than just a generic language interface. Specialization will be the norm. Initially, OpenAI used the fast route by tagging along with Microsoft, with Copilot offering integrations into many of Microsoft's core tools.

— [How OpenAI Can Secure Its Investments - A Playbook That Needs Microsoft](https://www.forbes.com/sites/lutzfinger/2023/08/18/the-potential-moat-of-openaihow-openai-can-secure-its-investments/) · _Forbes_

Question 53

Why was it difficult to measure LLM traffic conversion, and how did you solve it?

Accepted Answer

_Tracking LLM conversions is hard because users don't leave clean trails._

Good numbers are hard to come by because LLM traffic doesn't always leave a clean trail. Users might just copy and paste a product name and head to Amazon, making it hard to track. To get better data, we created a ChatGPT-style experience inside the site search of several e-commerce stores. This let us run A/B tests comparing regular keyword search with an AI-guided, conversational search experience. That's how we discovered the almost 9x higher conversion rate for the conversational approach.

— [Study Shows LLM Conversion Rate Is 9x Better — AEO Is Coming](https://www.forbes.com/sites/lutzfinger/2025/06/19/study-shows-llm-conversion-rate-is-9x-better---aeo-is-coming/) · _Forbes_

Q&A, Measurement

Can a well-designed prompt outperform OpenAI's o1 model?

Can we conclude that trust causes higher vaccination rates?

Did you find other factors that influenced vaccination rates besides trust?

How are companies currently using AI tools to measure employee contributions?

How did Varo use AI to reduce operational costs while improving customer service?

How did you isolate trust as a factor in COVID vaccination rates?

How did you solve the fairness problem in peer reviews for your Cornell data products course?

How did you use quizzes to improve the flipped classroom model?

How do you measure quality in healthcare data?

How does Think Labs AI use physics informed AI differently than general purpose models like ChatGPT?

How does your company Gooder AI address the gap between technical metrics and business value?

How has the shift from SEO to GEO changed what websites need to do for brands?

How have search queries changed with LLM-style interfaces compared to traditional search?

How much better does traffic from ChatGPT-style experiences convert compared to traditional search?

How much higher are conversion rates when using AI-powered search compared to traditional keyword search?

How much more valuable is LLM traffic compared to traditional search traffic?

How should brands handle product page personalization instead of using generic chatbots?

How should product builders think about data bias in AI systems?

What are the three broad measurement areas in today's healthcare?

What conversion rate difference did you find between LLM-driven traffic and traditional search traffic?

What did Ben learn at LinkedIn about predicting company success with data?

What did the data show about high trust versus low trust countries?

What did your analysis of ShopTalk 2025 vendors reveal about generative AI adoption in e-commerce?

What did your classroom study reveal about AI bias in tracking participation?

What does Marpai Connect measure in value-based care, and what do you de-emphasize?

What engagement methods did you test and which worked best?

What has been the impact of implementing data validation in your peer review process?

What is LIME and how does it help with explainable AI?

What is the 'Ask, Measure, Learn' framework?

What is the AI Effect and how does it relate to AGI?

What is the book Ask, Measure, Learn about?

What is the difference between virality and contagiousness?

What is value-based care and how does AI enable it?

What makes achieving general intelligence fundamentally different from doing many specialized tasks well?

What metrics do you use for data validation in peer review quality assurance?

What outcome metrics does Marpai Connect use to evaluate care effectiveness?

What pattern appears consistently in data about AI adoption and concern?

What percentage of healthcare costs are linked to reversible or manageable diseases?

What should I monitor to optimize my brand for AI assistants?

What unintended consequences can quality metrics create in healthcare?

What would Clubhouse need to do to understand conversations and improve content discovery?

Why do AI models still need human involvement even with unsupervised learning?

Why do LLMs convert better than search?

Why does Clubhouse's future success depend on using user data?

Why does creating more AI-generated content fail at generative engine optimization?

Why does traffic from AI assistants convert better than other channels?

Why don't monitoring dashboards work well for tracking brand presence in LLMs?

Why is eval data a moat?

Why is it important to measure appropriateness of medical interventions, not just quality?

Why is it so hard to know the quality of healthcare we receive?

Why is measuring social media so difficult despite all the tools available today?

Why is workflow integration important for OpenAI's feedback loops?

Why was it difficult to measure LLM traffic conversion, and how did you solve it?