AI-Driven Innovation Measurement: Testing the limits of Large Language Models and Knowledge Graphs for scaling the mapping of business innovations

Abstract

[EN] This work investigates the use of Large Language Models (LLMs) to identify innovations from web-scraped content, focusing on AI adaptation in Finland. The primary aim is to explore how advanced AI methods can support innovation measurement through unstructured data analysis. To achieve this, the study uses GPT-4o, a long context LLM, to extract relevant artifacts from web content, with a focus on entity identification and relationship extraction to generate knowledge graph (KG) structures. This research aims to understand how the combination of LLMs and KGs can provide a more comprehensive view of innovation landscapes. Preliminary findings indicate that LLMs effectively capture complex innovation-related information that traditional methods may overlook. However, LLM bias toward over-identifying artifacts poses challenges, which are addressed through additional filtration steps using LLM-as-a-judge evaluations and expert review. The results underscore the potential of LLMs to enhance innovation detection and measurement at scale, while also highlighting the need for human oversight in the process.