In Brief #9: Measuring AI-exposure in online job advertisements

Using a large-scale dataset of 182 million unique job postings from 27 Member States from 2022 to 2025, we developed and applied a multilingual, rule-based classification model that systematically detects whether positions require in-depth AI skills beyond basic application use.

Step 1: Developing a multilingual dictionary

A multilingual keyword dictionary formed the foundation of the model. It includes the 24 official languages of the EU as well as other region-specific languages such as Catalan, Luxembourgish and Turkish that are relevant in some EU Member States. Although these languages are spoken less frequently, they could theoretically appear in job postings in specific regions. The dictionary contains key terms like ‘machine learning’ and ‘artificial intelligence’ that were translated into all country-specific languages. In addition, the dictionary contains 1600 Englishlanguage entries that remain untranslated, as they primarily consist of the names of AI tools, frameworks or libraries. These keywords were compiled through an extensive literature review, complemented by computational linguistic methods and were then iteratively refined and validated by matching them against the online job postings throughout the analysis.

Step 2: Enhancing the rule-based classification model

As a rule-based classification approach like a dictionary always carries a risk of false-positive classifications, the model was enhanced with specific rules and context-dependent blacklists, which exclude ambiguous terms on a country-bycountry basis. To enhance the classification accuracy of the model, we created a test set of 27,000 job advertisements (1,000 per country) and manually compared the rule-based model’s classifications with additional large language model (LLM) annotations. For this purpose, we prompted a LLM to classify job postings that require in-depth AI skills. The comparison was followed by an iterative adjustment of the LLM prompt and the dictionary of the rule-based model.

Step 3: Evaluating the model

After optimising the LLM prompt and the rule-based classification model, an evaluation test set was generated. For each country, 500 job postings were presampled using the classification model, as the probability of encountering AIrelated job postings in the overall dataset is very low. This allowed for a much more accurate evaluation of the model’s precision. In addition, 500 job postings per country were added, selected completely at random.

As a result, 1,000 job postings per country – 27,000 in total – were included in the evaluation test set. The average F1-score, which measures the quality of the rulebased model classifications, was very high (0.96 with a maximum of 1.0) and balanced for all countries. Afterwards, the rule-based classification model was applied to the whole dataset of approximately 182 million job postings.