Evidence & evaluation

Portfolio Mapping: AI analysis at a global scale

Automating systematic analysis across large global portfolios, with human-labelled benchmarks, iterative validation and evaluator training.

Period
2026–present
Stage
Prototype applied; product in development
My role
AI workflow development and validation; product ownership; evaluator training

FAO Office of Evaluation · United Nations

Applied across global portfolios

Food and Agriculture Organization of the United Nations
Small-scale producers (BP4)
632reporting entries
Seeds programming
~1,000project documents
Technical Cooperation (TCP)
~1,500project documents

Application volumes as of November 2026

Screenshot of a global food-system intervention analysis, showing countries shaded by the number of activities identified.
Example analytical output: a global scan of food-system interventions. Screenshot of an activity-level map view.

Making global evidence usable

I develop automated AI workflows that make large collections of project records and documents available for systematic qualitative analysis. This enables global scans of interventions, patterns and gaps that would be impractical to assemble consistently by hand.

The work combines analytical-framework design, extraction and qualitative coding with large language models (LLMs), and traceable evidence outputs. I work with evaluators and an AI engineer to translate evaluation questions into workflows, source-linked analysis and dashboards.

Calibration and validation as a continuous loop

I evaluate AI performance against human-labelled datasets and use the disagreements to improve the analysis. Calibration is iterative: refine the codebook, prompts and decision rules, rerun the analysis, and compare the results with the human reference labels.

Precision, recall and F1 score help diagnose the quality of classifications. I examine which labels are supported, which relevant cases are missed, and where the framework or source material is ambiguous. These checks guide targeted error analysis and workflow refinement.

Quality assurance before, during and after analysis

  • Before: define the framework and boundary cases, establish human-labelled reference examples, and calibrate prompts and coding rules on a pilot batch.
  • During: check intermediate outputs, inspect source passages and difficult cases, and use review findings to refine prompts and analytical rules.
  • After: review sampled classifications and supporting evidence, investigate disagreement, and validate the interpretation of portfolio-wide patterns.

The prototype BP4 application included a stratified review of 60 entries using a second model and human judgement. The analysis mapped the portfolio’s scope and characteristics, with interpretation remaining under evaluator control.

Equipping evaluators to use AI well

I train evaluation teams in prompt engineering and practical AI literacy: understanding the analytical workflow, and interpreting and checking AI outputs. The aim is to help evaluators make better analytical decisions and use AI with an informed understanding of its strengths and limitations.

This includes user onboarding, practical workflow guidance, and feedback-led improvements to the application. Training and the validation process are part of the analytical work, alongside developing the software.

The foundation: OED’s first official AI application

I developed the prototype workflow behind OED’s first official application of AI in evaluations. That applied work is the foundation of my ongoing product work on Portfolio Mapping.

For the evaluation of FAO’s work on small-scale producers’ equitable access to resources (BP4), I translated programme priorities into an analytical framework, developed AI-assisted coding, and produced portfolio findings and visualizations. I worked with the lead evaluator on the framework and with colleagues on validation and interpretation.

FAO’s Office of Evaluation published an account of the application on 21 July 2026. The office authored that feature; the workflow development and analytical contribution described here are my work within the team.

From prototype to product

I now serve as product owner for Portfolio Mapping, developing reusable analysis workflows with evaluators and an AI engineer. Applications span BP4, Seeds programming and Technical Cooperation projects. The prototype has been applied in an evaluation; the broader product remains in development.

A small illustration

The evidence review below uses three invented passages to show why source checking matters. It does not reproduce Portfolio Mapping or represent FAO documents.

An illustrative evidence review

Does the evidence support the claim?

Source-linked analysis needs human judgment. These three invented passages show how a reviewer can separate an observation from a causal claim. This is a small illustration, using no research data.

Illustrative passage

In interviews, eight participants said their income had increased. The review did not include non-participants or collect baseline income data.
Read all examples and reviews

The programme improved household incomes.

In interviews, eight participants said their income had increased. The review did not include non-participants or collect baseline income data.

Insufficient evidence for a causal claim. Participants report an improvement, but there is no comparison or baseline. The passage supports a reported experience, rather than attribution to the programme.

Local partners helped design the research.

Before data collection, village representatives reviewed the interview guide and suggested changes to the timing and language of questions.

Supported within the passage. The passage describes a specific contribution by local representatives. It does not establish how representative the group was or how much influence they had.

The intervention reached all older residents.

The final attendance sheet lists 42 residents. The total number of eligible older residents was not recorded.

Coverage cannot be established. An attendance count gives the number reached. Without the eligible population, it cannot establish complete coverage.