AI for monitoring and evaluation
Ultramarine Work drafts the logic model, codes the transcripts, weights the survey and screens the literature, from the files already in your evaluation folder. Every number it returns names the file it came from.
KI-coding-Round-2.xlsxExample| Code | Informants | Excerpts | Example quote | Source |
|---|---|---|---|---|
| Employer partnerships | 15 | 41 | The employer told us what the certificate had to cover. | KI-04 |
| Childcare barriers | 12 | 27 | Classes ended at six, and daycare closed at five-thirty. | KI-11 |
| Coach turnover | 9 | 18 | I had three coaches in eight months. | KI-17 |
| Transport costs | 2 | 4 | The bus pass ran out before the program did. | KI-21 |
NoteOnly two of 22 informants raised transport costs, so report it as an emerging theme rather than a finding.
Ask it for
“Read the Pathways workforce proposal and build the logic model in the funder’s own format, with assumptions and the output and outcome indicators.”
Pathways-logic-model.xlsx“Code the 22 key informant transcripts against the codebook, with a quote under each code, and flag any theme carried by fewer than three informants.”
KI-coding-Round-2.xlsx“The March youth survey needs weighting to district enrollment totals, then a topline with margins of error, crosstabs by school level and significance tests.”
March-topline.xlsx“Screen the 340 abstracts against the inclusion criteria, log the reason for every full text you exclude, and keep the borderline calls on their own tab.”
PRISMA-screening.xlsx“Run the draft findings chapter past the evidence matrix and mark every claim that overreaches what the design can support.”
Findings-claim-check.pdf“Draft the IRB amendment for adding AI-assisted transcription to the protocol, and the consent form paragraph disclosing that AI is used.”
IRB-amendment-AI-transcription.docx
Inside the Program Evaluation & MEL mode
Program evaluation
Builds the logic model, drafts the comparison design a training impact claim rests on, and checks counts are unduplicated under the funder’s definition.
Survey design and analysis
Designs questionnaires with skip logic and a sampling plan, then weights the survey file to population targets for the topline, crosstabs and margins of error.
Qualitative research synthesis
Codes interviews and focus groups against your codebook, with a verbatim behind every code, and states how many informants each theme actually draws on.
Literature and evidence review
Documents the search so a colleague can rerun it, then screens abstracts against the inclusion criteria. It returns the PRISMA flow and the GRADE rating.
IRB submissions and informed consent
Drafts IRB applications, amendments and continuing reviews, and writes consent forms that lead with the key information section, in plain language.
- Office skills
- Excel, Word, PowerPoint, PDF
- Tools
- Read, edit and write files, Search files, Run code, Search and read the web, View images
- Connectors
- SharePoint, Google Drive, Gmail, and additional custom connectors
- House style
- Your fonts, layouts and rules, on every file
How it works
- 1
Open the evaluation folder
The folder holds the proposal, theory of change, indicator reference sheets, Kobo exports, transcripts, codebook and last quarter’s report template. Ultramarine Work reads them where they sit and follows the conventions those files use.
- 2
Name the deliverable
You can ask for a logic model in the funder’s columns, a coded transcript table, a weighted topline, a PRISMA flow or a findings chapter. Formulas stay live in the workbooks, so the weights and arithmetic are there for you to check.
- 3
Verify, then judge
Spot-check the codes against the transcripts and the figures against the exports before you weigh whether the evidence carries the claim. Ultramarine Work hands over a list of the entries it could not trace, and you write the judgment.
Who it’s for
Program evaluators, monitoring, evaluation and learning staff, impact measurement leads and independent evaluation consultants.
Questions
Will it invent findings or citations?
It can, the way any model can. The mode’s skills tell it to name the source file behind every number and statement, and to note what it could not find or reconcile instead of guessing. A fabricated finding that survives review undoes the report’s credibility, so check every claim against its source before anything goes out. The final responsibility stays with you.
Does our IRB need to approve AI in the analysis?
That call belongs to your IRB and your institution. Ultramarine Work can draft the application, amendment or continuing review and the informed consent forms, but content it works on is processed on Paper Instruments’ servers. Boards now commonly ask whether AI touches participant data, and they expect AI transcripts to be verified against the recordings, so confirm the arrangement before transcripts go in the folder.
Can it code interviews as well as NVivo?
It does the first pass, and you do the final coding. Put the codebook in the folder and it returns the coded transcripts with a segment behind every code, for you to review, merge and rename. It will miss code categories and generate near-duplicate codes, so the review is not a formality. Your interpretation decides what a theme means and whether it stays in the report.
Do we have to disclose AI use to the funder?
Increasingly they do, and what applies depends on your contract. UNDP’s evaluation guidelines require a disclosure of AI use in the evaluation report itself, some UN agencies require written approval before AI touches an evaluation, and the American Evaluation Association and the UK Evaluation Society expect disclosure and verification when AI is used. Ultramarine Work can draft the disclosure statement, and you decide what it says.
Where does participant data go, and is it kept confidential?
Ultramarine Work changes files only in the project’s folder. When a request needs it, it can read files elsewhere on your computer, never passwords, keys or app data, and you can turn that off in settings. The model runs on Paper Instruments’ servers, so the content it works on is sent there to be processed. Paper Instruments is SOC 2 Type II, and our security and compliance details are on our compliance page.