fix(author): problems that say what to write, for a chat without the type's file

Merged
AL
9d ago
+671-31

Closes #172.

A chat in a browser has the h5p-author skill's text, no type file and nothing to run, so it learns a content type's shape from the build's problems. Gemini in Chrome could not build a simple Interactive Book that way. This makes the problems say what to write, lets the build page hand the type's file back, and puts one sentence in the skill.

Measured

The weakest Claude model, Haiku 5.5, in that position: SKILL.md's text only, each reply built as the build page builds it, its problems pasted back in the page's words, up to four rounds. An Interactive Book, a Course Presentation, a Column, a Question Set and a Branching Scenario, three runs each. Every row but the last starts from the same 15 first replies.

At onceBy round 2By round 3By round 4Never
0.6.03 of 15812132
This PR's problems3 of 1512150
This PR's problems, and the type files pasted with the first of them3 of 15150
The skill's new sentence (a book, a page, slides; new first replies)9 of 9, from 0 of 9

Every package built, in every row, plays (h5p-verify) and holds what was asked for. The third row was run twice, once with a draft of the first message's wording, and each time the 12 runs that had failed built in the next round.

What changes

The problems (lib/generator.mjs, lib/build.mjs). What cost the rounds was a type's levels, not its fields. Three things are now said once, by their path and with what to write, then put where they belong and checked in the same build, with what is wrong inside named by where the source has it:

params.content[0]: is sub-content, but here it goes inside an object, in its field content: write params.content[0] as { "content": { "library": "H5P.AdvancedText", "params": { … } } }
params.slides: is a field of presentation, not of this level: write params.presentation.slides
params.chapters[0]: is the params of sub-content, without the sub-content around them: write params.chapters[0] as { "library": "H5P.Column", "params": { "content": … } }

The first was three or four problems an item, one of them a list of 35 libraries. The other two hid everything written inside them until the next build.

  • No count of a list the generator refused an item of. A book whose chapters were refused also had params.chapters: has 0 items, which was not so, and is the line Gemini quoted.
  • A field that takes several libraries names them all. It showed the first, and a scenario's texts came back as branching questions.
  • A scenario's question with no alternatives is not reported as a loop besides (lib/types.mjs).
  • A problem's list of fields names those an author writes, the interface texts as one. A book's was 39 names long.
  • loadLibraries loads the only library a field takes whether or not the source names it, which is what reads a chapter without its Column. Its translation is reported only where the content names it, as before.

The type files in the package (package.json, scripts/copy-type-files.mjs). prepack copies the skill's references/types/*.md into the package, each an export. The build page has the package and no skill: with them it can copy the type's file along with a source's problems. That page change is a separate pull request in missing-elements.github.io, after this is published.

The skill (skills/h5p-author/SKILL.md). Its line on sub-content says where a Column, a slide and a book hold theirs, and it gives the type files' address on GitHub for a chat that has its text and can open a link.

Tested

  • pnpm --filter @missing-elements/h5p-author test: 171 pass, the page's build in Chromium among them. New tests for each message, for the cases where nothing is guessed, for the list count, and for the copied type files.
  • pnpm --filter @missing-elements/h5p-skills test: 77 pass, so every type file's example still builds and plays.
  • The demo's tests/sweep-content.test.ts: 5 pass. The author's typecheck passes.

To release

h5p-author 0.7.0: two public members on ContentGenerator (moved, shortened) and a new export. Then the build page's pull request, and npx skills update -g for installed copies of the skill.

The other 23 listed types (added with b20ed8d)

The same test on the 23 content types the first run left out, two runs each: 46 first replies, each built package checked against what its task asked for.

Right at onceBy round 2By round 3By round 4Built, not what was askedNever built
0.6.016 of 4629343592
This PR's problems16 of 4629343673
This PR's problems, the type files handed back with the first ones16 of 4642434330
The type's file before the first reply (all 28 types, one run each)28 of 2800

So this PR is for a type's levels: a book, slides, a page, a scenario. On the other types it makes no difference, because there a weak model does not know the field names (options, sequenceImages, questionnaireElements) and a problem tells it one level's names a round. The type's file is what works for every type, which is what the build page hands back once this is released. Every package built in the last two rows plays (h5p-verify).

One misleading message of this PR's own showed up in that run and is fixed in b20ed8d: a Game Map source that wrote map for the picture was told it is a field of behaviour, whose map holds labels and fog. An object is now moved only into a group it writes a field of, not one of its interface texts (3b82230, cubic's finding on b20ed8d).


Summary by cubic

Fixes the h5p-author build's problems so they say what to write, which lets a chat that has only the skill's text build content it could not before.

Bug Fixes

  • Sub-content in the place of the group that holds it, a field a level above its group, and a library's params written without their sub-content are each reported once, with what to write, then checked in the same build, named by where the source has them.
  • An object moves to a group only where it writes one of the group's fields, so a Game Map's map written for the picture is not sent to behaviour.map, which holds the map's labels and fog.
  • A list the generator refused an item of is no longer counted; a field that takes several libraries names them all; a problem's field list names those an author writes; a scenario's question with no alternatives is not reported as a loop besides.
  • loadLibraries reads a field's options as the generator does, a name or { name }, so it loads the same one library the generator reads as a field's only one.

New Features

  • prepack copies the skill's type files into the package as references/types/<Type>.md, each an export, for the build page to hand back with a source's problems.
  • The skill's text now names where a Column, a slide and a book hold their sub-content, and gives the type files' address on GitHub.