Posts on Well-Shaped Words

Learn how to build a DuckLake, snapshot by snapshot

Thu, 27 Nov 2025 00:00:19 -0300

DuckDB is a democratizing influence on data analysis and engineering. Because its creators obviously care about both design and performance, the tool gives more people a chance to do complex data-processing tasks without the mental, financial, and computational overhead that would come with doing such jobs on a cloud platform.

As I get better at using DuckDB, I also become more attentive to the nuances of how data is represented in various formats and schemas. With better conceptual understanding, I get more ideas of how far I can go by using DuckDB as the main (if not only) tool in my stack. So, when the DuckDB team announced its new DuckLake format, I looked forward to learning a new mode of working with datasets.

This tutorial distills what I learned. I’m starting from a fresh slate, so I proceed methodically, taking note of how each component changes with each data operation.

Short conceptual background

Here’s a little bit about Ducklake and the source data that I’m using. Skip to procedure.

The components

I assume that the client and catalog coordinate to ensure the user reads and writes to the correct files.

Ducklake has three components to think about: the client, the catalog, and the data files.

The client. This is the SQL interface to model and query data.
The catalog. A transactional database that tracks metadata. Catalog data includes information about schema versions, data changes, and the correspondence between data files and their tables.
Storage. The data files that store all the rows of the DuckLake tables. In practice these are parquet files on a file system or in an object store.

When the client makes a statement that updates the schema, things change in the catalog. Any time the client adds or modifies data, things change in both the catalog and the data files. In this tutorial I try to highlight the corresponding changes that happen in each component with each modification.

The model

To be somewhat realistic, I chose a dataset of SEC filings. The SEC compiles all the statements they receive in a quarter in 4 related tables.

For this demo, I use only the num table, which records the value of transactions, and the sub table, which records information about submission and the company that made it.

Procedure

My configuration is as follows:

Client: DuckDB
Catalog: Sqlite
Storage: Files on local computer.

There are many catalog DBs to choose from. I originally used a DuckDB database for the catalog, but it can handle only one client connection. That became annoying surprisingly quickly, so I chose SQLite for a local, multi-client experience.

For a production DuckLake, Object storage is a more likely choice than my local disk.

Prerequisites

First, set up the structure for the source files.

In a fresh directory, download one of the quarterly zip files—I use 2025q1. This zip is over 100MB, so downloading may take a moment.
Create a directory called sources.
Unzip the zip to the sources directory.
```
unzip 2025q1.zip -d sources/
```

The file set up looks like this:

.
├── 2025q1.zip
└── sources
├── num.txt
├── pre.txt
├── readme.htm
├── sub.txt
└── tag.txt

Then install DuckDB and the relevant extensions:

Install DuckDB if you haven’t already.
Open a DuckDB client and install the DuckLake and SQLite extensions.
```
INSTALL ducklake;
INSTALL sqlite;
```

Attach the database

Attach the client to a new catalog database and specify the path to the data files:

ATTACH IF NOT EXISTS 'ducklake:sqlite:metadata.sqlite' AS ducklake_edgar
(DATA_PATH 'lake_files')

This creates a new catalog database file at metadata.sqlite.

├── 2025q1.zip
├── metadata.sqlite <-- catalog
└── sources
├── num.txt
├── pre.txt
├── readme.htm
├── sub.txt
└── tag.txt

The catalog initializes with all the DuckLake metadata tables. Some tables, like ducklake_snapshot and ducklake_schema, already have rows.

To query metadata tables, use the identifier __ducklake_metadata_<DUCKLAKE_ALIAS>.<TABLE>. For example, to query the ducklake_snapshot table:


FROM __ducklake_metadata_ducklake_edgar.ducklake_snapshot;

Appropriately, this “empty” snapshot is “Snapshot 0”:

┌─────────────┬───────────────────────────────┬────────────────┬─────────────────┬──────────────┐
│ snapshot_id │ snapshot_time │ schema_version │ next_catalog_id │ next_file_id │
│ int64 │ timestamp with time zone │ int64 │ int64 │ int64 │
├─────────────┼───────────────────────────────┼────────────────┼─────────────────┼──────────────┤
│ 0 │ 2025-11-22 10:40:04.206912-03 │ 0 │ 1 │ 0 │

Create a schema

To make a logical grouping of tables and relationships, create a schema.

USE ducklake_edgar;
CREATE SCHEMA IF NOT EXISTS fin_statements;
USE fin_statements;

Query the ducklake_snapshot_changes see if anything changed:

SELECT snapshot_id, changes_made
FROM __ducklake_metadata_ducklake_edgar.ducklake_snapshot_changes;

┌─────────────┬─────────────────────────────────┐
│ snapshot_id │ changes_made │
│ int64 │ varchar │
├─────────────┼─────────────────────────────────┤
│ 0 │ created_schema:"main" │
│ 1 │ created_schema:"fin_statements" │
└─────────────┴─────────────────────────────────┘

More information about the schema is in the ducklake_schema table.

Create tables

Now create the tables for the schema. These tables correspond to the num and sub tables from the source dataset and follow their model. To track data lineage, I also add loaded_at and source_file columns.

Notice also that I wrap the CREATE TABLE in BEGIN and COMMIT statements. Each snapshot is tied to a database transaction. So it’s all or nothing: a new snapshot is created only if the new tables are created, and each new table is created only if all the others are created too.

BEGIN;
CREATE TABLE IF NOT EXISTS submissions (
adsh VARCHAR,
cik INTEGER,
name VARCHAR,
sic SMALLINT,
countryba VARCHAR,
stprba VARCHAR,
cityba VARCHAR,
zipba VARCHAR,
bas1 VARCHAR,
bas2 VARCHAR,
baph VARCHAR,
countryma varchar,
stprma VARCHAR,
cityma VARCHAR,
zipma VARCHAR,
mas1 VARCHAR,
mas2 VARCHAR,
countryinc VARCHAR,
ein INTEGER,
former VARCHAR,
changed VARCHAR,
afs VARCHAR,
wksi BOOLEAN,
fye VARCHAR,
form VARCHAR,
period DATE,
fy SMALLINT,
fp VARCHAR,
filed DATE,
accepted DATETIME,
prevrpt BOOLEAN,
detail BOOLEAN,
instance VARCHAR,
nciks SMALLINT,
aciks VARCHAR,
source_file VARCHAR,
loaded_at TIMESTAMP
);
CREATE TABLE IF NOT EXISTS numeric_facts (
adsh VARCHAR,
tag VARCHAR,
version VARCHAR,
ddate DATE,
qtrs INTEGER,
uom VARCHAR,
segments VARCHAR,
coreg VARCHAR,
value DECIMAL(28,4),
footnote VARCHAR,
source_file VARCHAR,
loaded_at TIMESTAMP
);
CALL ducklake_edgar.set_commit_message('Matt Dodson', 'Creating the tables', extra_info => 'https://www.sec.gov/files/financial-statement-data-sets.pdf');
COMMIT;

I also use a special DuckLake function to add a commit message. To read this message and a summary of changes from one snapshot to the next, query the ducklake_snapshot_changes table:

FROM __ducklake_metadata_ducklake_edgar
.ducklake_snapshot_changes;


┌─────────────┬──────────────────────┬─────────────┬─────────────────────┬────────────────────────────────────────┐
│ snapshot_id │ changes_made │ author │ commit_message │ commit_extra_info │
│ int64 │ varchar │ varchar │ varchar │ varchar │
├─────────────┼──────────────────────┼─────────────┼─────────────────────┼────────────────────────────────────────┤
│ 0 │ created_schema:"ma… │ NULL │ NULL │ NULL │
│ 1 │ created_schema:"fi… │ NULL │ NULL │ NULL │
│ 2 │ created_table:"fin… │ Matt Dodson │ Creating the tables │ https://www.sec.gov/files/financial-… │
└─────────────┴──────────────────────┴─────────────┴─────────────────────┴────────────────────────────────────────┘

Besides the changes in the snapshot table, the catalog database now has data about the new tables. Query the ducklake_table table to see it:

SELECT table_id, begin_snapshot, table_name, path
FROM __ducklake_metadata_ducklake_edgar
.ducklake_table;

┌──────────┬────────────────┬───────────────┬────────────────┐
│ table_id │ begin_snapshot │ table_name │ path │
│ int64 │ int64 │ varchar │ varchar │
├──────────┼────────────────┼───────────────┼────────────────┤
│ 2 │ 2 │ numeric_facts │ numeric_facts/ │
│ 3 │ 2 │ submissions │ submissions/ │
└──────────┴────────────────┴───────────────┴────────────────┘

Notice that these tables also have a path. This path tracks where the data files for the table are in storage. But no files exist yet, because no data has been loaded into the tables.

Load data

Now load data from the source TXT files.

At the time of writing, Ducklake does not support primary keys. To avoid duplicate data, use the MERGE INTO statement to upsert data.

To explain the logic, here’s an annotated snippet of how the transformation into numeric_facts works:


MERGE INTO numeric_facts
USING (
SELECT
adsh::VARCHAR AS adsh,
tag::VARCHAR AS tag,
version::VARCHAR AS version,
STRPTIME(ddate::VARCHAR, '%Y%m%d')::DATE AS ddate,
qtrs::INTEGER AS qtrs,
uom::VARCHAR AS uom,
segments::VARCHAR AS segments,
coreg::VARCHAR AS coreg,
value::DECIMAL(28,4) AS value,
footnote::VARCHAR AS footnote,
'2025q1.zip'::VARCHAR AS source_file
FROM read_csv('sources/num.txt', delim='\t', header=true)
) AS source
ON numeric_facts.adsh = source.adsh
AND numeric_facts.tag = source.tag
AND numeric_facts.version = source.version
AND numeric_facts.ddate = source.ddate
AND numeric_facts.qtrs = source.qtrs
AND numeric_facts.uom = source.uom
AND COALESCE(numeric_facts.segments, '') = COALESCE(source.segments, '')
AND COALESCE(numeric_facts.coreg, '') = COALESCE(source.coreg, '')
WHEN MATCHED THEN DO NOTHING
WHEN NOT MATCHED THEN INSERT VALUES (
source.adsh, source.tag, source.version, source.ddate,
source.qtrs, source.uom, source.segments, source.coreg,
source.value, source.footnote,
source.source_file, CURRENT_TIMESTAMP
);

Merge the table numeric facts.
SELECT from the source file with some light transformation to validate data types.
Use the table’s composite key to check that the row does not already exist.
When checking, coalesce nulls to avoid inserting duplicates (I learned from experience).
When the keys don’t match, insert a new record into the table.

Here’s the full load statement.

BEGIN;
MERGE INTO submissions
USING (
SELECT
adsh::VARCHAR AS adsh,
cik::INTEGER AS cik,
name::VARCHAR AS name,
sic::SMALLINT AS sic,
countryba::VARCHAR AS countryba,
stprba::VARCHAR AS stprba,
cityba::VARCHAR AS cityba,
zipba::VARCHAR AS zipba,
bas1::VARCHAR AS bas1,
bas2::VARCHAR AS bas2,
baph::VARCHAR AS baph,
countryma::VARCHAR AS countryma,
stprma::VARCHAR AS stprma,
cityma::VARCHAR AS cityma,
zipma::VARCHAR AS zipma,
mas1::VARCHAR AS mas1,
mas2::VARCHAR AS mas2,
countryinc::VARCHAR AS countryinc,
ein::INTEGER AS ein,
former::VARCHAR AS former,
changed::VARCHAR AS changed,
afs::VARCHAR AS afs,
wksi::BOOLEAN AS wksi,
fye::VARCHAR AS fye,
form::VARCHAR AS form,
STRPTIME(period::VARCHAR, '%Y%m%d')::DATE AS period,
fy::SMALLINT AS fy,
fp::VARCHAR AS fp,
STRPTIME(filed::VARCHAR, '%Y%m%d')::DATE AS filed,
accepted::DATETIME AS accepted,
prevrpt::BOOLEAN AS prevrpt,
detail::BOOLEAN AS detail,
instance::VARCHAR AS instance,
nciks::SMALLINT AS nciks,
aciks::VARCHAR AS aciks,
'2025q1.zip'::VARCHAR AS source_file
FROM read_csv('sources/sub.txt', delim='\t', header=true)
) AS source
ON submissions.adsh = source.adsh
WHEN MATCHED THEN DO NOTHING
WHEN NOT MATCHED THEN INSERT VALUES (
source.adsh, source.cik, source.name, source.sic,
source.countryba, source.stprba, source.cityba, source.zipba,
source.bas1, source.bas2, source.baph,
source.countryma, source.stprma, source.cityma, source.zipma,
source.mas1, source.mas2, source.countryinc, source.ein,
source.former, source.changed, source.afs, source.wksi,
source.fye, source.form, source.period, source.fy, source.fp,
source.filed, source.accepted, source.prevrpt, source.detail,
source.instance, source.nciks, source.aciks,
source.source_file, CURRENT_TIMESTAMP
);
-- Numeric facts
MERGE INTO numeric_facts
USING (
SELECT
adsh::VARCHAR AS adsh,
tag::VARCHAR AS tag,
version::VARCHAR AS version,
STRPTIME(ddate::VARCHAR, '%Y%m%d')::DATE AS ddate,
qtrs::INTEGER AS qtrs,
uom::VARCHAR AS uom,
segments::VARCHAR AS segments,
coreg::VARCHAR AS coreg,
value::DECIMAL(28,4) AS value,
footnote::VARCHAR AS footnote,
'2025q1.zip'::VARCHAR AS source_file
FROM read_csv('sources/num.txt', delim='\t', header=true)
) AS source
ON numeric_facts.adsh = source.adsh
AND numeric_facts.tag = source.tag
AND numeric_facts.version = source.version
AND numeric_facts.ddate = source.ddate
AND numeric_facts.qtrs = source.qtrs
AND numeric_facts.uom = source.uom
AND COALESCE(numeric_facts.segments, 'none') = COALESCE(source.segments, 'none')
AND COALESCE(numeric_facts.coreg, 'none') = COALESCE(source.coreg, 'none')
WHEN MATCHED THEN DO NOTHING
WHEN NOT MATCHED THEN INSERT VALUES (
source.adsh, source.tag, source.version, source.ddate,
source.qtrs, source.uom, source.segments, source.coreg,
source.value, source.footnote,
source.source_file, CURRENT_TIMESTAMP
);
CALL ducklake_edgar.set_commit_message('Matt Dodson', 'Loaded data from 2025q1.zip');
COMMIT;

Now the tables have data. The ducklake_table_stats table provides a summary:

FROM __ducklake_metadata_ducklake_edgar
.ducklake_table_stats;

┌──────────┬──────────────┬─────────────┬─────────────────┐
│ table_id │ record_count │ next_row_id │ file_size_bytes │
│ int64 │ int64 │ int64 │ int64 │
├──────────┼──────────────┼─────────────┼─────────────────┤
│ 2 │ 3658551 │ 3658551 │ 118205163 │
│ 3 │ 6231 │ 6231 │ 826336 │
└──────────┴──────────────┴─────────────┴─────────────────┘

Loading data also created the data path, lake_files, as a directory. And this path has parquet files:

.
├── lake_files
│   └── fin_statements
│   ├── numeric_facts
│   │   └── ducklake-019abaa3-3089-7fd2-ad3d-fbe1e49c555a.parquet
│   └── submissions
│   └── ducklake-019abaa3-2feb-7b74-9ea0-7e7f3ed49182.parquet

These parquet files are where the data exists. Query the metadata about them through the ducklake_data_file table:

SELECT data_file_id, table_id, file_size_bytes, path, begin_snapshot, end_snapshot
FROM __ducklake_metadata_ducklake_edgar
.ducklake_data_file;

┌──────────────┬──────────┬─────────────────┬───────────────────────────────────────────────────────┬────────────────┬──────────────┐
│ data_file_id │ table_id │ file_size_bytes │ path │ begin_snapshot │ end_snapshot │
│ int64 │ int64 │ int64 │ varchar │ int64 │ int64 │
├──────────────┼──────────┼─────────────────┼───────────────────────────────────────────────────────┼────────────────┼──────────────┤
│ 0 │ 2 │ 118205163 │ ducklake-019abaa3-3089-7fd2-ad3d-fbe1e49c555a.parquet │ 3 │ NULL │
│ 1 │ 3 │ 826336 │ ducklake-019abaa3-2feb-7b74-9ea0-7e7f3ed49182.parquet │ 3 │ NULL │
└──────────────┴──────────┴─────────────────┴───────────────────────────────────────────────────────┴────────────────┴──────────────┘

Time to query the actual data.

Query

Query the top three firms that reported the most profit on their annual report for 2024-12-31 (and who filed an annual report in this batch of SEC data).

This query joins the two tables to relate the numeric entries with the information about the company that submitted the entry.

SELECT name, value / 1000000 "net_income_millions", ddate period_end, filed date_filed
FROM numeric_facts num
INNER JOIN submissions sub ON num.adsh = sub.adsh
WHERE tag = 'NetIncomeLoss'
AND form='10-K'
AND uom='USD' AND segments IS NULL
AND ddate='2024-12-31'
ORDER BY net_income_millions DESC
LIMIT 3;


┌────────────────────────┬─────────────────────┬────────────┬────────────┐
│ name │ net_income_millions │ period_end │ date_filed │
│ varchar │ double │ date │ date │
├────────────────────────┼─────────────────────┼────────────┼────────────┤
│ ALPHABET INC. │ 100118.0 │ 2024-12-31 │ 2025-02-05 │
│ BERKSHIRE HATHAWAY INC │ 88995.0 │ 2024-12-31 │ 2025-02-24 │
│ META PLATFORMS, INC. │ 62360.0 │ 2024-12-31 │ 2025-01-30 │
└────────────────────────┴─────────────────────┴────────────┴────────────┘

The actual process of working with and cleaning this dataset is a topic of its own. In short, firms have different fiscal years and apparently have full discretion over how they tag each financial statement. Take care when comparing data across firms and submissions.

Anyway, this article is about DuckLake, so let’s get back to the metadata.

Delete

Intel has performed quite poorly and frankly I’d prefer to think it never existed.

Let’s delete all Intel entries from the numeric_facts table:

BEGIN;
DELETE FROM numeric_facts
USING submissions sub
WHERE numeric_facts.adsh = sub.adsh
AND sub.name ILIKE '%INTEL%';
CALL ducklake_edgar.set_commit_message('Matt Dodson', 'Getting rid of Intel data');
COMMIT;

To confirm:

SELECT count(*) FROM numeric_facts num
INNER JOIN submissions sub
ON sub.adsh = num.adsh
WHERE sub.name ILIKE '%INTEL%';

┌──────────────┐
│ count_star() │
│ int64 │
├──────────────┤
│ 0 │
└──────────────┘

This delete creates a new snapshot:

SELECT snapshot_id, changes_made, commit_message
FROM __ducklake_metadata_ducklake_edgar
.ducklake_snapshot_changes
ORDER BY snapshot_id DESC
LIMIT 1;

┌─────────────┬──────────────────────┬───────────────────────────┐
│ snapshot_id │ changes_made │ commit_message │
│ int64 │ varchar │ varchar │
├─────────────┼──────────────────────┼───────────────────────────┤
│ 4 │ deleted_from_table:2 │ Getting rid of Intel data │
└─────────────┴──────────────────────┴───────────────────────────┘

Interesting. The delete also creates a new file in storage.

lake_files/fin_statements/numeric_facts
├── ducklake-019abaa3-3089-7fd2-ad3d-fbe1e49c555a.parquet
└── ducklake-019abac9-2488-7301-be20-09fd7b27f0d6-delete.parquet

For a record of these delete files, query the ducklake_delete_file table:

SELECT delete_file_id, table_id, delete_count
FROM __ducklake_metadata_ducklake_edgar
.ducklake_delete_file;

┌────────────────┬──────────┬──────────────┐
│ delete_file_id │ table_id │ delete_count │
│ int64 │ int64 │ int64 │
├────────────────┼──────────┼──────────────┤
│ 2 │ 2 │ 4885 │
└────────────────┴──────────┴──────────────┘

In summary, all data changes are tracked at each snapshot. This might be useful.

Time travel

As it turns out, my delete operation was overzealous. With my sloppy WHERE clause, I deleted not only the records for Intel but also all records for all companies with something about “intelligence” in their name. Besides, I discovered that deleting unpleasant data does not necessarily delete unpleasant emotions.

Fortunately, DuckLake snapshots preserve the data changes. Using the time travel feature, I can query old snapshots.

From the snapshot_changes table, I see that the delete happened in Snapshot 4. To get the data before the delete, query with AT (VERSION => 3):

SELECT count(*) FROM numeric_facts num AT (VERSION => 3)
INNER JOIN submissions sub
ON sub.adsh = num.adsh
WHERE sub.name ILIKE '%INTEL%';

┌──────────────┐
│ count_star() │
│ int64 │
├──────────────┤
│ 4885 │
└──────────────┘

You can review all changes from one snapshot to the other with the Data change feed function.

Recover

Here’s a short way to restore numeric_facts to the table from an old snapshot (thanks to a user on the DuckDB discord for the tip).

BEGIN;
CREATE OR REPLACE TABLE numeric_facts AS (FROM numeric_facts AT (version => 3));
CALL ducklake_edgar.set_commit_message('Matt Dodson', 'Going back to the way things were');
COMMIT;

Find old data files

With all the changes, some of my data files are no longer needed. You can expect this by checking the ducklake_data_file.

For example, the data in file 0 was relevant between snapshots 3 and 5:

FROM __ducklake_metadata_ducklake_edgar.ducklake_data_file
WHERE end_snapshot IS NOT NULL;

┌──────────────┬──────────┬────────────────┬──────────────┬
│ data_file_id │ table_id │ begin_snapshot │ end_snapshot │
│ int64 │ int64 │ int64 │ int64 │
├──────────────┼──────────┼────────────────┼──────────────┤
│ 0 │ 2 │ 3 │ 5 │
└──────────────┴──────────┴────────────────┴──────────────┘

Maintain

One thing that surprised me at first was that until the user takes action, the number of total data files grows with almost every write operation. Besides delete files, inserting more data into a table adds a data file (and the catalog tracks how the rows start and end across files)

Now I understand that this write-only approach is the normal way of working with parquet and data lakes. Besides preserving history and read and write performance, object storage is cheap.

Nevertheless, the structure of the data files can weigh on performance and costs. The DuckDB extension has a number of maintenance functions to manage this.

Final thoughts

I enjoyed learning about DuckLake. Even if I just continue to use it locally, the catalog and multi-client experience make it a nice alternative to a “classic” DuckDB database file.

With no implementation experience and only a vague idea of how data lakes and catalogs work in general, it took me some time to wrap my head around how DuckLake works as an application and data architecture. I feel I have a solid intuition of how things work, and I hope this tutorial has helped you feel the same way.

Ideas to go further

To learn more about how DuckLake works, try these operations and explore how the changes manifest in the catalog and storage.

Load more data
Document your tables with comments
Do an UPDATE (check for snapshot expirations)
Alter a table (what metadata tables will change?)
Change the config

For ideas about using Ducklake in the wider data engineering ecosystem, you can try these:

Change the storage to S3 or something similar
Go further with the downstream transformations, building marts and views that can actually serve for financial screening and analysis
Incorporate transformation tools like dbt or orchestration tools like Dagster to build full data pipelines

Good luck!

Make software tutorials that stick

Sat, 09 Aug 2025 12:51:21 -0300

Learning is deeper and more durable when it’s effortful.
—Make It Stick

This year I developed a course on an abstruse standard about manufacturing systems. To prepare for the work, I read Make It Stick: The Science of Successful Learning, an accessible and practical book by cognitive scientists that synthesizes various studies in memory and learning.

While designing a course isn’t the same as writing documentation, the principles overlap. Tutorials, in particular, help people learn through practice.

Many of the principles from Make It Stick, then, also apply to making more effective tutorials. Here are the tips I gleaned to design a better learning experience. All quoted text is from the book.

Start with context

Start every tutorial with a small narrative that frames a plausible problem that can be solved by applying a new skill. As the tutorial progresses, your learner should actively participate in the construction of a solution.

Putting new knowledge into a larger context helps learning.

Skills and information are easier to recall if the learner can position what they’ve learned into a wider body of knowledge or memorable narrative. Plus, if the example is plausible and common, the narrative serves as a training simulation for the problems that the learner will meet in real-world contexts.

Have them try first

Before demonstrating solutions, give learners a chance to guess. In practice, you could do something as simple as starting a section with a true-or-false or multiple-choice question. Of course, with a little JavaScript, the possibilities get much more interactive.

Trying to solve a problem before being taught the solution leads to better learning, even when errors are made in the attempt.

Guessing makes learners think about the problem and primes them to pay attention to the answer. And as long as you have some mechanism to give feedback, an incorrect guess isn’t a bad thing: it’s an opportunity.

Making mistakes and correcting them builds the bridges to advanced learning.

Weave different tasks into the overall story

A long tutorial likely incorporates more than one task. Rather than order the tasks in sequence, interleave them.

When you’re adept at extracting the underlying principles or “rules” that differentiate types of problems, you’re more successful at picking the right solutions in unfamiliar situations.

Sequential vs. braided task sequencing. With interleaved tasks, spaced repetition comes for free.

For example, an introduction to SQL likely covers not only the SELECT statement, but also UPDATE, CREATE, and so on. The conventional approach proceeds sequentially, exhausting all there is to say about one topic before turning to another. The more effective approach weaves these topics together. For example, the tutorial might teach learners first how to select a table, then update it, and then create a new table before finally rotating back to SELECT to query the new data.

Learning is more effective when the learner has a chance practice working on the problem domain from various angles. This strategy also has the added effect of providing spaced recall.

Reincorporate old information

As the tutorial progresses, revisit topics that you introduced earlier.

Retrieval practice—recalling facts or concepts or events from memory—is a more effective learning strategy than review by rereading.

Even if the tutorial doesn’t last long, a little bit of spaced repetition can help the learners retain important aspects of it their long-term memory. Recall is a critical part of applying skills, but you have to stop thinking about something in the first place to recall it later.

Together with the interleaving technique, spacing information out helps the learner adapt the tutorial to their problem.

if you space out practice at a task and get a little rusty between sessions, or you interleave the practice of two or more subjects,retrieval is harder and feels less productive, but the effort produces longer lasting learning and enables more versatile application of it in later settings.

Build a mock test at the end

A small assessment at the end can help reinforce the concepts. This is the second layer of retrieval practice.

In virtually all areas of learning, you build better mastery when you use testing as a tool to identify weaknesses and bring up your areas of weakness.

Testing in this case is not about trying to assess whether the learner knows something; it’s about helping learners remember and avoid a false sense of mastery.

Ask the user to elaborate

After giving them a chance to practice, ask the learners some questions to get them thinking about how the information applies to their wider body of information.

If you practice elaboration, there’s no known limit to how much you can learn. Elaboration is the process of giving new material meaning by expressing it in your own words and connecting it with what you already know.

Presumably your learners are taking the tutorial because they think the skill that it imparts might apply to some broader problem in their lives. Ask them to think about what that application is.

You could even add an input text field for answers. Even if no one reads their answers, the very act of writing to elaborate topic helps them assimilate it into their wider body of knowledge.

For learners and designers, more effort brings better results (and more effort)

A skill takes time to learn, and time is our most precious resource. Things that are easy-to-learn are appealing objects of our attention. Ironically, though, what’s easiest to learn is hardest to remember. Durable skills are hard-won.

However, designers of learning experiences can use these techniques to diminish the tradeoff between ease and memorability. Efficiently designed tutorials can be both inviting to the learner and challenging enough to help them retain the information. Of course, I don’t deny that designing tutorials in this way requires more effort on the part of the writer as well. But all things excellent are as difficult as they are rare.

The need to understand that when learning is hard, you’re doing important work. To understand that striving and setbacks, as in any action video game or new BMX bike stunt, are essential if you are to surpass your current level of performance toward true expertise.

Shell tools for writers

Sun, 02 Mar 2025 16:11:17 -0300

The command line is a text-based interface, and writing is a text-based activity. Shell tools, therefore, are writing tools. Here are some examples that have served me.

Find and replace

Replace all instances of an old phrase within a set of MD files. This works best on a Git repo so that you can review the diffs before committing any changes.

old_term="relref \"/"
new_term="ref \"/"
find . -name "*.md" -exec sed -i "s/$old_term/$new_term/g" {} +

Count and sort frequency across files

Some changes are too contextual for global search and replace. If the fixes are made per page and incrementally, you can maximize early impact by starting on the pages that require the most changes.

This grep | sort | awk pipeline:

Searches for a term.
Sorts the list by frequency.
Filters out files with 0 occurrences.

grep "relref \"/" -irc content/ \
| sort -t : -k 2 --reverse \
| awk -F : '$2 != 0'

Sort all errors by frequency

Very close in form to the preceding, this sorts all errors in a log file. With over 200 errors to fix, this ensures time isn’t wasted having to edit the same page multiple times to fix errors.

First, direct stderr to a file:

hugo server 2> relref-errors.log

Then sort the file by the third field.

sort -t : --key=3 relref-errors.log

However it was pretty messy to read.

So rather than filtering I just used awk to make a CSV and then used duckdb to sort.

echo "file,location,ref" > tmp.csv \
&& awk -F: '{print $3"\""","$4":"$5","$2}' relref-errors.log >> tmp.csv

Filter JSON array of links checked

The Linkinator JSON format outputs a status report of every link on a site.

Most links work and some of the broken ones stem from 500 or 429 errors, which don’t reflect link quality. So I can use jq to filter only the links that have a status of 404

linkinator --recurse --format JSON \
"https://docs.rhize.com" \
| jq '.links[] | select(.status == 404)'

Round a float to decimal places with Hugo Shortcodes

Sat, 28 Dec 2024 17:39:24 -0300

The built-in Go math.Round function rounds to integers. So, to round a number to decimals using Hugo, you need to write a small shortcode.

The following examples all use the technique described in Round float to any precision in Go.

A “conventional” solution uses a shortcode that is available to all pages.
The inline examples create a rounding function with define.

Shortcode to round floats

This shortcode is defined in layouts/shortcodes/roundFlt.html. It takes two arguments:

The number to round
The decimal precision (default 2)

{{ $num := (.Get 0 | float) }}
{{ $precision := default 2 (.Get 1 ) }}
{{ $ratio := math.Pow 10 $precision }}
{{ $product := math.Product $ratio $num }}
{{ div (math.Round $product) $ratio }}

Example call:


- 2 decimal places: {{< roundFlt 3.141592653589793238 >}}
- 5 decimal places: {{< roundFlt 3.141592653589793238 5 >}}

Renders as:

2 decimal places (default): 3.14

5 decimal places: 3.14159

Inline shortcodes scoped to one block

If you want to scope the shortcode to a page, you can use inline shortcodes with the same logic from the preceding section. However, inline shortcodes cannot have nested shortcodes. If you want to use the rounding function within more shortcode logic, you can use the template and define functions.

Besides showing another technique, I include these because inline shortcodes have interesting use cases for pages that involve numerical analysis and presentation—for examples, read Hugo inline spreadsheets.

Fixed precision

The least complex method keeps precision fixed to a certain number—in this case, 4.

{{ $precision := 4 }}
{{ define "rnd" }}
{{ $ratio := math.Pow 10 4 }}
{{ $product := math.Product $ratio . }}
{{ div (math.Round $product) $ratio }}
{{ end }}
Pi rounded to {{ $precision }} digits:
{{ template "rnd" math.Pi }}

Renders as:

Pi rounded to 4 digits:

3.1416

Variable precision

If you want variable precision, you can put the value and precision in a dict. This is more verbose than the other examples.


{{ define "rnd" }}
{{ $num := .val }}
{{ $pow := default 2 .precision }}
{{ $ratio := math.Pow 10 $pow }}
{{ $product := math.Product $ratio $num }}
{{- div (math.Round $product) $ratio }}
{{ end }}
Pi to 5 digits:
{{- template "rnd" (dict "val" math.Pi "precision" 5 ) }}

Renders as

Pi to 5 digits:

3.14159

Hugo inline spreadsheet

Fri, 20 Dec 2024 16:00:42 -0300

With Hugo inline shortcodes, you calculate values and build complex, dynamic tables without ever leaving the Markdown page. I discovered this technique when I wrote my post to test the accuracy of a sample of ChatGPT–generated values. Rather than copy my calculations into the page, I realized I could use variables and Hugo built-in Math functions to keep the writing and computation in a single place.

I’ve written before about inline shortcodes, but this use case might be the most interesting one I’ve found. As an authoring experience, I find it intuitive to calculate values directly beside the accompanying prose. And if these calculations or data presentations are unique to one page, separation of concerns is no concern of mine.

This is something akin to literate programming. Surely this technique won’t replace Jupyter notebooks, but it’s an interesting use case for Hugo pages that involve numerical research or demonstration.

Here are some examples.

Examples of Inline calculation

All the numbers in the following sections were calculated from an inline shortcode in the page. Open the accordion after the table to read the code.

Calculate summary statistics

Consider the following array of numbers: [100.12 140.2 134 150 102].

I can use an inline shortcode to calculate summary statistics and populate a Markdown table.

statistic	value
sum	749.53
mean	124.92
variance	348.04
std	18.66

{{ $cost := slice 100.12 140.2 134.0 150.0 102.0 }}
{{ $cost := slice 100.12 140.2 134.0 150.0 102.0 123.21 }}
{{ $sum := math.Sum $cost }}
{{ $mean := div $sum (len $cost) }}
{{ $sumdiff := 0.0 }}
{{ range $cost }}
{{ $sumdiff = add $sumdiff (pow (sub . $mean) 2) }}
{{ end }}
{{ $variance := div $sumdiff (len $cost) }}
{{ $std := math.Sqrt $variance }}
| statistic | value |
|-----------|------------------------------------|
| sum | {{ printf "%.2f" $sum }} |
| mean | {{ printf "%.2f" $mean }} |
| variance | {{ printf "%.2f" $variance }} |
| std | {{ printf "%.2f" $std }} |

Use slices to compute a table of dynamic values

This example compares two slices to create a “spreadsheet”. Some conditional formatting changes the CSS of the derived cells whenever the value is less than 0. The table footer gives a summary.

In this case, I use an HTML table rather than a Markdown one. This is to avoid figuring out how to trim whitespace correctly to keep a single continuous Markdown table across a range of generated values.

Cost	Sale Price	Difference
100.12	135.00	34.88
140.20	135.00	-5.20
134.00	135.00	1.00
150.00	200.00	50.00
102.00	129.99	27.99
123.21	115.15	-8.06
Total Profit		100.61


{{ $cost := slice 100.12 140.2 134.0 150.0 102.0 123.21 }}
{{ $price := slice 135.0 135.0 135.0 200.0 129.99 115.15 }}
{{ $differences := slice }}
<table>
<thead>
<tr>
<th>Cost</th>
<th>Sale Price</th>
<th>Difference</th>
</tr>
</thead>
<tbody>
{{ range $index, $costItem := $cost }}
{{ $priceItem := index $price $index }}
{{ $difference := sub $priceItem $costItem }}
{{ $differences = $differences | append $difference }}
<tr>
<td>{{ printf "%.2f" $costItem }}</td>
<td>{{ printf "%.2f" $priceItem }}</td>
<td>{{ if lt $difference 0}}
<span style="color:red"> {{ printf "%.2f" $difference }}</span>
{{else}}
{{ printf "%.2f" $difference }}
</td>
{{end}}
</tr>
{{ end }}
</tbody>
<tfoot>
<tr>
<td colspan="2" style="text-align: right; font-weight: bold;">Total Profit</td>
<td style="font-weight: bold;">{{ printf "%.2f" (math.Sum $differences) }}</td>
</tr>
</tfoot>
</table>

The HTML table could probably be more semantic, but this seems like a good proof of concept.

Embed calculated values in LaTeX formulas

If you use Latex in Hugo, you can substitute the variables in your formulas with variables from an inline shortcode.

The formula for margin of error is as follows: $$ z_{\frac{\alpha}{2}}\cdot\sqrt{\dfrac{ {\hat p}(1-{\hat p})}{n}} $$

In this case, our variables have the following values:

Statistic	Value
Sample size, $$ n $$	91
Sample proportion, $$ {\hat p} $$	0.78
Z critical value, $$z_{\frac{\alpha}{2}}$$	1.96

So our margin of error is calculated as follows:

$$ 1.96 \cdot \sqrt{\dfrac{ 0.78(1-0.78)}{ 91 }} = 0.0851 $$

In this case, variables in the inline shortcode store or calculate all values. So if I change the value of a variable, the values in the table, formula, and derived margin of error all change, too.


The formula for margin of error is as follows:
$$ z_{\frac{\alpha}{2}}\cdot\sqrt{\dfrac{ {\hat p}(1-{\hat p})}{n}} $$
{{ $total := 91.0 }}
{{ $errors := 20.0 }}
{{ $zcrit := 1.96 }}
{{ $correct := (sub $total $errors) }}
{{ $error_rate := (div $errors $total) }}
{{ $correct_rate := (div $correct $total ) }}
{{ $std_err :=
(math.Sqrt
(div (math.Product $error_rate $correct_rate)
($total))
)
}}
{{ $margin_err := math.Product $zcrit $std_err }}
In this case, our variables have the following values:
| Statistic | Value |
|----------------|--------------|
| Sample size, $$ n $$ | {{ $total }} |
| Sample proportion, $$ {\hat p} $$ | {{printf "%.2f" $correct_rate }} |
| [Z critical value](https://www.criticalvaluecalculator.com/blog/understanding-zscore-and-zcritical-value-in-statistics-a-comprehensive-guide), $$z_{\frac{\alpha}{2}}$$ | {{ printf "%.2f" $zcrit }} |
So our calculation is
$$ {{ $zcrit }} \cdot \sqrt{\dfrac{ {{printf "%.2f" $correct_rate}}(1-{{printf "%.2f" $correct_rate}})}{ {{$total}} }} $$
Which gives us a **margin of error of {{printf "%.4f" $margin_err}}**.

Limitations and ideas for development

This is a fun and flexible way to centralize concerns for work that involves numbers, prose, and Hugo.

The main limitation I found was that I had to be careful to make sure all source values were floats, not integers. Otherwise, I got unexpected values. This also means that I used a lot of printf statements to round values to a few decimals. A more robust method to handle these issues would be to pipe the numbers into the cast.ToFloat function and write a shortcode to round to decimals.

If you are doing high-precision calculation, this is probably not viable. Please do not plan works of civil engineering using Hugo inline shortcodes.

For future work, one might experiment with:

Using remote data to work with source data in a CSV or JSON file
Using the Store pad to persist values across different shortcodes in the page—or, alternatively, putting the entire page inside an inline shortcode.

There’s also much room to extend the presentation of the data, perhaps with custom inline JS or CSS.

Is ChatGPT a reliable research assistant? Building a books database

Tue, 17 Dec 2024 17:58:07 -0300

LLMs are good at processing unstructured data, like internet pages and research papers, and at generating structured output, like code and JSON files. The devil is in the generated details. If they aren’t correct, what’s the use?

Testing the correctness of generated data is the goal of my informal experiment, which uses ChatGPT to build a database of information about classic books. The input is long list of books that Harold Bloom considered to comprise the Western Canon. The goal output is a table of these books with additional columns for information that wasn’t present in the initial list.

Let’s ignore the validity of the list itself and consider its worth as source material to test an LLM’s capabilities as a research assistant:

It would be boring to manually add contextual information for every single one of these 1300+ works
Almost all the books in the list have extensive discussion in the wider body of written literature, meaning the model’s training corpus likely includes some basic contextual information about each one.

So I used ChatGPT to build me a books database.

Results summary

Jump to full results | Source files

Though validation errors arose, ChatGPT did reasonably well at following my prompt to create a CSV file using the input values and specified schema. The categorical values, for the enum Form field and freeform Genre field, seem reasonable at a glance, but all I did was glance at them.

To evaluate the quantity of the reported year, I tested a sample of 100 randomly selected rows. The sample accuracy rate was 78%, and I estimate the population accuracy-rate parameter to be between 70% and 87%.

For this specific case, I’d say the LLM worked well enough as a research assistant for my unserious work. With so many potential errors, however, it is unreliable for work with meaningful stakes. Nevertheless, I see ways to improve the setup. And, as an assistant, the bot may be a better fit for helping researchers categorize and filter, rather than delivering high-precision values and facts.

Method

To generate the data, I used the default version of ChatGPT 4o Mini that existed on November 24th, 2024. I did not log in.

To inspect and clean up the data, I used DuckDB to create a table from the generated CSV files.

Goal

The main goal was to test how well ChatGPT could date a list of literary works for me. I also wanted to see how it does in classifying them by Genre and Medium.

Prompt

I refined the prompt in iterations. The added instructions tell the story of all the issues that arose in the prior attempts.

First prompt:

Turn this into a single CSV file in the format
Author, Title, Form (enum: novel, poetry, drama, philosophy/essay), Genre, Year, Country

These first prompts returned many poor-quality and invalid fields. After a few iterations, my prompt was something like this:

Turn this into a single CSV file.
Each book needs its own row
Only enclose fields in " if they contain a comma.
The schema is as follows. Let year be an integer.
If year is indeterminate, leave empty.
All other fields must have values.
Author, Title, Form (enum: novel, poetry, drama, philosophy/essay, story collection), Genre, Year, Country

This worked pretty well, enough that I generated all rows and started checking the accuracy of the Year field from a sample. This exposed a fatal flaw in my experimental design: many of these works, in particular the collections of stories and poetry and the works from antiquity, are impossible to precisely date.

So I came up with my final attempt:


Turn this into a single CSV file.
Each book needs its own row
Only enclose fields in " if they contain a comma.
The schema is as follows:
Author, Title, Form, Genre, Year, Country
Form is an enum with values ['Drama','Novel','Story collection','Philosophy/essay','Poetry','Memoire/Autobiography']
Let year be an integer.
If form is drama, use year it was first performed.
Otherwise, use year first published.
If multiple volumes, use year the first volume was published.
if you cannot determine year, leave blank.
I expect many collections and ancient works to have empty year fields.

At the end of the each prompt, I appended an unedited snippet of Bloom’s list. In the first attempts, I pasted too many items and ChatGPT occasionally hallucinated additional rows. In the final attempt, I prompted about 100 rows at a time.

I made four CSV files, one for each “Age” as classified by Bloom.

Exploration and clean up

As I went through attempts, I used DuckDB to explore the data. Sometimes obvious structural issues surfaced: all titles for a particular author were in one semi-colon-delimited row; a cluster of rows had large swaths of empty fields. When stuff like this happened, I rewrote my prompt and tried again.

For the final table, I added a column to record the literary “age” before joining the CSV files in one table. I updated the final fields to fix minor validation issues and remove whitespace. The most common nuisances were CSV validation errors for columns with commas in their values, such as The Mandrake, A Comedy. I also converted all BC years to negative integers.

Results

I assess how well ChatGPT did by a few criteria:

Did it structure the information as asked?
Did it add information that was accurate?

For the “added information” part, ChatGPT added three new data points for each book¹:

Year
Form
Genre

In the following sections, I’ll discuss how well ChatGPT did at each task.

Semi-structured prose to CSV

ChatGPT ingested a large semi-structured prose list and turned it into a CSV file. This definitely saved some labor. However, as the list already had a regular structure, this task also might have been done with some old-fashioned regex.

The structure of the data mostly conformed to my expectations, though it took some trial and error to validate the CSV output.

A future experiment might try to extract data from completely unstructured prose into a structured table of facts.

Quality of output for `form` field

Most of the time, ChatGPT reported the Form according to one of the values specified in the prompt.

However, it also repeatedly added new values to some works, which was against instructions and created invalid table inserts. In some cases, I might have preferred ChatGPT’s suggestion—I especially wish I included the value history—but this editorialization went outside of the prompt.

From 1300 books, 228 genres

Unlike the Form field, which was explicitly an enum, the Genre field could take any value.

ChatGPT created 228 genres. Only 118 genres were used for more than one title, and only 26 genres were used for more than 10 titles. This distribution makes some thin categories, but I do like the specificity of some of the singleton genres I saw, such as “Andalusian folk poetry.”

select genre, count(genre)
from books
GROUP BY all
order by count(genre) desc;

Genre	count(genre)
Modernist fiction	213
Modernist poetry	191
Tragedy	78
Comedy	50
Realist novel	49
Contemporary poetry	45
Historical fiction	41
Lyric poetry	37
Romantic poetry	31
Literary criticism	31
Postmodern fiction	31
Short stories	24
Modernist drama	24
Epic	24
Realist fiction	23
Philosophy	22
Romantic novel	20
Satire	17
Autobiography	15
Symbolist poetry	13
Drama	13
Historical novel	13
Fantasy	13
Modernist novel	12
Social novel	12
Magical realism	12
Adventure novel	9
Victorian poetry	9
Victorian novel	9
Poetry	9
Surrealist poetry	9
Gothic novel	9
Essays	8
Political fiction	7
Science fiction	7
Fantasy fiction	7
Romance	7
Epic poetry	7
Satirical fiction	7
Philosophical novel	7
Contemporary drama	7
Absurdist play	6
Satirical novel	6
Philosophical fiction	6
Historical drama	5
Epistolary novel	5
Political philosophy	4
Pre-Raphaelite poetry	4
Folk tales	4
Adventure fiction	4
Yiddish poetry	4
Religious text	4
Philosophical essay	4
Elegy	4
Historical writing	4
Absurdist theatre	4
Biography	4
Psychological fiction	4
Yiddish literature	4
Picaresque	4
Baroque poetry	4
Jewish fiction	4
American poetry	4
Religious poetry	3
Novel	3
Comic novel	3
Experimental fiction	3
Art criticism	3
Autobiographical novel	3
Autobiographical fiction	3
Mystical poetry	3
War poetry	3
Memoir	3
Ethics	3
Fiction	3
Neoclassical poetry	3
Existentialist fiction	3
Romantic	3
History	3
Religious philosophy	2
Bildungsroman	2
Art history	2
Devotional poetry	2
Aesthetics	2
Education	2
Satirical poetry	2
Philosophical text	2
Symbolism	2
Surrealist fiction	2
Futurist poetry	2
True crime	2
Sonnet	2
Pastoral poetry	2
Philosophical drama	2
Feminist literature	2
Crime fiction	2
Feminist fiction	2
Socialist realism	2
Sentimental novel	2
Psychological novel	2
Expressionist poetry	2
Symbolist fiction	2
Novella	2
Philosophical treatise	2
Aestheticism	2
Naturalist fiction	2
Diary	2
Modernist theory	2
Philosophical essays	2
Naturalist drama	2
Industrial novel	2
Dystopian fiction	2
Poetics	2
Theology	2
Mythology	2
Lyrical poetry	2
Autobiographical	2
Political drama	2
Science and art notes	1
Social philosophy	1
Arthurian legend	1
Epigram	1
Ballad opera	1
Ballads	1
Prose poetry	1
Comic verse	1
Decadent poetry	1
Satirical tales	1
Travel literature	1
Historical study	1
Existentialist poetry	1
Philosophical prose	1
Dadaism	1
Humor	1
Political thriller	1
Allegory	1
Religious	1
Philosophical tale	1
Political commentary	1
Travel	1
Philosophical poetry	1
Christmas fiction	1
Tragicomedy	1
Gothic tales	1
Political novel	1
War novel	1
Experimental poetry	1
Feminist philosophy	1
Religious fiction	1
Political poetry	1
Gothic fiction	1
Western	1
Fantasy drama	1
Translation	1
Detective novel	1
Modernist essays	1
Metaphysical poetry	1
Nature Writing	1
Masque	1
Utopian literature	1
Dialogue	1
Dramatic monologue	1
Aesthetic novel	1
Operetta	1
Mystery novel	1
Surrealist drama	1
Utopian philosophy	1
Travel poetry	1
Sonnet sequence	1
Spiritual	1
Autobiographical essay	1
Mystery	1
Nonsense verse	1
Letters	1
Travel writing	1
Holocaust poetry	1
Adventure	1
Periodical essays	1
Maxims	1
Epigrams	1
Philosophical fragments	1
Scottish poetry	1
Children’s literature	1
Religious novel	1
Coming-of-age novel	1
Neorealism	1
Philosophical dialogue	1
Surrealist play	1
Existentialist novel	1
Anthropological theory	1
Futurist drama/poetry	1
Romantic fiction	1
Post-colonial fiction	1
Modernist philosophy	1
Surreal fiction	1
Poetry/Drama	1
Memoire/Autobiography	1
Political theory	1
Tragedy/Comedy	1
Political	1
Fables	1
Aesthetic theory	1
Folklore	1
Utopian novel	1
Romantic opera	1
Historical account	1
Surrealism	1
Art theory	1
Theater theory	1
Literary fiction	1
Psychological thriller	1
poetry	1
Andalusian folk poetry	1
Philosophical memoir	1
Modernist prose	1
Yiddish fiction	1
Jewish memoir	1
Fable	1
Social criticism	1
Fairy tales	1
Modernism	1
Literary memoir	1
Existential fiction	1
Holocaust fiction	1
Post-apartheid fiction	1
Negritude	1
Existentialist play	1
War fiction	1

Evaluating the quality of the generated genres is a topic for another post. I didn’t see anything obviously absurd or wrong but I didn’t look hard.

Accuracy of reported `Year`

When I started this experiment, the Year field seemed the most objective and easiest to fact check. So Year is the only field I validated for this first experiment.

As checking all 1300 fields for errors would defeat the purpose of having a research assistant, I used DuckDB to give me a 100-field sample:

CREATE TABLE sample
as SELECT Author, Title, "Year"
FROM BOOKS
USING SAMPLE 100;

My fact-checking technique was irresponsibly informal: first I looked for the work on English Wikipedia. If the work and its date didn’t appear in a page or in the “Works” section of the author, I checked Goodreads, Google Books, and the Internet Archive.

Of the 100-work sample, 91 had dates that I could verify. If you’re interested in the 9 I couldn’t verify, you can read the comments:

select GPT_correct, count(GPT_correct) from "sample"
group by all;

─────────────┬────────────────────┐
│ GPT_correct │ count(GPT_correct) │
│ boolean │ int64 │
├─────────────┼────────────────────┤
│ false │ 20 │
│ true │ 71 │
└─

D select * from "sample"
where GPT_correct IS NULL;

| Author | Title | Year | GPT_correct | correction_comments |
|-------------------------------|---------------------------|-----:|-------------|-------------------------------------|
| Yeats, W. B. (William Butler) | Mythologies | 1938 | | not sure how to date |
| Angelos Sikelianos | Selected Poems | 1960 | | not sure about this |
| Fernando Pessoa | Always | | | not sure about title |
| Kevin Hart | Peniel and Other Poems | 1986 | | not sure about title |
| Robert Penn Warren | Selected Poems | 1997 | | Likely indeterminate, but ambiguous |
| Ronald Firbank | Five Novels | 1924 | | ambiguous |
| Németh Laszlo | Guilt | 1942 | | not sure about title |
| Francis Ponge | Things: Selected Writings | | | ambiguous |
| Lucretius | The Way Things Are | -55 | | not sure how to date |

This left a sample of 91 works. Of these, ChatGPT incorrectly reported the year of 20 rows. “Incorrect” in this case means either factually wrong (e.g. reported 2003 when the first publication was 2001) or against the prompt’s instructions to leave ambiguous fields blank.

Here are the results:

SELECT * FROM sample
WHERE GPT_correct IS False;

Author	Title	Year	GPT_correct	correction_comments
Thomas Hardy	Collected Poems	1928	false	1919
Philippe de Commynes	Memoirs	1489	false	1498 or indeterminate
Miroslav Holub	The Fly	1972	false	indeterminate or 1987 (year of translation)
Léonie Adams	Poems: A Selection	1953	false	1954
Günter Eich	Moles	1964	false	indeterminate: english edition dates from 66 to 72
Alfred Cord	A Call in the Midst of the Crowd	2005	false	1978
Georges Bataille	Blue of Noon	1955	false	1957
Sholem Asch	East River	1922	false	1946
Heinrich von Kleist	Stories	1810	false	indeterminate
Paul Celan	Poems	1991	false	indeterminate
Vachel Lindsay	Collected Poems	1931	false	indeterminate
François Villon	Poems	1400	false	indeterminate
Denis Johnson	Fiskadoro	1999	false	1985
Anthony Burgess	Nothing Like the Sun	1961	false	1964
Jacob and Wilhelm Grimm	Fairy Tales		false	First published 1812
Witold Gombrowicz	Three Novels	1967	false	indeterminate or 1978 (year of translation and compiliation)
Hugo von Hofmannsthal	Poems and Verse Plays	1990	false	indeterminate
Seneca, Lucius Annaeus	Tragedies, particularly Medea and Hercules Furens	-65	false	indeterminate
Ernest Hemingway	Complete Short Stories	1938	false	indeterminate or 1987
Robert Garnier	Mark Antony	1582	false	1578

Estimating accuracy rate

The sample year rate has the following statistics:

Statistic	Value
Total fields	91
Errors	20
Sample accuracy rate	0.78%
Standard error	0.04
Sample distribution interval	[0.70,0.87]

In other words, using this method to build a confidence interval for a proportion, we can say with 95% confidence that the range [0.70,0.87] covers the true accuracy rate.

Check my work

Discussion

Even with all the time spent rewriting prompts and fixing validation issues, ChatGPT built the books CSV database in far less time than it would have taken manually.

The results for the added fields are mixed. From my cursory checks, the Genre and Form fields are at least reasonable. The error rate of the Year field, though, is too high to be reliable.

In this case, I judge my ChatGPT-built database to good enough for unserious work. If you were using this database just to discover some books that a specific critic recommended and wanted to filter by some crude categorizations and fuzzy dates, the data quality is acceptable. If, however, you needed to do precise research, all the generated values are too suspicious to be acceptable.

Ideas for extended research

I believe there is potential in using LLMs to create bespoke databases; I’m not sure how easy it is to generate a very good one. Part of the quality issues in my DB arose from my haphazard design, and but another part stems from the inherent instability of ChatGPT’s answers. For this reason, the build-a-research-database use case might work better for exploration and rough filtering—say filtering by Genre— than for retrieving precise values.

If anything, this generated table can serve as a baseline to try to improve. Here are some ideas to improve this experiment:

Have a clear hypothesis and criteria for how to test quality
Think carefully about the specific values to generate and whether they really are objectively verifiable
Test this quality on a small sample first to discover issues with the prompt
If creating a large database, build some automation tools to check for basic issues and reprompt (things like adding all works for an author on one row happened multiple times)
Refine the interaction with the LLM: better prompting, better use and understanding of context windows, changing temperature, and so on.

I also have some ideas to extend this use case:

Normalize the facts in different tables. For the input, create a table with a primary key. Then make that a foreign key in the tables where information needs to be generated. Pass the LLM the schema and have it build the database.
Require a source for each reported fact
If the database is public, offer a way for users to fact check and edit values, similar to Wikipedia.

The journey was more fun than the result

My books DB may be a dud, but the process of making it was fun:

I got to use DuckDB for the first time..
I researched a ton of books I never heard of:
I now have a database for future learning projects, e.g. to build a simple web app or practice normalizing a data model.
I discovered tons of issues in how I thought about creating an LLM test for a certain use case
I gained a new appreciation for how difficult it is to manage information precisely. All of my table’s values are subject for interpretation.

it also added “Country” for each author. This information was usually in the prompt, but sometimes this was regional or politically ambigious. In these cases ChatGPT picked a country. I saw no errors there. ↩︎

Auto-generate glossaries With Hugo

Wed, 20 Mar 2024 21:53:58 -0500

Hugo can use data files, like JSON and YAML, to make templates. This feature is often useful for technical writing:

The data file can serve as a source of truth, and writers can use its properties wherever they please.
For long repetitive structures, the format needs to be written only once in the shortcode, rather than once for every item.

This topic shows some shortcodes that use a YAML file as a single source of terminology truth in a Hugo site. These templates save some formatting tedium and ensure that terminology definitions stay consistent across different pages and presentations.

Steps to generate a glossary

The procedure is as follows:

Write the glossary to something like data/termbase.yaml.
Write a template that reads the YAML file and makes a formatted list.
Extend its usefulness by adding however many properties you want to the data file.

Write the YAML file

First create the file to be your source of truth:

In the data directory, create a YAML file called something like data/termbase.yaml.
Create an array of term objects. For a glossary, you need at least term and definition properties (choose whatever key names you want).
Optionally, add other properties to extend the templating possibilities (subsequent sections give examples).

- term: Abies
definition: A genus of coniferous trees, properly called fir.
category: nature
- term: Write the Docs
definition: A great place to discuss docs
abbr: WtD
category: "docs"
- term: Mean time to repair
definition: The average time it takes to repair a failed component in a system
abbr: MTTR
category: ""
- term: Montiform
definition: Resembling a mountain in form
category: "nature"
- term: Mozilla Developer Network
definition: A great place to read docs
abbr: MDN
category: "docs"
- term: Oryctology
definition: An old name for mineralogy and geology.
category: "nature"
- term: Request for comment
definition: A publication from the standards-setting bodies of the internet
abbr: RFC
category: "protocols"
- term: Request for proposal
definition: A call for proposals
abbr: RFP
- term: Transmission Control Protocol
definition: A bidirectional protocol to deliver a reliable, ordered, and error-checked stream of octets between applications.
abbr: TCP
category: "protocols"
- term: Topaz
definition: A mineral occuring in rhombic prisms, generally yellowish and pellucid
category: "nature"
- term: User Datagram Protocol
definition: A connectionless communication protocol.
abbr: UDP
category: "protocols"

Alphabetizing the list is probably a good idea, but the sort function can do that for you.

Write the glossary template

Next, write a template to generate a glossary page. This is a good time to use an HTML description list.

{{- $arg := (.Get 0) -}}
{{- range sort .Site.Data.termbase "term" -}}
<dl>
<dt>{{- .term }}{{ with .abbr }} ({{ . -}}){{ end -}}
</dt>
<dd>{{- .definition -}}</dd>
</dl>
{{- end -}}

In this particular template, if the term has an abbr property, it prints an abbreviation in parenthesis next to the definition. But it works fine without abbreviations, too. Feel free to copy and paste.

Use the shortcode in a page

This shortcode renders the preceding YAML file as follows:

More uses of the termbase

With a single source of terminology truth, you can add more shortcodes to manage terminology across an entire site. Here are some ways.

Reuse the definitions in tooltips

Each entry in the preceding YAML file has properties for term and definition—that’s all the data you need to create a tooltip that expands a definition on hover. I started using this quite often in my work on docs.rhize.com.

This shortcode takes one argument: either the term or, if you have one, the abbreviation. It creates an HTML abbr( or “tooltip”) on the page.

<!--- variables based on shortcode argument -->
{{- $entry := (.Get 0) -}}
<!-- range over data file -->
{{- range $.Site.Data.termbase -}}
<!-- Find matches -->
{{- if eq ( .abbr | lower) ( $entry | lower ) -}}
<abbr title="{{- .term -}}. {{ .definition | plainify -}}"> {{- .abbr -}}</abbr>
{{- else if eq ( .term | lower) ( $entry | lower ) -}}
<abbr title="{{ .definition | plainify -}}"> {{- $entry -}}</abbr>
{{- end -}}
{{- end -}}

When called, it prints the term along with a tooltip that expands its definition. For example:

Some knowledge of {{< abbr "tcp" >}}
is useful to anyone who works with networks.

Renders as:

Some knowledge of TCP is useful to anyone who works with networks.

To get even fancier, you could make an abbreviation that expands dynamically.

Make mini-glossaries

If you notice, most words in the termbase have a category property. You could use this to create mini glossaries.

This requires only two more evaluations in the preceding glossary creator:

Accept one argument in the call
Print the term and definition only if its category property matches the argument.

{{- $entry := (.Get 0) -}}

{{- range sort .Site.Data.termbase "term" -}}
{{- if (eq .category $entry) -}}
{{- .term }} {{ with .abbreviation }} ({{ . -}}) {{- end -}} 
{{- .definition -}}
{{- end -}}
{{- end -}}

Now I’ll make some mini-glosses:

#### Nature gloss:
{{< mini-gloss "nature" >}}
#### Protocol gloss:
{{< mini-gloss "protocols" >}}
#### Docs gloss:
{{< mini-gloss "docs" >}}

These render as follows:

Add supportive links to the glossary

Perhaps you want to link some glossary entries to their most relevant documentation topic. For this, you can just add an array of links in a ReadMore property, then modify the glossary shortcode to print them at the end of the html <dd>.

Ways to improve

These shortcodes are unsophisticated. I wouldn’t be surprised to learn of ways to write the logic more efficiently or separate concerns more elegantly within the Hugo architecture. Still, I have used or am actively using all these strategies in my professional work, and they’ve worked well enough.

But I’d be happy to learn of better ways, and I’ll update this doc to acknowledge any help. Some ideas to iterate:

In the mini-gloss, make the categories property an array so one term could display in multiple mini-glossaries. I think this involves managing the state of an outer context from an inner one.
Give the abbreviation smarter capitalization.

At some point, a huge YAML file would be a pain to maintain: I wouldn’t want to write a whole dictionary like this. Some ideas to improve the UX:

Add validation of the data file.
Combine multiple files into one gloss (the opposite of the mini-gloss example)
Add some interface to manage entries without directly editing the data file

Hugo also has an open issue to add glossaries, so maybe this doc will be outdated someday. Either way, I like the flexibility of using a data file to manage terms.

Write up: WTD Atlantic, 2023

Wed, 27 Sep 2023 09:03:38 -0300

Earlier this month, I presented a talk at Write the Docs called “Graphs, not trees: a ground-up approach to fixing a docs site.”

Thanks to Dennis Dawson for this great sketch

Before I write up my thoughts, let me thank some people:

Yesica Torrico for being the design brain behind the visuals, style, and overall organization
Sviatoslav Abakumov for giving an incisive critique of a quite crappy first version.

I also thank the conference organizers, who did a great job curating talks and running a tight ship; the speakers, who all gave interesting talks; and of course the participants, who made this feel as interactive as an in-person conference. Through the chat, I really felt that the audience was an active participant in each talk.

Preparation

It started with an . I realized how inchoate my idea was only when I wrote the . Then the was accepted, and I thought what the hell did I write?

From here the process went through a few drafts, starting with a little storyboard and then an outlined script. The content of the images changed the content of the speech; for example, I originally used that man-donkey image only because it amused me, but after I referenced it a few more times, it became a through line of my whole narrative.

One month out, I recorded a loose, semi-improvised talk and requested feedback from my friend Sviatoslav. He pointed out a number of dead ends and faulty arguments. So I went back and wrote a script (ironically, now that I knew what I wanted to say, I used a trunk-and-branch approach to re-design my argument). Responding to this feedback made the presentation much tighter.

Tools

Effective communication does the most with its medium. Since the presentation was pre-recorded, my medium involved not only voice and image but also moving tracks that can be layered, edited, and re-watched. My original dream had animation and multi-track audio and video, all disciplines I know nothing about—even just making suitable audio was above me—and soon I hauled that dream away. ¹

I do think the RevealJS medium gave me some flexibility to animate and make transitions that worked with the content.

To control the slides, I used KDE connect.

Presentations

The best part of the presentation was the live chat. I was pumped up!

All the talks were great. My only complaint is that there were too many good choices.

I also enjoyed the format of the parallel unconference sessions. I liked to speedily cycle through the rooms, a unique benefit of virtual conferences. The equivalent in real life would be to constantly barge into new rooms, listen for five seconds, leave, then return five minutes later to pop my head in for another five seconds.

Lessons

I’d never done a talk like this before. If I could do it over, I’d try to get a good recording environment before I did anything else. I underestimated how hard it is to get good lighting and sound, and by the time I recorded, it was too late to change environments. I also realized I had a lot of “ums” and “kind of, like, you know, a sort of…”, all noise that obscures the signal.

More advice for my past self:

Don’t underestimate how hard it is to learn a new technology. RevealJS, audio, and video all presented new challenges (especially the latter two).
Get feedback from multiple perspectives.
Get feedback on an early, rough form. More polished versions might limit ideas.
Rewrite and practice—but leave room for improvisation.
Let the material dictate the story.
Start with the story, then the images, then rework the story. Add all the little finishing details at the end. Words are way easier to change than images.

Pretty bland and generic, to be honest. I think I knew all this before I did the presentation, but now I really know.

Wrap up

I had never attended a professional conference, and I wasn’t expecting much from the virtual format. But the process ended up being wonderfully interactive, and there were so many smart ideas.

This was probably the most fun moment of my technical-writing career.

For the line “Google is not a tower, it’s a subway”, I spent hours animating the Google logo to look like a train. After watching multiple times to understand what I was trying to represent, my designer colleague asked why I was showing a freight train when I’m talking about a subway.

Besides looking terrible, this “Google subway train” would give everyone in the station carbon-monoxide poisoning.

↩︎

A shortcode for dynamic abbreviations

Mon, 03 Jul 2023 10:06:11 -0300

This shortcode ensures that an abbreviation is always defined on a page and always has an HTML abbr element. It also saves writers from writing verbose, error-prone HTML and needing to track when a term first appears on a page.

For example, this text uses a shortcode:

_{{< abbr "mttr" >}}_ is a basic measure
of maintainability in repairable systems.
If two systems have equal types
of failure and equal failure rates,
the system with the lower {{< abbr "mttr" >}}
has higher availability.
Let's say it one more time for no reason:
{{< abbr "mttr" >}}.

On the page, it renders as follows. Notice the abbreviations have tooltips:

is a basic measure of maintainability in repairable systems. If two systems have equal types of failure and equal failure rates, the system with the lower has higher availability. Let’s say it one more time for no reason: .

Abbreviations should generally be avoided. But when they are necessary, this shortcode improves the experience for both readers and writers.

Hugo requirements

The following examples are ready to copy and paste. To use them, change the terms and abbreviations for your lexis.

But, to modify these or make your own versions, it helps to know the following Hugo concepts:

Variables. Required both to make custom variables and to understand how to use shortcode, page, and .Scratch variables.
Functions. Required to make templates to read data and conditionally write text.
Context. Required to understand how Hugo interprets a variable reference.

Steps to implement a conditional shortcode

The general procedure is as follows:

Define a term and its abbreviation. To facilitate reuse, store the data in variables or key-value pairs.
```
{{ $term := "Mozilla Developer Network" }}
{{ $abbreviation := "MDN" }}
```

Use another variable to give each term a unique counter.

{{ $count_id := print ( $abbreviation | lower ) "_count"}}

In a shortcode, create a conditional statement based on the counter’s value. If the count has started, write the HTML abbreviation.
```
{{ if gt ( .Page.Scratch.Get $count_id) 0 }}
<abbr title="{{- $term -}}">{{- $abbreviation -}}</abbr>
{{ .Scratch.Get $count_id }}
```
In the else block, write the full phrase with its abbreviation. Then use the scratch pad to set the counter to 1.
```
{{ else }}
{{ $term }} ({{ $abbreviation }})
{{ .Page.Scratch.Set ($count_id) 1 }}
{{ end }}
```

I combined these snippets in a shortcode located at layouts/shortcodes/abbreviations/mdn.html. I can call it as follows:

> Do you know the {{< abbreviations/mdn >}} docs?
>
> Yes! I love the {{< abbreviations/mdn >}} docs.

It renders as follows:

Do you know the Mozilla Developer Network (MDN) docs?

Yes! I love the MDN docs.

However, if you have multiple terms, this shortcode has avoidable boilerplate. The following sections describe some better, more general solutions.

Design decisions (and examples)

Hugo is flexible about how it handles variables and logic, so you have many ways to implement this shortcode. The following sections outline some approaches to take. When possible, I recommend keeping terms in a separate data file.

Keep terms in a data file

One method is to use a data file to store your terms in an array. I like this approach because it separates concerns and because the list can be reused as the source of other templated pages (like glossaries).

Write your abbreviations in data/terms.yaml:

- term: Transmission Control Protocol
abbreviation: TCP
- term: User Datagram Protocol
abbreviation: UDP

Range over the abbreviations in a shortcode.


<!--- variables based on shortcode argument -->
{{ $entry := (.Get 0 | lower) }}
{{ $count_id := print $entry "_count"}}
<!-- range over data file -->
{{ range $.Site.Data.terms }}
<!-- Find matches -->
{{ if eq ( .abbreviation | lower) $entry }}
{{ if gt ( $.Page.Scratch.Get ( $count_id )) 0 }}
<abbr title="{{- .term -}}">{{- .abbreviation -}}</abbr>
{{ else }}
{{ .term }} ({{ .abbreviation }})
{{ $.Page.Scratch.Set ( $count_id ) 1 }}
{{ end }}
{{ end }}
{{ end }}

Call it using the abbreviation as an argument.
```
The {{< abbreviations "tcp" >}} is not the
{{< abbreviations "udp" >}}.
Again,
{{< abbreviations "tcp" >}} is not
{{< abbreviations "udp" >}}.
```
This renders as:

The Transmission Control Protocol (TCP) is not the User Datagram Protocol (UDP) . Again, TCP is not UDP .

The drawback is that it requires a little more conditional coding, and the syntax to call it is a bit more complex.

A shortcode for each file

If you want each acronym to use its own file or shortcode call, try this approach:

In a partial, write the reusable logic. This file is called abbreviation.html:

 {{ $count_id := print ( .acronym | lower ) "_count"}}
{{ if gt ( .context.Scratch.Get $count_id) 0 }}
<abbr title="{{ .term }}">{{- .acronym -}}</abbr>
{{ .Scratch.Get $count_id }}
{{ else }}
{{ .term }} ({{ .acronym }})
{{ .context.Scratch.Set ($count_id) 1 }}
{{ end }}

In a shortcode, define the term and abbreviation, and then call the partial.

 {{ $term := "transmission control protocol" }}
{{ $abbreviation := "tcp" }}
<!-- reusable boilerplate -->
{{ partial "abbreviation.html" (dict "term" $term "abbreviation" $abbreviation "context" .page ) }}

The call syntax is slightly simpler, but I don’t like much else about it. Still, I’m happy I figured out how to do this because now I know how to pass variables and context to a partial (thanks to help from the Hugo forums).

In-line

If you plan to use the shortcode exactly once, you can use an inline shortcode and define terms in the page parameters. Setting terms in the page parameters probably has a number of reasonable uses, but I can imagine only two times where an inline shortcode makes sense for this use case:

If you are writing a long blog with acronyms that you never plan to use again.
If you are writing in a modularized set-up where it’s difficult to change the layouts or data directories.

Nevertheless, if you want to do this, read the inline example.

Discussion and links

This shortcode is handy for docs where abbreviations for specialized terms are often used. Perhaps you could combine it with a prose linter to enforce formatting.

On broader note, this shortcode hints at something more powerful: a way to conditionally format text based on a mutable page state. Now that I’ve figured out these small examples, maybe I’ll discover some greater use of the Hugo scratch pad.

For more reading, these helped me make this post.

Blog posts about Hugo .Scratch
- Hugo Scratch explained
- Hugo scope in a template
The forum post I made about how to share logic and variables between shortcodes.

"Allows You To" considered harmful

Wed, 21 Jun 2023 22:11:19 -0300

The Forward feature allows you to forward messages…

This is one of the most common constructions in technical documents—and one of the least useful. In fact, you can always avoid the phrase “feature allows you to verb” and your docs will be better off:

The phrase often incorrectly implies an intermediary between a behavior and its result.
Even when an intermediary is necessary, the semantic meaning of allows is ambiguous.
Even when the phrase is appropriate, it has a shorter alternative, lets you.

That’s the gist. The following sections elaborate with examples, citations, and proposed alternatives.

Reasons to avoid “allows you to”

In short, the phrase always creates inaccuracy, imprecision, or unnecessary text.

It’s often inaccurate

Along with similar constructions like lets you or enables you to, allows you to can create an unnecessary grammatical layer between a feature and its behavior. In linguistic terms, such verbs are examples of catenative verbs, meaning they subordinate other verbs in a chain. More specifically, they are examples of causatives, verbs that “cause” some other action. ¹

But these constructions often describe behavior that has no secondary effect. Let’s return to this text, lifted from an old version of the WhatsApp documentation (now fixed): ²

The Forward feature allows you to forward messages from a group or individual chat to another group or individual chat.

Is this true? Does the forward feature allow me to forward messages, or does it forward messages? Why not describe behavior directly?

The Forward feature forwards messages…

Maybe that example sounds awkwardly redundant, but unnecessary catenative verbs commonly intrude in all sorts of technical description. For example, this is from the MDN JavaScript reference.

Window.cancelIdleCallback()

Enables you to cancel a callback previously scheduled with Window.requestIdleCallback.

What does the method do: cancel callbacks or enable cancelling a callback elsewhere? Though this example uses enables instead of allows, the effect is the same: unnecessary words and increased ambiguity.

Its meaning is often ambiguous

Sometimes a writer really does want to describe a feature as a means to some end. In this case, a catenative verb might be appropriate. For example, these are from the MDN docs:

Making a copy like this allows you to effectively use the request/response again while varying the init options

The Twitter API allows you to do things like displaying your latest tweets on your website.

Both of these sentences describe one action (making a copy, calling the Twitter API) that can help users accomplish some later action (reusing, displaying). So causatives seem appropriate. But even in these cases, are phrases like allows you to or lets you really the best constructions?

If you look up allow or let in any dictionary, you’ll find the first definition is always about giving permission. ³ Isn’t it patronizing and inaccurate to think of a software object as giving me, a human, permission to display tweets on a website? (I could just take a screenshot and bypass the API). Of course, allows you to also has a more general meaning (I discuss that in the last section). But in software docs, where many readers are reading in a foreign language or using a machine translator, the least ambiguous option is best.

“Lets you” is shorter

Maybe you find my previous arguments unconvincing. Besides, some software features really do configure permissions. Either way, lets you is still a better construction. The reason is undeniable: it’s shorter.

Good technical writing should be as short as possible. Semantically, the following sentences are essentially identical: ³

- The `authenticate` endpoint allows you to view resources according to your credentials.
+ The `authenticate` endpoint lets you view resources according to your credentials.

But lets you uses five fewer characters. If you can transmit the same meaning with less text, you should: boost signal, reduce noise.

In a single sentence, such an improvement is small and rather inconsequential. But small changes add up, and technical communicators should try to standardize quality at all levels.

I am not alone in this idea: the Google Style Guide also recommends lets over allows. ⁴

Alternatives

Instead of allows you to and other intermediaries, tailor your vocabulary and syntax to the goals of your text.

In task-based docs, front the action

In a doc that describes procedures, consider inverting your syntax. That is, place the result before the command. For example, a doc called “How to use Nginx” could have a phrase like this.

To stop the server, run nginx -s quit.

Beginning with an infinitive of purpose helps readers scan for what they want to do (since readers scan in an F-shaped pattern).

In references, put behavior before context

Descriptions in references should be spartan. For example, a doc called “Nginx CLI reference” might have something like this:

nginx -s quit

Stops the server

If it’s important to communicate goals, you could provide context in a second sentence. For example:

The OCR API processes a batch of images and detects common visual features. These groupings can serve as search filters for a large image database.

Use the imperative

This is often the simplest and most direct way to describe the purpose of a tool. For example, the Whatsapp doc quoted at the beginning now reads as follows:

Use the Forward feature to forward messages from an individual or group chat to another individual or group chat.

Some writer at Facebook also didn’t like allows you to!

But who cares about such a common and tiny phrase?

True, this issue is small in the grand scheme of things. Good docs won’t be spoiled by an allows you to (MDN is proof). But these details do matter if you think systematically about creating quality documentation. If allows you to creates an unnecessary intermediary between an actor and action, it also creates inaccuracy, and technical communicators should always Try to tell the truth. Even when it’s appropriate, shorter alternatives exist, and technical texts should be as simple as possible and no simpler.

It’s also true that the phrase is common. When it comes to language in ordinary life, I’m a descriptivist. And I must acknowledge that allows you to is a perfectly common expression, ⁵ very often used without any sub-text about giving permission. However, technical writing represents a highly constrained use of language, where brevity and clarity are more important than “naturalness.”

Avoid allows you to. This isn’t a big issue, but it’s an easy one to fix, and better is better.

Pullum et al. The Cambridge Grammar of the English Language. Chapter 14, “Non-finite and verbless clauses.” ↩︎
At least, from the quoted text in this Stack Exchange Post, I infer that the WhatsApp page used to say “allows you to.” ↩︎
For example, the Cambridge dictionary entry for allow says “To give permission.” The entry for let says “To allow.” ↩︎ ↩︎
“Instead, use lets you.” ↩︎
For informal evidence, the expression has over 16 thousand occurrences in the Youglish corpus. ↩︎

Sentence repair: technical edits for two academic texts

Sat, 17 Jun 2023 23:15:24 -0300

I approach editing with the same creed as a doctor: first do no harm. My constant doubt is I’ll over-edit and change the meaning. This is why I have researched set of practices: to reduce guessing and have a standardized set of quality improvements.

However, when reading for fun, I sometimes can’t stop myself from trying to fix sentences that confuse me. Here are two examples of how a writer of software documentation would fix two texts that have nothing to do with software.

Wait, who did Spinoza write for?

This little primer on Spinoza is clear and well-written, but this particular sentence is hard to understand.

It may surprise you to hear that Spinoza’s Ethics was written just as much for a non-expert audience in the twenty-first century as for the philosophical world of the seventeenth.

—Beth Lord, Spinoza’s Ethics

It’s hard to diagnose exactly why: at 33 words, it’s a bit long, and the negation, “non-expert” is avoidable. But no major flaw jumps out. Yet it’s hard to understand the sentence’s relationships.

Having read this sentence a few times, I take it to literally mean that Spinoza wrote this book for two audiences:

The philosophical world of the 17th century
Non-experts of the 21st century.

The surrounding context, though, doesn’t give Spinoza so much foresight. Rather, Lord wants to say that the audience of this book have always been experts, “steeped in the philosophical tradition.” I think the confusion comes from the length and the long phrases.

How about one of these? One focuses on the difficulty of reading it, and another focuses on the publication audience.

To emphasize the difficulty of the text:

It may surprise you to hear that Spinoza’s Ethics has always been for a specialist audience. A non-expert of the seventeenth century would find the book just as difficult to read as a non-expert of the twenty-first century would.

To emphasize who the work was published for:

It may surprise you to hear that Spinoza’s Ethics were written for an expert audience. This was true when it was published for the philosophical world of the seventeenth century, and it’s just as true today.

This is an example where expanding might make a sentence easier to parse. In the next example, cutting text makes the sentence clearer.

Oracular or plain verbose?

This example is trickier. Though shorter, it has many more phrases that I try to avoid. Yet I think fans of this work would find my proposed edits disagreeable.

A building or town will only be alive to the extent that it is governed by the timeless way.

—Christopher Alexander, The Timeless Way of Building

First, a note on style: from the little I’ve read, Christopher Alexander seems to like these orbicular ways of writing, and perhaps some readers find this style imbues a “timelessness” of its own. Furthermore, the passive construction arranges the sentence to end with timeless way, which perhaps reinforces his central subject.

In an essay, such rhetorical flair is the author’s prerogative. But in a piece of technical writing, I’d chop this down. Though the sentence is short, it could be shorter:

Will only be alive. Why use the future? Was this false at the time of writing?
Be alive. Why use an adjective when a verb exists?
Only be. What is only modifying? The verb be or the extent?
To the extent that. Doesn’t only imply to the extent?
Is governed by. Is the passive functionally necessary?

Let’s chop it down progressively. I’ll let you decide where I went too far.

First, removing the passive and future doesn’t change meaning at all, but it brings the character count from 93 to 87.

A building or town is only alive to the extent that it is governed by the timeless way.

Removing the passive gets it down to 82.

A building or town is only alive to the extent that the timeless way governs it.

Next, does the distinction between “it is alive” and “it lives” matter here? If not, we can get the character count down to 78.

A building or town lives only to the extent that the timeless way governs it.

Aren’t “only” and “to the extent” redundant? Maybe only acts as necessary emphasis. But if not, removing only gets it to 73.

A building or town lives to the extent that the timeless way governs it.

Or maybe we could just compress it in a relative clause. Even restoring the passive, this version brings the character count down to 64.

A building or town lives only when governed by the timeless way.

It seems like Alexander wants to make a boundary in time and space delimited by the “the timeless way”: life and timeless governance on one side, death and ephemeral governance on the other. Perhaps his “to the extent that” conveys this boundary better than my truncated version does.

Have I changed the essential meaning too much?

A note on style and audience

As an editor, I’d be much more comfortable proposing my change to Beth Lord’s text than I would to Christopher Alexander’s. Lord’s book is about philosophy, but her goals are much closer to those of a technical writer: make a difficult topic easier to approach. Alexander’s work is in some sense a philosophy of its own, which specifically mimics works like the Tao Te Ching.

In any case, I’m not being totally fair to either author, because they have different audiences and style considerations than I do. People rarely settle in to enjoy a technical text as they would for a favorite book.

The audience of my technical texts are usually reading to accomplish something else, and a large majority of them may be reading in a second language and under stressful work environments. For this reason, I impose a minimalist, even brutalist, style on my technical texts, a style that would ruin the work of many great prose writers. As a technical editor, I would not be able to ethically operate on the works of, for example, Annie Dillard or Sir Thomas Browne.

Uses for Hugo inline shortcodes

Tue, 09 May 2023 00:00:00 +0000

A typical Hugo setup designates one directory for content (the text that people read) and another for presentation configuration (the logic that controls how the text is organized on the page). Such separation of concerns is quite sensible, but it adds some overhead to the writing process.

As it turns out, Hugo also supports inline shortcodes, defined directly in the content file. This means you can stay in your markdown file and still create shortcodes for templates, variables, and data mutations on the content within.

Inline shortcodes might be handy in a few cases.

When the shortcode is scoped to one page: This is the essential circumstance. If the shortcode is to be reused across content files, it should be shared in the layouts/shortcodes dir.
When the build and content repos are separated: If the Hugo build system is modularized across different repos, an inline shortcode might be more convenient.
For quick prototyping: Sometimes, opportunities to template or reuse text are revealed only after writers start creating. With inline shortcodes, writers could prototype the shortcode in the markdown file, then abstract it out later.
For fun: It’s liberating to write shortcodes on the fly—no need to think about how things fit into the bigger picture.

This post documents ways to use inline shortcodes for common technical-writing problems. All the shortcodes here also work as “normal” standalone functions. So, again, if you want to reuse a shortcode here, remove the inline suffix and add it to layouts/shortcodes instead.

Requirements

To use inline shortcodes, you must:

Use Hugo version 0.52.0 or later
Enable inline shortcodes in your Hugo config:
```
enableInlineShortcodes: true
```
Inline shortcodes execute code. Enable them only if you trust your authors and reviewers.

Syntax

To declare an inline shortcode:

Name your shortcode in opening and closing tags. Append .inline to the tag name.

Within the shortcode tag, write the value that the shortcode will return.

<!--- opening tag -->
{{< hello.inline >}}
<!--- content -->
Hello, you!
<!--- closing tag -->
{{< /hello.inline >}}

The first time you use the shortcode, the value renders. For example, the preceding snippet renders as follows:

Hello, you!

On subsequent calls, use one tag with a self-closing /.

{{< hello.inline />}}

Like all shortcodes, inline shortcodes can use parameters, site or page data, and built-in Hugo functions.

Examples

These examples show inline shortcodes to solve some manual tasks that arise in my work as a technical writer.

Embed dynamic calculations

This has its own post: Hugo inline spreadsheet.

Expand acronym names

This first example uses a variable to avoid typing verbose text.

The HTML abbr element has a title element that expands when the cursor hovers over it. Here’s the example from the Mozilla Developer Network docs:

Ashok's joke made me
<abbr title="Laugh Out Loud">LOL</abbr>
big time.

That HTML renders as follows:

Ashok’s joke made me LOL big time.

It’s a chore to write that HTML each time, and the chance of a typo increases with each use. If you need to expand this acronym on only one page, you could use an inline shortcode to minimize typing.

The first time you use an acronym shortcode,
you type more,
{{< lol-abbr.inline >}}
<abbr title="laugh out loud">lol</abbr>
{{< /lol-abbr.inline >}}
On subsequent uses, you type less,
{{< lol-abbr.inline />}}.

The preceding snippet renders as follows:

The first time you use an acronym shortcode, you type more, lol

On subsequent uses, you type less, lol .

Use page frontmatter

For another way to make variables, you can add a new property to your frontmatter. The inline shortcode can access the value as a variable scoped to the page.

For example, you could use frontmatter to keep track of the last application version tested, and then use an inline shortcode to automatically update version references within the docs.

Add the version name in a frontmatter property:
```
last_version: Arctic effulgence
```

Reference the frontmatter value within your inline shortcode as $.Page.Params.<key-name>.

{{% frontmatter.inline %}}
This procedure was last tested on version
`{{ $.Page.Params.last_version }}`
{{% /frontmatter.inline %}}

This shortcode renders as follows:

This procedure was last tested on version Arctic effulgence

The text processing could get much more complex if you want to mutate the data, or if your frontmatter includes arrays or nested object.

Make a page-specific template

You can also use inline shortcodes to template repetitive phrasing (or lists or tables) in a document.

A template could apply to even single sentences. For example, a page of conceptual content may end each section with links to learn more.

Write a snippet sentence with placeholder values.

This template uses variables for author names and their id within the Project Gutenberg online library. For clarity, I’m using named parameters like author instead of positional arguments.
```
For more by {{ .Get "author" }}, visit the author's
<a href="({{ $.Page.Params.library_url }}/{{ .Get "id" }})"
Gutenberg page. </a>
```

Wrap the template in an inline shortcode tag, with values for the parameters.

{{< learn-more.inline author="Emily Dickinson" id="996" >}}
For more by {{ .Get "author" }},
visit the author's
<a href=
"{{ $.Page.Params.library_url }}/{{ .Get "id" }}">
Gutenberg page. </a>
{{< /learn-more.inline >}}

This shortcode generates the following HTML snippet.

For more by Emily Dickinson, visit the author’s Gutenberg page.

After declaring it once, I can reuse with new properties:

{{% learn-more.inline author="Sappho" id="32618" /%}}
{{% learn-more.inline author="shakespeare" id="65" /%}}

In the final rendering, these shortcodes appear as follows:

For more by Sappho, visit the author's Gutenberg page.

For more by Shakespeare, visit the author's Gutenberg page.

Make a counter

Sometimes, a tutorial has multiple sections and a required order. I generally avoid numbering these sections, because many things could go wrong: I could miscount, more steps could be added later, and so on.

An inline shortcode can serve as a counter to dynamically increment steps. For this one, I create two shortcodes:

The counter-reset starts or resets the counter, step. It uses the .Scratch function to store page data.
```
{{% counter-reset.inline %}}
{{ .Page.Scratch.Set "step" 1 }}
{{% /counter-reset.inline %}}
```

The increment shortcode prints the value of the counter and then increments it.

{{% increment.inline %}}
{{ .Page.Scratch.Get "step" */}}
{{ .Page.Scratch.Add "step" 1 */}}
{{% /increment.inline %}}
<!--- call it again -->
{{% increment.inline %}}

For demonstration purposes, I’ll incorporate the counter with a template, wrapping the increment function around text to describe a procedure.

{{% counter-reset.inline %}}
{{- .Page.Scratch.Set "step" 1 -}}
{{% /counter-reset.inline %}}
# How to run a race
{{% increment.inline "start at the beginning" /*%}}
## Step {{ .Page.Scratch.Get "step" }}: {{ .Get 0 }}
{{- .Page.Scratch.Add "step" 1 -}}
{{%/* /increment.inline %}}
{{% increment.inline "run to the middle" /%}}
{{% increment.inline "finish strong" /%}}
# How to technically write
<!--- reset counter -->
{{% counter-reset.inline /%}}
{{% increment.inline "say what you're going to say" /%}}
{{% increment.inline "say it" /%}}
{{% increment.inline "say what you said" /%}}


# How to run a race
## Step 1: start at the beginning
## Step 2: run to the middle
## Step 3: finish strong
# How to technically write
<!--- reset counter -->
## Step 1: say what you're going to say
## Step 2: say it
## Step 3: say what you said

Synthesis: First the full phrase, then abbreviate with tooltip

This example uses one shortcode to write the entire phrase the first time a term is used, then to write the abbreviation with a tooltip all other uses. This way, you can move acronyms around and always be sure that they are defined the first time.

This combines techniques from previous sections.

The page frontmatter defines a word and its acronym.
The shortcode uses the Scratch pad to determine whether the acronym has been used.

This page has the following frontmatter:

lol:
term: Laugh out loud
acronym: LOL

The following shortcode uses with to check whether the shortcode has already been used, then use that as a condition to print different strings.

The first time, I'll write an inline shortcode
and expand the entire phrase.
And that phrase is...
<!-- define acronym -->
- {{< acro.inline >}}
<!-- use with to if check word exists -->
{{ with .Page.Scratch.Get "lol_count" }}
<!-- If it exists, write abbr -->
<abbr title="{{ $.Page.Params.lol.term }}">
{{ $.Page.Params.lol.acronym }}</abbr>
<!-- If not, write whole term, then indicate it was used by setting counter -->
{{ else }}
{{ $.Page.Params.lol.term }} ({{ $.Page.Params.lol.acronym }})
<!-- change state of variable to indicate use -->
{{ .Page.Scratch.Set "lol_count" 1 }}
{{ end }}
{{< /acro.inline >}}
Now I'll use it a few more times:
- 2nd
{{< acro.inline />}}
- 3rd
{{< acro.inline />}}
- 4th
{{< acro.inline />}}

Here is how it renders:

The first time, I’ll write an inline shortcode and expand the entire phrase. And that phrase is…

laugh out loud (LOL)

Now I’ll use it a few more times.

2nd LOL
3rd LOL
4th LOL

Discussion

As I wrote this post, I learned much more about shortcodes in general than I did about inline shortcodes.

Inline shortcodes are fun to write, but they can turn a page into a soupy, tangled mess (for example, the source for this page has started to get confusing to navigate). Let me reiterate one more time: if you want to reuse a shortcode on another page, add it to your layouts. Besides that, some other drawbacks exist:

They make the document more complicated to understand.
They have some technical limitations. For example, you can’t nest them.
They return values when declared, so you can’t write your functions in one part of the page and use them in another.

That said, I think inline shortcodes can be pretty handy, and I’ll probably start using them more. I especially see using them to template repetitive pages and to prototype.

Hugo issue #4011. This issue is where the inline shortcodes were proposed. Aside from historical interest, it has a good discussion about naming and design decisions.
Hugo Shortcode docs. Canonical docs about all shortcodes, including inline ones.
Hugo functions. All these are available in your inline shortcodes.

How to use shortcode headings in the Hugo TOC

Thu, 20 Apr 2023 00:00:00 +0000

When I write headings, I always consider how they’ll look in the auto-generated table of contents (TOC). If the headings are descriptive, the TOC creates a powerful navigation tool, providing:

A high-level summary of the overall document
A set of entry points to help readers find the sections that most relate to them.

I wrote a long post, Most common edits in technical documentation. Because it was so long and regularly structured, I wrote a shortcode to template the content from different data files. When my shortcode content didn’t appear in my TOC on this Hugo site, I was sad.

But I accidentally found a solution: write the shortcode in markdown.

Solution: markdown headings

In short:

Use markdown headings in the shortcode (that’s ## instead of <h2>).
Delimit the shortcodes with the percent sign (that’s % instead of <).

Shortcodes with the % delimiter render before the page. I don’t know why the headings wouldn’t work with a percent delimiter and HTML headings, but this is what I’ve found. For a demonstration, read the next section.

Live demonstration

I’ve written two shortcodes with level-three headings. The only difference is heading style:

The html-headings shortcode uses HTML.

<h3>HTML heading, {{ (.Get 0) }}-delimited shortcode</h3>

The markdown-headings shortcode uses markdown.

### Markdown heading, {{ (.Get 0) }}-delimited shortcode

Now, I’ll put the following snippet directly in the file that you’re reading.


{{< toc-demo/html-headings "bracket" >}}
This heading doesn't appear in the TOC.
{{< toc-demo/markdown-headings "bracket" >}}
No heading is created because the angle bracket
causes the shortcode to be processed as HTML.
{{% toc-demo/html-headings "percent" %}}
This heading doesn't appear in the TOC.
{{% toc-demo/markdown-headings "percent" %}}
This heading _appears_ in TOC.
Scroll back up and check yourself!
![Drawing of a confused Jackie Chan](/images/jackie-what.jpg)

Here it comes:

HTML heading, bracket-delimited shortcode

This heading doesn’t appear in the TOC.

### Markdown heading, bracket-delimited shortcode

No heading is created because the angle bracket causes the shortcode to be processed as HTML.

HTML heading, percent-delimited shortcode

This heading doesn’t appear in the TOC.

Markdown heading, percent-delimited shortcode

This heading appears in TOC. Scroll back up and check yourself!

Discussion

I’m not sure why only Markdown headings work. As far as I can gather, before v0.55, no shortcode content could appear in a TOC, because the markdown parser didn’t process shortcodes in any way. Somehow, though, my HTML headings still get skipped by the part of the application that generates the TOC.

EDIT: with a bit of looking, I believe some better solutions might be to use unsafe = true in the config, or perhaps, in more complex shortcodes, using RawContent. If I investigate more, I’ll make a new post.

This seems to be a common problem in Hugo. Here are some relevant links and issues that I could find.

Hugo 0.55 release notes: Shortcodes revisted
Pulls:
- 2323: Handle TOC before handling shortcodes
- 5702: Make page an interface
Issues:
- 6690: Shortcode way of including md cause headings missed in TOC
Community posts. I think better solutions are here!
- Applying headings in shortcodes to TOC
- URL/TOC issues

SEO basics are all you need: too much "optimization" worsens docs

Mon, 07 Mar 2022 07:59:32 -0300

For a long time, I knew nothing about search-engine optimization. I avoided learning about it. I thought that SEO was something for marketers and other mercantile types, not for technical-writing “purists” like me.

Of course, this was a silly line of thinking: technical writers should care about SEO. Why wouldn’t I want to make my docs findable? Everybody searches!

Besides, how could I dismiss SEO if I didn’t know how it worked? Now that I’ve learned a bit, I can say some SEO is a good thing, but too much will harm your docs.

Standard disclaimer: I’m ignorant

I’m far from an SEO expert. I’ve read two books¹ and the Google search essentials.

Besides, I believe that in the long run, making any page “perfect” for search engines is a doomed strategy. What is optimal today may be suboptimal tomorrow. I explain this a bit more at the end of this article.

The good parts

It’s very important to support searching. Even if you have no commercial interest, don’t you want people to find what you write? Fortunately, the official recommendations of SEO are perfectly harmonious with the guidelines of good doc writing.

SEO essentials are doc essentials

The following guidelines are just good practices for writing:

Have descriptive headers.
Make accessible pages.
Break up your content and make it scannable.
Use descriptive link text.
Promote docs in other channels (e.g. on Twitter)
Write good content that people want to link to.
Write good content that people want to link to (I know I repeated myself).

Metadata is important too

Some of the more advanced concepts are good ideas for docs too.

You can reuse a good meta description as a page summary, both for your site and social shares.²
If you already use structured authoring, using Schema.org and JSON-LD might help you classify your content.

You can use the sitemap to create visualizations of your information flow.

The bad parts: trying to hack SEO

All documentation should be readable and findable. A little SEO should help readers find your docs on search engines. But if you blindly follow the latest recommendations, they’ll struggle to actually read the text.

Here are some 2022 SEO best practices™ that create bad docs.

Dubious: intentionally create long pages

In SEO jargon, dwell time is how long a user stays on a site.³ The idea is that, if a user stays on a page for a long time, then that page is probably useful and interesting. In practice, people try to increase dwell time by padding out content.

Perhaps that makes sense for blog content, but “lengthy” is pretty much an anti-goal for documentation writers. I don’t want readers to stay on my documentation site for a long time. I want them to get in, find what they need, leave quickly, and get on with their lives.

Fortunately, I don’t think many documentarians are trying to make arbitrarily long docs. But many web-content writers are. In fact, I would say that the goal of “dwell time” has created far more poor, bloated content than useful longform content.

Use Google to search for a recipe of your favorite food.
Click any result in the first page.

Is the page longer than a recipe in a book? Are the actual ingredients and procedures buried two-thirds of the way down the page?

If there’s a topic that’s worthy of more than a few short paragraphs, then the writer’s big challenge is figuring out what information to leave out, not how to cram more in.

Dubious: use a lot of synonyms

Some SEO guidelines I’ve looked at say to sprinkle synonymous and related terms throughout your content to help the latent semantic indices(LSIs). An LSI is essentially a thesaurus. It helps search engines return results for “on-premises” when you search for “self-hosted”.⁴

Technical communication should use language that is precise and unambiguous. Ideally, one term to match to one thing. Terminology is a hard thing to nail down, and coherent documentation should use consistent names.

However, it does make sense to incorporate some synonyms, precisely to support searching.

In introductory sentences that start like “In some places, this term is also called…”
In some kind of thesaurus or other controlled vocabulary.

In other places, a document that uses different terms for the same thing is just confusing. In a world where “namespace” already means so many different things, why increase ambiguity?

Be wary of for-profit SEO advice

SEO is an arms race. Google tries to figure out what makes good content; marketers figure that out and exploit it; Google penalizes overuse of the metric and comes up with something else. If you try to be “optimal,” you might end up with an unreadable site that has bad SEO anyway. It doesn’t help that ninety-five percent of the SEO advice that you’ll find is either spam or a scam.

Ninety-five percent of SEO content falls within the spam-scam continuum.

A lot of these SEO “best practice” sites don’t even know what they are talking about. If they did, they’d probably keep their knowledge hidden. Even Google engineers use machine learning to support results, which means there’s always some part of the rankings that is determined within a black box.

Good SEO and Good docs

As I work on larger sites, I’ve come to appreciate how important it is to support searching.

However, for your docs, don’t worry too much about SEO. Instead, make useful, accessible content that people want to link to.

For SEO, I also have a great long-term strategy. Make useful, accessible content that people want to link to.

SEO in 2022, by Adam Clarke, and Ultimate Guide to Link Building, by Garrett French and Eric Ward. ↩︎
This site uses the description as a top-level summary. ↩︎
I’ve also seen it called “stay time” and “retention”. ↩︎
Google apparently doesn’t use LSI, but something more advanced https://backlinko.com/hub/seo/lsi ↩︎

Unbundle that giant OpenAPI file!

Thu, 03 Feb 2022 08:47:01 -0300

Awesome! 10,000 lines of YAML boilerplate

Have you ever said “I love reading 10k-line YAML files?” If you have, I suspect you’re a robot.

Because if you’re a human, there’s a good chance you don’t love reading 10k-line YAML files. And if you’ve ever worked on an large OpenAPI spec, there’s also a good chance you’ve gotten lost, at least once, in thousands of lines of YAML spaghetti.

Fortunately, it doesn’t need to be that way. Because many parsers support multi-file references, you can unbundle the YAML file into individual pieces.

Even if the parser you use looks in only one file, it’s simple to take a multi-file definition and bundle it into a single megafile, sure to please a hungry machine.

Unbundling solves multiple documentation issues

In a system with large or multiple REST APIs, there are some common documentation problems that come up:

The organization might need separate public and private versions.
The organization might want to present multiple APIs that have a large overlap in their underlying schema.
If the definition is hand-written, the writer might be sad about working with a YAML mega-file.
Maybe, if the writer uses the same hardware that I use, a mega-YAML file might even cause their text editor to lag and crash.

Fortunately, all these problems have one solution:

Split the definition up into smaller files

Redoc to the rescue

Redoc is pretty well-known for their API docs. Besides their redoc-cli tool, they also make openapi, a command-line utility that bundles, unbundles, previews, and lints docs.

This demo uses openapi. The topic focuses on the bundling and unbundling, but the other features are nice too.

Tutorial: Unbundle a large YAML file

In this demo, I’m going to:

Inspect an openapi.yaml file
Use the openapi CLI tool to unbundle it into small pieces.
Separate the API into two top-level definitions.
Rebundle the API as two distinct documents.

This demonstration should be reproducible. If you can use the command line, feel free to follow along. You’ll need to use npm to install the openapi tool (linked in the preceding section).

Find a large file

First we need to find a heavy-duty API spec. The OpenAPI directory is a good place to look.

I’m going to choose an API from NASA. At 373 lines, it’s a flyweight in the world of OpenAPI definitions. But it’s enough for a demonstration.

Download your API file. You can do it with cUrl like this:

curl -o openapi.yaml https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/nasa.gov/asteroids%20neows/3.4.0/openapi.yaml

Inspect its contents. This API is quite slim. It has just a few endpoints:
- /api. Calling this endpoint returns the OpenAPI specification (recursive!).
- /api/projects. This lists all projects
- /api/projects/{id}. This returns information about a specific project.

Let’s pretend this mega API file has two audiences:

Tech writers, who are interested in only OpenAPI YAML files.
Project managers, who are interested in only projects

For this demonstration, I’m going to create two different API definitions. This way we can have a special dedicated API for the tech writers, and another one for the project managers.

Unbundle the file

Make sure your file is in a clean directory.

.
└── openapi.yaml
0 directories, 1 file

Create a directory for the output. I call mine unbundled.

mkdir unbundled

.
├── openapi.yaml
└── unbundled
1 directory, 1 file

Run the openapi split command.

openapi split openapi.yaml --outDir unbundled/

Document: openapi.yaml is successfully split
and all related files are saved to the directory: unbundled/
openapi.yaml: split processed in 134ms

This tool is fast, even on very big files.

Explore the new contents of the bundled directory. There are many more files now, one for each path, and one for each reusable schema.

.
├── openapi.yaml # The original file
└── unbundled # the unbundled directory
├── components # The re-usable schema
│   └── schemas
│   ├── closeoutDocument.yaml
│   ├── coInvestigator.yaml
│   ├── destination.yaml
│   ├── file.yaml
│   ├── libraryItem.yaml
│   ├── organization.yaml
│   ├── principalInvestigator.yaml
│   ├── programDirector.yaml
│   ├── programManager.yaml
│   ├── projectManager.yaml
│   ├── project.yaml
│   ├── technologyArea.yaml
│   └── workLocation.yaml
├── openapi.yaml #the new, top-level definiton
└── paths #the paths
├── api@projects{.format}.yaml
├── api@projects@{id}{.format}.yaml
└── api.yaml
4 directories, 18 files

Split the definition into two APIs

We’re going to make two APIs, one for tech writers, and one for project managers.

Inspect the unbundled/openapi.yaml file. It now has only 50 lines. That’s because the schemas and endpoints are tucked away in their own files. The paths property now looks like this:

paths:
/api:
$ref: paths/api.yaml
'/api/projects/{id}{.format}':
$ref: 'paths/api@projects@{id}{.format}.yaml'
'/api/projects{.format}':
$ref: 'paths/api@projects{.format}.yaml

If you are familiar with the $ref keyword, this should look familiar—it’s the same principle, but instead of referencing a position in the file, you’re now referencing a directory.

To make two API definitions, copy the top-level definition to another file. Rename both files, if you want.

cp unbundled/openapi.yaml unbundled/writersAPI.yaml
mv unbundled/openapi.yaml unbundled/projectsAPI.yaml

Open writersAPI.yaml. Delete the two /projects paths, and their references.
Open projectsAPI.yaml. Delete the /api path, and its reference.

That’s it! Now you have two separate API definitions, which share two underlying schema. Furthermore, your specification is slim and DRY. You won’t have to wrangle any YAML monsters, but if different paths use the same schema, you can update multiple definitions by updating only one schema.

Rebundle the file

Some parsers can handle multi-line definitions. Redoc is an example (I guess that’s not a surprise).

Others can read only one file. If that’s the case for you, you can still work off an unbundled file. But you’ll just need to bundle it when you’re done.

To keep the directory neat, make a directory for bundles.

mkdir bundled

Use the openapi bundle command to create new, single-file definitions from your multi-file definitions.

openapi bundle unbundled/writersAPI.yaml -o bundled/BundledWritersAPI.yaml
openapi bundle unbundled/projectsAPI.yaml -o bundled/BundledProjectsAPI.yaml

Now you have all kinds of API definitions. They don’t all contain the same information, but they are all made from the same source files.

.
├── bundled
│   ├── bundledProjectsAPI.yaml
│   └── bundledWritersAPI.yaml
├── openapi.yaml
└── unbundled
├── components
│   └── schemas
│   ├── closeoutDocument.yaml
│   ├── coInvestigator.yaml
│   ├── destination.yaml
│   ├── file.yaml
│   ├── libraryItem.yaml
│   ├── organization.yaml
│   ├── principalInvestigator.yaml
│   ├── programDirector.yaml
│   ├── programManager.yaml
│   ├── projectManager.yaml
│   ├── project.yaml
│   ├── technologyArea.yaml
│   └── workLocation.yaml
├── paths
│   ├── api@projects{.format}.yaml
│   ├── api@projects@{id}{.format}.yaml
│   └── api.yaml
├── projectsAPI.yaml
└── writersAPI.yaml
5 directories, 21 files

Where to go from here?

I’ve written a few OpenAPI definitions by hand. At the beginning, it usually is easier to just write in one file. At some point, though, that file is going to get unwieldy. At that time, think about unbundling.

Besides writer comfort, unbundled definitions are extremely handy for creating multiple API definitions from a single source. For example, instead of “Writers” and “Projects,” you might have “Internal” and “External”.

You can also use a CI to automate the bundling process. I had one client with public and private API documents. Their public docs were built using a tool that didn’t support multi-file definitions.

Using Github actions, I made a “script” that:

Bundled the file in the internal repo
Sent the bundled file to a public repo, where it was turned into documentation.

More than a year later, I see the public docs are still getting automatically updated. Nice!

Posts on Well-Shaped Words

Learn how to build a DuckLake, snapshot by snapshot

Short conceptual background

The components

The model

Procedure

Prerequisites

Attach the database

Create a schema

Create tables

Load data

Query

Delete

Time travel

Recover

Find old data files

Maintain

Final thoughts

Ideas to go further

Make software tutorials that stick

Start with context

Have them try first

Weave different tasks into the overall story

Reincorporate old information

Build a mock test at the end

Ask the user to elaborate

For learners and designers, more effort brings better results (and more effort)

Shell tools for writers

Find and replace

Count and sort frequency across files

Sort all errors by frequency

Filter JSON array of links checked

Round a float to decimal places with Hugo Shortcodes

Shortcode to round floats

Inline shortcodes scoped to one block

Fixed precision

Variable precision

Hugo inline spreadsheet

Examples of Inline calculation

Calculate summary statistics

Use slices to compute a table of dynamic values

Embed calculated values in LaTeX formulas

Limitations and ideas for development

Is ChatGPT a reliable research assistant? Building a books database

Results summary

Method

Goal

Prompt

Exploration and clean up

Results

Semi-structured prose to CSV

Quality of output for form field

From 1300 books, 228 genres

Accuracy of reported Year

Estimating accuracy rate

Discussion

Ideas for extended research

The journey was more fun than the result

Auto-generate glossaries With Hugo

Steps to generate a glossary

Write the YAML file

Write the glossary template

Use the shortcode in a page

More uses of the termbase

Reuse the definitions in tooltips

Make mini-glossaries

Nature gloss:

Protocol gloss:

Docs gloss:

Add supportive links to the glossary

Ways to improve

Write up: WTD Atlantic, 2023

Preparation

Tools

Presentations

Lessons

Wrap up

A shortcode for dynamic abbreviations

Hugo requirements

Steps to implement a conditional shortcode

Quality of output for `form` field

Accuracy of reported `Year`