GitHub Pages Custom Domain Setup
Configuring a custom domain for GitHub Pages requires coordinating DNS, repo settings, build tool config, and deployment mode — each can silently break the others.
All Lessons (125)
Live Infrastructure for Integration Testing
When local services are already running, skip mocks and test the real pipeline end-to-end
Adapter Pattern for Multi-Cloud Portability
Abstract base classes with minimal interfaces let the same RAG pipeline run on four different cloud providers without conditional logic in business code.
Code Review as Requirements Source
Systematic triage of code review findings produces a traceable requirements document — turning ad hoc observations into prioritized, implementable work.
Five-Stage Design-to-Execution Workflow
A repeatable workflow — Design, PDR, Plan, Execute, Commit — with table-driven task tracking and one-commit-per-phase discipline, applied across 18 project phases.
Lazy Imports for Optional Cloud Dependencies
Deferring cloud SDK imports to runtime lets the same codebase run with or without any given SDK installed, and enables testing without real dependencies.
Phased Multi-Cloud Infrastructure as Code
Three cloud stacks (AWS, Azure, GCP) built in separate phases with OIDC federation, avoiding cross-cloud coupling while sharing a common authentication pattern.
RAG Corpus Chunking Strategy
Splitting documents at H2 headings with stable IDs and content hashes produces predictable, debuggable chunks that support incremental re-indexing.
Rule-Based Gap Detection Without ML
Seven heuristic rules detect when a RAG corpus can't answer a query, without training data or a classifier — transparent and debuggable, with known trade-offs.
GitHub Pages Build Pipeline
Designing a GitHub Actions workflow that harvests, validates, builds, indexes, and deploys a static site.
Harvester Design Decisions
Key choices in building the lesson harvester — recursive scanning, path-based slugs, and integrated export generation.
Static Search with Pagefind
Integrating Pagefind for full-text search on a static site with no backend.
Validation Severity Model
Why the validator uses ERROR/WARNING/INFO levels and why warnings never fail the build.
Block 3: Calendar Optimization & Phase 4 Lessons
- 020 Lesson 020: DuckDB executemany Hangs — Use PyArrow Bulk Insert
- 021 Lesson 021: Calendar as Portfolio Optimization, Not Top-N Ranking
- 022 Lesson 022: Heuristic Month-Fit Scoring Without Text Metadata
- 023 Lesson 023: Maximum Marginal Relevance for Diversity-Aware Selection
- 024 Lesson 024: Hungarian Algorithm for Optimal Assignment
- 025 Lesson 025: Multiple Selection Methods as Baselines
- 026 Lesson 026: Formalizing De Facto Dependencies
- 027 Lesson 027: Migration Ordering and Apply-on-Use Gaps
Block 4: Synthetic Validation & S3-S4 Lessons
Block 5: Web App & Interactive Tooling Lessons
- 031 Lesson 031: Read-Only DB Connections for Web Layers
- 032 Lesson 032: Startup Cache for Interactive Scoring
- 033 Lesson 033: Vanilla JS SPA Without a Build Step
- 034 Lesson 034: Reusing Query Modules Across CLI and Web
- 035 Lesson 035: Design System Portability via Tokens
- 036 Lesson 036: Linter Rules vs. Framework Idioms
- 037 Lesson 037: Static Site Generation via Fetch Shim
- 038 Lesson 038: CI Path Portability and Release Artifacts
Block 6: Vision, Curation & AI-Assisted Workflow Lessons
- 039 Lesson 039: Mock Tagger Pattern for Vision Pipeline Testing
- 040 Lesson 040: Controlled Vocabulary as Schema Contract
- 041 Lesson 041: Utility Function Design for Synthetic Voting Bias
- 042 Lesson 042: Lift as the Primary Bias Detection Metric
- 043 Lesson 043: PII Sanitization in Static JSON Exports
- 044 Lesson 044: Acceptance Tests as Executable Specifications
- 045 Lesson 045: Embedding-Based Deduplication for Image Collection Curation
- 046 Lesson 046: Lazy Imports for Deployment Compatibility
- 047 Lesson 047: CLIP Zero-Shot as a Database Column Factory
- 048 Lesson 048: Greedy Max-Min Diversity Selection
- 049 Lesson 049: Drag-and-Drop as the Simplest Viable Interaction
- 050 Lesson 050: Connected Components for Transitive Deduplication
- 051 Lesson 051: Sigmoid Calibration for Domain-Specific CLIP Scores
- 052 Lesson 052: Incremental Feature Extraction Over Full Re-runs
- 053 Lesson 053: Audit-First Design
- 054 Lesson 054: Phased Autonomous Execution Plans
- 055 Lesson 055: Session Continuity via Documentation Artifacts
- 056 Lesson 056: Environment Self-Interference in AI-Assisted Development
- 057 Lesson 057: Test-Gated Commits at Scale
- Lessons Planned
Block 1
- Lesson: PEP 8 Compliance in Data Engineering Pipelines
- Lesson: Algorithm Selection — sklearn KMeans vs Pillow Quantize for Dominant Colors
- Lesson: Batch Database Operations — INSERT/UPDATE Patterns at Scale
- Lesson: Choosing k for k-Means Clustering in a Constrained Selection Problem
- Lesson: Concurrent HTTP Downloads with Connection Pooling
- Lesson: Disjoint Data Populations Breaking Multimodal Joins
- Lesson: DuckDB Single-Writer Constraint in Concurrent Pipelines
- Lesson: Per-Record Overhead That Doesn't Matter at 10 Rows Kills You at 12,000
- Lesson: Resume-Safe Pipeline Design — Surviving Interrupts and Partial Failures
- Lesson: Debugging with Surrogate Key Ranges
- Lesson: Building with Synthetic Data Before Real Data Arrives
Lesson 012: Bayesian Beta-Binomial Smoothing
The Artemis vote system shows 50 random images per ballot and asks voters to pick 5 favorites. With 500 ballots across 1...
Lesson 013: Elo Rating for Image Comparison
The Artemis pairwise voting mode shows two images side by side and asks "which is better?" This produces binary outcomes...
Lesson 014: Bradley-Terry-Luce and When to Skip It
We have pairwise comparison data (image A beats image B) and want the best possible strength estimates. Bradley-Terry-Lu...
Lesson 015: Borda Count for Ranked Voting
The Artemis category voting mode asks voters to rank their top 3 images within a category. We need to convert these part...
Lesson 016: Krippendorff's Alpha for Sparse Agreement
We want to measure whether voters agree on which images are good. With 100 voters and 12,217 images, the voter-image mat...
Lesson 017: Composite Scoring with Heterogeneous Signals
We have three different types of preference data — batch selection rates, Elo ratings from pairwise comparisons, and Bor...
Lesson 018: Run-ID Partitioned Scoring
The scoring pipeline will be re-run as new vote data arrives, as scoring methods are tuned, or as bugs are fixed. Each r...
Lesson 019: NULL as Honest Missing Data
Several columns in the scoring output have no meaningful value for most images. Only 200 images have Elo scores (from 2,...
Lesson 020: DuckDB executemany Hangs — Use PyArrow Bulk Insert
DuckDB's executemany with parameterized INSERT statements can hang indefinitely at scale (10K+ rows). Replacing it with ...
Lesson 021: Calendar as Portfolio Optimization, Not Top-N Ranking
When selecting a fixed-size collection where the items must work together (a calendar, a playlist, a portfolio, a menu),...
Lesson 022: Heuristic Month-Fit Scoring Without Text Metadata
When images lack text metadata (titles, descriptions, captions), month or season suitability can still be approximated f...
Lesson 023: Maximum Marginal Relevance for Diversity-Aware Selection
Greedy Maximum Marginal Relevance (MMR) is the practical default for selecting a diverse, high-quality subset from a lar...
Lesson 024: Hungarian Algorithm for Optimal Assignment
When you need to assign N items to N slots where each item-slot pair has a fitness score, the Hungarian algorithm gives ...
Lesson 025: Multiple Selection Methods as Baselines
When building an optimizer, always generate multiple candidate solutions using different methods — including at least on...
Lesson 026: Formalizing De Facto Dependencies
A dependency that's imported in production code but missing from the package manifest is a time bomb. It works on the de...
Lesson 027: Migration Ordering and Apply-on-Use Gaps
When a database migration creates a table that new code writes to, the migration must be applied before the code runs — ...
Lesson 028: Chi-Squared Tests for Bias Detection at Small Scale
We planted known biases in synthetic vote data — 10% of voters had position bias (preferring earlier-displayed images), ...
Lesson 029: Ground-Truth Recovery as Optimizer Validation
We have a calendar optimizer that selects 13 images from 12,217 using a weighted objective function (popularity, diversi...
Lesson 030: Reliability Delta as Noise Measurement
We know that 20% of synthetic voters are intentionally noisy (10% position-biased, 10% random). We compute Krippendorff'...
Lesson 031: Read-Only DB Connections for Web Layers
When an embedded database (DuckDB, SQLite) serves both a batch pipeline and an interactive web app, the web layer should...
Lesson 032: Startup Cache for Interactive Scoring
When an interactive web app needs sub-100ms responses from a scoring function that depends on large lookup tables, load ...
Lesson 033: Vanilla JS SPA Without a Build Step
A hash-routed single-page application built with vanilla JavaScript, ES modules, and dynamic import() can deliver a func...
Lesson 034: Reusing Query Modules Across CLI and Web
When a CLI pipeline and a web API need the same data, import the query functions directly rather than duplicating SQL. A...
Lesson 035: Design System Portability via Tokens
A design system built on CSS custom properties (design tokens) can be shared across completely independent frontends — s...
Lesson 036: Linter Rules vs. Framework Idioms
When a linter rule flags code that follows a framework's official pattern, suppress the rule per-line with noqa rather t...
Lesson 037: Static Site Generation via Fetch Shim
A FastAPI + JavaScript SPA can be deployed to GitHub Pages without rewriting frontend code by using a fetch shim — a sma...
Lesson 038: CI Path Portability and Release Artifacts
When a project develops on Windows but deploys via CI on Linux, hardcoded paths like D:/artemis/warehouse.duckdb will fa...
Lesson 039: Mock Tagger Pattern for Vision Pipeline Testing
The vision tagging pipeline uses Qwen2.5-VL (a 7B-parameter vision-language model) to classify image attributes. Running...
Lesson 040: Controlled Vocabulary as Schema Contract
The vision tagging pipeline needs a consistent set of image attributes shared across five components: the vision model p...
Lesson 041: Utility Function Design for Synthetic Voting Bias
Synthetic vote generation needs to produce votes that exhibit detectable attribute-based bias while remaining statistica...
Lesson 042: Lift as the Primary Bias Detection Metric
Block-aware statistics need a metric that answers: "does this voting block select images with attribute X more than expe...
Lesson 043: PII Sanitization in Static JSON Exports
The static site serves pre-built JSON files from a public URL. The warehouse database contains voter surrogate keys (vot...
Lesson 044: Acceptance Tests as Executable Specifications
The biased voting blocks pipeline spans six components: config validation, vote generation, attribute analysis, cluster ...
Lesson 045: Embedding-Based Deduplication for Image Collection Curation
When working with a large image collection from an automated source, assume near-duplicates dominate the pool until prov...
Lesson 046: Lazy Imports for Deployment Compatibility
Import heavy dependencies inside the function that uses them, not at module scope. A module-level import numpy means eve...
Lesson 047: CLIP Zero-Shot as a Database Column Factory
A single CLIP model, used for zero-shot classification against descriptive text prompts, functions as a general-purpose ...
Lesson 048: Greedy Max-Min Diversity Selection
To select k items that maximally represent the diversity within a group, iteratively pick the item most distant from all...
Lesson 049: Drag-and-Drop as the Simplest Viable Interaction
When the user's mental model is "put this thing in that slot," drag-and-drop is less code and more intuitive than altern...
Lesson 050: Connected Components for Transitive Deduplication
When deduplicating by pairwise similarity, use graph connected components to group items — not naive pair-based merging....
Lesson 051: Sigmoid Calibration for Domain-Specific CLIP Scores
CLIP logits have domain-specific distributions. Converting them to meaningful [0,1] confidence scores requires a sigmoid...
Lesson 052: Incremental Feature Extraction Over Full Re-runs
When adding new features to an existing collection, delete-and-rewrite only the new columns rather than re-processing ev...
Lesson 053: Audit-First Design
Before writing any code for a new feature, produce a written audit of the existing codebase: what exists, what can be re...
Lesson 054: Phased Autonomous Execution Plans
Breaking large projects into numbered, independently shippable phases — each with explicit entry criteria, exit criteria...
Lesson 055: Session Continuity via Documentation Artifacts
When working across multiple AI-assisted sessions, continuity must be encoded in files, not in conversation history. A s...
Lesson 056: Environment Self-Interference in AI-Assisted Development
An AI coding assistant that launches background processes (dev servers, database connections, build watchers) will fight...
Lesson 057: Test-Gated Commits at Scale
Gate every commit on a passing test suite, not on "the feature looks done." With 1,500+ tests across a project, the suit...
Lesson: Algorithm Selection — sklearn KMeans vs Pillow Quantize for Dominant Colors
The visual feature extraction pipeline ran sklearn's KMeans on every thumbnail to find 5 dominant colors. Each call took...
Lesson: Batch Database Operations — INSERT/UPDATE Patterns at Scale
Multiple pipeline stages in Artemis started with per-row INSERT or UPDATE patterns that worked fine during development (...
Lesson: Building with Synthetic Data Before Real Data Arrives
The Artemis project needed voter preference data to build its statistical models and calendar optimizer. But real vote d...
Lesson: Choosing k for k-Means Clustering in a Constrained Selection Problem
We needed to cluster 12,217 Artemis II mission photos into visually distinct groups. The clusters serve a specific downs...
Lesson: Concurrent HTTP Downloads with Connection Pooling
The original thumbnail downloader processed 12,217 images sequentially. Each download created a new httpx.Client instanc...
Lesson: Debugging with Surrogate Key Ranges
When investigating why multimodal clustering produced zero results, the breakthrough came from a simple query: sql SELEC...
Lesson: Disjoint Data Populations Breaking Multimodal Joins
Multimodal clustering required images to have both CLIP image embeddings AND text embeddings. The intersection of these ...
Lesson: DuckDB Single-Writer Constraint in Concurrent Pipelines
While developing the concurrent thumbnail downloader, the DuckDB warehouse file (warehouse.duckdb) became locked by the ...
Lesson: PEP 8 Compliance in Data Engineering Pipelines
Python scripts in the Artemis project span multiple roles: XML/JSON migration, schema validation, metadata enrichment, t...
Lesson: Per-Record Overhead That Doesn't Matter at 10 Rows Kills You at 12,000
The original thumbnail downloader worked flawlessly on 5 images during development. When scaled to 12,217 images, it was...
Lesson: Resume-Safe Pipeline Design — Surviving Interrupts and Partial Failures
The thumbnail download process was killed multiple times during development — once to change the rate limit, once to adj...
Lessons Planned
Future lessons identified from the Claude Code usage insights analysis (64 sessions, 150 commits, May 2026). These are p...
AI-Graded Content Validation
Large question banks authored by multiple sources (human or AI) accumulate factual errors that are invisible to structur...
Answer Position Bias
When humans author multiple-choice questions, the correct answer tends to cluster in certain positions (often B or C). T...
Building a Codebase Review Skill
A structured review skill turns the ad-hoc "look at this code and tell me what's wrong" request into a repeatable, evide...
Building a Lessons Skill for Claude Code
A Claude Code skill file is a structured prompt that turns a repeatable workflow into a single slash command. The skill'...
Building a Phase Execution Skill
A phased plan is only as good as its execution discipline. A /phase skill automates the mechanical parts of plan executi...
Bulk Metadata Enrichment Scripts
When hundreds of data records need the same type of update (adding titles, categories, tags, or enriched descriptions), ...
Client-Side State Persistence with localStorage
localStorage can serve as a full persistence layer for client-side applications when the data is user-specific, the data...
Code Review Driven Remediation
A whole-codebase code review is only as valuable as the remediation that follows it. The review itself produces a findin...
Content Quality Auditing at Scale
When you have hundreds or thousands of content items authored by different sources at different times, quality varies wi...
Content Security Policy for Static Sites
A Content Security Policy (CSP) is achievable on a static site without server-side headers by using a <meta> tag. The ch...
Design System Migration
Migrating an existing multi-page site to a design system is a page-by-page operation, not a big-bang rewrite. The design...
Design-First Development
Writing a design document and a Physical Design Requirements (PDR) document before coding catches architectural mistakes...
Equivalence Testing During Format Migration
When migrating a data format (XML to JSON) that feeds a rendering pipeline, the only way to prove the migration is corre...
Hint Quality as a Spectrum
A progressive hint system (brief nudge → full explanation → deep-dive knowledge) is more pedagogically effective than a ...
Integration Testing a DOM Application with jsdom
A browser-based application that uses DOM APIs (querySelector, innerHTML, addEventListener) can be integration-tested in...
Legacy Artifact Removal
After a migration, the old system's artifacts (files, code, tests, scripts) must be actively removed in a deliberate cle...
Lessons Learned as a Practice
Systematically extracting lessons from project work — and writing them as standalone documents — turns ephemeral experie...
Phased Release Planning
Breaking large features into ordered phases — each independently shippable, each ending with a commit — transforms ambit...
Provider-Agnostic Plugin Architecture
When a system needs to support multiple "providers" (vendors, brands, data sources) that share the same behavior but dif...
Scaling Content Without Scaling Complexity
Adding 50+ exams across 10 providers to a quiz application required zero changes to the core quiz engine, data loader, o...
Schema Enforcement at the Data Layer
Adding runtime schema validation to your data loading layer catches entire categories of bugs that would otherwise surfa...
Schema Variant Consolidation
When multiple people or processes author data files for the same system without a shared schema, variant schemas emerge....
Static Site as Application Platform
A full-featured application (quiz engine, progress persistence, scoring, results dashboards, 10 providers, 50+ exams) ca...
Testing Provider Detection Logic
When critical logic is embedded in a class that's hard to test (DOM-coupled UI class), developers sometimes copy the log...
Verbatim Answer Leakage in Hints
When hints contain the exact text of the correct answer choice, they short-circuit learning. The learner reads the hint,...
XML Entity Encoding Pitfalls
XML entity encoding bugs (Q&A vs Q&A) are the most common class of data corruption in XML content pipelines. They're...
XML to JSON Migration
When migrating a live data format (XML to JSON), the key risk is not the conversion itself — it's proving that the new f...
XSS in Trusted-Data Applications
Using innerHTML to render content from "your own" data files (XML, JSON, markdown) is an XSS vulnerability even when the...
Choosing the Right Similarity Algorithm
Before choosing a similarity algorithm, understand whether your data uses binary membership (item has feature or doesn't...
Crosswalk and Taxonomy Evolution
Occupation codes are not stable identifiers across taxonomy revisions. The same SOC code can refer to different occupati...
Data Quality Traps in Government Sources
Government data sources contain artifacts of their internal production processes — temp files in archives, renamed colum...
Derived Metrics from Base Observations
Base observations are source truth; derived values are computed artifacts. Mixing them in the same table creates ambigui...
Design System Cross-Pollination
When two projects share an author, the stronger design system should inform the weaker one — but adopting visual feel is...
Dimensional Modeling for Labor Data
A four-layer warehouse architecture (raw, staging, core, marts) with strict separation of concerns at each layer produce...
Extract Patterns for Government APIs
Federal data sources are not designed for programmatic access. They block bare HTTP requests, publish in heterogeneous f...
Fetch Shim Architecture
A static site can replicate a dynamic API by intercepting JavaScript fetch() calls and redirecting them to pre-built JSO...
Geography Comparison Pitfalls
Geographic wage comparisons are inherently incomplete: nominal gaps do not account for cost-of-living differences, suppr...
Idempotent Pipeline Design
Data pipelines fail — downloads timeout, parsers hit unexpected formats, database connections drop. Idempotency (running...
Inflation Adjustment with CPI
Comparing nominal wages across years is misleading because the dollar's purchasing power changes over time. Converting t...
Multi-Vintage Query Pitfalls
Once a warehouse holds multiple vintages of the same dataset, every query must explicitly decide whether it wants the la...
Ranked Movers and Outlier Interpretation
Percentage changes on small bases are statistically volatile and can dominate ranked lists even when the absolute econom...
Schema Drift Detection
Government data sources change column names, add or remove columns, and retype columns between releases — often without ...
Static Site Generation
A server-side web application can be deployed to a static hosting platform by pre-rendering every page and API response ...
Testing and Deployment
Separating tests by their infrastructure requirements — fixtures-only, in-memory server, real database — lets CI run fas...
The Federal Labor Data Landscape
When building an analytical warehouse from multiple federal data products, the single most important architectural decis...
The Multi-Vintage Challenge
When loading multiple vintages of the same dataset, dimension tables must deduplicate on business key alone — not on bus...
Thread-Safe Database Connections
When a web framework dispatches synchronous endpoint handlers to a thread pool, a shared database connection will produc...
Time-Series Normalization
Fact tables store snapshots — single measurements at single points in time. Time-series analysis requires a separate nor...
UI-Data Alignment
A web application that shows buttons, links, or filters for data that does not exist creates a worse experience than one...