When your serverless backend can't run PDF libraries (no headless Chrome, no native modules), generate the PDF in the browser instead. The server renders an HTML page with the data embedded, and client-side JavaScript builds and downloads the PDF. The user doesn't notice the difference.
D1 is serverless SQLite hosted by Cloudflare. It gives you a real SQL database accessible only from your Worker — no connection strings, no connection pooling, no database server to manage. For small apps, it eliminates the entire database operations layer.
Cloudflare Workers are serverless JavaScript functions that run at the edge — no server to manage, no container to configure. They wake up on each request, execute, and sleep. For small to medium web apps, they replace traditional backend servers entirely.
In a serverless workflow where `wrangler deploy` pushes code directly to production, deployment is decoupled from git. This means there's no CI/CD pipeline gating production changes — the developer must impose their own discipline. Treat every production deploy like a `git push --force`: require exp...
When you add a database column via direct SQL instead of a migration file, your dev environment won't have it. The code works in production (where you ran the SQL) but crashes in dev (where the column doesn't exist). Always use migration files, even for "quick" schema changes.
When piping a value to a CLI tool that stores secrets, `echo` adds a trailing newline that becomes part of the stored value. This silently breaks any credential that's compared byte-for-byte — OAuth client IDs, API tokens, webhook secrets. Always use `printf` instead.
Google OAuth2 lets users sign in with their Google account. Your server redirects to Google, Google authenticates the user, and redirects back with a code. You exchange the code for the user's email. The entire flow is four HTTP calls and requires no client-side SDK.
When users can sign up via OAuth (Google, Apple, etc.), they bypass your signup form — and any required fields on it. If your app requires data that OAuth doesn't provide (a phone number, a company name, a role), you need a gate between login and the main app that collects it before proceeding.
When a phone number appears in your product, decide early whether it's an identity (the account itself) or data (a field on a record). Conflating the two creates the wrong data model, the wrong auth flow, and forces users into a single-phone-per-account constraint that doesn't match reality.
How you describe your product to users determines which features you build, which fields you show, and which language your code uses. A positioning document written before implementation saves more engineering time than any technical design doc because it eliminates features before they're built.
You can ship a production web app in a single day using technologies you've never touched before — if you choose technologies that have small API surfaces, skip the tutorials, and build the real thing from the start. The fastest way to learn a tool is to use it on a real problem, not to study it in...
Twilio Verify is a two-API-call service for phone number verification. You call "send code," Twilio texts a 6-digit code to the phone. You call "check code" with what the user entered, Twilio tells you if it matches. You never see, store, or manage the code yourself.
Wrangler is the CLI tool that manages the entire Cloudflare Workers lifecycle — creating databases, setting secrets, deploying code, tailing logs, and managing environments. It replaces what would otherwise be a CI/CD pipeline, a deployment script, and a cloud console.
The Artemis vote system shows 50 random images per ballot and asks voters to pick 5 favorites. With 500 ballots across 12,217 images, most images are shown only 1-2 times. A raw selection rate of "1 out of 1 shown = 100%" is meaningless — it tells you nothing about whether the image is actually pref...
The Artemis pairwise voting mode shows two images side by side and asks "which is better?" This produces binary outcomes (winner / loser) for specific pairs, not absolute ratings. We need to convert these relative comparisons into a single continuous strength score per image that can be combined wit...
We have pairwise comparison data (image A beats image B) and want the best possible strength estimates. Bradley-Terry-Luce (BTL) is the textbook model for this — it's more principled than Elo. But with 2,000 comparisons across 12,217 images, we chose to skip BTL entirely. This lesson explains what B...
The Artemis category voting mode asks voters to rank their top 3 images within a category. We need to convert these partial rankings into numeric scores that can be aggregated across voters and combined with batch and pairwise preference signals.
We want to measure whether voters agree on which images are good. With 100 voters and 12,217 images, the voter-image matrix is >98% missing — most voters never saw most images. Standard agreement metrics require complete matrices. We need a reliability measure that handles extreme sparsity.
We have three different types of preference data — batch selection rates, Elo ratings from pairwise comparisons, and Borda scores from category rankings. Each covers a different subset of images, uses a different scale, and captures a different aspect of preference. Most images have data from only o...
The scoring pipeline will be re-run as new vote data arrives, as scoring methods are tuned, or as bugs are fixed. Each run produces a full set of scores for all 12,217 images. If each run overwrites the previous scores, we lose the ability to compare methods, audit changes, or roll back to a known-g...
Several columns in the scoring output have no meaningful value for most images. Only ~200 images have Elo scores (from 2,000 pairwise votes). Only ~150 images have Borda scores (from 250 category rankings). The BTL model wasn't run at all. Fleiss' kappa and Kendall's W can't be computed with incompl...
DuckDB's `executemany` with parameterized INSERT statements can hang indefinitely at scale (10K+ rows). Replacing it with a PyArrow table and `INSERT INTO ... SELECT * FROM tbl` completes the same work in under a second. When DuckDB is your warehouse, bulk writes should go through its columnar inges...
When selecting a fixed-size collection where the items must work together (a calendar, a playlist, a portfolio, a menu), the problem is constrained set optimization — not top-N ranking. The best collection often contains none of the individually top-ranked items, because collection-level properties...
When images lack text metadata (titles, descriptions, captions), month or season suitability can still be approximated from visual features alone — color temperature, brightness, contrast, and content flags. The signal is coarse (3-4 seasonal buckets, not 13 distinct months) but sufficient to preven...
Greedy Maximum Marginal Relevance (MMR) is the practical default for selecting a diverse, high-quality subset from a large pool. At each step, it picks the item that maximizes quality minus similarity to already-selected items. It runs in O(K × N) time, requires no optimization library, and naturall...
When you need to assign N items to N slots where each item-slot pair has a fitness score, the Hungarian algorithm gives the provably optimal assignment in O(N^3) time. For small N (≤50), it runs in microseconds and eliminates the need for greedy heuristics, manual tuning, or iterative search. Use sc...
When building an optimizer, always generate multiple candidate solutions using different methods — including at least one naive baseline. The baseline proves the optimizer adds value. The alternatives expose the trade-off frontier. Without baselines, you can't distinguish "good optimization" from "e...
A dependency that's imported in production code but missing from the package manifest is a time bomb. It works on the developer's machine (where the package was installed for something else) and fails on fresh installs, CI, or new team members. Audit imports against declared dependencies whenever ad...
When a database migration creates a table that new code writes to, the migration must be applied before the code runs — not just before the next CLI invocation. If the code path that triggers the write doesn't call `apply_migrations()`, the table won't exist at runtime, even though the migration fil...
We planted known biases in synthetic vote data — 10% of voters had position bias (preferring earlier-displayed images), 20% had visual-drama bias (preferring dramatic images). We need statistical tests that can detect these biases with only 100 voters and 500 ballots, without requiring heavy statist...
We have a calendar optimizer that selects 13 images from 12,217 using a weighted objective function (popularity, diversity, month-fit, cover-fit, redundancy penalty). The optimizer reports an objective score, but a high score doesn't prove the optimizer is selecting the *right* images — it could be...
We know that 20% of synthetic voters are intentionally noisy (10% position-biased, 10% random). We compute Krippendorff's alpha on all voters and get a moderate value (~0.52). But how much of the low agreement is caused by these noisy voters vs. genuine preference diversity among neutral voters? We...
When an embedded database (DuckDB, SQLite) serves both a batch pipeline and an interactive web app, the web layer should open the database in read-only mode. This avoids writer-lock conflicts entirely and makes the architecture self-documenting: the web app *cannot* mutate the warehouse, by construc...
When an interactive web app needs sub-100ms responses from a scoring function that depends on large lookup tables, load those tables into memory at startup rather than querying the database per request. The cache size is bounded (you know exactly what's in the warehouse), startup cost is a one-time...
A hash-routed single-page application built with vanilla JavaScript, ES modules, and dynamic `import()` can deliver a functional multi-page experience — navigation, pagination, filtering, modals, live API calls — with zero build toolchain. For internal tools and single-user apps, this eliminates npm...
When a CLI pipeline and a web API need the same data, import the query functions directly rather than duplicating SQL. Add the serialization layer (Pydantic models, JSON responses) at the API boundary, not in the query module. The query module returns plain Python objects (dataclasses, dicts, tuples...
A design system built on CSS custom properties (design tokens) can be shared across completely independent frontends — static HTML pages, vanilla JS SPAs, embedded widgets — by copying two files. The tokens provide visual consistency without requiring a shared component library, a build system, or a...
When a linter rule flags code that follows a framework's official pattern, suppress the rule per-line with `noqa` rather than restructuring the code. Linter rules encode general best practices; framework idioms encode domain-specific patterns that intentionally violate those practices. Restructuring...
A FastAPI + JavaScript SPA can be deployed to GitHub Pages without rewriting frontend code by using a **fetch shim** — a small JavaScript interceptor injected into `index.html` that redirects API calls to pre-generated JSON files and handles filtering, sorting, and pagination client-side. The build...
When a project develops on Windows but deploys via CI on Linux, hardcoded paths like `D:/artemis/warehouse.duckdb` will fail silently or crash. Every path that differs between dev and CI must be configurable via environment variable. Similarly, large binary dependencies (databases, model weights) sh...
The vision tagging pipeline uses Qwen2.5-VL (a 7B-parameter vision-language model) to classify image attributes. Running the real model requires a GPU, takes seconds per image, and produces non-deterministic outputs. The full pipeline — config loading, tagging, derived label computation, DB persiste...
The vision tagging pipeline needs a consistent set of image attributes shared across five components: the vision model prompt, the attribute parser/validator, the database schema, the voting block config, and the cluster labeling engine. If any component uses an attribute code the others don't recog...
Synthetic vote generation needs to produce votes that exhibit detectable attribute-based bias while remaining statistically plausible. A biased voter block that always votes for images with specific attributes produces trivially detectable (and unrealistic) bias. A block with too much noise produces...
Block-aware statistics need a metric that answers: "does this voting block select images with attribute X more than expected?" Raw selection counts don't work because blocks have different sizes. Rate differences (block rate - global rate) are hard to interpret when base rates vary widely. The metri...
The static site serves pre-built JSON files from a public URL. The warehouse database contains voter surrogate keys (`voter_sk`), hashed voter IDs (`voter_public_hash`), random seeds, config hashes, and local file paths. None of these should appear in public-facing JSON. The sanitization must be rel...
The biased voting blocks pipeline spans six components: config validation, vote generation, attribute analysis, cluster analysis, score/calendar impact, and static export. Unit tests cover each component in isolation, but the interesting behaviors — "does a biased block produce detectable lift in th...
When working with a large image collection from an automated source, assume near-duplicates dominate the pool until proven otherwise. Embedding cosine similarity with connected-component grouping reduces a collection to its unique members in minutes, but the threshold choice dramatically affects the...
Import heavy dependencies inside the function that uses them, not at module scope. A module-level `import numpy` means every consumer of that module — including lightweight build scripts, CI pipelines, and serverless functions — must have numpy installed, even if they never call the code path that n...
A single CLIP model, used for zero-shot classification against descriptive text prompts, functions as a general-purpose column generator for structured databases. Each new prompt produces a new confidence column — no training, no fine-tuning, no labeled data. The cost of adding a column is one forwa...
To select k items that maximally represent the diversity within a group, iteratively pick the item most distant from all already-selected items. This greedy max-min approach is O(n×k), produces near-optimal diversity in practice, and avoids the NP-hard max-dispersion problem entirely.
When the user's mental model is "put this thing in that slot," drag-and-drop is less code and more intuitive than alternatives like dropdowns, search dialogs, or multi-step wizards. The key is spatial co-visibility: the source pool and target slots must be on screen simultaneously so the user can se...
When deduplicating by pairwise similarity, use graph connected components to group items — not naive pair-based merging. Pairwise similarity is not transitive in theory (A~B and B~C doesn't guarantee A~C), but for near-duplicates in practice, transitivity holds and connected components correctly gro...
CLIP logits have domain-specific distributions. Converting them to meaningful [0,1] confidence scores requires a sigmoid transform calibrated to the actual logit range in your image collection. A universal threshold doesn't work — the sigmoid center and scale must be tuned empirically by examining l...
When adding new features to an existing collection, delete-and-rewrite only the new columns rather than re-processing everything. The key enabler is tagging each row with its source (model version, label source, attribute code) so that surgical deletes and inserts are possible without touching exist...
Before writing any code for a new feature, produce a written audit of the existing codebase: what exists, what can be reused, where new code slots in. The audit document prevents reimplementing existing functionality and identifies the exact extension points — saving more time than it costs to write...
Breaking large projects into numbered, independently shippable phases — each with explicit entry criteria, exit criteria, and a commit checkpoint — transforms ambitious multi-session work from a coordination problem into a queue of self-contained tasks. The plan file is both the work instruction and...
When working across multiple AI-assisted sessions, continuity must be encoded in files, not in conversation history. A startup document, a plan file with status tracking, and a project CLAUDE.md that reflects current state eliminate ramp-up overhead and prevent context loss from session clears and c...
An AI coding assistant that launches background processes (dev servers, database connections, build watchers) will fight with its own previous instances over shared resources like ports and file locks. Explicit cleanup before each launch — kill orphan processes, release locks, verify port availabili...
Gate every commit on a passing test suite, not on "the feature looks done." With 1,500+ tests across a project, the suite catches regressions that visual inspection misses — wrong column names, broken imports, type mismatches, off-by-one errors. The test suite is the contract for "this commit is saf...
When serving DuckDB through a multi-threaded web framework (FastAPI/uvicorn), never share a single connection object across concurrent request handlers. Instead, call `conn.cursor()` to create a per-request cursor. DuckDB's Python driver does not support concurrent queries on the same connection fro...
When a dashboard metric can be computed either from a mutable state flag or from an immutable record table, always derive it from the immutable source. Mutable flags reflect the *current* state, which may not be the state your metric is trying to describe. Immutable fact tables preserve the *histori...
Adding a brief "why this page matters" block at the top of every page in a data application transforms it from an internal tool into a self-guided case study. A single sentence of context lets a reviewer understand what they're looking at without reading documentation or having the author present to...
When the same project-level number (image count, cluster count, lesson count) appears in multiple frontend modules, centralize it in a single metadata object. Better still, fetch live counts from the API at render time and use the centralized constant only as a fallback. Hardcoded numbers scattered...
Add a numbered "review this project in N minutes" path to the homepage of any portfolio project or case study. Without explicit guidance, reviewers wander randomly through pages and miss the strongest parts of the work. A curated path ensures every reviewer sees the same narrative arc, regardless of...
Any frontend page that fires multiple `fetch()` calls via `Promise.all()` is an implicit concurrency test for the backend. If your API endpoints work individually via `curl` but fail when the browser loads a page that hits them simultaneously, you have a shared-state concurrency bug — not a data or...
A single-page application rendered entirely in JavaScript is invisible to search engine crawlers that don't execute JS. Adding a `<noscript>` block with the project's core content — title, summary, key links, and attribution — provides a crawlable baseline that costs minutes to implement and ensures...
The visual feature extraction pipeline ran sklearn's `KMeans` on every thumbnail to find 5 dominant colors. Each call took ~147ms per image. For 12,217 images, that's ~30 minutes of CPU time on dominant color extraction alone — a feature that contributes a single JSON column to the feature table.
Multiple pipeline stages in Artemis started with per-row INSERT or UPDATE patterns that worked fine during development (5-10 rows) but became bottlenecks at full scale (12,000+ rows). The per-row pattern appeared in three places:
The Artemis project needed voter preference data to build its statistical models and calendar optimizer. But real vote data from ArtemisTimeline.com wasn't yet available — the vote export hadn't been requested, and the site's API only exposes aggregate leaderboards, not raw ballots.
We needed to cluster 12,217 Artemis II mission photos into visually distinct groups. The clusters serve a specific downstream purpose: ensuring the final 13-image calendar selection has visual diversity. The choice of k (number of clusters) directly affects whether the optimization can produce a goo...
The original thumbnail downloader processed 12,217 images sequentially. Each download created a new `httpx.Client` instance, which meant a fresh TCP connection and TLS handshake for every single request — all to the same Cloudflare R2 CDN endpoint. At 0.1s rate limiting plus ~50-200ms connection ove...
When investigating why multimodal clustering produced zero results, the breakthrough came from a simple query:
Multimodal clustering required images to have both CLIP image embeddings AND text embeddings. The intersection of these two sets was empty — 0 images qualified. The clustering silently logged a warning and returned 0 results. The pipeline appeared to work, but an entire analysis dimension produced n...
While developing the concurrent thumbnail downloader, the DuckDB warehouse file (`warehouse.duckdb`) became locked by the download process. Any attempt to check progress, run `artemis-pipeline status`, or open a second connection failed with:
Python scripts in the Artemis project span multiple roles: XML/JSON migration, schema validation, metadata enrichment, test harnesses, and lesson harvesting. Without a consistent style standard, each script drifts toward the author's (or AI assistant's) habits — camelCase here, inconsistent indentat...
The original thumbnail downloader worked flawlessly on 5 images during development. When scaled to 12,217 images, it was unacceptably slow — not because of network latency, but because of per-image overhead that was invisible at small scale.
The thumbnail download process was killed multiple times during development — once to change the rate limit, once to adjust the timeout, once at the user's request. Each time, the question was: how much progress was lost? Can we pick up where we left off?
In the Astro ecosystem, plugin packages (`@astrojs/*`) release independently of the core framework and frequently break peer dependency compatibility. Pin plugin versions explicitly and test upgrades in isolation rather than accepting latest.
When multiple domains share identical page structures but differ only in subject matter, model the variation as typed content collections and render everything through shared components. The architecture's value comes from enforcing uniform structure via schemas while allowing unlimited content vari...
GitHub Pages deployment with static site generators has three independently-failing configuration points — workflow file location, CNAME record, and site URL in the build config — and all three must be correct simultaneously. A deploy that "almost works" is usually missing exactly one of these.
When building a platform that serves N variants of the same structure, start with a single consolidated site that treats variation as data, not as separate projects. Late consolidation — after scaffolding N separate sites — is expensive and produces a massive, risky changeset.
Astro's scoped `<style>` blocks do not penetrate MDX `<Content />` output. Any styles that need to reach MDX-rendered HTML must live in global CSS or use `:global()` selectors. This is a framework-level constraint, not a bug to work around.
When building a system that will serve N instances of the same pattern, build one instance end-to-end first — from scaffold through deployment — before replicating. The prototype surfaces architectural assumptions that only become visible under real content, real routing, and real build constraints.
Relative links in templated multi-section static sites break silently when page nesting depth varies. Use a systematic link strategy — either always-absolute paths from the site root, or a helper that resolves relative to the current topic — rather than hand-coding relative hrefs in content files.
Astro's `site` and `base` config fields look like they do the same thing but serve completely different purposes. Getting them wrong produces a site that deploys successfully but generates incorrect canonical URLs, broken sitemaps, or broken routing -- and the failure mode changes depending on wheth...
A site can build and run perfectly in dev mode while harboring type errors that fail `astro check` in CI. If CI gates on type checking (and it should), these latent errors become deploy blockers that surface only after pushing -- never during local development.
When migrating a static site to a hosting platform that serves from a subdirectory (e.g., `username.github.io/repo/`), every hardcoded internal link breaks. The migration isn't done when the deploy workflow is green -- it's done when every `href`, asset path, and client-side route has been audited f...
When integrating with multiple external APIs that share a common pipeline contract, define an abstract base class that handles cross-cutting concerns (rate limiting, timeouts, credential redaction, error classification) and requires subclasses to implement only the source-specific logic (`fetch` and...
When building a system that depends on external data sources, templates, or configuration-driven behavior, ship the configuration files before the code that consumes them. This forces you to validate your data model against real requirements before investing in implementation, and it makes each subs...
When deduplicating records from heterogeneous sources with varying ID reliability, use a priority-ordered cascade of match strategies — from strongest (source-native IDs) to weakest (fuzzy metadata). Check each level in order and stop at the first match. This avoids both false negatives (missed dupl...
SQLite's FTS5 extension provides production-quality full-text search without an external service. The key to making it work reliably is sync triggers (not application-level writes), `rowid`-based joins (not column joins), and treating the FTS table as a read-only projection of the main table.
Mock-based tests validate your code's logic, not your assumptions about the external API. When an adapter passes all mock tests but fails against the real API, the bug is almost always in the mock — you encoded incorrect assumptions about field names, response structure, or protocol behavior.
For a full-stack application built from scratch, a strict bottom-up phase order — schema, models, data, services, pipeline, API, UI, export — with one commit per phase and a green test suite at each boundary, produces a codebase where every layer is testable in isolation and integration bugs surface...
When scaling a plugin architecture, ship configuration and data files first (before any code), tier new plugins by API complexity, and close with registry-level consistency tests. This ordering catches integration mismatches early and keeps each phase independently shippable.
When ranking records from heterogeneous sources, decompose the score into independent components with explicit weights, each normalized to 0.0–1.0. This makes the scoring system auditable (you can explain why a record scored high), tunable (change one weight without affecting others), and extensible...
When stripping a codebase down to a subset of its functionality, remove in dependency order — packages first, then CLI registrations, then migrations, then dependencies, then tests, then deployment artifacts. Each commit should leave the system runnable, not just compilable.
When extracting structured identifiers (model names, product versions, package names) from unstructured text, a dictionary of known values with normalization beats regex-only extraction. Regex handles the general case; the dictionary handles the important cases correctly.
When a CLI tool supports multiple authentication methods with different billing paths, scripts that invoke it must explicitly select the intended billing path — otherwise, environment variable precedence silently routes charges to the wrong budget.
When a data pipeline has multiple interacting failure modes, writing a design document that catalogs all errors before fixing any of them produces better fixes than addressing errors one at a time. The design doc reveals which failures share root causes and which fixes would conflict.
Shared browser instances in async code need explicit synchronization at creation time and explicit cleanup at shutdown. Without both, you get leaked browser processes from races and resource warnings from incomplete teardown — problems that surface only under concurrent load, not in unit tests.
A git revert is a signal that the original change had a design gap — not just a bug. When you revert, don't just re-implement the same approach with a fix; use the revert as a forcing function to write down what the original approach missed before trying again.
Never bundle additive changes (new sources) with destructive changes (dropping existing pages) in one commit. If a rollback is needed, you lose the additions along with the removals — and untangling them under pressure wastes time.
When primary collection methods fail due to anti-bot defenses (Cloudflare, JS rendering), Google News RSS feeds provide a reliable fallback that requires no browser automation — but RSS item bodies are often useless title echoes that need enrichment from the actual article pages.
SQLite supports exactly one concurrent writer. When an async pipeline shares a database with a long-running server process, the fix is architectural (serialize writers) — not a PRAGMA tweak. WAL mode reduces contention but does not eliminate it.
When building reports that combine deterministic data extraction with LLM synthesis, split them into two explicit stages: a repeatable extraction step that produces a structured intermediate file, and a separate synthesis step that feeds that file to the LLM. This makes each stage independently test...
Large question banks authored by multiple sources (human or AI) accumulate factual errors that are invisible to structural validation. Using an LLM to independently attempt each question blind — without seeing the answer key — and then comparing its answer to the stored correct answer, surfaces wron...
When humans author multiple-choice questions, the correct answer tends to cluster in certain positions (often B or C). Test-takers learn this pattern and use it as a guessing heuristic. Randomizing answer positions eliminates this bias and makes the quiz a better learning tool.
A structured review skill turns the ad-hoc "look at this code and tell me what's wrong" request into a repeatable, evidence-based audit that produces the same quality of findings regardless of who runs it or when. The skill's value comes from its taxonomy of problem categories (derived from real iss...
A Claude Code skill file is a structured prompt that turns a repeatable workflow into a single slash command. The skill's power comes from clearly separating modes (read-only vs write), defining explicit quality contracts for outputs, and providing the AI with enough heuristics to make judgment call...
A phased plan is only as good as its execution discipline. A `/phase` skill automates the mechanical parts of plan execution — picking the next task, timestamping start/completion, verifying work, committing atomically — so the human (or AI) can focus on doing the actual work rather than maintaining...
When hundreds of data records need the same type of update (adding titles, categories, tags, or enriched descriptions), writing a dedicated Python script that reads a manifest and patches the data files is orders of magnitude faster and more reliable than manual editing. The script is disposable, bu...
localStorage can serve as a full persistence layer for client-side applications when the data is user-specific, the data volume is small, and there is no multi-device sync requirement. The key challenges are key design, migration of storage formats, and graceful handling of storage limits and corrup...
A whole-codebase code review is only as valuable as the remediation that follows it. The review itself produces a findings document. The remediation requires a separate phased plan that prioritizes findings by severity, groups them into shippable phases, and tracks each fix to completion with test v...
When you have hundreds or thousands of content items authored by different sources at different times, quality varies wildly unless you define measurable thresholds and audit systematically. The audit itself is more valuable than the fixes it produces — it turns "the hints feel thin" into "22 of 33...
A Content Security Policy (CSP) is achievable on a static site without server-side headers by using a `<meta>` tag. The challenge is crafting a policy that's strict enough to block XSS but permissive enough to allow legitimate functionality — especially ES module imports from CDNs and inline styles...
Migrating an existing multi-page site to a design system is a page-by-page operation, not a big-bang rewrite. The design system (tokens + components) must be complete and proven on one reference page before touching others. The migration ends with deleting the old stylesheets — if the old CSS files...
Writing a design document and a Physical Design Requirements (PDR) document before coding catches architectural mistakes when they're cheapest to fix. The design doc explores the problem space; the PDR specifies the physical implementation. Skipping either leads to rework: skipping design means buil...
When migrating a data format (XML to JSON) that feeds a rendering pipeline, the only way to prove the migration is correct is to run both formats through the pipeline and compare the outputs field-by-field. Unit tests of the new loader are necessary but insufficient — they prove the new code works,...
A progressive hint system (brief nudge → full explanation → deep-dive knowledge) is more pedagogically effective than a single "show answer" button. But each level must serve a distinct purpose with a measurable quality bar, or they collapse into three versions of the same thin content.
A browser-based application that uses DOM APIs (querySelector, innerHTML, addEventListener) can be integration-tested in Node.js using jsdom, without launching a real browser. This is faster than Playwright/Selenium and simpler to set up, but requires dependency injection to decouple the application...
After a migration, the old system's artifacts (files, code, tests, scripts) must be actively removed in a deliberate cleanup pass — they don't disappear on their own. The removal is safe only when you can prove the new system is fully operational, and the cleanup itself requires a plan because the o...
Systematically extracting lessons from project work — and writing them as standalone documents — turns ephemeral experience into a durable knowledge base. The practice is most valuable when it is automated enough to be low-friction (discovery from git history) but requires human judgment for what ac...
Breaking large features into ordered phases — each independently shippable, each ending with a commit — transforms ambitious work into manageable steps with explicit progress tracking. The phase plan is both a work queue and an audit trail.
When a system needs to support multiple "providers" (vendors, brands, data sources) that share the same behavior but differ in branding and content, the architecture should make adding a new provider a data-only operation with minimal code changes. The code that distinguishes providers should be con...
Adding 50+ exams across 10 providers to a quiz application required zero changes to the core quiz engine, data loader, or results page. The architecture held because the provider abstraction was clean, the data format was standardized, and provider-specific logic was confined to a single function an...
Adding runtime schema validation to your data loading layer catches entire categories of bugs that would otherwise surface as confusing UI glitches. The cost is a one-time schema definition and a few lines of validation code. The payoff is immediate, clear error messages instead of silent wrong beha...
When multiple people or processes author data files for the same system without a shared schema, variant schemas emerge. The variants look similar enough to pass casual inspection but differ in element names, nesting structure, or attribute naming — causing parser failures on some files but not othe...
A full-featured application (quiz engine, progress persistence, scoring, results dashboards, 10 providers, 50+ exams) can be built with vanilla HTML, CSS, and ES6 modules — no framework, no build step, no server. This approach trades developer convenience (hot reload, component abstractions, state m...
When critical logic is embedded in a class that's hard to test (DOM-coupled UI class), developers sometimes copy the logic into the test file and test the copy instead. This creates a dangerous illusion of coverage: the tests pass, but they're not testing the real code. When the real code diverges f...
When hints contain the exact text of the correct answer choice, they short-circuit learning. The learner reads the hint, sees the answer verbatim, and selects it without understanding why it's correct. This is a subtle content defect that is invisible in manual review but easy to detect programmatic...
XML entity encoding bugs (`Q&A` vs `Q&A`) are the most common class of data corruption in XML content pipelines. They're invisible in many editors, they pass casual visual inspection, and they cause parse failures that manifest as "the file won't load" with no useful error message. Any pipeline...
When migrating a live data format (XML to JSON), the key risk is not the conversion itself — it's proving that the new format produces identical behavior. The migration succeeded because the conversion was treated as a pipeline problem (convert, validate, prove equivalence) rather than a rewrite.
Using `innerHTML` to render content from "your own" data files (XML, JSON, markdown) is an XSS vulnerability even when the data is self-authored today. The threat model changes when the data pipeline changes: content contributions, bulk imports from external sources, or AI-generated content can all...
When a system must produce multiple visual representations of the same architecture, build a single normalized graph model and derive all outputs from it. Renderers that read the same model cannot drift from each other; renderers that maintain their own state always will.
When building a knowledge graph that must support regeneration, deduplication, and cross-system references, enforce a structured ID format from day one. An ID like `entity_type.domain.name` is simultaneously human-readable, machine-parseable, and stable across re-extraction — properties that free-fo...
When building multiple rendering backends for the same data model, define the visual language (colors, shape semantics, edge style categories) once and map it to each renderer's syntax independently. Visual consistency across output formats is easy to lose when renderers are built in isolation; a sh...
When building a multi-phase system, track progress at the row level within each phase (Open → Started → Completed with timestamps), commit only when an entire phase is green, and never batch multiple phases into one commit. This granularity makes it possible to resume mid-phase, measure velocity, an...
When building a query or configuration system, provide a registry of named presets for the common cases and a full custom endpoint for everything else. Presets give users instant value without learning the schema; the custom path preserves full flexibility for power users.
In a split-stack project (separate frontend and backend processes on different ports), configure the frontend dev server to proxy API requests to the backend rather than hardcoding backend URLs or relying on CORS alone. The proxy eliminates cross-origin issues during development, keeps the frontend...
When building an entity extraction pipeline, implement rule-based heuristics first and defer LLM-assisted extraction until the deterministic baseline is tested and measured. The rule-based layer gives you a reproducible, cost-free, fast foundation that LLM extraction can extend — not replace.
When deploying a static site to a subdirectory path (e.g., `github.io/project/` instead of a custom domain root), every internal link must be prefixed with the base path. Setting the framework's `site` config is not enough — you must also set `base`, and every hardcoded absolute `href` in components...
When CI workflows hand-maintain `pip install` commands that duplicate what `pyproject.toml` already declares, the two lists will drift. New dependencies added to `pyproject.toml` will be missing in CI, causing build failures that can't be reproduced locally. The fix is to use `pip install .` so `pyp...
A GitHub Actions workflow that deploys to GitHub Pages will fail on the first run if Pages is not enabled in the repository settings. The workflow will build successfully but the deploy step returns a 404 — "Ensure GitHub Pages has been enabled." This is a configuration prerequisite, not a code bug,...
Running the same lint, format, and test checks locally before pushing catches failures that would otherwise require a push-fix-push cycle through CI. The cost of a local preflight is seconds; the cost of a CI round-trip is minutes plus noise (failed build notifications, red badges, extra commits). A...
Running a systematic, category-driven code review after implementation is complete catches a class of issues that per-phase testing and acceptance criteria miss. Per-phase verification asks "does this phase work?" — a structured review asks "what's wrong across the whole codebase?" The two are compl...
Any UI that displays LLM-generated text has two untrusted input sources: the user's query and the model's response. Both must be sanitized before DOM insertion. The model's output is especially dangerous because developers intuitively trust "their own backend" — but the LLM's response is no more tru...
Before choosing a similarity algorithm, understand whether your data uses binary membership (item has feature or doesn't) or continuous scores (item has every feature at varying levels). Set-based metrics like Jaccard collapse to a constant when every item has every feature — the signal is in the sc...
Occupation codes are not stable identifiers across taxonomy revisions. The same SOC code can refer to different occupations in different versions, and naively comparing values across revisions produces misleading results. A crosswalk — an explicit mapping from old codes to new codes with cardinality...
Government data sources contain artifacts of their internal production processes — temp files in archives, renamed columns between releases, duplicate hierarchical rows, suppressed values that look like nulls but carry legal meaning, and CDN configurations that reject non-browser HTTP clients. Defen...
Base observations are source truth; derived values are computed artifacts. Mixing them in the same table creates ambiguity about whether a number is a measurement or a calculation. Separating them into distinct tables — with explicit derivation methods and base-metric linkage — makes the distinction...
When two projects share an author, the stronger design system should inform the weaker one — but adopting visual feel is a different task than adopting architecture. Port the tokens and typography; don't port the rendering pipeline.
A four-layer warehouse architecture (raw, staging, core, marts) with strict separation of concerns at each layer produces a system where raw data is always recoverable, business meaning is assigned in exactly one place, and analytical queries never need to understand source formats.
Federal data sources are not designed for programmatic access. They block bare HTTP requests, publish in heterogeneous formats, embed preamble rows in spreadsheets, and experience periodic outages around major releases. A robust extract layer must handle all of these realities with browser-like head...
A static site can replicate a dynamic API by intercepting JavaScript `fetch()` calls and redirecting them to pre-built JSON files. The key technique is a monkey-patch of the global `fetch` function that routes API URLs to static file paths, with client-side filtering for search and client-side compo...
Geographic wage comparisons are inherently incomplete: nominal gaps do not account for cost-of-living differences, suppressed cells create invisible holes in small-occupation maps, and the same query pattern must work across national, state, and metro levels without separate code paths. A dimension-...
Data pipelines fail — downloads timeout, parsers hit unexpected formats, database connections drop. Idempotency (running the same operation twice produces the same result as running it once) must be designed into every layer: delete-before-insert for facts, check-before-insert for dimensions, and gr...
Comparing nominal wages across years is misleading because the dollar's purchasing power changes over time. Converting to constant dollars using CPI-U deflation separates genuine labor market shifts from background price-level changes and is essential for any multi-vintage wage trend analysis.
Once a warehouse holds multiple vintages of the same dataset, every query must explicitly decide whether it wants the latest snapshot or all history. Forgetting this decision produces silent data quality bugs — duplicate rows, empty columns, or misleading percentages — that look correct at the SQL l...
Percentage changes on small bases are statistically volatile and can dominate ranked lists even when the absolute economic impact is trivial. Any ranked-change display must show both percentage and absolute values so users can distinguish genuine labor market shifts from small-sample noise.
Government data sources change column names, add or remove columns, and retype columns between releases — often without notice. A pipeline that assumes a fixed schema will silently break or load garbage. Proactive drift detection at the staging boundary turns silent corruption into a loud, actionabl...
A server-side web application can be deployed to a static hosting platform by pre-rendering every page and API response as files, then injecting a JavaScript fetch shim that transparently redirects API calls to the corresponding JSON files. The application's JavaScript never knows it's running on a...
Separating tests by their infrastructure requirements — fixtures-only, in-memory server, real database — lets CI run fast on every push while reserving expensive real-data validation for local runs. The deployment pipeline then layers lint, format, test, build, and deploy into a strict sequence wher...
When building an analytical warehouse from multiple federal data products, the single most important architectural decision is identifying the stable external key that connects them. For labor data, that key is the Standard Occupational Classification (SOC) code — every design decision flows from tr...
When loading multiple vintages of the same dataset, dimension tables must deduplicate on business key alone — not on business key plus source release. Including the source release in dimension lookups gives the same real-world entity different surrogate keys in each vintage, making cross-vintage joi...
When a web framework dispatches synchronous endpoint handlers to a thread pool, a shared database connection will produce intermittent wrong results — not errors, but silently incorrect data. The fix is per-thread connections via `threading.local()`, with a global override path for test injection.
Fact tables store snapshots — single measurements at single points in time. Time-series analysis requires a separate normalization step that aligns snapshots across periods into a conformed schema with explicit metric definitions, and a further separation between base observations and derived series...
A web application that shows buttons, links, or filters for data that does not exist creates a worse experience than one that simply omits them. Every UI element that implies data availability must be backed by a runtime or build-time check that the data actually exists.