ROADMAP.md

  1# Roadmap
  2
  3Findings from a full review on 2026-09-10, ordered for work. Each item is
  4sized to be one gitbay issue and one merge request. Sizes: S is under an
  5hour, M is a few hours, L needs a design pass first.
  6
  7Verified on: local build from `main` at 028903a, live instance
  8art.krz.sh (v1.4.0), devianter v0.3.4. DeviantArt's WAF blocks the
  9reviewing machine's egress IP, so DA-backed pages were checked on the live
 10instance.
 11
 12Existing gitbay issues: #1 cleanup, #2 search filters, #3 description
 13parsing, #4 instance checker, #5 Makefile, #6 emote bug. They are mapped
 14below where they overlap.
 15
 16## Status
 17
 18Everything below except 5.3 and 5.5 shipped in v1.5.0 (2026-09-11), with
 19v1.5.1 fixing the release build. Merge requests !6 through !25 on gitbay.
 20Still open: #33 (LibRedirect submission), #1, #2, #3 and #6 from the
 21original list, which need real DeviantArt payloads to work from.
 22
 23## Ordering principle
 24
 25Frontend proxies get blocked upstream. The existing throttle
 26(`app/httpclient.go`) spreads requests out; it does not reduce them. Tier 1
 27reduces them. Tier 0 lands first because everything after it needs CI.
 28Tier 2's escaping fix lands before other template work because it touches
 29every template and builder.
 30
 31## Upstream cost today
 32
 33Nothing from DeviantArt's JSON API is cached. Only wixmp media is cached,
 34and only when `cache.enabled` is on, which defaults to off.
 35
 36| Page | Upstream calls per view |
 37|---|---|
 38| Post | 2 API (deviation, comments) + 1 avatar per commenter |
 39| User about | 2 API (gruser, comments) + avatars |
 40| Gallery, favourites, search, DD | 1 API, plus thumbnails when the media cache is off |
 41| `/api/random` | up to 3 searches |
 42| Atom feed | 1 API per poll, per reader |
 43
 44Avatars come from a.deviantart.net and are fetched fresh on every view.
 45Concurrent requests for the same page each go upstream. No response carries
 46`Cache-Control`. There is no robots.txt and no per-client rate limit.
 47
 48## Tier 0: process
 49
 50### 0.1 CI on merge requests (S, #7)
 51
 52Done in !6.
 53
 54No pipeline runs `go vet`, `go test` or `golangci-lint`. The only workflow
 55is the GitHub release build; gitbay has zero builds. `.golangci.yml` exists
 56but is never run.
 57
 58Fix: add a gitbay build that runs vet, test with `-race`, and golangci-lint
 59on every MR. Check `gitbay build --help` and `ssh git@gitbay.org help build`
 60for the pipeline format. Optionally mirror the same job to
 61`.github/workflows/` so the GitHub mirror shows status too.
 62
 63Verify: an MR with a failing test shows a failed build.
 64
 65## Tier 1: reduce upstream requests
 66
 67### 1.1 Cache DeviantArt API responses (L, #8)
 68
 69Done in !8.
 70
 71Add an in-memory cache in front of every devianter call: DD, search,
 72deviation, gruser, gallery, favourites, comments. Keyed by endpoint plus
 73arguments. Bounded by entry count or bytes. Singleflight so concurrent
 74requests for one key share a single upstream call. Per-endpoint TTLs, in
 75config with defaults on the order of: DD and search a few minutes,
 76deviations longer, comments shorter.
 77
 78This is the item the TODO at `app/cache.go:3` names. It makes feed polling
 79and `/api/random` close to free.
 80
 81Design first: cache interface, key derivation, TTL config keys, eviction,
 82what the hit/miss log looks like. Write the spec to
 83`docs/superpowers/specs/` and review before code.
 84
 85Files: new `app/apicache.go`, call sites in `app/wrapper.go`,
 86`app/api.go`, `app/api_json.go`, config in `app/config.go`, docs in
 87`SETUP.md`.
 88
 89Verify: unit tests for TTL expiry, singleflight, and bounded size. Two
 90sequential requests for one page make one upstream call.
 91
 92### 1.2 Cache avatars and emotes, add Cache-Control (M, #9)
 93
 94Done in !9.
 95
 96`Emojitar` in `app/wrapper.go` fetches from a.deviantart.net or
 97e.deviantart.net on every request and never stores the result. Route it
 98through the same disk and memory cache path as `DownloadAndSendMedia`.
 99
100Add `Cache-Control` headers: long `max-age` with `immutable` on
101`/media/file` (token-signed wixmp URLs do not change), a day on avatars and
102emotes, a day on `/stylesheet` and `/favicon.ico`, a short `max-age` on HTML
103matching the API cache TTL.
104
105Depends on: nothing, but pairs with 1.1.
106
107Verify: second avatar request is served without an upstream fetch; headers
108present in `curl -I` output.
109
110### 1.3 robots.txt and per-client rate limit (S, #10)
111
112Done in !10.
113
114Serve `/robots.txt` disallowing `/search`, `/api`, `/group_user`,
115`/media` and any path with `?p=`. Add a per-client-IP token bucket ahead of
116the upstream throttle so one crawler cannot consume the whole DA budget and
117turn it into latency for everyone else. Honour `X-Forwarded-For` only when
118the request came from a configured trusted proxy.
119
120Files: `app/router.go`, new `app/ratelimit.go`, `app/config.go`,
121`SETUP.md`.
122
123Verify: test that N+1 requests from one address within the window get 429.
124
125### 1.4 Fewer calls per page (M, #11)
126
127Done in !11.
128
129Post view is two API calls because comments are fetched inline. Move
130comments behind a link (`/post/{author}/{name}/comments` or `?comments=1`)
131so the default post view is one call. Same for the user about page.
132
133Rebuild `/api/random` to pick from cached DD or search results (1.1) instead
134of issuing up to three fresh searches per hit.
135
136Depends on: 1.1.
137
138Verify: post view makes exactly one upstream call in a test with a fake
139transport.
140
141### 1.5 Media cache on by default (S, #12)
142
143Done in !12.
144
145A proxying instance with no cache re-fetches every image from wixmp on every
146view. Set `cache.enabled: true` in the built-in defaults in `app/config.go`
147and in `config.example.json`, with a sane `lifetime` and `max-size`. Keep
148`memcache` off. Document the change in `SETUP.md`.
149
150Verify: fresh start with no config writes to the cache directory.
151
152## Tier 2: security and correctness
153
154### 2.1 Escape template output (M, #13)
155
156Done in !7.
157
158`app/util.go` imports `text/template`. Nothing interpolated is escaped: the
159search query in `static/html/search.htm` and `head.htm`, and every DA
160username, title and description written by `DeviationList`,
161`ParseComments`, `BuildUserPlate` and `ParseDescription`. The CSP blocks
162scripts but not markup, inline styles, meta refresh or injected forms; any
163title containing `<` corrupts the page.
164
165Fix: switch to `html/template`; wrap the pre-built HTML fragments in
166`template.HTML`; escape strings in the Go builders with
167`html.EscapeString` and attribute-escape URLs. Add tests that a query and a
168title containing `"><b>` render as text.
169
170Land before other template work.
171
172### 2.2 Restore the user About branch (S, #14)
173
174Done in !13.
175
176`app/wrapper.go:35` has `else if false`, inherited from upstream commit
177048bb47. Registration date, interests, social links and bio never render for
178users. Find out why it was disabled (likely a devianter struct change),
179restore the branch, add a test with a fixture.
180
181### 2.3 Group search pagination (S, #15)
182
183Done in !13.
184
185`app/wrapper.go:274` increments the page and requests offset `10*page`, so
186page two starts at result 20 and results 10 to 19 are never shown. The nav
187bar also shows the incremented number. Use `10*(page-1)` and do not mutate
188`s.Page` before `NavBase`.
189
190### 2.4 Emojitar writes a body after 404 (S, #16)
191
192Done in !9.
193
194`app/wrapper.go:344` lacks a `return` after `ReturnHTTPError(404)`.
195
196### 2.5 Valid Atom feed (S, #17)
197
198Done in !15.
199
200`DeviationList` in `app/parsers.go` emits no feed-level `<id>` or
201`<updated>`, bare integer entry ids, RFC 1123 `<published>` instead of RFC
2023339, and `media:thumbinal`. Verified on the live feed. Fix all five and add
203a test that parses the output with an Atom library or checks the required
204elements.
205
206### 2.6 `-c` bounds check (S, #18)
207
208Done in !13.
209
210`app/cli.go:29` checks `len(a) >= 2` instead of `n+1 < len(a)`;
211`skunkyart -x -c` panics.
212
213### 2.7 Sanitize the 502 page (S, #19)
214
215Done in !7.
216
217`Error` in `app/util.go` writes the upstream error, including the full
218CloudFront block page, into an `<h3>` unescaped. Truncate to one line and
219escape. Folds into 2.1 if done together.
220
221### 2.8 Parse templates once (S, #20)
222
223Done in !13.
224
225`ExecuteTemplate` calls `ParseFS` on every request. Parse at startup;
226supply the per-request `T` function through the data struct or a per-request
227`Funcs` clone. Template errors then fail at boot instead of as 500s.
228
229## Tier 3: config, docs, i18n
230
231### 3.1 Config-less start and default alignment (S, #21)
232
233Done in !16.
234
235`ExecuteConfig` exits if `config.json` is missing even though defaults
236exist. Start with defaults when no `-c` is given and the default file is
237absent. Align the built-in `nsfw: true` with the example's `false`, or
238document why they differ.
239
240### 3.2 Cache documentation (S, #22)
241
242Done in !16.
243
244`SETUP.md`: `update-interval` is in seconds (the example scans every 5s);
245the `d` unit works but is unlisted; `y` is 360 days; exceeding `max-size`
246deletes the whole cache directory; `lifetime: null` in the example.
247
248### 3.3 API and search type docs (S, #23)
249
250Done in !16.
251
252`API.md` says `t` is text search; devianter defines it as tag. The
253"Folders" option in `static/html/gruser.htm` maps to `f`, which is
254favourites. Fix the doc and rename or remove the option.
255
256### 3.4 i18n coverage (M, #24)
257
258Done in !17.
259
260Go-built HTML hardcodes English: comment headers, "In reply to",
261pagination, folder and content headings, "No results", "[ TEXT ]".
262`gruser.htm` section headings and the index blurb are untranslated. Every
263template declares `lang="en"`. `Languages()` in `app/i18n.go` is unused.
264Move the strings into the catalogues, set `lang` from the resolved
265language, and either use or remove `Languages()`.
266
267### 3.5 systemd unit (S, #25)
268
269Done in !18.
270
271`services/skunkyart.example.service` uses `Directory=` (not a valid key),
272placeholder paths, and says it was never tested. Write a working unit with
273`WorkingDirectory`, `User`, `DynamicUser` or a dedicated user,
274`NoNewPrivileges`, and `Restart=on-failure`. Test it once on a Linux host.
275
276### 3.6 SETUP.md structure (S, #26)
277
278Done in !16.
279
280The nginx section sits between config keys; `theme` and `language` come
281after it. Reorder: config keys, units, reverse proxy.
282
283### 3.7 README (S, #27)
284
285Done in !18.
286
287Add: endpoints and what they do, running the binary without Docker with
288the service files, what `REDIRECTS.md` is for (redirector rules), and a
289screenshot.
290
291## Tier 4: UI
292
293### 4.1 Viewport and mobile CSS (S, #28)
294
295Done in !19.
296
297`static/html/head.htm` and `index.htm` use `initial-scale=0.4` and
298`height=device-height`; `skunky.css` then compensates with
299`* { font-size: 120% }` in portrait. Use `width=device-width,
300initial-scale=1` and adjust the portrait rules to match. Check on a phone
301width before and after.
302
303### 4.2 Accessibility (S, #29)
304
305Done in !19.
306
307Listing and avatar images in `DeviationList`, `ParseComments` and
308`BuildUserPlate` have no `alt`. The post page has no heading element for
309the title. Add both.
310
311### 4.3 Index stylesheet (S, #30)
312
313Done in !19.
314
315`static/html/index.htm` carries an inline stylesheet duplicating layout
316rules. Move it into `skunky.css`.
317
318## Tier 5: identity and reach
319
320### 5.1 One canonical forge (S, #31)
321
322Done in !20.
323
324Origin and issues are on gitbay; releases, the image, Dependabot, the
325instances.json fetch at `app/util.go:64`, the About page "Report an issue"
326link, the index source link, and the `--add-instance` exit message all
327point at GitHub. Decide which is canonical. If gitbay: fetch
328`instances.json` from gitbay, point the links there, keep the GitHub mirror
329for the image build only. If GitHub stays the public face: say so in the
330README and leave the links.
331
332### 5.2 Instance checker (M, issue #4, #32)
333
334Done in !21.
335
336A scheduled job that fetches each instance's `/api/instance` and marks dead
337ones in `INSTANCES.md`, or a CI job that fails when one is down.
338
339### 5.3 LibRedirect listing (S, #33)
340
341Open. The two upstream pull requests are written up on #33.
342
343`REDIRECTS.md` already describes the URL mapping. Check whether LibRedirect
344lists SkunkyArt with the dead upstream instances and submit the fork and
345art.krz.sh. This is the cheapest way to get users.
346
347### 5.4 Makefile and binary releases (S, issue #5, #34)
348
349Done in !22.
350
351Targets for build with the embed tag and version stamp, test, lint.
352Publish binaries alongside the image on release tags.
353
354### 5.5 Existing issues
355
356- #1 cleanup: the TODOs at `app/parsers.go:222` and `app/cache.go:3`; the
357  second is 1.1. `sendMedia` in `app/api.go` duplicates `ParseMedia`'s
358  magic string offsets (`[21:]`, `dot+11`); share one function.
359- #2 search filters: blocked on what devianter exposes; scope after 1.1.
360- #3 description parsing: `ParseDescription` drops `header-two`, ordered
361  lists and nested styles. Needs fixtures from real descriptions.
362- #6 emote bug: the `a.Val[8:9] == "e"` and `[37:len-4]` offsets in the
363  HTML branch of `ParseDescription`. Parse the URL instead of slicing.
364
365## Merge record
366
367Merged into main in this order on 2026-09-11, each stacked on the one
368before: !6 (0.1), !7 (2.1, 2.7), !8 (1.1), !9 (1.2, 2.4), !10 (1.3),
369!11 (1.4), !12 (1.5), !13 (2.2, 2.3, 2.6, 2.8), !15 (2.5), !16 (3.1,
3703.2, 3.3, 3.6), !17 (3.4), !18 (3.5, 3.7), !19 (4.1, 4.2, 4.3), !20
371(5.1), !21 (5.2), !22 (5.4). Then !23 (release string), !24 (Go 1.26 in
372the image and binaries builds) and !25 (no VCS stamping) for the
373release itself.
374
375Two things the stack taught: lint on macOS never compiles the Linux-only
376files, so run `GOOS=linux golangci-lint run` before pushing; and `go get`
377can raise the go directive in go.mod, so check the Dockerfile and
378workflow images still match it.