ROADMAP.md

Roadmap

Findings from a full review on 2026-09-10, ordered for work. Each item is sized to be one gitbay issue and one merge request. Sizes: S is under an hour, M is a few hours, L needs a design pass first.

Verified on: local build from main at 028903a, live instance art.krz.sh (v1.4.0), devianter v0.3.4. DeviantArt's WAF blocks the reviewing machine's egress IP, so DA-backed pages were checked on the live instance.

Existing gitbay issues: #1 cleanup, #2 search filters, #3 description parsing, #4 instance checker, #5 Makefile, #6 emote bug. They are mapped below where they overlap.

Status

Everything below except 5.3 shipped in v1.5.0 (2026-09-11), with v1.5.1 fixing the release build. Six patch releases followed on 2026-09-11 and 2026-09-12 from running the public instance; see "After v1.5.0" at the end. Of the original six issues, #1, #3, #4, #5 and #6 are closed by merged work and #2 is closed as not possible for a guest session. The one open issue is #33, the LibRedirect submission, which needs the maintainer's account.

Ordering principle

Frontend proxies get blocked upstream. The existing throttle (app/httpclient.go) spreads requests out; it does not reduce them. Tier 1 reduces them. Tier 0 lands first because everything after it needs CI. Tier 2's escaping fix lands before other template work because it touches every template and builder.

Upstream cost today

Nothing from DeviantArt's JSON API is cached. Only wixmp media is cached, and only when cache.enabled is on, which defaults to off.

Page	Upstream calls per view
Post	2 API (deviation, comments) + 1 avatar per commenter
User about	2 API (gruser, comments) + avatars
Gallery, favourites, search, DD	1 API, plus thumbnails when the media cache is off
/api/random	up to 3 searches
Atom feed	1 API per poll, per reader

Avatars come from a.deviantart.net and are fetched fresh on every view. Concurrent requests for the same page each go upstream. No response carries Cache-Control. There is no robots.txt and no per-client rate limit.

Tier 0: process

0.1 CI on merge requests (S, #7)

Done in !6.

No pipeline runs go vet, go test or golangci-lint. The only workflow is the GitHub release build; gitbay has zero builds. .golangci.yml exists but is never run.

Fix: add a gitbay build that runs vet, test with -race, and golangci-lint on every MR. Check gitbay build --help and ssh git@gitbay.org help build for the pipeline format. Optionally mirror the same job to .github/workflows/ so the GitHub mirror shows status too.

Verify: an MR with a failing test shows a failed build.

Tier 1: reduce upstream requests

1.1 Cache DeviantArt API responses (L, #8)

Done in !8.

Add an in-memory cache in front of every devianter call: DD, search, deviation, gruser, gallery, favourites, comments. Keyed by endpoint plus arguments. Bounded by entry count or bytes. Singleflight so concurrent requests for one key share a single upstream call. Per-endpoint TTLs, in config with defaults on the order of: DD and search a few minutes, deviations longer, comments shorter.

This is the item the TODO at app/cache.go:3 names. It makes feed polling and /api/random close to free.

Design first: cache interface, key derivation, TTL config keys, eviction, what the hit/miss log looks like. Write the spec to docs/superpowers/specs/ and review before code.

Files: new app/apicache.go, call sites in app/wrapper.go, app/api.go, app/api_json.go, config in app/config.go, docs in SETUP.md.

Verify: unit tests for TTL expiry, singleflight, and bounded size. Two sequential requests for one page make one upstream call.

1.2 Cache avatars and emotes, add Cache-Control (M, #9)

Done in !9.

Emojitar in app/wrapper.go fetches from a.deviantart.net or e.deviantart.net on every request and never stores the result. Route it through the same disk and memory cache path as DownloadAndSendMedia.

Add Cache-Control headers: long max-age with immutable on /media/file (token-signed wixmp URLs do not change), a day on avatars and emotes, a day on /stylesheet and /favicon.ico, a short max-age on HTML matching the API cache TTL.

Depends on: nothing, but pairs with 1.1.

Verify: second avatar request is served without an upstream fetch; headers present in curl -I output.

1.3 robots.txt and per-client rate limit (S, #10)

Done in !10.

Serve /robots.txt disallowing /search, /api, /group_user, /media and any path with ?p=. Add a per-client-IP token bucket ahead of the upstream throttle so one crawler cannot consume the whole DA budget and turn it into latency for everyone else. Honour X-Forwarded-For only when the request came from a configured trusted proxy.

Files: app/router.go, new app/ratelimit.go, app/config.go, SETUP.md.

Verify: test that N+1 requests from one address within the window get 429.

1.4 Fewer calls per page (M, #11)

Done in !11.

Post view is two API calls because comments are fetched inline. Move comments behind a link (/post/{author}/{name}/comments or ?comments=1) so the default post view is one call. Same for the user about page.

Rebuild /api/random to pick from cached DD or search results (1.1) instead of issuing up to three fresh searches per hit.

Depends on: 1.1.

Verify: post view makes exactly one upstream call in a test with a fake transport.

1.5 Media cache on by default (S, #12)

Done in !12.

A proxying instance with no cache re-fetches every image from wixmp on every view. Set cache.enabled: true in the built-in defaults in app/config.go and in config.example.json, with a sane lifetime and max-size. Keep memcache off. Document the change in SETUP.md.

Verify: fresh start with no config writes to the cache directory.

Tier 2: security and correctness

2.1 Escape template output (M, #13)

Done in !7.

app/util.go imports text/template. Nothing interpolated is escaped: the search query in static/html/search.htm and head.htm, and every DA username, title and description written by DeviationList, ParseComments, BuildUserPlate and ParseDescription. The CSP blocks scripts but not markup, inline styles, meta refresh or injected forms; any title containing < corrupts the page.

Fix: switch to html/template; wrap the pre-built HTML fragments in template.HTML; escape strings in the Go builders with html.EscapeString and attribute-escape URLs. Add tests that a query and a title containing "><b> render as text.

Land before other template work.

2.2 Restore the user About branch (S, #14)

Done in !13.

app/wrapper.go:35 has else if false, inherited from upstream commit 048bb47. Registration date, interests, social links and bio never render for users. Find out why it was disabled (likely a devianter struct change), restore the branch, add a test with a fixture.

2.3 Group search pagination (S, #15)

Done in !13.

app/wrapper.go:274 increments the page and requests offset 10*page, so page two starts at result 20 and results 10 to 19 are never shown. The nav bar also shows the incremented number. Use 10*(page-1) and do not mutate s.Page before NavBase.

2.4 Emojitar writes a body after 404 (S, #16)

Done in !9.

app/wrapper.go:344 lacks a return after ReturnHTTPError(404).

2.5 Valid Atom feed (S, #17)

Done in !15.

DeviationList in app/parsers.go emits no feed-level <id> or <updated>, bare integer entry ids, RFC 1123 <published> instead of RFC 3339, and media:thumbinal. Verified on the live feed. Fix all five and add a test that parses the output with an Atom library or checks the required elements.

2.6 -c bounds check (S, #18)

Done in !13.

app/cli.go:29 checks len(a) >= 2 instead of n+1 < len(a); skunkyart -x -c panics.

2.7 Sanitize the 502 page (S, #19)

Done in !7.

Error in app/util.go writes the upstream error, including the full CloudFront block page, into an <h3> unescaped. Truncate to one line and escape. Folds into 2.1 if done together.

2.8 Parse templates once (S, #20)

Done in !13.

ExecuteTemplate calls ParseFS on every request. Parse at startup; supply the per-request T function through the data struct or a per-request Funcs clone. Template errors then fail at boot instead of as 500s.

Tier 3: config, docs, i18n

3.1 Config-less start and default alignment (S, #21)

Done in !16.

ExecuteConfig exits if config.json is missing even though defaults exist. Start with defaults when no -c is given and the default file is absent. Align the built-in nsfw: true with the example's false, or document why they differ.

3.2 Cache documentation (S, #22)

Done in !16.

SETUP.md: update-interval is in seconds (the example scans every 5s); the d unit works but is unlisted; y is 360 days; exceeding max-size deletes the whole cache directory; lifetime: null in the example.

3.3 API and search type docs (S, #23)

Done in !16.

API.md says t is text search; devianter defines it as tag. The "Folders" option in static/html/gruser.htm maps to f, which is favourites. Fix the doc and rename or remove the option.

3.4 i18n coverage (M, #24)

Done in !17.

Go-built HTML hardcodes English: comment headers, "In reply to", pagination, folder and content headings, "No results", "[ TEXT ]". gruser.htm section headings and the index blurb are untranslated. Every template declares lang="en". Languages() in app/i18n.go is unused. Move the strings into the catalogues, set lang from the resolved language, and either use or remove Languages().

3.5 systemd unit (S, #25)

Done in !18.

services/skunkyart.example.service uses Directory= (not a valid key), placeholder paths, and says it was never tested. Write a working unit with WorkingDirectory, User, DynamicUser or a dedicated user, NoNewPrivileges, and Restart=on-failure. Test it once on a Linux host.

3.6 SETUP.md structure (S, #26)

Done in !16.

The nginx section sits between config keys; theme and language come after it. Reorder: config keys, units, reverse proxy.

3.7 README (S, #27)

Done in !18.

Add: endpoints and what they do, running the binary without Docker with the service files, what REDIRECTS.md is for (redirector rules), and a screenshot.

Tier 4: UI

4.1 Viewport and mobile CSS (S, #28)

Done in !19.

static/html/head.htm and index.htm use initial-scale=0.4 and height=device-height; skunky.css then compensates with * { font-size: 120% } in portrait. Use width=device-width, initial-scale=1 and adjust the portrait rules to match. Check on a phone width before and after.

4.2 Accessibility (S, #29)

Done in !19.

Listing and avatar images in DeviationList, ParseComments and BuildUserPlate have no alt. The post page has no heading element for the title. Add both.

4.3 Index stylesheet (S, #30)

Done in !19.

static/html/index.htm carries an inline stylesheet duplicating layout rules. Move it into skunky.css.

Tier 5: identity and reach

5.1 One canonical forge (S, #31)

Done in !20.

Origin and issues are on gitbay; releases, the image, Dependabot, the instances.json fetch at app/util.go:64, the About page "Report an issue" link, the index source link, and the --add-instance exit message all point at GitHub. Decide which is canonical. If gitbay: fetch instances.json from gitbay, point the links there, keep the GitHub mirror for the image build only. If GitHub stays the public face: say so in the README and leave the links.

5.2 Instance checker (M, issue #4, #32)

Done in !21.

A scheduled job that fetches each instance's /api/instance and marks dead ones in INSTANCES.md, or a CI job that fails when one is down.

5.3 LibRedirect listing (S, #33)

Open. The two upstream pull requests are written up on #33.

REDIRECTS.md already describes the URL mapping. Check whether LibRedirect lists SkunkyArt with the dead upstream instances and submit the fork and art.krz.sh. This is the cheapest way to get users.

5.4 Makefile and binary releases (S, issue #5, #34)

Done in !22.

Targets for build with the embed tag and version stamp, test, lint. Publish binaries alongside the image on release tags.

5.5 Existing issues

• #1 cleanup: the TODOs at app/parsers.go:222 and app/cache.go:3; the second is 1.1. sendMedia in app/api.go duplicates ParseMedia's magic string offsets ([21:], dot+11); share one function.
• #2 search filters: blocked on what devianter exposes; scope after 1.1.
• #3 description parsing: ParseDescription drops header-two, ordered lists and nested styles. Needs fixtures from real descriptions.
• #6 emote bug: the a.Val[8:9] == "e" and [37:len-4] offsets in the HTML branch of ParseDescription. Parse the URL instead of slicing.

Merge record

Merged into main in this order on 2026-09-11, each stacked on the one before: !6 (0.1), !7 (2.1, 2.7), !8 (1.1), !9 (1.2, 2.4), !10 (1.3), !11 (1.4), !12 (1.5), !13 (2.2, 2.3, 2.6, 2.8), !15 (2.5), !16 (3.1, 3.2, 3.3, 3.6), !17 (3.4), !18 (3.5, 3.7), !19 (4.1, 4.2, 4.3), !20 (5.1), !21 (5.2), !22 (5.4). Then !23 (release string), !24 (Go 1.26 in the image and binaries builds) and !25 (no VCS stamping) for the release itself.

Two things the stack taught: lint on macOS never compiles the Linux-only files, so run GOOS=linux golangci-lint run before pushing; and go get can raise the go directive in go.mod, so check the Dockerfile and workflow images still match it.

After v1.5.0

Everything here came out of deploying v1.5 to art.krz.sh and watching it.

v1.5.1 (!24, !25)

go.mod had moved to Go 1.26 when x/sync came in while the Dockerfile and the binaries job still used 1.25, so the v1.5.0 tag built no image. Both now match go.mod. The tarball job also failed on VCS stamping inside the build container, fixed with -buildvcs=false.

v1.5.2 (!27, !28)

The instance's VPN exit was banned by DeviantArt's WAF and every DeviantArt-backed page 502d until someone restarted the stack.

• API cache entries past their TTL are kept for api-cache.stale (default 1h) and served when upstream fails or answers 403 or 429. A block starts a one minute backoff during which the instance stops asking.
• /api/random answered 401 with proxying on: the media signing token was inside the path. Fixed, and Download sits behind a seam.
• The session bootstrap runs before the listener opens, so the first seconds after a restart no longer 502.

Operationally: the instance moved off the VPN to its own address, which was clean at the time, and the container got a real log driver (it had none, which hid every error line).

v1.5.3 (!29)

The direct address was banned within an hour. upstream.min-interval-ms and upstream.max-concurrent replace the source constants; the instance runs at 1000 ms and one in flight. The ban lifted after about seventy minutes. The instance also raised api-cache.ttl to 30 minutes and stale to a day, and a Cloudflare managed-challenge rule now covers non-browser clients on search, post and profile paths.

v1.5.4 (!30, !31)

Cache rotation trims the oldest files down to max-size instead of emptying the directory, and never touches the directory itself, which on a bind mount logged unlinkat and mkdir errors every pass (#36). The per-platform stat files went with it. x/net bumped (#35).

v1.5.5 (!32)

A crawler walking post pages at 70 a minute produced 733 upstream timeouts in ten minutes while the address stayed unbanned: each queued request still took its interval turn after its client had timed out. The throttle now honours the request context, and sheds a request that would queue longer than 20 seconds with a 503 and Retry-After. The instance's rate-limit dropped to 20 per minute, burst 10.

v1.5.6 (!33, !34)

The real cause of #3: DeviantArt's editor stores descriptions and comments as a document tree ({"version":1,"document":...}), not Draft.js blocks, so every current description rendered empty. A new renderer covers the node set seen in 92 live payloads: paragraphs, headings, lists, quotes, code, breaks, rules, text marks, emotes (#6), embedded artworks, GIF embeds and mentions. Media and post URLs are parsed instead of sliced by offset (#1), and old HTML emotes map by image name.

Closed without code

#2 search filters: DeviantArt's guest search ignores every order value the site itself uses, and the deviations endpoint redirects guests. Nothing to expose. #4 and #5 were covered by the instance checker and the Makefile.

Lessons

• DeviantArt bans an address on volume, not on whether it is a VPN. Pacing, caching and shedding at the instance are what keep it clean; rotating exits only buys an hour.
• A restart empties the in-memory API cache, so a ban right after a deploy has nothing stale to serve. Deploy when the instance is quiet.
• Log driver none is a trap. Every incident here was diagnosed from lines that driver would have dropped.
• Fetch real payloads from a host DeviantArt accepts before touching a parser; the format had changed under the old one.