!533 · krz/gitbay

.gitbay/wiki/Parity.org +10 −1

		@@ -188,6 +188,8 @@ rather than the one the web page shows.
188	188	\| one commit with its patch \| yes \| yes \| yes \|
189	189	\| blame \| yes \| yes \| yes \|
190	190	\| search file contents \| yes \| yes \| yes \|
	191	\| find a definition (symbols) \| yes \| yes \| no \|
	192	\| jump to definition from a file \| n/a \| yes \| no \|
191	193	\| compare two refs \| yes \| yes \| yes \|
192	194	\| branches and tags \| yes \| yes \| yes \|
193	195	\| wiki (read) \| yes \| yes \| yes \|
		@@ -323,6 +325,11 @@ now render the way the reference does. go-org is not yet on the corpus.
323	325	\| activity feed \| yes \| yes \| yes \|
324	326	\| command reference \| yes \| no \| n/a \|
325	327
	328	=repo symbols= and =/{owner}/{repo}/symbols= answer from the same
	329	store query and ranking. Jumping to a definition is a link on a name in
	330	the blob view, which a terminal has no place for; =repo symbols= with
	331	the name is its CLI form.
	332
326	333	=explore= is the listing without a query; =repo search= is the one with.
327	334	=search= is both plus the title and body of every issue and merge
328	335	request the caller can read, over FTS5; the web serves it at =/search=
		@@ -464,6 +471,7 @@ when there is none.
464	471	\| worker queues \| yes \| yes \| no \|
465	472	\| runners \| yes \| no \| no \|
466	473	\| repository list, archive, visibility \| yes \| no \| no \|
	474	\| rebuild a symbol index \| yes \| no \| no \|
467	475	\| audit log \| yes \| no \| no \|
468	476	\| instance statistics \| yes \| no \| no \|
469	477
		@@ -489,7 +497,8 @@ key, the other cannot be undone.
489	497
490	498	* Pagination
491	499
492		=issue list=, =mr list=, =repo list=, =feed= and =build list= take
	500	=issue list=, =mr list=, =repo list=, =feed=, =build list= and =repo
	501	symbols= take
493	502	=--limit <n>= and =--cursor <c>=. Cursors are opaque; each page carries
494	503	the next one. Without the flags a list stays as it was, so existing
495	504	scripts are unchanged — for =build list= that means the newest fifty

.gitbay/wiki/Users.org +48

		@@ -242,12 +242,60 @@ gitbay repo settings git-daemon you/project on # expose over git://
242	242	gitbay repo topics add you/project cli forge # free-form tags, shown on the web
243	243	gitbay repo search forge # find repos by name/description/topic
244	244	gitbay repo grep you/project "some string" # literal git grep over the default branch
	245	gitbay repo symbols you/project Dispatch # where a name is defined, from the symbol index
245	246	gitbay repo pin you/project # pin to your web dashboard
246	247	gitbay repo unpin you/project
247	248	gitbay repo archive you/project # read-only: pushes and issue/MR
248	249	gitbay repo unarchive you/project # writes refused, browsing intact
249	250	#+end_src
250	251
	252	A push to the default branch queues a symbol index of its tree, built in
	253	the background after the push completes; a tree that is already indexed
	254	is not indexed again. =repo symbols <owner/name> [--kind k] <query>=
	255	lists definitions whose name starts with the query (two characters
	256	at least), exact matches
	257	first, case-sensitive before case-insensitive, as name, kind, path and
	258	line; =--limit= and =--cursor= page it, and without them it stops at
	259	200 rows. A Go method is listed as =Type.Method= and also matches on
	260	=Method=. Only the default branch is indexed: =--ref= accepts another
	261	ref only when its tree is the indexed one. The index follows read
	262	access: a repository you cannot read has no symbols either.
	263
	264	What is indexed:
	265
	266	\| language \| kinds \|
	267	\|--------------------------------+-----------------------------------------------------------\|
	268	\| Go (parsed) \| function, method, type, const, var \|
	269	\| Swift \| function, class, struct, enum, interface (protocol), type \|
	270	\| Rust \| function, struct, enum, interface (trait), type, module, const, macro \|
	271	\| Python \| function, method, class \|
	272	\| JavaScript, TypeScript \| function, class, interface, type, enum, const \|
	273	\| C and C++ headers (=.h=, =.hpp=) \| function (prototype), struct, enum, class, type, macro \|
	274	\| shell (=.sh=, =.bash=, =.zsh=) \| function \|
	275	\| org, Markdown \| section (a heading) \|
	276
	277	Languages other than Go are matched a line at a time, so unusual
	278	definition shapes are missed. Files over 1 MiB, anything under
	279	=vendor/= or =node_modules/=, =_gen.go=, =.pb.go=, Go files marked
	280	=Code generated ... DO NOT EDIT.=, and =*.min.js= are skipped, and a
	281	name longer than 256 bytes is dropped. One run stops at 200,000
	282	symbols, 32 MiB of names and paths, or two minutes, and publishes what
	283	it found as a partial index that says which bound it reached.
	284
	285	A new index is written while the previous one stays in use, and
	286	replaces it in one step. A run that cannot build an index records why
	287	and leaves the previous index current; the same tree is tried again
	288	once after an hour, or when a push changes the tree. A repository that
	289	has not been pushed to since the index existed has none until its next
	290	push to the default branch. =admin symbols reindex <owner/name>=
	291	rebuilds one regardless.
	292
	293	On the web, a file viewed at the indexed tree (the default branch's
	294	head, or any commit with the same tree) links each name the index
	295	holds: to its definition when there is one, to the results page at
	296	=/{owner}/{repo}/symbols?q=<name>= when there are several. The file's
	297	own definitions are listed above its source.
	298
251	299	Import from another forge — git data first, then optionally the issue
252	300	and PR history from GitHub or any Forgejo instance such as Codeberg
253	301	(issues keep state/labels/comments; PRs land as closed or merged MRs

CHANGELOG.org +10

		@@ -54,6 +54,16 @@ anything beyond "replace the binary and restart" is needed.
54	54	attributes it emits; MathML written as raw HTML is still stripped.
55	55	Anything outside the subset, and math past 1000 expressions or
56	56	256 KiB of TeX in one document, shows as source (#294).
	57	- Code navigation. A push or merge to the default branch queues a symbol
	58	index of its tree, built by a background worker and keyed by the tree
	59	id, so an unchanged tree is not indexed again. =repo symbols
	60	<owner/name> <query>= finds definitions by name or prefix, and the
	61	blob view links names to their definitions, with a results page at
	62	=/{owner}/{repo}/symbols= and a per-file symbol list. Go is parsed;
	63	Swift, Rust, Python, JavaScript/TypeScript, C headers, shell, org and
	64	Markdown are matched by line. =admin symbols reindex= forces a
	65	rebuild. Migration: =symbol_indexes=, =symbols= and
	66	=symbol_requests=. (#293)
57	67
58	68	* v1.38.0 — 2026-09-29
59	69

cmd/gitbay/main.go +4

		@@ -146,6 +146,9 @@ func newRoot() *cobra.Command {
146	146	group("mr", "merge requests in any repository (audited)",
147	147	pass("prune", passOpts{server: []string{"admin", "mr", "prune"}}),
148	148	),
	149	group("symbols", "symbol indexes",
	150	pass("reindex", passOpts{server: []string{"admin", "symbols", "reindex"}}),
	151	),
149	152	),
150	153	manCmd(root),
151	154	)
		@@ -553,6 +556,7 @@ func repoCmd() *cobra.Command {
553	556	pass("fork", passOpts{server: []string{"repo", "fork"}, needsRepo: true}),
554	557	pass("search", passOpts{server: []string{"repo", "search"}}),
555	558	pass("grep", passOpts{server: []string{"repo", "grep"}, needsRepo: true}),
	559	pass("symbols", passOpts{server: []string{"repo", "symbols"}, needsRepo: true}),
556	560	pass("diff", passOpts{server: []string{"repo", "diff"}, needsRepo: true}),
557	561	pass("tree", passOpts{server: []string{"repo", "tree"}, needsRepo: true}),
558	562	pass("cat", passOpts{server: []string{"repo", "cat"}, needsRepo: true}),

cmd/gitbay/summaries_gen.go +2

		@@ -17,6 +17,7 @@ var summaries = map[string]string{
17	17	"admin runners remove": "drop a key's runner heartbeat row, e.g. one that polled once by mistake (instance admins)",
18	18	"admin runners": "the build queue and runner accounts: last poll, scope, the build each holds (instance admins)",
19	19	"admin stats": "instance statistics: counts and per-repository disk usage",
	20	"admin symbols reindex": "rebuild a repository's symbol index, even when its tree is already indexed (instance admins)",
20	21	"admin user create": "create an account, optionally with a key and a verified address (instance admins)",
21	22	"admin user delete": "delete an account that anchors nothing (keys, emails and sessions go with it)",
22	23	"admin user demote": "remove instance admin from an account (never the last one)",
		@@ -207,6 +208,7 @@ var summaries = map[string]string{
207	208	"repo settings visibility": "set repository visibility",
208	209	"repo settings website": "set the repository website",
209	210	"repo show": "show repository details",
	211	"repo symbols": "find where a name is defined, from the default branch's symbol index",
210	212	"repo topics add": "add topics",
211	213	"repo topics remove": "remove topics",
212	214	"repo topics": "list topics",

cmd/gitbayd/main.go +4

		@@ -37,6 +37,7 @@ import (
37	37	"gitbay.org/gitbay/internal/seal"
38	38	"gitbay.org/gitbay/internal/sshd"
39	39	"gitbay.org/gitbay/internal/store"
	40	"gitbay.org/gitbay/internal/symbols"
40	41	"gitbay.org/gitbay/internal/toolpath"
41	42	"gitbay.org/gitbay/internal/webhook"
42	43	)
		@@ -228,6 +229,9 @@ func serveCmd() *cobra.Command {
228	229	go deps.New(st, cfg, func(owner, name string) string {
229	230	return control.RepoDir(cfg.Server.Root, owner, name)
230	231	}, buildinfo.String()).Run(whCtx)
	232	go symbols.New(st, func(owner, name string) string {
	233	return control.RepoDir(cfg.Server.Root, owner, name)
	234	}).Run(whCtx)
231	235
232	236	// One pack-generation budget for SSH, smart HTTP and git://.
233	237	// Anonymous clients ("ip:" principals) share all but one

e2e/readonly_test.go +8

		@@ -130,6 +130,7 @@ func TestReadOnlyCommandsWriteNothing(t *testing.T) {
130	130	"repo cat": {"alice/app", "f.go"},
131	131	"repo blame": {"alice/app", "f.go"},
132	132	"repo grep": {"alice/app", "hello"},
	133	"repo symbols": {"alice/app", "app"},
133	134	"repo diff": {"alice/app", "main", "feat"},
134	135	"repo commit": {"alice/app", sha},
135	136	"repo download": {"alice/app"},
		@@ -186,6 +187,13 @@ func TestReadOnlyCommandsWriteNothing(t *testing.T) {
186	187	// an SGR sequence escaping into plain output.
187	188	binaryOutput := map[string]bool{"repo download": true}
188	189
	190	// The push to main asked for a symbol index; the worker writing it
	191	// mid-loop would be blamed on whichever read ran then.
	192	waitFor(t, "symbol index", func() bool {
	193	_, _, code := inst.ssh(t, aliceKey, "", "repo", "symbols", "alice/app", "app")
	194	return code == 0
	195	})
	196
189	197	dbPath := filepath.Join(inst.root, "gitbay.db")
190	198	before := dbFingerprint(t, dbPath)
191	199	for _, cmd := range control.Commands() {

e2e/symbols_test.go added +66

		@@ -0,0 +1,66 @@
	1	package e2e
	2
	3	import (
	4	"encoding/json"
	5	"os"
	6	"path/filepath"
	7	"strings"
	8	"testing"
	9	)
	10
	11	// A push to the default branch is indexed in the background, and
	12	// repo symbols then finds a Go function where it is defined (#293).
	13	func TestSymbolsIndexAfterPush(t *testing.T) {
	14	t.Setenv("GITBAY_SYMBOLS_TICK", "200ms")
	15	inst := startInstanceWith(t, "[web]\nmode = \"accounts\"\n")
	16	aliceKey := inst.newKey(t, "alice")
	17	bobKey := inst.newKey(t, "bob")
	18	inst.admin(t, "admin", "user", "create", "alice", "--key", aliceKey+".pub")
	19	inst.admin(t, "admin", "user", "create", "bob", "--key", bobKey+".pub")
	20	if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app", "--private"); code != 0 {
	21	t.Fatalf("repo create: %s", errOut)
	22	}
	23
	24	if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "symbols", "alice/app", "Serve"); code != 3 \|\|
	25	!strings.Contains(errOut, "no symbol index") {
	26	t.Fatalf("before any push: exit %d: %s", code, errOut)
	27	}
	28
	29	work := t.TempDir()
	30	env := inst.gitEnv(aliceKey)
	31	mustGit(t, work, env, "clone", inst.sshURL("alice/app"), "w")
	32	dir := filepath.Join(work, "w")
	33	os.WriteFile(filepath.Join(dir, "server.go"), []byte("package app\n\n// Serve runs.\nfunc Serve() {}\n"), 0o644)
	34	mustGit(t, dir, env, "checkout", "-q", "-b", "main")
	35	mustGit(t, dir, env, "add", ".")
	36	mustGit(t, dir, env, "commit", "-q", "-m", "base")
	37	mustGit(t, dir, env, "push", "-q", "origin", "main")
	38
	39	var out string
	40	waitFor(t, "symbol index", func() bool {
	41	var code int
	42	out, _, code = inst.ssh(t, aliceKey, "", "repo", "symbols", "alice/app", "Serve", "--json")
	43	return code == 0
	44	})
	45	var res struct {
	46	Data []struct {
	47	Name string `json:"name"`
	48	Kind string `json:"kind"`
	49	Path string `json:"path"`
	50	Line int `json:"line"`
	51	} `json:"data"`
	52	}
	53	if err := json.Unmarshal([]byte(out), &res); err != nil {
	54	t.Fatalf("%v: %s", err, out)
	55	}
	56	if len(res.Data) != 1 \|\| res.Data[0].Name != "Serve" \|\| res.Data[0].Kind != "function" \|\|
	57	res.Data[0].Path != "server.go" \|\| res.Data[0].Line != 4 {
	58	t.Fatalf("repo symbols: %s", out)
	59	}
	60
	61	// The index follows read access: to bob the repository does not exist.
	62	if _, errOut, code := inst.ssh(t, bobKey, "", "repo", "symbols", "alice/app", "Serve"); code != 3 \|\|
	63	!strings.Contains(errOut, "not found") {
	64	t.Fatalf("outsider: exit %d: %s", code, errOut)
	65	}
	66	}

internal/control/import.go +1

		@@ -173,6 +173,7 @@ func runRepoImport(c *Ctx, args []string) int {
173	173	gitutil.SetHead(dir, branch)
174	174	c.Store.UpdateDefaultBranch(id, branch)
175	175	}
	176	c.Store.RequestSymbolIndex(id, false)
176	177
177	178	c.Store.RecordEvent(id, c.User.ID, "repo.imported", fmt.Sprintf(`{"from":%q}`, from))
178	179	type out struct {

internal/control/mr.go +1

		@@ -1611,6 +1611,7 @@ func mergeMR(c *Ctx, repo store.Repo, mr store.MR, strategy string) int {
1611	1611	ProcessCommitMessages(c.Store, dir, repo, c.User.ID, c.Scope, targetSHA, newSHA)
1612	1612	ProcessMRDescription(c.Store, repo, mr, c.User.ID, c.Scope)
1613	1613	RecordLandedCommits(c.Store, dir, repo, targetSHA, newSHA)
	1614	c.Store.RequestSymbolIndex(repo.ID, false)
1614	1615	}
1615	1616	// A merge moves the ref directly, so it never reaches post-receive and
1616	1617	// none of the ref-update work fires on its own. The event webhooks

internal/control/refsupdated.go +2

		@@ -51,6 +51,8 @@ func RefsUpdated(st *store.Store, cfg config.Config, repoID, userID int64, scope
51	51	dir := RepoDir(cfg.Server.Root, pushedRepo.OwnerName, pushedRepo.Name)
52	52	ProcessCommitMessages(st, dir, pushedRepo, userID, scope, u.Old, u.New)
53	53	RecordLandedCommits(st, dir, pushedRepo, u.Old, u.New)
	54	// The symbol index is rebuilt by its worker, never here.
	55	st.RequestSymbolIndex(pushedRepo.ID, false)
54	56	}
55	57	// A branch push with a .gitbay/ci.yml queues one build per job.
56	58	if pushedRepoErr == nil && !u.IsDelete {

internal/control/repo.go +1

		@@ -790,6 +790,7 @@ func runSetDefaultBranch(c *Ctx, args []string) int {
790	790	if err := c.Store.UpdateDefaultBranch(repo.ID, branch); err != nil {
791	791	return c.fail(protocol.ExitFailure, "%v", err)
792	792	}
	793	c.Store.RequestSymbolIndex(repo.ID, false)
793	794	return c.emit(map[string]string{"default_branch": branch}, func(w io.Writer) {
794	795	fmt.Fprintf(w, "default branch of %s is now %s\n", repo.Path(), branch)
795	796	})

internal/control/symbols.go added +186

		@@ -0,0 +1,186 @@
	1	package control
	2
	3	import (
	4	"errors"
	5	"fmt"
	6	"io"
	7	"strconv"
	8	"strings"
	9
	10	"gitbay.org/gitbay/internal/gitutil"
	11	"gitbay.org/gitbay/internal/policy"
	12	"gitbay.org/gitbay/internal/protocol"
	13	"gitbay.org/gitbay/internal/store"
	14	"gitbay.org/gitbay/internal/symbols"
	15	)
	16
	17	func init() {
	18	register(Command{Path: []string{"repo", "symbols"},
	19	Summary: "find where a name is defined, from the default branch's symbol index",
	20	Usage: "repo symbols <owner/name> [--ref <ref>] [--kind <kind>] [--limit <n>] [--cursor <c>] <query>",
	21	Flags: []Flag{
	22	{"--ref", "<ref>", "a ref whose tree is the indexed one; only the default branch is indexed", "the default branch"},
	23	{"--kind", "<kind>", "only this kind: " + strings.Join(symbols.Kinds, ", "), ""},
	24	{"--limit", "<n>", "rows per page", ""},
	25	{"--cursor", "<c>", "continue from the previous page", ""},
	26	},
	27	Examples: []string{
	28	"repo symbols krz/gitbay Dispatch",
	29	"repo symbols krz/gitbay --kind method Ctx.",
	30	},
	31	ReadOnly: true, Run: runRepoSymbols})
	32	register(Command{Path: []string{"admin", "symbols", "reindex"},
	33	Summary: "rebuild a repository's symbol index, even when its tree is already indexed (instance admins)",
	34	Usage: "admin symbols reindex <owner/name>",
	35	Examples: []string{"admin symbols reindex krz/gitbay"},
	36	Run: runAdminSymbolsReindex})
	37	}
	38
	39	// symbolsUnpaged caps a listing given without --limit or --cursor.
	40	const symbolsUnpaged = maxPageLimit
	41
	42	type symbolOut struct {
	43	Name string `json:"name"`
	44	Kind string `json:"kind"`
	45	Path string `json:"path"`
	46	Line int `json:"line"`
	47	}
	48
	49	func runRepoSymbols(c *Ctx, args []string) int {
	50	args, p, code := parsePageFlags(c, args, "symbol", false)
	51	if code >= 0 {
	52	return code
	53	}
	54	f, err := c.parseArgs(args, flagSpec{Values: []string{"--ref", "--kind"}, MaxPos: 2, Usage: c.Cmd.Usage})
	55	if err != nil {
	56	return c.fail(protocol.ExitUsage, "%v", err)
	57	}
	58	path, query, ref, kind := f.pos(0), f.pos(1), f.Value("--ref"), f.Value("--kind")
	59	if path == "" \|\| query == "" {
	60	return c.usage()
	61	}
	62	if len(query) < MinSymbolQuery \|\| len(query) > maxQueryLen {
	63	return c.fail(protocol.ExitUsage, "query must be %d to %d characters", MinSymbolQuery, maxQueryLen)
	64	}
	65	if kind != "" && !symbols.ValidKind(kind) {
	66	return c.fail(protocol.ExitUsage, "--kind must be one of %s", strings.Join(symbols.Kinds, ", "))
	67	}
	68	repo, code := resolveRepo(c, path, policy.CanRead)
	69	if code >= 0 {
	70	return code
	71	}
	72	idx, err := c.Store.SymbolIndexFor(repo.ID)
	73	if errors.Is(err, store.ErrNotFound) {
	74	if f, ferr := c.Store.SymbolFailureFor(repo.ID); ferr == nil {
	75	return c.fail(protocol.ExitFailure, "the symbol index of %s could not be built: %s", repo.Path(), f.Note)
	76	}
	77	return c.fail(protocol.ExitNotFound, "%s has no symbol index yet; one is built after a push to %s", repo.Path(), repo.DefaultBranch)
	78	} else if err != nil {
	79	return c.fail(protocol.ExitFailure, "%v", err)
	80	}
	81	if ref != "" && ref != repo.DefaultBranch {
	82	dir := RepoDir(c.Cfg.Server.Root, repo.OwnerName, repo.Name)
	83	sha, err := gitutil.ResolveRef(dir, ref)
	84	if err != nil \|\| !IndexedTree(dir, sha, idx) {
	85	return c.fail(protocol.ExitNotFound, "only the default branch, %s, is indexed; %s is not at the indexed tree", repo.DefaultBranch, ref)
	86	}
	87	}
	88	var after int64
	89	if p.key != "" {
	90	cursorIdx, id, ok := ParseSymbolCursor(p.key)
	91	if !ok {
	92	return c.fail(protocol.ExitUsage, "bad cursor")
	93	}
	94	if cursorIdx != idx.ID {
	95	return c.fail(protocol.ExitUsage, "%s", StaleSymbolCursor)
	96	}
	97	after = id
	98	}
	99	limit := p.queryLimit()
	100	if !p.active {
	101	limit = symbolsUnpaged + 1
	102	}
	103	rows, err := c.Store.SearchSymbols(idx.ID, query, kind, limit, after)
	104	if err != nil {
	105	return c.fail(protocol.ExitFailure, "%v", err)
	106	}
	107	capped := !p.active && len(rows) > symbolsUnpaged
	108	if capped {
	109	rows = rows[:symbolsUnpaged]
	110	}
	111	rows, next := trimPage(p, rows, "symbol", func(r store.SymbolRow) string {
	112	return SymbolCursor(idx.ID, r.ID)
	113	})
	114	var ds []symbolOut
	115	for _, r := range rows {
	116	ds = append(ds, symbolOut{r.Name, r.Kind, r.Path, r.Line})
	117	}
	118	return c.emitPage(p, ds, next, func(w io.Writer) {
	119	tb := c.table(w, "NAME", "KIND", "LOCATION")
	120	for _, d := range ds {
	121	tb.row(cRef(d.Name), cText(d.Kind), cFlex(fmt.Sprintf("%s:%d", d.Path, d.Line)))
	122	}
	123	tb.flush()
	124	if capped {
	125	fmt.Fprintf(c.Stderr, "first %d matches; page with --limit and --cursor\n", symbolsUnpaged)
	126	}
	127	if idx.State == "partial" {
	128	fmt.Fprintf(c.Stderr, "the index is partial: %s\n", idx.Note)
	129	}
	130	})
	131	}
	132
	133	// MinSymbolQuery is the shortest query repo symbols and the results page
	134	// take: one character matches too much of an index to be worth a page.
	135	const MinSymbolQuery = 2
	136
	137	// StaleSymbolCursor is the refusal for a cursor taken from an index that
	138	// has since been replaced.
	139	const StaleSymbolCursor = "the symbol index was rebuilt since that cursor; start again without it"
	140
	141	// SymbolCursor is the paging key after row id of index indexID, the form
	142	// ParseSymbolCursor reads.
	143	func SymbolCursor(indexID, id int64) string {
	144	return strconv.FormatInt(indexID, 10) + "." + strconv.FormatInt(id, 10)
	145	}
	146
	147	// ParseSymbolCursor reads "<index id>.<row id>". The index id makes a
	148	// cursor from before a rebuild fail rather than page through the new
	149	// index from an unrelated row.
	150	func ParseSymbolCursor(key string) (int64, int64, bool) {
	151	a, b, ok := strings.Cut(key, ".")
	152	if !ok {
	153	return 0, 0, false
	154	}
	155	idx, err1 := strconv.ParseInt(a, 10, 64)
	156	id, err2 := strconv.ParseInt(b, 10, 64)
	157	return idx, id, err1 == nil && err2 == nil && id > 0
	158	}
	159
	160	// IndexedTree reports whether commit sha has the tree idx was built from:
	161	// the default branch's head when the index is current, or any other
	162	// commit with the same content. The blob view uses it to decide whether
	163	// its names can link into the index.
	164	func IndexedTree(dir, sha string, idx store.SymbolIndex) bool {
	165	if sha == idx.Commit {
	166	return true
	167	}
	168	tree, err := gitutil.ResolveTree(dir, sha)
	169	return err == nil && tree == idx.Tree
	170	}
	171
	172	func runAdminSymbolsReindex(c *Ctx, args []string) int {
	173	if len(args) != 1 {
	174	return c.usage()
	175	}
	176	repo, code := adminRepo(c, args[0])
	177	if code >= 0 {
	178	return code
	179	}
	180	if err := c.Store.RequestSymbolIndex(repo.ID, true); err != nil {
	181	return c.fail(protocol.ExitFailure, "%v", err)
	182	}
	183	return c.emit(map[string]string{"repo": repo.Path(), "state": "queued"}, func(w io.Writer) {
	184	fmt.Fprintf(w, "queued a rebuild of the symbol index of %s\n", repo.Path())
	185	})
	186	}

internal/control/symbols_test.go added +144

		@@ -0,0 +1,144 @@
	1	package control
	2
	3	import (
	4	"bytes"
	5	"encoding/json"
	6	"strings"
	7	"testing"
	8
	9	"gitbay.org/gitbay/internal/protocol"
	10	"gitbay.org/gitbay/internal/store"
	11	)
	12
	13	func symbolsFixture(t testing.T) (store.Store, store.Repo, store.User, store.User) {
	14	t.Helper()
	15	st, err := store.Open(":memory:")
	16	if err != nil {
	17	t.Fatal(err)
	18	}
	19	t.Cleanup(func() { st.Close() })
	20	if err := st.MigrateUp(); err != nil {
	21	t.Fatal(err)
	22	}
	23	aliceID, _ := st.CreateUser("alice", false)
	24	bobID, _ := st.CreateUser("bob", false)
	25	id, err := st.CreateRepo("user", aliceID, "secret", "private")
	26	if err != nil {
	27	t.Fatal(err)
	28	}
	29	repo, _ := st.RepoByID(id)
	30	var rows []store.SymbolRow
	31	for _, n := range []string{"Parse", "ParseAll", "ParseArgs", "Parser", "parse"} {
	32	rows = append(rows, store.SymbolRow{Name: n, Key: n, Kind: "function", Path: "p.go", Line: len(rows) + 1})
	33	}
	34	rows = append(rows, store.SymbolRow{Name: "Parser.Run", Key: "Run", Kind: "method", Path: "p.go", Line: 40})
	35	if _, err := st.ReplaceSymbolIndex(store.SymbolIndex{RepoID: id, Commit: "c0", Tree: "t0", State: "ok"}, rows); err != nil {
	36	t.Fatal(err)
	37	}
	38	return st, repo,
	39	store.User{ID: aliceID, Username: "alice"},
	40	store.User{ID: bobID, Username: "bob"}
	41	}
	42
	43	type symbolPage struct {
	44	Data struct {
	45	Items []symbolOut `json:"items"`
	46	Next string `json:"next"`
	47	} `json:"data"`
	48	}
	49
	50	func runSymbols(t testing.T, st store.Store, u store.User, argv ...string) (int, string, string) {
	51	t.Helper()
	52	var out, errOut bytes.Buffer
	53	c := &Ctx{User: u, Scope: "full", Store: st, Stdout: &out, Stderr: &errOut, JSON: true}
	54	c.Cfg.Server.Root = t.TempDir()
	55	code := Dispatch(c, argv)
	56	if code != protocol.ExitOK {
	57	// In JSON mode a refusal is the output's error field.
	58	return code, out.String(), out.String() + errOut.String()
	59	}
	60	return code, out.String(), errOut.String()
	61	}
	62
	63	func TestRepoSymbolsPages(t *testing.T) {
	64	st, repo, alice, _ := symbolsFixture(t)
	65	var names []string
	66	cursor := ""
	67	for pages := 0; ; pages++ {
	68	argv := []string{"repo", "symbols", repo.Path(), "Parse", "--limit", "2"}
	69	if cursor != "" {
	70	argv = append(argv, "--cursor", cursor)
	71	}
	72	code, out, errOut := runSymbols(t, st, alice, argv...)
	73	if code != protocol.ExitOK {
	74	t.Fatalf("exit %d: %s", code, errOut)
	75	}
	76	var p symbolPage
	77	if err := json.Unmarshal([]byte(out), &p); err != nil {
	78	t.Fatalf("%v: %s", err, out)
	79	}
	80	for _, it := range p.Data.Items {
	81	names = append(names, it.Name)
	82	}
	83	if p.Data.Next == "" {
	84	break
	85	}
	86	if pages > 5 {
	87	t.Fatal("paging does not end")
	88	}
	89	cursor = p.Data.Next
	90	}
	91	if got, want := strings.Join(names, " "), "Parse ParseAll ParseArgs Parser Parser.Run parse"; got != want {
	92	t.Fatalf("got %q, want %q", got, want)
	93	}
	94
	95	// A cursor from before a rebuild is refused rather than followed.
	96	st.ReplaceSymbolIndex(store.SymbolIndex{RepoID: repo.ID, Commit: "c1", Tree: "t1", State: "ok"},
	97	[]store.SymbolRow{{Name: "Parse", Key: "Parse", Kind: "function", Path: "p.go", Line: 1}})
	98	code, _, errOut := runSymbols(t, st, alice, "repo", "symbols", repo.Path(), "Parse", "--cursor", cursor)
	99	if code != protocol.ExitUsage \|\| !strings.Contains(errOut, "rebuilt") {
	100	t.Fatalf("stale cursor: exit %d: %s", code, errOut)
	101	}
	102	}
	103
	104	func TestRepoSymbolsRefusals(t *testing.T) {
	105	st, repo, alice, bob := symbolsFixture(t)
	106	for _, tc := range []struct {
	107	name string
	108	user store.User
	109	argv []string
	110	code int
	111	msg string
	112	}{
	113	{"outsider sees no repository", bob, []string{"repo", "symbols", repo.Path(), "Parse"}, protocol.ExitNotFound, "not found"},
	114	{"another ref", alice, []string{"repo", "symbols", repo.Path(), "--ref", "feature", "Parse"}, protocol.ExitNotFound, "only the default branch"},
	115	{"unknown kind", alice, []string{"repo", "symbols", repo.Path(), "--kind", "widget", "Parse"}, protocol.ExitUsage, "--kind"},
	116	{"no query", alice, []string{"repo", "symbols", repo.Path()}, protocol.ExitUsage, ""},
	117	{"one-character query", alice, []string{"repo", "symbols", repo.Path(), "P"}, protocol.ExitUsage, "2 to"},
	118	{"reindex needs an admin", alice, []string{"admin", "symbols", "reindex", repo.Path()}, protocol.ExitDenied, "admin"},
	119	} {
	120	t.Run(tc.name, func(t *testing.T) {
	121	code, _, errOut := runSymbols(t, st, tc.user, tc.argv...)
	122	if code != tc.code \|\| !strings.Contains(errOut, tc.msg) {
	123	t.Fatalf("exit %d, want %d with %q: %s", code, tc.code, tc.msg, errOut)
	124	}
	125	})
	126	}
	127
	128	code, _, errOut := runSymbols(t, st, alice, "repo", "symbols", repo.Path(), "--ref", repo.DefaultBranch, "--kind", "method", "Run")
	129	if code != protocol.ExitOK {
	130	t.Fatalf("default branch by name: exit %d: %s", code, errOut)
	131	}
	132	}
	133
	134	func TestAdminSymbolsReindexQueuesAForcedBuild(t *testing.T) {
	135	st, repo, _, _ := symbolsFixture(t)
	136	code, _, errOut := runSymbols(t, st, rootUser(t, st), "admin", "symbols", "reindex", repo.Path())
	137	if code != protocol.ExitOK {
	138	t.Fatalf("exit %d: %s", code, errOut)
	139	}
	140	reqs, err := st.SymbolRequests()
	141	if err != nil \|\| len(reqs) != 1 \|\| reqs[0].RepoID != repo.ID \|\| !reqs[0].Force {
	142	t.Fatalf("requests = %+v, %v", reqs, err)
	143	}
	144	}

internal/gitutil/treeblobs.go added +117

		@@ -0,0 +1,117 @@
	1	package gitutil
	2
	3	import (
	4	"bufio"
	5	"bytes"
	6	"context"
	7	"errors"
	8	"fmt"
	9	"io"
	10	"os/exec"
	11	"strconv"
	12	"strings"
	13
	14	"gitbay.org/gitbay/internal/toolpath"
	15	)
	16
	17	// ListBlobs lists every blob in a tree, recursively, with its size. Name
	18	// is the path from the tree's root. Paths are read NUL-separated, so no
	19	// file name is quoted or split.
	20	func ListBlobs(ctx context.Context, dir, tree string) ([]TreeEntry, error) {
	21	out, err := exec.CommandContext(ctx, toolpath.Look("git"), "-C", dir,
	22	"ls-tree", "-r", "-l", "-z", "--end-of-options", tree).Output()
	23	if err != nil {
	24	return nil, fmt.Errorf("ls-tree %s: %w", tree, err)
	25	}
	26	var entries []TreeEntry
	27	for _, rec := range bytes.Split(out, []byte{0}) {
	28	meta, name, ok := strings.Cut(string(rec), "\t")
	29	if !ok {
	30	continue
	31	}
	32	f := strings.Fields(meta)
	33	if len(f) != 4 \|\| f[1] != "blob" {
	34	continue
	35	}
	36	// ls-tree prints a size of BAD for a blob it cannot read.
	37	size, err := strconv.ParseInt(f[3], 10, 64)
	38	if err != nil {
	39	return nil, fmt.Errorf("ls-tree %s: cannot read %s", tree, name)
	40	}
	41	entries = append(entries, TreeEntry{Mode: f[0], Type: f[1], SHA: f[2], Size: size, Name: name})
	42	}
	43	return entries, nil
	44	}
	45
	46	// CatBlobs reads blobs by id through one `git cat-file --batch`, calling
	47	// fn with each one's index in shas and its contents, in order. fn
	48	// returning false stops the read. Cancelling parent stops it too, and is
	49	// reported as parent's error.
	50	func CatBlobs(parent context.Context, dir string, shas []string, fn func(i int, data []byte) bool) error {
	51	ctx, cancel := context.WithCancel(parent)
	52	defer cancel()
	53	cmd := exec.CommandContext(ctx, toolpath.Look("git"), "-C", dir, "cat-file", "--batch")
	54	stdin, err := cmd.StdinPipe()
	55	if err != nil {
	56	return err
	57	}
	58	stdout, err := cmd.StdoutPipe()
	59	if err != nil {
	60	return err
	61	}
	62	if err := cmd.Start(); err != nil {
	63	return err
	64	}
	65	go func() {
	66	w := bufio.NewWriter(stdin)
	67	for _, s := range shas {
	68	if _, err := fmt.Fprintln(w, s); err != nil {
	69	break
	70	}
	71	}
	72	w.Flush()
	73	stdin.Close()
	74	}()
	75	r := bufio.NewReader(stdout)
	76	readErr := func() error {
	77	for i := range shas {
	78	header, err := r.ReadString('\n')
	79	if err != nil {
	80	return err
	81	}
	82	// <sha> <type> <size>, or <sha> missing
	83	f := strings.Fields(header)
	84	if len(f) != 3 {
	85	return fmt.Errorf("cat-file: %s", strings.TrimSpace(header))
	86	}
	87	size, err := strconv.ParseInt(f[2], 10, 64)
	88	if err != nil {
	89	return fmt.Errorf("cat-file: %s", strings.TrimSpace(header))
	90	}
	91	data := make([]byte, size+1) // the object and its trailing newline
	92	if _, err := io.ReadFull(r, data); err != nil {
	93	return err
	94	}
	95	if !fn(i, data[:size]) {
	96	return errStopped
	97	}
	98	}
	99	return nil
	100	}()
	101	if readErr != nil {
	102	cancel()
	103	}
	104	io.Copy(io.Discard, stdout)
	105	waitErr := cmd.Wait()
	106	switch {
	107	case readErr == errStopped:
	108	return nil
	109	case parent.Err() != nil:
	110	return parent.Err()
	111	case readErr != nil:
	112	return readErr
	113	}
	114	return waitErr
	115	}
	116
	117	var errStopped = errors.New("stopped")

internal/hookd/symbols_test.go added +63

		@@ -0,0 +1,63 @@
	1	package hookd
	2
	3	import (
	4	"context"
	5	"os"
	6	"path/filepath"
	7	"testing"
	8
	9	"gitbay.org/gitbay/internal/control"
	10	"gitbay.org/gitbay/internal/policy"
	11	"gitbay.org/gitbay/internal/store"
	12	"gitbay.org/gitbay/internal/symbols"
	13	)
	14
	15	// A push to the default branch asks for a symbol index and builds none
	16	// itself; a push to another branch asks for nothing. The worker builds
	17	// the index afterwards.
	18	func TestPostReceiveRequestsSymbolIndex(t *testing.T) {
	19	f := newShapeFixture(t)
	20	requested := func() bool {
	21	t.Helper()
	22	reqs, err := f.st.SymbolRequests()
	23	if err != nil {
	24	t.Fatal(err)
	25	}
	26	return len(reqs) == 1 && reqs[0].RepoID == f.repo.ID && !reqs[0].Force
	27	}
	28
	29	f.git(f.src, "checkout", "-q", "-b", "side")
	30	side := f.appCommit("side")
	31	f.sync()
	32	f.srv.postReceive(Request{RepoID: f.repo.ID, UserID: f.uid, Scope: "full", Updates: []policy.RefUpdate{
	33	{Ref: "refs/heads/side", Old: zeroSHA40, New: side}}})
	34	if requested() {
	35	t.Fatal("a push to a side branch asked for a symbol index")
	36	}
	37
	38	f.git(f.src, "checkout", "-q", "main")
	39	os.WriteFile(filepath.Join(f.src, "main.go"), []byte("package main\n\nfunc Serve() {}\n"), 0o644)
	40	f.git(f.src, "add", ".")
	41	f.git(f.src, "commit", "-q", "-m", "go")
	42	head := f.sha("HEAD")
	43	f.sync()
	44	f.srv.postReceive(Request{RepoID: f.repo.ID, UserID: f.uid, Scope: "full", Updates: []policy.RefUpdate{
	45	{Ref: "refs/heads/main", Old: f.base, New: head}}})
	46	if !requested() {
	47	t.Fatal("a push to the default branch did not ask for a symbol index")
	48	}
	49	if _, err := f.st.SymbolIndexFor(f.repo.ID); err != store.ErrNotFound {
	50	t.Fatalf("the push built an index itself: %v", err)
	51	}
	52
	53	w := symbols.New(f.st, func(owner, name string) string { return control.RepoDir(f.root, owner, name) })
	54	w.Sweep(context.Background())
	55	x, err := f.st.SymbolIndexFor(f.repo.ID)
	56	if err != nil \|\| x.Commit != head \|\| x.State != "ok" {
	57	t.Fatalf("index after the sweep = %+v, %v", x, err)
	58	}
	59	rows, _ := f.st.SearchSymbols(x.ID, "Serve", "", 0, 0)
	60	if len(rows) != 1 \|\| rows[0].Path != "main.go" \|\| rows[0].Line != 3 {
	61	t.Fatalf("Serve = %+v", rows)
	62	}
	63	}

internal/httpd/routes.go +1

		@@ -65,6 +65,7 @@ func (s *Server) Routes() []Route {
65	65	Route{Method: "GET", Pattern: "/{owner}/{repo}/blob/{ref}/{path...}", Handler: s.blob},
66	66	Route{Method: "GET", Pattern: "/{owner}/{repo}/blame/{ref}/{path...}", Handler: s.blame},
67	67	Route{Method: "GET", Pattern: "/{owner}/{repo}/search", Handler: s.search},
	68	Route{Method: "GET", Pattern: "/{owner}/{repo}/symbols", Handler: s.symbolsPage},
68	69	Route{Method: "GET", Pattern: "/{owner}/{repo}/milestones", Handler: s.milestones},
69	70	Route{Method: "GET", Pattern: "/{owner}/{repo}/labels", Handler: s.labels},
70	71	Route{Method: "GET", Pattern: "/{owner}/{repo}/wiki", Handler: s.wiki},

internal/httpd/symbols.go added +157

		@@ -0,0 +1,157 @@
	1	package httpd
	2
	3	import (
	4	"errors"
	5	"fmt"
	6	"html/template"
	7	"net/http"
	8	"net/url"
	9	"regexp"
	10	"strconv"
	11	"strings"
	12
	13	"gitbay.org/gitbay/internal/control"
	14	"gitbay.org/gitbay/internal/store"
	15	"gitbay.org/gitbay/internal/symbols"
	16	)
	17
	18	// nameSpan is a name token as chroma writes it with classes: n, nx, nf,
	19	// nc and the rest of the Name family. Keywords, literals and punctuation
	20	// carry other classes and are never linked.
	21	var nameSpan = regexp.MustCompile(`<span class="n[a-z]?">([\p{L}_$][\p{L}\p{N}_$]*)</span>`)
	22
	23	// escapePath escapes each segment of a slash-separated path for a URL.
	24	func escapePath(p string) string {
	25	parts := strings.Split(p, "/")
	26	for i, s := range parts {
	27	parts[i] = url.PathEscape(s)
	28	}
	29	return strings.Join(parts, "/")
	30	}
	31
	32	// blobSymbols finds the viewed file's symbols and links the names in its
	33	// highlighted source to their definitions, when commit, the ref being
	34	// viewed as the page resolved it, has the indexed tree. A name defined once links to that line at the same
	35	// ref; one defined more than once links to the results page. Names the
	36	// index does not hold stay plain. The whole page costs one lookup of the
	37	// file's distinct names, however many times each appears.
	38	func (s *Server) blobSymbols(p repoPage, commit, filePath string, code template.HTML) (template.HTML, []store.SymbolRow) {
	39	idx, err := s.st.SymbolIndexFor(p.Repo.ID)
	40	if err != nil \|\| !control.IndexedTree(p.Dir, commit, idx) {
	41	return code, nil
	42	}
	43	list, _ := s.st.SymbolsInFile(idx.ID, filePath)
	44	if code == "" {
	45	return code, list
	46	}
	47	seen := map[string]bool{}
	48	var keys []string
	49	for _, m := range nameSpan.FindAllStringSubmatch(string(code), -1) {
	50	if !seen[m[1]] {
	51	seen[m[1]] = true
	52	keys = append(keys, m[1])
	53	}
	54	}
	55	if len(keys) == 0 {
	56	return code, list
	57	}
	58	targets, err := s.st.SymbolTargets(idx.ID, keys)
	59	if err != nil \|\| len(targets) == 0 {
	60	return code, list
	61	}
	62	repoPath := "/" + p.Repo.Path()
	63	ref := escapePath(p.Ref)
	64	linked := nameSpan.ReplaceAllStringFunc(string(code), func(span string) string {
	65	name := nameSpan.FindStringSubmatch(span)[1]
	66	t, ok := targets[name]
	67	if !ok {
	68	return span
	69	}
	70	href := repoPath + "/symbols?q=" + url.QueryEscape(name)
	71	if t.Count == 1 {
	72	href = repoPath + "/blob/" + ref + "/" + escapePath(t.Path) + "#L" + strconv.Itoa(t.Line)
	73	}
	74	return `<a class="sym" href="` + template.HTMLEscapeString(href) + `">` + span + `</a>`
	75	})
	76	return template.HTML(linked), list
	77	}
	78
	79	// symbolsPageSize is the results page's page length.
	80	const symbolsPageSize = 100
	81
	82	// symbolsPage lists the definitions matching a query, ranked as `repo
	83	// symbols` ranks them, from the same store query.
	84	func (s Server) symbolsPage(w http.ResponseWriter, r http.Request) {
	85	p, ok := s.repoFor(w, r, "")
	86	if !ok {
	87	return
	88	}
	89	p.Tab = "search"
	90	q := strings.TrimSpace(r.URL.Query().Get("q"))
	91	kind := r.URL.Query().Get("kind")
	92	if !symbols.ValidKind(kind) {
	93	kind = ""
	94	}
	95	var (
	96	rows []store.SymbolRow
	97	next string
	98	problem string
	99	note string
	100	)
	101	idx, err := s.st.SymbolIndexFor(p.Repo.ID)
	102	switch {
	103	case errors.Is(err, store.ErrNotFound):
	104	problem = "no symbol index yet; one is built after a push to " + p.Repo.DefaultBranch
	105	if f, ferr := s.st.SymbolFailureFor(p.Repo.ID); ferr == nil {
	106	problem = "the symbol index could not be built: " + f.Note
	107	}
	108	case err != nil:
	109	http.Error(w, "internal error", http.StatusInternalServerError)
	110	return
	111	case q == "":
	112	case len(q) < control.MinSymbolQuery \|\| len(q) > 200:
	113	problem = fmt.Sprintf("query must be %d to 200 characters", control.MinSymbolQuery)
	114	default:
	115	if idx.State == "partial" {
	116	note = "the index is partial: " + idx.Note
	117	}
	118	var after int64
	119	if a := r.URL.Query().Get("after"); a != "" {
	120	cursorIdx, id, ok := control.ParseSymbolCursor(a)
	121	switch {
	122	case !ok:
	123	problem = "bad cursor"
	124	case cursorIdx != idx.ID:
	125	problem = control.StaleSymbolCursor
	126	}
	127	after = id
	128	}
	129	if problem != "" {
	130	break
	131	}
	132	rows, err = s.st.SearchSymbols(idx.ID, q, kind, symbolsPageSize+1, after)
	133	if err != nil {
	134	http.Error(w, "internal error", http.StatusInternalServerError)
	135	return
	136	}
	137	if len(rows) > symbolsPageSize {
	138	rows = rows[:symbolsPageSize]
	139	v := url.Values{"q": {q}, "after": {control.SymbolCursor(idx.ID, rows[len(rows)-1].ID)}}
	140	if kind != "" {
	141	v.Set("kind", kind)
	142	}
	143	next = "/" + p.Repo.Path() + "/symbols?" + v.Encode()
	144	}
	145	}
	146	s.render(w, "symbols.html", struct {
	147	repoPage
	148	Query string
	149	Kind string
	150	Kinds []string
	151	Problem string
	152	Note string
	153	Rows []store.SymbolRow
	154	Next string
	155	Commit string
	156	}{p, q, kind, symbols.Kinds, problem, note, rows, next, idx.Commit})
	157	}

internal/httpd/symbols_test.go added +139

		@@ -0,0 +1,139 @@
	1	package httpd
	2
	3	import (
	4	"context"
	5	"os"
	6	"os/exec"
	7	"path/filepath"
	8	"strings"
	9	"testing"
	10
	11	"gitbay.org/gitbay/internal/config"
	12	"gitbay.org/gitbay/internal/control"
	13	"gitbay.org/gitbay/internal/store"
	14	"gitbay.org/gitbay/internal/symbols"
	15	)
	16
	17	func symbolGit(t *testing.T, dir string, args ...string) {
	18	t.Helper()
	19	cmd := exec.Command("git", args...)
	20	cmd.Dir = dir
	21	cmd.Env = append(os.Environ(),
	22	"GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null",
	23	"GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.test",
	24	"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.test")
	25	if out, err := cmd.CombinedOutput(); err != nil {
	26	t.Fatalf("git %v: %v\n%s", args, err, out)
	27	}
	28	}
	29
	30	// symbolServer serves alice/app (public) and alice/secret (private), each
	31	// with main indexed and a side branch whose tree differs.
	32	func symbolServer(t testing.T) Server {
	33	t.Helper()
	34	st, err := store.Open(":memory:")
	35	if err != nil {
	36	t.Fatal(err)
	37	}
	38	t.Cleanup(func() { st.Close() })
	39	if err := st.MigrateUp(); err != nil {
	40	t.Fatal(err)
	41	}
	42	uid, err := st.CreateUser("alice", false)
	43	if err != nil {
	44	t.Fatal(err)
	45	}
	46	root := t.TempDir()
	47	src := filepath.Join(root, "src")
	48	symbolGit(t, root, "init", "-q", "-b", "main", "src")
	49	files := map[string]string{
	50	"main.go": "package main\n\nfunc main() { helper(); dup(); unknown() }\n\nfunc helper() {}\n",
	51	"a/dup.go": "package a\n\nfunc dup() {}\n",
	52	"b/dup.go": "package b\n\nfunc dup() {}\n",
	53	}
	54	for name, body := range files {
	55	os.MkdirAll(filepath.Join(src, filepath.Dir(name)), 0o755)
	56	os.WriteFile(filepath.Join(src, name), []byte(body), 0o644)
	57	}
	58	symbolGit(t, src, "add", ".")
	59	symbolGit(t, src, "commit", "-q", "-m", "base")
	60	symbolGit(t, src, "checkout", "-q", "-b", "side")
	61	os.WriteFile(filepath.Join(src, "side.go"), []byte("package main\n"), 0o644)
	62	symbolGit(t, src, "add", ".")
	63	symbolGit(t, src, "commit", "-q", "-m", "side")
	64
	65	w := symbols.NewWith(st, func(owner, name string) string { return control.RepoDir(root, owner, name) }, 0)
	66	for _, r := range []struct{ name, vis string }{{"app", "public"}, {"secret", "private"}} {
	67	id, err := st.CreateRepo("user", uid, r.name, r.vis)
	68	if err != nil {
	69	t.Fatal(err)
	70	}
	71	dir := control.RepoDir(root, "alice", r.name)
	72	os.MkdirAll(filepath.Dir(dir), 0o755)
	73	symbolGit(t, root, "init", "-q", "--bare", dir)
	74	symbolGit(t, src, "push", "-q", dir, "main", "side")
	75	if _, err := w.Index(context.Background(), id, false); err != nil {
	76	t.Fatal(err)
	77	}
	78	}
	79	cfg := config.Default()
	80	cfg.Server.Root = root
	81	cfg.Server.SiteURL = "https://forge.test/"
	82	return New(cfg, st, nil)
	83	}
	84
	85	func TestBlobLinksIndexedNames(t *testing.T) {
	86	h := symbolServer(t).Handler()
	87	w := get(t, h, "/alice/app/blob/main/main.go", nil)
	88	if w.Code != 200 {
	89	t.Fatalf("blob: %d", w.Code)
	90	}
	91	body := w.Body.String()
	92	for _, want := range []string{
	93	// defined once: straight to the line, at the ref being viewed
	94	`<a class="sym" href="/alice/app/blob/main/main.go#L5"><span class="nf">helper</span></a>`,
	95	// defined twice: to the results page
	96	`<a class="sym" href="/alice/app/symbols?q=dup"><span class="nf">dup</span></a>`,
	97	// the per-file list
	98	`2 symbols in this file`,
	99	} {
	100	if !strings.Contains(body, want) {
	101	t.Errorf("blob page lacks %s", want)
	102	}
	103	}
	104	if strings.Contains(body, `>unknown</span></a>`) {
	105	t.Error("a name the index does not hold is linked")
	106	}
	107
	108	// A branch at another tree is not what the index describes.
	109	w = get(t, h, "/alice/app/blob/side/main.go", nil)
	110	if w.Code != 200 \|\| strings.Contains(w.Body.String(), `class="sym"`) \|\| strings.Contains(w.Body.String(), "in this file") {
	111	t.Errorf("side branch: %d, links or list present", w.Code)
	112	}
	113	}
	114
	115	func TestSymbolsPage(t *testing.T) {
	116	h := symbolServer(t).Handler()
	117	w := get(t, h, "/alice/app/symbols?q=dup", nil)
	118	body := w.Body.String()
	119	if w.Code != 200 \|\| !strings.Contains(body, "a/dup.go:3") \|\| !strings.Contains(body, "b/dup.go:3") {
	120	t.Fatalf("results page: %d\n%s", w.Code, body)
	121	}
	122	w = get(t, h, "/alice/app/symbols?q=dup&kind=type", nil)
	123	if strings.Contains(w.Body.String(), "a/dup.go:3") {
	124	t.Error("kind filter ignored")
	125	}
	126	if w := get(t, h, "/alice/app/symbols?q=d", nil); !strings.Contains(w.Body.String(), "query must be 2 to 200 characters") {
	127	t.Error("a one-character query was searched")
	128	}
	129	// A cursor from an index that has been replaced says so rather than
	130	// showing an empty page.
	131	if w := get(t, h, "/alice/app/symbols?q=dup&after=999999.1", nil); !strings.Contains(w.Body.String(), "rebuilt since that cursor") {
	132	t.Errorf("stale cursor:\n%s", w.Body.String())
	133	}
	134	for _, p := range []string{"/alice/secret/symbols?q=dup", "/alice/secret/blob/main/main.go"} {
	135	if w := get(t, h, p, nil); w.Code != 404 {
	136	t.Errorf("%s anonymously: %d, want 404", p, w.Code)
	137	}
	138	}
	139	}

internal/httpd/web.go +14 −2

		@@ -694,7 +694,14 @@ func (s Server) blob(w http.ResponseWriter, r http.Request) {
694	694	}
695	695	p.Tab = "files"
696	696	filePath := strings.Trim(r.PathValue("path"), "/")
697		data, err := gitutil.ReadBlob(p.Dir, p.Ref, filePath, maxRenderBytes+1)
	697	// The file and its symbol links are read from one commit, even if the
	698	// ref moves while the page renders.
	699	commit, err := gitutil.ResolveRef(p.Dir, p.Ref)
	700	if err != nil {
	701	s.notFound(w, r)
	702	return
	703	}
	704	data, err := gitutil.ReadBlob(p.Dir, commit, filePath, maxRenderBytes+1)
698	705	if err != nil {
699	706	s.notFound(w, r)
700	707	return
		@@ -706,6 +713,10 @@ func (s Server) blob(w http.ResponseWriter, r http.Request) {
706	713	if !binary && !image {
707	714	codeHTML = highlight(filePath, data)
708	715	}
	716	var fileSymbols []store.SymbolRow
	717	if !binary && !image {
	718	codeHTML, fileSymbols = s.blobSymbols(p, commit, filePath, codeHTML)
	719	}
709	720	// Markdown and org render like a README, with the source one click
710	721	// away; ?view=source shows the text instead.
711	722	renderable := markupFile(filePath) && !binary
		@@ -752,8 +763,9 @@ func (s Server) blob(w http.ResponseWriter, r http.Request) {
752	763	Rendered bool // this response shows the rendering
753	764	RenderedHTML template.HTML
754	765	Nav fileNav
	766	Symbols []store.SymbolRow
755	767	}{p, cs, base, filePath, filePath, "blob", binary, image, len(data), lines,
756		entry.Mode == "100755", entry.Mode == "120000", branches, codeHTML, renderable, rendered, renderedHTML, nav})
	768	entry.Mode == "100755", entry.Mode == "120000", branches, codeHTML, renderable, rendered, renderedHTML, nav, fileSymbols})
757	769	}
758	770
759	771	// releases lists tag-anchored releases with notes and assets.

internal/mirror/mirror.go +3

		@@ -84,6 +84,9 @@ func (w *Worker) sweep() {
84	84	w.St.SetMirrorResult(m.ID, err.Error())
85	85	} else {
86	86	w.St.SetMirrorResult(m.ID, "")
	87	if m.Direction == "pull" {
	88	w.St.RequestSymbolIndex(m.RepoID, false)
	89	}
87	90	}
88	91	}
89	92	}

internal/store/migrations/0072_symbols.down.sql added +4

		@@ -0,0 +1,4 @@
	1	DROP TABLE symbol_requests;
	2	DROP TABLE symbol_failures;
	3	DROP TABLE symbols;
	4	DROP TABLE symbol_indexes;

internal/store/migrations/0072_symbols.up.sql added +64

		@@ -0,0 +1,64 @@
	1	-- Symbol indexes (#293): the definitions found in the tree of a
	2	-- repository's default branch head. tree is the key: a head whose tree
	3	-- is the current index's needs no new one.
	4	--
	5	-- An index is written in chunks while its state is building, which no
	6	-- read sees; publishing it makes it ok or partial (a bound was reached,
	7	-- note says which) and retires the previous one in one short
	8	-- transaction. Building and retired rows are deleted, symbols first and
	9	-- in chunks, by the worker. At most one index per repository is current.
	10	--
	11	-- AUTOINCREMENT because a replaced index must not hand its id to the
	12	-- next: a paging cursor names the index it was taken from.
	13	CREATE TABLE symbol_indexes (
	14	id INTEGER PRIMARY KEY AUTOINCREMENT,
	15	repo_id INTEGER NOT NULL REFERENCES repos(id) ON DELETE CASCADE,
	16	commit_sha TEXT NOT NULL,
	17	tree TEXT NOT NULL,
	18	state TEXT NOT NULL CHECK (state IN ('building', 'ok', 'partial', 'retired')),
	19	note TEXT NOT NULL DEFAULT '',
	20	files INTEGER NOT NULL DEFAULT 0,
	21	symbols INTEGER NOT NULL DEFAULT 0,
	22	built_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
	23	);
	24	CREATE INDEX symbol_indexes_repo ON symbol_indexes(repo_id, state);
	25	CREATE UNIQUE INDEX symbol_indexes_current ON symbol_indexes(repo_id)
	26	WHERE state IN ('ok', 'partial');
	27	-- name is what is listed, key the name as written at a use site (they
	28	-- differ for Go methods: Type.Method and Method). lname and lkey are
	29	-- their lower-case forms, for case-insensitive prefix ranges.
	30	CREATE TABLE symbols (
	31	id INTEGER PRIMARY KEY,
	32	index_id INTEGER NOT NULL REFERENCES symbol_indexes(id) ON DELETE CASCADE,
	33	name TEXT NOT NULL,
	34	lname TEXT NOT NULL,
	35	key TEXT NOT NULL,
	36	lkey TEXT NOT NULL,
	37	kind TEXT NOT NULL,
	38	path TEXT NOT NULL,
	39	line INTEGER NOT NULL
	40	);
	41	CREATE INDEX symbols_lname ON symbols(index_id, lname);
	42	CREATE INDEX symbols_lkey ON symbols(index_id, lkey);
	43	CREATE INDEX symbols_key ON symbols(index_id, key);
	44	CREATE INDEX symbols_path ON symbols(index_id, path, line);
	45	-- The last run that could not build an index, kept apart so the current
	46	-- index stays in place. Cleared by the next index published.
	47	CREATE TABLE symbol_failures (
	48	repo_id INTEGER PRIMARY KEY REFERENCES repos(id) ON DELETE CASCADE,
	49	tree TEXT NOT NULL,
	50	note TEXT NOT NULL,
	51	failed_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
	52	);
	53	-- Repositories waiting for the index worker. seq counts requests, so a
	54	-- push that lands while a build runs leaves its request in place; force
	55	-- rebuilds even when the tree is the indexed one. not_before defers a
	56	-- retry after a failure; attempts counts those retries.
	57	CREATE TABLE symbol_requests (
	58	repo_id INTEGER PRIMARY KEY REFERENCES repos(id) ON DELETE CASCADE,
	59	seq INTEGER NOT NULL DEFAULT 1,
	60	force INTEGER NOT NULL DEFAULT 0,
	61	attempts INTEGER NOT NULL DEFAULT 0,
	62	not_before TEXT NOT NULL DEFAULT '',
	63	requested_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
	64	);

internal/store/symbols.go added +412

		@@ -0,0 +1,412 @@
	1	package store
	2
	3	import (
	4	"database/sql"
	5	"errors"
	6	"fmt"
	7	"strings"
	8	"unicode/utf8"
	9	)
	10
	11	// SymbolIndex is one symbol index of a repository. The states are
	12	// described with the table's schema; reads only ever see ok or partial.
	13	type SymbolIndex struct {
	14	ID int64
	15	RepoID int64
	16	Commit string
	17	Tree string
	18	State string // building \| ok \| partial \| retired
	19	Note string
	20	Files int
	21	Symbols int
	22	BuiltAt string
	23	}
	24
	25	// SymbolFailure is the last run that could not build an index.
	26	type SymbolFailure struct {
	27	Tree string
	28	Note string
	29	FailedAt string
	30	}
	31
	32	// SymbolRow is one definition. Key is the name as written where it is
	33	// used; see internal/symbols.
	34	type SymbolRow struct {
	35	ID int64
	36	Name string
	37	Key string
	38	Kind string
	39	Path string
	40	Line int
	41	}
	42
	43	// SymbolTarget is where a key is defined: Count definitions, the first of
	44	// them at Path and Line.
	45	type SymbolTarget struct {
	46	Count int
	47	Path string
	48	Line int
	49	}
	50
	51	// SymbolRequest is a repository waiting for the index worker.
	52	type SymbolRequest struct {
	53	RepoID int64
	54	Seq int64
	55	Force bool
	56	Attempts int
	57	}
	58
	59	// RequestSymbolIndex queues a repository for the index worker. A request
	60	// already waiting is bumped, so one taken by a running build is not
	61	// cleared when that build ends, and a deferred retry becomes due now;
	62	// force is kept once set.
	63	func (s *Store) RequestSymbolIndex(repoID int64, force bool) error {
	64	_, err := s.DB.Exec(`
	65	INSERT INTO symbol_requests (repo_id, force) VALUES (?, ?)
	66	ON CONFLICT (repo_id) DO UPDATE SET seq = seq + 1,
	67	force = MAX(force, excluded.force), attempts = 0, not_before = '',
	68	requested_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`, repoID, force)
	69	return err
	70	}
	71
	72	// SymbolRequests lists the requests that are due, oldest first.
	73	func (s *Store) SymbolRequests() ([]SymbolRequest, error) {
	74	rows, err := s.DB.Query(`
	75	SELECT repo_id, seq, force, attempts FROM symbol_requests
	76	WHERE not_before = '' OR not_before <= strftime('%Y-%m-%dT%H:%M:%fZ','now')
	77	ORDER BY requested_at, repo_id`)
	78	if err != nil {
	79	return nil, err
	80	}
	81	defer rows.Close()
	82	var out []SymbolRequest
	83	for rows.Next() {
	84	var r SymbolRequest
	85	if err := rows.Scan(&r.RepoID, &r.Seq, &r.Force, &r.Attempts); err != nil {
	86	return nil, err
	87	}
	88	out = append(out, r)
	89	}
	90	return out, rows.Err()
	91	}
	92
	93	// DoneSymbolRequest clears a request the worker has handled, unless it was
	94	// requested again since it was read.
	95	func (s *Store) DoneSymbolRequest(r SymbolRequest) error {
	96	_, err := s.DB.Exec("DELETE FROM symbol_requests WHERE repo_id = ? AND seq = ?", r.RepoID, r.Seq)
	97	return err
	98	}
	99
	100	// DeferSymbolRequest keeps a request for another attempt after seconds,
	101	// unless it was requested again since it was read.
	102	func (s *Store) DeferSymbolRequest(r SymbolRequest, seconds int) error {
	103	_, err := s.DB.Exec(`
	104	UPDATE symbol_requests SET force = 0, attempts = attempts + 1,
	105	not_before = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?)
	106	WHERE repo_id = ? AND seq = ?`, fmt.Sprintf("+%d seconds", seconds), r.RepoID, r.Seq)
	107	return err
	108	}
	109
	110	const symbolIndexCols = `id, repo_id, commit_sha, tree, state, note, files, symbols, built_at`
	111
	112	func scanSymbolIndex(row *sql.Row) (SymbolIndex, error) {
	113	var x SymbolIndex
	114	err := row.Scan(&x.ID, &x.RepoID, &x.Commit, &x.Tree, &x.State, &x.Note, &x.Files, &x.Symbols, &x.BuiltAt)
	115	if errors.Is(err, sql.ErrNoRows) {
	116	return x, ErrNotFound
	117	}
	118	return x, err
	119	}
	120
	121	// SymbolIndexFor returns a repository's current index, or ErrNotFound
	122	// when none has been published.
	123	func (s *Store) SymbolIndexFor(repoID int64) (SymbolIndex, error) {
	124	return scanSymbolIndex(s.DB.QueryRow(`SELECT `+symbolIndexCols+`
	125	FROM symbol_indexes WHERE repo_id = ? AND state IN ('ok', 'partial')`, repoID))
	126	}
	127
	128	// SymbolIndexByID returns an index in any state.
	129	func (s *Store) SymbolIndexByID(id int64) (SymbolIndex, error) {
	130	return scanSymbolIndex(s.DB.QueryRow(`SELECT `+symbolIndexCols+`
	131	FROM symbol_indexes WHERE id = ?`, id))
	132	}
	133
	134	// BeginSymbolIndex creates an index in the building state, which no read
	135	// sees until PublishSymbolIndex.
	136	func (s *Store) BeginSymbolIndex(repoID int64, commit, tree string) (int64, error) {
	137	res, err := s.DB.Exec(`
	138	INSERT INTO symbol_indexes (repo_id, commit_sha, tree, state)
	139	VALUES (?, ?, ?, 'building')`, repoID, commit, tree)
	140	if err != nil {
	141	return 0, err
	142	}
	143	return res.LastInsertId()
	144	}
	145
	146	// symbolInsertRows is how many rows one INSERT carries: eight columns
	147	// each, well under SQLite's variable limit.
	148	const symbolInsertRows = 200
	149
	150	// AddSymbols writes rows into an index being built, in one transaction.
	151	// The caller keeps each call to a few thousand rows so the write lock is
	152	// held briefly.
	153	func (s *Store) AddSymbols(indexID int64, syms []SymbolRow) error {
	154	tx, err := s.DB.Begin()
	155	if err != nil {
	156	return err
	157	}
	158	defer tx.Rollback()
	159	for len(syms) > 0 {
	160	n := min(len(syms), symbolInsertRows)
	161	var q strings.Builder
	162	q.WriteString("INSERT INTO symbols (index_id, name, lname, key, lkey, kind, path, line) VALUES ")
	163	args := make([]any, 0, n*8)
	164	for i, r := range syms[:n] {
	165	if i > 0 {
	166	q.WriteString(",")
	167	}
	168	q.WriteString("(?,?,?,?,?,?,?,?)")
	169	args = append(args, indexID, r.Name, strings.ToLower(r.Name), r.Key, strings.ToLower(r.Key), r.Kind, r.Path, r.Line)
	170	}
	171	if _, err := tx.Exec(q.String(), args...); err != nil {
	172	return err
	173	}
	174	syms = syms[n:]
	175	}
	176	return tx.Commit()
	177	}
	178
	179	// PublishSymbolIndex makes a built index the repository's current one, as
	180	// ok or partial, and retires the one it replaces, in one short
	181	// transaction. A recorded failure is cleared.
	182	func (s *Store) PublishSymbolIndex(x SymbolIndex) error {
	183	tx, err := s.DB.Begin()
	184	if err != nil {
	185	return err
	186	}
	187	defer tx.Rollback()
	188	if _, err := tx.Exec(`
	189	UPDATE symbol_indexes SET state = 'retired'
	190	WHERE repo_id = ? AND state IN ('ok', 'partial')`, x.RepoID); err != nil {
	191	return err
	192	}
	193	res, err := tx.Exec(`
	194	UPDATE symbol_indexes SET state = ?, note = ?, files = ?,
	195	symbols = (SELECT COUNT(*) FROM symbols WHERE index_id = ?),
	196	built_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')
	197	WHERE id = ? AND repo_id = ? AND state = 'building'`,
	198	x.State, x.Note, x.Files, x.ID, x.ID, x.RepoID)
	199	if err != nil {
	200	return err
	201	}
	202	if n, _ := res.RowsAffected(); n != 1 {
	203	return fmt.Errorf("symbol index %d is not being built", x.ID)
	204	}
	205	if _, err := tx.Exec("DELETE FROM symbol_failures WHERE repo_id = ?", x.RepoID); err != nil {
	206	return err
	207	}
	208	return tx.Commit()
	209	}
	210
	211	// symbolDeleteRows is how many symbols one purge transaction deletes.
	212	const symbolDeleteRows = 5000
	213
	214	// PurgeSymbolIndexes deletes a repository's indexes that are not current:
	215	// retired ones, and building ones left by a run that did not finish. The
	216	// symbols go a chunk per transaction, then the index row.
	217	func (s *Store) PurgeSymbolIndexes(repoID int64) error {
	218	rows, err := s.DB.Query(`SELECT id FROM symbol_indexes
	219	WHERE repo_id = ? AND state IN ('building', 'retired')`, repoID)
	220	if err != nil {
	221	return err
	222	}
	223	var ids []int64
	224	for rows.Next() {
	225	var id int64
	226	if err := rows.Scan(&id); err != nil {
	227	rows.Close()
	228	return err
	229	}
	230	ids = append(ids, id)
	231	}
	232	rows.Close()
	233	if err := rows.Err(); err != nil {
	234	return err
	235	}
	236	for _, id := range ids {
	237	for {
	238	res, err := s.DB.Exec(`DELETE FROM symbols WHERE id IN
	239	(SELECT id FROM symbols WHERE index_id = ? LIMIT ?)`, id, symbolDeleteRows)
	240	if err != nil {
	241	return err
	242	}
	243	if n, _ := res.RowsAffected(); n == 0 {
	244	break
	245	}
	246	}
	247	if _, err := s.DB.Exec(`DELETE FROM symbol_indexes
	248	WHERE id = ? AND state IN ('building', 'retired')`, id); err != nil {
	249	return err
	250	}
	251	}
	252	return nil
	253	}
	254
	255	// ReplaceSymbolIndex builds, publishes and purges in one call: the
	256	// worker's sequence, for a caller holding every row already.
	257	func (s *Store) ReplaceSymbolIndex(x SymbolIndex, syms []SymbolRow) (int64, error) {
	258	id, err := s.BeginSymbolIndex(x.RepoID, x.Commit, x.Tree)
	259	if err != nil {
	260	return 0, err
	261	}
	262	for len(syms) > 0 {
	263	n := min(len(syms), 5000)
	264	if err := s.AddSymbols(id, syms[:n]); err != nil {
	265	return 0, err
	266	}
	267	syms = syms[n:]
	268	}
	269	x.ID = id
	270	if err := s.PublishSymbolIndex(x); err != nil {
	271	return 0, err
	272	}
	273	return id, s.PurgeSymbolIndexes(x.RepoID)
	274	}
	275
	276	// RecordSymbolFailure records a run that could not build an index. The
	277	// current index, if any, stays current.
	278	func (s *Store) RecordSymbolFailure(repoID int64, tree, note string) error {
	279	_, err := s.DB.Exec(`
	280	INSERT INTO symbol_failures (repo_id, tree, note) VALUES (?, ?, ?)
	281	ON CONFLICT (repo_id) DO UPDATE SET tree = excluded.tree, note = excluded.note,
	282	failed_at = strftime('%Y-%m-%dT%H:%M:%fZ','now')`, repoID, tree, note)
	283	return err
	284	}
	285
	286	// SymbolFailureFor returns the last failure, or ErrNotFound.
	287	func (s *Store) SymbolFailureFor(repoID int64) (SymbolFailure, error) {
	288	var f SymbolFailure
	289	err := s.DB.QueryRow(`SELECT tree, note, failed_at FROM symbol_failures WHERE repo_id = ?`, repoID).
	290	Scan(&f.Tree, &f.Note, &f.FailedAt)
	291	if errors.Is(err, sql.ErrNoRows) {
	292	return f, ErrNotFound
	293	}
	294	return f, err
	295	}
	296
	297	// SymbolFailureRecent reports whether tree failed within the last
	298	// seconds.
	299	func (s *Store) SymbolFailureRecent(repoID int64, tree string, seconds int) (bool, error) {
	300	var n int
	301	err := s.DB.QueryRow(`SELECT COUNT(*) FROM symbol_failures
	302	WHERE repo_id = ? AND tree = ?
	303	AND failed_at > strftime('%Y-%m-%dT%H:%M:%fZ', 'now', ?)`,
	304	repoID, tree, fmt.Sprintf("-%d seconds", seconds)).Scan(&n)
	305	return n > 0, err
	306	}
	307
	308	// SearchSymbols finds the symbols whose name or key starts with q, ignoring
	309	// case, ranked: an exact match first, then a prefix match, each
	310	// case-sensitive before case-insensitive, then by name, path and line.
	311	// after is the id of the last row of the previous page, 0 for the first;
	312	// an id that is no longer in the result gives an empty page. kind filters
	313	// when not empty.
	314	func (s *Store) SearchSymbols(indexID int64, q, kind string, limit int, after int64) ([]SymbolRow, error) {
	315	lq := strings.ToLower(q)
	316	hi := lq + "\U0010FFFF"
	317	n := utf8.RuneCountInString(q)
	318	lim := -1
	319	if limit > 0 {
	320	lim = limit
	321	}
	322	rows, err := s.DB.Query(`
	323	WITH m AS (
	324	SELECT id, name, key, kind, path, line,
	325	CASE WHEN name = ? OR key = ? THEN 0
	326	WHEN substr(name, 1, ?) = ? OR substr(key, 1, ?) = ? THEN 1
	327	WHEN lname = ? OR lkey = ? THEN 2
	328	ELSE 3 END AS rank
	329	FROM symbols
	330	WHERE index_id = ?
	331	AND ((lname >= ? AND lname < ?) OR (lkey >= ? AND lkey < ?))
	332	AND (? = '' OR kind = ?)
	333	)
	334	SELECT id, name, key, kind, path, line FROM m
	335	WHERE ? = 0 OR (rank, name, path, line, id) >
	336	(SELECT rank, name, path, line, id FROM m WHERE id = ?)
	337	ORDER BY rank, name, path, line, id
	338	LIMIT ?`,
	339	q, q, n, q, n, q, lq, lq,
	340	indexID, lq, hi, lq, hi, kind, kind,
	341	after, after, lim)
	342	if err != nil {
	343	return nil, err
	344	}
	345	return scanSymbols(rows)
	346	}
	347
	348	// SymbolsInFile lists one file's symbols in line order.
	349	func (s *Store) SymbolsInFile(indexID int64, path string) ([]SymbolRow, error) {
	350	rows, err := s.DB.Query(`
	351	SELECT id, name, key, kind, path, line FROM symbols
	352	WHERE index_id = ? AND path = ? ORDER BY line, id`, indexID, path)
	353	if err != nil {
	354	return nil, err
	355	}
	356	return scanSymbols(rows)
	357	}
	358
	359	func scanSymbols(rows *sql.Rows) ([]SymbolRow, error) {
	360	defer rows.Close()
	361	var out []SymbolRow
	362	for rows.Next() {
	363	var r SymbolRow
	364	if err := rows.Scan(&r.ID, &r.Name, &r.Key, &r.Kind, &r.Path, &r.Line); err != nil {
	365	return nil, err
	366	}
	367	out = append(out, r)
	368	}
	369	return out, rows.Err()
	370	}
	371
	372	// symbolKeysPerQuery bounds the IN list of one SymbolTargets query.
	373	const symbolKeysPerQuery = 500
	374
	375	// SymbolTargets looks up where each of keys is defined, case-sensitively.
	376	// A key with no definition is absent from the result.
	377	func (s *Store) SymbolTargets(indexID int64, keys []string) (map[string]SymbolTarget, error) {
	378	out := map[string]SymbolTarget{}
	379	for len(keys) > 0 {
	380	n := min(len(keys), symbolKeysPerQuery)
	381	args := make([]any, 0, n+1)
	382	args = append(args, indexID)
	383	for _, k := range keys[:n] {
	384	args = append(args, k)
	385	}
	386	// SQLite takes the bare columns of an aggregate query with MIN
	387	// from the row that holds the minimum: the first definition.
	388	rows, err := s.DB.Query(`
	389	SELECT key, COUNT(*), path, line, MIN(id) FROM symbols
	390	WHERE index_id = ? AND key IN (?`+strings.Repeat(",?", n-1)+`)
	391	GROUP BY key`, args...)
	392	if err != nil {
	393	return nil, err
	394	}
	395	for rows.Next() {
	396	var k string
	397	var t SymbolTarget
	398	var id int64
	399	if err := rows.Scan(&k, &t.Count, &t.Path, &t.Line, &id); err != nil {
	400	rows.Close()
	401	return nil, err
	402	}
	403	out[k] = t
	404	}
	405	rows.Close()
	406	if err := rows.Err(); err != nil {
	407	return nil, err
	408	}
	409	keys = keys[n:]
	410	}
	411	return out, nil
	412	}

internal/store/symbols_test.go added +93

		@@ -0,0 +1,93 @@
	1	package store
	2
	3	import (
	4	"strings"
	5	"testing"
	6	)
	7
	8	func symbolFixture(t testing.T) (Store, int64) {
	9	t.Helper()
	10	s := open(t)
	11	if err := s.MigrateUp(); err != nil {
	12	t.Fatal(err)
	13	}
	14	uid, err := s.CreateUser("alice", false)
	15	if err != nil {
	16	t.Fatal(err)
	17	}
	18	repoID, err := s.CreateRepo("user", uid, "app", "public")
	19	if err != nil {
	20	t.Fatal(err)
	21	}
	22	idx, err := s.ReplaceSymbolIndex(SymbolIndex{RepoID: repoID, Commit: "c", Tree: "t", State: "ok"}, []SymbolRow{
	23	{Name: "handlers", Key: "handlers", Kind: "var", Path: "b.go", Line: 1},
	24	{Name: "Server.Handle", Key: "Handle", Kind: "method", Path: "a.go", Line: 9},
	25	{Name: "Handle", Key: "Handle", Kind: "function", Path: "z.go", Line: 3},
	26	{Name: "handle", Key: "handle", Kind: "function", Path: "c.go", Line: 2},
	27	{Name: "HandleFunc", Key: "HandleFunc", Kind: "function", Path: "a.go", Line: 20},
	28	{Name: "Other", Key: "Other", Kind: "type", Path: "a.go", Line: 30},
	29	})
	30	if err != nil {
	31	t.Fatal(err)
	32	}
	33	return s, idx
	34	}
	35
	36	func symbolNames(rows []SymbolRow) string {
	37	var out []string
	38	for _, r := range rows {
	39	out = append(out, r.Name)
	40	}
	41	return strings.Join(out, " ")
	42	}
	43
	44	// Exact before prefix, case-sensitive before not, then by name; a method
	45	// matches on the name it is called by as well as Type.Method.
	46	func TestSearchSymbolsRanks(t *testing.T) {
	47	s, idx := symbolFixture(t)
	48	rows, err := s.SearchSymbols(idx, "Handle", "", 0, 0)
	49	if err != nil {
	50	t.Fatal(err)
	51	}
	52	if got, want := symbolNames(rows), "Handle Server.Handle HandleFunc handle handlers"; got != want {
	53	t.Fatalf("got %q, want %q", got, want)
	54	}
	55	rows, _ = s.SearchSymbols(idx, "Handle", "method", 0, 0)
	56	if got := symbolNames(rows); got != "Server.Handle" {
	57	t.Fatalf("kind filter: %q", got)
	58	}
	59	rows, _ = s.SearchSymbols(idx, "server.", "", 0, 0)
	60	if got := symbolNames(rows); got != "Server.Handle" {
	61	t.Fatalf("Type. prefix: %q", got)
	62	}
	63
	64	// Paging walks the same order without repeats.
	65	var paged []SymbolRow
	66	var after int64
	67	for {
	68	page, err := s.SearchSymbols(idx, "handle", "", 2, after)
	69	if err != nil {
	70	t.Fatal(err)
	71	}
	72	if len(page) == 0 {
	73	break
	74	}
	75	paged = append(paged, page...)
	76	after = page[len(page)-1].ID
	77	}
	78	all, _ := s.SearchSymbols(idx, "handle", "", 0, 0)
	79	if symbolNames(paged) != symbolNames(all) \|\| len(all) != 5 {
	80	t.Fatalf("paged %q, all %q", symbolNames(paged), symbolNames(all))
	81	}
	82	}
	83
	84	func TestSymbolTargets(t *testing.T) {
	85	s, idx := symbolFixture(t)
	86	got, err := s.SymbolTargets(idx, []string{"Handle", "Other", "missing"})
	87	if err != nil {
	88	t.Fatal(err)
	89	}
	90	if len(got) != 2 \|\| got["Handle"].Count != 2 \|\| got["Other"] != (SymbolTarget{1, "a.go", 30}) {
	91	t.Fatalf("targets = %+v", got)
	92	}
	93	}

internal/symbols/extract.go added +355

		@@ -0,0 +1,355 @@
	1	// Package symbols finds definitions in source files and keeps one index of
	2	// them per repository, for `repo symbols` and the blob view's links from a
	3	// name to where it is defined.
	4	//
	5	// The indexer is pure Go, with no external tagger. Go is parsed with
	6	// go/parser; the other languages are matched a line at a time against a
	7	// small table of anchored patterns, which finds the common definition
	8	// shapes and misses the unusual ones.
	9	//
	10	// Go function, method (named Type.Method), type, const, var
	11	// Swift function, class, struct, enum, interface (protocol), type (typealias)
	12	// Rust function, struct, enum, interface (trait), type, module, const, macro
	13	// Python function, method (an indented def), class
	14	// JavaScript, TS function, class, interface, type, enum, const
	15	// C and C++ headers function (prototype), struct, enum, class, type (typedef), macro (#define)
	16	// Shell function
	17	// Org, Markdown section (a heading)
	18	//
	19	// The index covers the tree of the default branch's head and is keyed by
	20	// that tree's id, so a push that leaves the tree as it was is not indexed
	21	// again. Worker builds it in the background after a push to the default
	22	// branch.
	23	package symbols
	24
	25	import (
	26	"bytes"
	27	"go/ast"
	28	"go/parser"
	29	"go/token"
	30	"path"
	31	"regexp"
	32	"strings"
	33	)
	34
	35	// Kinds are the symbol kinds the indexer records, in the order they are
	36	// listed.
	37	var Kinds = []string{
	38	"function", "method", "class", "struct", "enum", "interface",
	39	"type", "const", "var", "module", "macro", "section",
	40	}
	41
	42	// ValidKind reports whether k is one of Kinds.
	43	func ValidKind(k string) bool {
	44	for _, v := range Kinds {
	45	if v == k {
	46	return true
	47	}
	48	}
	49	return false
	50	}
	51
	52	// Symbol is one definition in one file. Name is what is listed; Key is the
	53	// name as it is written where the symbol is used, which differs from Name
	54	// only for Go methods (Name "Server.Handle", Key "Handle").
	55	type Symbol struct {
	56	Name string
	57	Key string
	58	Kind string
	59	Line int
	60	}
	61
	62	// MaxFileBytes is the largest file the indexer reads.
	63	const MaxFileBytes = 1 << 20
	64
	65	// Skip reports whether a file is left out of the index before it is read:
	66	// too large, vendored, generated by its name, or in a language the indexer
	67	// does not know.
	68	func Skip(p string, size int64) bool {
	69	if size > MaxFileBytes \|\| Language(p) == "" {
	70	return true
	71	}
	72	for _, seg := range strings.Split(path.Dir(p), "/") {
	73	if seg == "vendor" \|\| seg == "node_modules" {
	74	return true
	75	}
	76	}
	77	base := path.Base(p)
	78	return strings.HasSuffix(base, "_gen.go") \|\| strings.HasSuffix(base, ".pb.go") \|\|
	79	strings.HasSuffix(base, ".min.js")
	80	}
	81
	82	var languages = map[string]string{
	83	".go": "go",
	84	".swift": "swift",
	85	".rs": "rust",
	86	".py": "python",
	87	".js": "js",
	88	".mjs": "js",
	89	".cjs": "js",
	90	".jsx": "js",
	91	".ts": "js",
	92	".tsx": "js",
	93	".mts": "js",
	94	".h": "c",
	95	".hh": "c",
	96	".hpp": "c",
	97	".hxx": "c",
	98	".sh": "shell",
	99	".bash": "shell",
	100	".zsh": "shell",
	101	".org": "org",
	102	".md": "markdown",
	103	".markdown": "markdown",
	104	}
	105
	106	// Language names the indexer's language for a path, or "" when it has
	107	// none.
	108	func Language(p string) string {
	109	return languages[strings.ToLower(path.Ext(p))]
	110	}
	111
	112	// generatedGo is the marker `go generate` tools write, per the Go
	113	// convention for generated files.
	114	var generatedGo = regexp.MustCompile(`(?m)^// Code generated .* DO NOT EDIT\.$`)
	115
	116	// MaxNameBytes is the longest name the index keeps. A longer one is
	117	// dropped: no real definition is named that way, and a file of them is a
	118	// way to fill the index.
	119	const MaxNameBytes = 256
	120
	121	// Extract returns the definitions in one file, in line order.
	122	func Extract(p string, data []byte) []Symbol {
	123	var syms []Symbol
	124	switch lang := Language(p); lang {
	125	case "":
	126	return nil
	127	case "go":
	128	syms = extractGo(data)
	129	case "org", "markdown":
	130	syms = extractHeadings(lang, data)
	131	default:
	132	syms = extractLines(rules[lang], data)
	133	}
	134	kept := syms[:0]
	135	for _, s := range syms {
	136	if len(s.Name) <= MaxNameBytes && len(s.Key) <= MaxNameBytes {
	137	kept = append(kept, s)
	138	}
	139	}
	140	return kept
	141	}
	142
	143	func extractGo(data []byte) []Symbol {
	144	head := data
	145	if len(head) > 4096 {
	146	head = head[:4096]
	147	}
	148	if generatedGo.Match(head) {
	149	return nil
	150	}
	151	fset := token.NewFileSet()
	152	// A file that does not parse still yields the declarations before the
	153	// error.
	154	f, _ := parser.ParseFile(fset, "", data, parser.SkipObjectResolution)
	155	if f == nil {
	156	return nil
	157	}
	158	var out []Symbol
	159	add := func(id *ast.Ident, name, kind string) {
	160	if id == nil \|\| id.Name == "_" {
	161	return
	162	}
	163	out = append(out, Symbol{Name: name, Key: id.Name, Kind: kind, Line: fset.Position(id.Pos()).Line})
	164	}
	165	for _, d := range f.Decls {
	166	switch d := d.(type) {
	167	case *ast.FuncDecl:
	168	if d.Recv == nil \|\| len(d.Recv.List) == 0 {
	169	add(d.Name, d.Name.Name, "function")
	170	continue
	171	}
	172	recv := receiverName(d.Recv.List[0].Type)
	173	if recv == "" {
	174	add(d.Name, d.Name.Name, "method")
	175	continue
	176	}
	177	add(d.Name, recv+"."+d.Name.Name, "method")
	178	case *ast.GenDecl:
	179	for _, spec := range d.Specs {
	180	switch s := spec.(type) {
	181	case *ast.TypeSpec:
	182	add(s.Name, s.Name.Name, "type")
	183	case *ast.ValueSpec:
	184	kind := "var"
	185	if d.Tok == token.CONST {
	186	kind = "const"
	187	}
	188	for _, n := range s.Names {
	189	add(n, n.Name, kind)
	190	}
	191	}
	192	}
	193	}
	194	}
	195	return out
	196	}
	197
	198	// receiverName is the type a method is declared on, without a pointer or
	199	// type parameters.
	200	func receiverName(e ast.Expr) string {
	201	for {
	202	switch t := e.(type) {
	203	case *ast.StarExpr:
	204	e = t.X
	205	case *ast.ParenExpr:
	206	e = t.X
	207	case *ast.IndexExpr:
	208	e = t.X
	209	case *ast.IndexListExpr:
	210	e = t.X
	211	case *ast.Ident:
	212	return t.Name
	213	default:
	214	return ""
	215	}
	216	}
	217	}
	218
	219	// rule is one line pattern: the first submatch is the name.
	220	type rule struct {
	221	re *regexp.Regexp
	222	kind string
	223	}
	224
	225	func r(kind, pattern string) rule { return rule{regexp.MustCompile(pattern), kind} }
	226
	227	// ident is a name in the languages the line rules cover.
	228	const ident = `([A-Za-z_$][A-Za-z0-9_$]*)`
	229
	230	var rules = map[string][]rule{
	231	"swift": {
	232	r("function", `^\s(?:@\w+\s+)(?:(?:public\|private\|fileprivate\|internal\|open\|static\|class\|final\|override\|mutating\|nonisolated\|async\|convenience\|required)\s+)*func\s+`+ident),
	233	r("class", `^\s(?:@\w+\s+)(?:(?:public\|private\|fileprivate\|internal\|open\|final)\s+)*(?:class\|actor)\s+`+ident),
	234	r("struct", `^\s(?:@\w+\s+)(?:(?:public\|private\|fileprivate\|internal)\s+)*struct\s+`+ident),
	235	r("enum", `^\s(?:@\w+\s+)(?:(?:public\|private\|fileprivate\|internal\|indirect)\s+)*enum\s+`+ident),
	236	r("interface", `^\s(?:@\w+\s+)(?:(?:public\|private\|fileprivate\|internal)\s+)*protocol\s+`+ident),
	237	r("type", `^\s(?:(?:public\|private\|fileprivate\|internal)\s+)typealias\s+`+ident),
	238	},
	239	"rust": {
	240	r("function", `^\s(?:pub(?:$[^)]$)?\s+)?(?:(?:const\|async\|unsafe\|extern(?:\s+"[^"]")?)\s+)fn\s+`+ident),
	241	r("struct", `^\s(?:pub(?:$[^)]$)?\s+)?struct\s+`+ident),
	242	r("enum", `^\s(?:pub(?:$[^)]$)?\s+)?enum\s+`+ident),
	243	r("interface", `^\s(?:pub(?:$[^)]$)?\s+)?(?:unsafe\s+)?trait\s+`+ident),
	244	r("type", `^\s(?:pub(?:$[^)]$)?\s+)?type\s+`+ident),
	245	r("module", `^\s(?:pub(?:$[^)]$)?\s+)?mod\s+`+ident),
	246	r("const", `^\s(?:pub(?:$[^)]$)?\s+)?(?:const\|static(?:\s+mut)?)\s+`+ident+`\s*:`),
	247	r("macro", `^\s*macro_rules!\s+`+ident),
	248	},
	249	"python": {
	250	r("function", `^(?:async\s+)?def\s+`+ident),
	251	r("method", `^\s+(?:async\s+)?def\s+`+ident),
	252	r("class", `^\s*class\s+`+ident),
	253	},
	254	"js": {
	255	r("function", `^\s(?:export\s+)?(?:default\s+)?(?:async\s+)?function\s\?\s`+ident),
	256	r("class", `^\s*(?:export\s+)?(?:default\s+)?(?:abstract\s+)?class\s+`+ident),
	257	r("interface", `^\s*(?:export\s+)?(?:declare\s+)?interface\s+`+ident),
	258	r("type", `^\s(?:export\s+)?(?:declare\s+)?type\s+`+ident+`\s(?:<[^=]>)?\s=`),
	259	r("enum", `^\s*(?:export\s+)?(?:declare\s+)?(?:const\s+)?enum\s+`+ident),
	260	r("const", `^(?:export\s+)?const\s+`+ident+`\s(?::[^=])?=`),
	261	},
	262	"c": {
	263	r("macro", `^\s#\sdefine\s+`+ident),
	264	r("struct", `^\s(?:typedef\s+)?struct\s+`+ident+`\s\{`),
	265	r("enum", `^\s(?:typedef\s+)?enum\s+(?:class\s+)?`+ident+`\s(?::[^{]*)?\{`),
	266	r("class", `^\sclass\s+`+ident+`\s(?::[^{;]*)?\{`),
	267	r("type", `^\stypedef\s+[^;(]?\b`+ident+`\s*;`),
	268	r("type", `^\s}\s`+ident+`\s*;`),
	269	r("function", `^[A-Za-z_][\w\s\&:<>,]?[\s\&]`+ident+`\s$[^;{]$\s(?:const\s*)?;`),
	270	},
	271	"shell": {
	272	r("function", `^\sfunction\s+([A-Za-z_][A-Za-z0-9_:.-])`),
	273	r("function", `^\s([A-Za-z_][A-Za-z0-9_:.-])\s\s(?:\{\|$)`),
	274	},
	275	}
	276
	277	// keywords are never names, though a rule can capture one: Swift's
	278	// `class var x` reads as a class named var, C's `if (x);` as a prototype.
	279	var keywords = map[string]bool{
	280	"return": true, "if": true, "while": true, "for": true, "switch": true,
	281	"sizeof": true, "else": true, "case": true, "do": true, "goto": true,
	282	"var": true, "let": true, "func": true, "static": true,
	283	}
	284
	285	func extractLines(rs []rule, data []byte) []Symbol {
	286	var out []Symbol
	287	line := 0
	288	for len(data) > 0 {
	289	line++
	290	var text []byte
	291	if i := bytes.IndexByte(data, '\n'); i >= 0 {
	292	text, data = data[:i], data[i+1:]
	293	} else {
	294	text, data = data, nil
	295	}
	296	if len(text) > 1000 {
	297	continue
	298	}
	299	for _, ru := range rs {
	300	m := ru.re.FindSubmatch(text)
	301	if m == nil {
	302	continue
	303	}
	304	name := string(m[1])
	305	if keywords[name] {
	306	continue
	307	}
	308	out = append(out, Symbol{Name: name, Key: name, Kind: ru.kind, Line: line})
	309	break
	310	}
	311	}
	312	return out
	313	}
	314
	315	var (
	316	mdHeading = regexp.MustCompile(`^#{1,6}\s+(.+?)\s#\s*$`)
	317	orgHeading = regexp.MustCompile(`^\+\s+(.+?)\s$`)
	318	)
	319
	320	// extractHeadings lists headings as sections, skipping what sits inside a
	321	// code block, where a line starting with # or * is code.
	322	func extractHeadings(lang string, data []byte) []Symbol {
	323	var out []Symbol
	324	inBlock := false
	325	for i, text := range strings.Split(string(data), "\n") {
	326	trimmed := strings.TrimSpace(text)
	327	if lang == "markdown" {
	328	if strings.HasPrefix(trimmed, "```") \|\| strings.HasPrefix(trimmed, "~~~") {
	329	inBlock = !inBlock
	330	continue
	331	}
	332	} else {
	333	lower := strings.ToLower(trimmed)
	334	if strings.HasPrefix(lower, "#+begin_") {
	335	inBlock = true
	336	continue
	337	}
	338	if strings.HasPrefix(lower, "#+end_") {
	339	inBlock = false
	340	continue
	341	}
	342	}
	343	if inBlock \|\| len(text) > 1000 {
	344	continue
	345	}
	346	re := mdHeading
	347	if lang == "org" {
	348	re = orgHeading
	349	}
	350	if m := re.FindStringSubmatch(text); m != nil {
	351	out = append(out, Symbol{Name: m[1], Key: m[1], Kind: "section", Line: i + 1})
	352	}
	353	}
	354	return out
	355	}

internal/symbols/extract_test.go added +187

		@@ -0,0 +1,187 @@
	1	package symbols
	2
	3	import (
	4	"fmt"
	5	"strings"
	6	"testing"
	7	)
	8
	9	// got renders symbols as "kind name key line", one per entry, key left
	10	// out when it equals the name.
	11	func got(syms []Symbol) []string {
	12	var out []string
	13	for _, s := range syms {
	14	if s.Key != s.Name {
	15	out = append(out, fmt.Sprintf("%s %s %s %d", s.Kind, s.Name, s.Key, s.Line))
	16	continue
	17	}
	18	out = append(out, fmt.Sprintf("%s %s %d", s.Kind, s.Name, s.Line))
	19	}
	20	return out
	21	}
	22
	23	func TestExtract(t *testing.T) {
	24	tests := []struct {
	25	path string
	26	src string
	27	want []string
	28	}{
	29	{"a.go", `package a
	30
	31	// Doc.
	32	func Top() {}
	33
	34	type Server struct{}
	35
	36	type (
	37	List[T any] struct{}
	38	ID int
	39	)
	40
	41	func (s *Server) Handle() {}
	42	func (l List[T]) Push(v T) {}
	43	func (List[T]) Len() int { return 0 }
	44
	45	const Max = 3
	46	const (
	47	A, B = 1, 2
	48	_ = 9
	49	)
	50
	51	var global int
	52	`, []string{
	53	"function Top 4", "type Server 6", "type List 9", "type ID 10",
	54	"method Server.Handle Handle 13", "method List.Push Push 14", "method List.Len Len 15",
	55	"const Max 17", "const A 19", "const B 19", "var global 23",
	56	}},
	57	{"broken.go", "package a\n\nfunc Before() {}\n\nfunc (\n", []string{"function Before 3"}},
	58	{"gen.go", "// Code generated by stringer; DO NOT EDIT.\n\npackage a\n\nfunc X() {}\n", nil},
	59	{"a.swift", `import Foundation
	60
	61	public final class Store {
	62	static func make() -> Store { Store() }
	63	class var shared: Store { make() }
	64	}
	65	struct Point { var x: Int }
	66	enum Mode { case a }
	67	protocol Drawable {}
	68	actor Cache {}
	69	typealias Handler = () -> Void
	70	@MainActor func onMain() {}
	71	`, []string{
	72	"class Store 3", "function make 4", "struct Point 7", "enum Mode 8",
	73	"interface Drawable 9", "class Cache 10", "type Handler 11", "function onMain 12",
	74	}},
	75	{"lib.rs", `pub fn open() {}
	76	pub(crate) async fn fetch() {}
	77	struct Inner;
	78	pub enum Kind { A }
	79	pub trait Store {}
	80	type Result<T> = std::result::Result<T, Error>;
	81	mod tests {}
	82	pub const LIMIT: usize = 3;
	83	static mut COUNT: u32 = 0;
	84	macro_rules! bail {}
	85	impl Store for Inner {}
	86	`, []string{
	87	"function open 1", "function fetch 2", "struct Inner 3", "enum Kind 4",
	88	"interface Store 5", "type Result 6", "module tests 7", "const LIMIT 8",
	89	"const COUNT 9", "macro bail 10",
	90	}},
	91	{"app.py", `import os
	92
	93	def main():
	94	pass
	95
	96	class Client:
	97	async def get(self):
	98	return helper()
	99
	100	async def run():
	101	pass
	102	`, []string{"function main 3", "class Client 6", "method get 7", "function run 10"}},
	103	{"web.ts", `export function render() {}
	104	export default async function load() {}
	105	function* gen() {}
	106	export class View {}
	107	export interface Props {}
	108	export type ID = string;
	109	export type Map<K, V> = Record<K, V>;
	110	export const enum Color { Red }
	111	export const LIMIT = 10;
	112	const handler = () => {};
	113	const inner = 1;
	114	`, []string{
	115	"function render 1", "function load 2", "function gen 3", "class View 4",
	116	"interface Props 5", "type ID 6", "type Map 7", "enum Color 8",
	117	"const LIMIT 9", "const handler 10",
	118	}},
	119	{"api.h", `#ifndef API_H
	120	#define API_H
	121	#define MAX_LEN 64
	122	struct buffer {
	123	int len;
	124	};
	125	typedef struct buffer buffer_t;
	126	typedef enum mode { MODE_A } mode_t;
	127	int buffer_open(const char *path);
	128	static inline void buffer_data(buffer_t b);
	129	typedef struct {
	130	int x;
	131	} point;
	132	#endif
	133	`, []string{
	134	"macro API_H 2", "macro MAX_LEN 3", "struct buffer 4", "type buffer_t 7",
	135	"enum mode 8", "function buffer_open 9", "function buffer_data 10", "type point 13",
	136	}},
	137	{"deploy.sh", `#!/bin/sh
	138	build() {
	139	go build ./...
	140	}
	141	function release {
	142	build
	143	}
	144	ship_it ()
	145	{
	146	release
	147	}
	148	`, []string{"function build 2", "function release 5", "function ship_it 8"}},
	149	{"README.md", "# Title\n\nText\n\n```sh\n# not a heading\n```\n\n## Usage ##\n", []string{"section Title 1", "section Usage 9"}},
	150	{"notes.org", "* Top\n#+begin_src sh\n* not a heading\n#+end_src\n** Child heading\n", []string{"section Top 1", "section Child heading 5"}},
	151	{"data.json", `{"a": 1}`, nil},
	152	}
	153	for _, tc := range tests {
	154	t.Run(tc.path, func(t *testing.T) {
	155	g := got(Extract(tc.path, []byte(tc.src)))
	156	if strings.Join(g, "\n") != strings.Join(tc.want, "\n") {
	157	t.Errorf("got:\n%s\nwant:\n%s", strings.Join(g, "\n"), strings.Join(tc.want, "\n"))
	158	}
	159	})
	160	}
	161	}
	162
	163	func TestSkip(t *testing.T) {
	164	tests := []struct {
	165	path string
	166	size int64
	167	skip bool
	168	}{
	169	{"main.go", 100, false},
	170	{"main.go", MaxFileBytes + 1, true},
	171	{"main.go", MaxFileBytes, false},
	172	{"vendor/x/y.go", 10, true},
	173	{"web/node_modules/a/index.js", 10, true},
	174	{"internal/vendors/y.go", 10, false},
	175	{"zz_gen.go", 10, true},
	176	{"api.pb.go", 10, true},
	177	{"static/app.min.js", 10, true},
	178	{"static/app.js", 10, false},
	179	{"image.png", 10, true},
	180	{"Makefile", 10, true},
	181	}
	182	for _, tc := range tests {
	183	if got := Skip(tc.path, tc.size); got != tc.skip {
	184	t.Errorf("Skip(%q, %d) = %v, want %v", tc.path, tc.size, got, tc.skip)
	185	}
	186	}
	187	}

internal/symbols/worker.go added +247

		@@ -0,0 +1,247 @@
	1	package symbols
	2
	3	import (
	4	"context"
	5	"errors"
	6	"fmt"
	7	"log/slog"
	8	"os"
	9	"time"
	10
	11	"gitbay.org/gitbay/internal/gitutil"
	12	"gitbay.org/gitbay/internal/store"
	13	)
	14
	15	// Bounds on one index run, and how it writes.
	16	const (
	17	// DefaultMaxSymbols is where a repository's index stops growing.
	18	DefaultMaxSymbols = 200_000
	19	// DefaultMaxBytes bounds the names, keys and paths one index holds, so
	20	// a tree of long names cannot fill the database within the count.
	21	DefaultMaxBytes = 32 << 20
	22	// DefaultMaxTime is how long one run may take.
	23	DefaultMaxTime = 2 * time.Minute
	24	// DefaultChunkRows is how many symbols one write transaction carries.
	25	DefaultChunkRows = 5000
	26	// DefaultBackoff is how long a failed tree waits before it is tried
	27	// again.
	28	DefaultBackoff = time.Hour
	29	)
	30
	31	// Worker builds the index for each repository that has asked for one:
	32	// post-receive and the merge path ask after the default branch moves, and
	33	// `admin symbols reindex` asks with force. One repository at a time.
	34	type Worker struct {
	35	St *store.Store
	36	RepoDir func(owner, name string) string
	37	Tick time.Duration
	38	MaxSymbols int
	39	MaxBytes int
	40	MaxTime time.Duration
	41	ChunkRows int
	42	Backoff time.Duration
	43	// chunkHook runs after each chunk is written; tests read the index
	44	// mid-build through it.
	45	chunkHook func(indexID int64)
	46	}
	47
	48	func New(st store.Store, repoDir func(owner, name string) string) Worker {
	49	tick := 5 * time.Second
	50	if v := os.Getenv("GITBAY_SYMBOLS_TICK"); v != "" {
	51	if d, err := time.ParseDuration(v); err == nil {
	52	tick = d
	53	}
	54	}
	55	return NewWith(st, repoDir, tick)
	56	}
	57
	58	// NewWith is New with the tick given, and every bound at its default.
	59	func NewWith(st store.Store, repoDir func(owner, name string) string, tick time.Duration) Worker {
	60	return &Worker{St: st, RepoDir: repoDir, Tick: tick,
	61	MaxSymbols: DefaultMaxSymbols, MaxBytes: DefaultMaxBytes, MaxTime: DefaultMaxTime,
	62	ChunkRows: DefaultChunkRows, Backoff: DefaultBackoff}
	63	}
	64
	65	// Run sweeps until ctx ends.
	66	func (w *Worker) Run(ctx context.Context) {
	67	t := time.NewTicker(w.Tick)
	68	defer t.Stop()
	69	for {
	70	select {
	71	case <-ctx.Done():
	72	return
	73	case <-t.C:
	74	w.Sweep(ctx)
	75	}
	76	}
	77	}
	78
	79	// Sweep handles every due request once. A request is cleared whatever
	80	// the outcome, except that a first failure is kept for one retry after
	81	// Backoff: a failure is retried once, not in a loop, and again only when
	82	// something asks.
	83	func (w *Worker) Sweep(ctx context.Context) {
	84	reqs, err := w.St.SymbolRequests()
	85	if err != nil {
	86	slog.Error("symbols: listing requests", "err", err)
	87	return
	88	}
	89	for _, req := range reqs {
	90	if ctx.Err() != nil {
	91	return
	92	}
	93	failed, err := w.Index(ctx, req.RepoID, req.Force)
	94	if ctx.Err() != nil {
	95	return // shutting down: the request stays for the next start
	96	}
	97	if err != nil {
	98	slog.Warn("symbols: indexing", "repo", req.RepoID, "err", err)
	99	}
	100	if failed && req.Attempts == 0 {
	101	w.St.DeferSymbolRequest(req, int(w.Backoff.Seconds()))
	102	continue
	103	}
	104	w.St.DoneSymbolRequest(req)
	105	}
	106	}
	107
	108	// Index brings one repository's index up to its default branch's head.
	109	// A head whose tree is already indexed is left alone unless force is set,
	110	// and so is one whose tree failed within Backoff. The new index is
	111	// written in chunks while no read can see it, then published in one
	112	// short transaction. A run cut short by a bound publishes what it found
	113	// as partial; one that cannot build records the failure and leaves the
	114	// current index in place, reporting failed. The error is for the log.
	115	func (w *Worker) Index(ctx context.Context, repoID int64, force bool) (failed bool, err error) {
	116	repo, err := w.St.RepoByID(repoID)
	117	if errors.Is(err, store.ErrNotFound) {
	118	return false, nil
	119	} else if err != nil {
	120	return false, err
	121	}
	122	// A run that stopped part way left its building index behind.
	123	if err := w.St.PurgeSymbolIndexes(repo.ID); err != nil {
	124	return false, err
	125	}
	126	dir := w.RepoDir(repo.OwnerName, repo.Name)
	127	commit, err := gitutil.ResolveRef(dir, "refs/heads/"+repo.DefaultBranch)
	128	if err != nil {
	129	return false, nil // no default branch yet: nothing to index
	130	}
	131	tree, err := gitutil.ResolveTree(dir, commit)
	132	if err != nil {
	133	return false, err
	134	}
	135	if !force {
	136	if cur, err := w.St.SymbolIndexFor(repo.ID); err == nil && cur.Tree == tree {
	137	return false, nil
	138	}
	139	if recent, err := w.St.SymbolFailureRecent(repo.ID, tree, int(w.Backoff.Seconds())); err != nil \|\| recent {
	140	return false, err
	141	}
	142	}
	143	x := store.SymbolIndex{RepoID: repo.ID, Commit: commit, Tree: tree, State: "ok"}
	144	syms, files, runErr := w.collect(ctx, dir, tree)
	145	if ctx.Err() != nil {
	146	return false, ctx.Err()
	147	}
	148	x.Files = files
	149	switch {
	150	case errors.Is(runErr, errSymbolCap):
	151	x.State, x.Note = "partial", fmt.Sprintf("stopped at %d symbols", w.MaxSymbols)
	152	case errors.Is(runErr, errByteBudget):
	153	x.State, x.Note = "partial", fmt.Sprintf("stopped at %d bytes of names and paths", w.MaxBytes)
	154	case errors.Is(runErr, context.DeadlineExceeded):
	155	x.State, x.Note = "partial", fmt.Sprintf("stopped after %s", w.MaxTime)
	156	case runErr != nil:
	157	if err := w.St.RecordSymbolFailure(repo.ID, tree, runErr.Error()); err != nil {
	158	return true, err
	159	}
	160	return true, fmt.Errorf("%s failed: %v", repo.Path(), runErr)
	161	}
	162	if x.ID, err = w.St.BeginSymbolIndex(repo.ID, commit, tree); err != nil {
	163	return false, err
	164	}
	165	for len(syms) > 0 {
	166	if ctx.Err() != nil {
	167	return false, ctx.Err() // the next run purges what was written
	168	}
	169	n := min(len(syms), w.ChunkRows)
	170	if err := w.St.AddSymbols(x.ID, syms[:n]); err != nil {
	171	return false, err
	172	}
	173	syms = syms[n:]
	174	if w.chunkHook != nil {
	175	w.chunkHook(x.ID)
	176	}
	177	}
	178	if err := w.St.PublishSymbolIndex(x); err != nil {
	179	return false, err
	180	}
	181	if err := w.St.PurgeSymbolIndexes(repo.ID); err != nil {
	182	return false, err
	183	}
	184	if x.State != "ok" {
	185	return false, fmt.Errorf("%s %s: %s", repo.Path(), x.State, x.Note)
	186	}
	187	return false, nil
	188	}
	189
	190	var (
	191	errSymbolCap = errors.New("symbol cap reached")
	192	errByteBudget = errors.New("byte budget reached")
	193	)
	194
	195	// collect reads every indexable blob in tree and extracts its symbols,
	196	// stopping at the symbol cap, the byte budget or the time bound with what
	197	// it has.
	198	func (w *Worker) collect(ctx context.Context, dir, tree string) ([]store.SymbolRow, int, error) {
	199	ctx, cancel := context.WithTimeout(ctx, w.MaxTime)
	200	defer cancel()
	201	blobs, err := gitutil.ListBlobs(ctx, dir, tree)
	202	if err != nil {
	203	if ctx.Err() != nil {
	204	return nil, 0, ctx.Err()
	205	}
	206	return nil, 0, err
	207	}
	208	var paths, shas []string
	209	for _, b := range blobs {
	210	if b.Mode == "120000" \|\| b.Mode == "160000" \|\| Skip(b.Name, b.Size) {
	211	continue
	212	}
	213	paths = append(paths, b.Name)
	214	shas = append(shas, b.SHA)
	215	}
	216	var out []store.SymbolRow
	217	files, used := 0, 0
	218	var stop error
	219	err = gitutil.CatBlobs(ctx, dir, shas, func(i int, data []byte) bool {
	220	if gitutil.IsBinary(data) {
	221	return true
	222	}
	223	files++
	224	for _, s := range Extract(paths[i], data) {
	225	if len(out) >= w.MaxSymbols {
	226	stop = errSymbolCap
	227	return false
	228	}
	229	// The key is counted too: it is stored beside the name.
	230	n := len(s.Name) + len(s.Key) + len(paths[i])
	231	if used+n > w.MaxBytes {
	232	stop = errByteBudget
	233	return false
	234	}
	235	used += n
	236	out = append(out, store.SymbolRow{Name: s.Name, Key: s.Key, Kind: s.Kind, Path: paths[i], Line: s.Line})
	237	}
	238	return true
	239	})
	240	if stop != nil {
	241	return out, files, stop
	242	}
	243	if err != nil && ctx.Err() == nil {
	244	return nil, files, err
	245	}
	246	return out, files, err
	247	}

internal/symbols/worker_test.go added +390

		@@ -0,0 +1,390 @@
	1	package symbols
	2
	3	import (
	4	"context"
	5	"fmt"
	6	"os"
	7	"os/exec"
	8	"path/filepath"
	9	"strings"
	10	"testing"
	11	"time"
	12
	13	"gitbay.org/gitbay/internal/store"
	14	)
	15
	16	type fixture struct {
	17	t *testing.T
	18	st *store.Store
	19	repo store.Repo
	20	src string
	21	bare string
	22	w *Worker
	23	}
	24
	25	func newFixture(t testing.T) fixture {
	26	t.Helper()
	27	st, err := store.Open(":memory:")
	28	if err != nil {
	29	t.Fatal(err)
	30	}
	31	t.Cleanup(func() { st.Close() })
	32	if err := st.MigrateUp(); err != nil {
	33	t.Fatal(err)
	34	}
	35	uid, err := st.CreateUser("alice", false)
	36	if err != nil {
	37	t.Fatal(err)
	38	}
	39	id, err := st.CreateRepo("user", uid, "app", "public")
	40	if err != nil {
	41	t.Fatal(err)
	42	}
	43	repo, err := st.RepoByID(id)
	44	if err != nil {
	45	t.Fatal(err)
	46	}
	47	root := t.TempDir()
	48	f := &fixture{t: t, st: st, repo: repo, src: filepath.Join(root, "src"), bare: filepath.Join(root, "app.git")}
	49	f.w = NewWith(st, func(owner, name string) string { return f.bare }, 0)
	50	f.git(root, "init", "-q", "-b", repo.DefaultBranch, "src")
	51	f.git(root, "init", "-q", "--bare", f.bare)
	52	return f
	53	}
	54
	55	func (f *fixture) git(dir string, args ...string) string {
	56	f.t.Helper()
	57	cmd := exec.Command("git", args...)
	58	cmd.Dir = dir
	59	cmd.Env = append(os.Environ(),
	60	"GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null",
	61	"GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.test",
	62	"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.test")
	63	out, err := cmd.CombinedOutput()
	64	if err != nil {
	65	f.t.Fatalf("git %v: %v\n%s", args, err, out)
	66	}
	67	return strings.TrimSpace(string(out))
	68	}
	69
	70	// commit writes files (an empty content removes one) and pushes the
	71	// default branch to the bare repository.
	72	func (f *fixture) commit(files map[string]string) string {
	73	f.t.Helper()
	74	for name, content := range files {
	75	p := filepath.Join(f.src, name)
	76	if content == "" {
	77	os.Remove(p)
	78	continue
	79	}
	80	os.MkdirAll(filepath.Dir(p), 0o755)
	81	if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
	82	f.t.Fatal(err)
	83	}
	84	}
	85	f.git(f.src, "add", "-A")
	86	f.git(f.src, "commit", "-q", "--allow-empty", "-m", "c")
	87	f.git(f.src, "push", "-q", "--force", f.bare, "HEAD:refs/heads/"+f.repo.DefaultBranch)
	88	return f.git(f.src, "rev-parse", "HEAD")
	89	}
	90
	91	func (f *fixture) sweep(force bool) store.SymbolIndex {
	92	f.t.Helper()
	93	if err := f.st.RequestSymbolIndex(f.repo.ID, force); err != nil {
	94	f.t.Fatal(err)
	95	}
	96	f.w.Sweep(context.Background())
	97	if reqs, _ := f.st.SymbolRequests(); len(reqs) != 0 {
	98	f.t.Fatalf("request left after a sweep: %+v", reqs)
	99	}
	100	x, err := f.st.SymbolIndexFor(f.repo.ID)
	101	if err != nil {
	102	f.t.Fatal(err)
	103	}
	104	return x
	105	}
	106
	107	func (f *fixture) names(x store.SymbolIndex) []string {
	108	f.t.Helper()
	109	rows, err := f.st.SearchSymbols(x.ID, "", "", 0, 0)
	110	if err != nil {
	111	f.t.Fatal(err)
	112	}
	113	var out []string
	114	for _, r := range rows {
	115	out = append(out, r.Path+":"+r.Name)
	116	}
	117	return out
	118	}
	119
	120	func TestWorkerIndexesDefaultBranch(t *testing.T) {
	121	f := newFixture(t)
	122	head := f.commit(map[string]string{
	123	"main.go": "package main\n\nfunc Hello() {}\n",
	124	"vendor/dep/dep.go": "package dep\n\nfunc Vendored() {}\n",
	125	"gen/zz_gen.go": "package gen\n\nfunc Generated() {}\n",
	126	"big/big.go": "package big\n\nfunc Big() {}\n" + strings.Repeat("//\n", MaxFileBytes/3),
	127	"docs/guide.md": "# Guide\n",
	128	"assets/logo.png": "\x89PNG\r\n",
	129	"node_modules/x.js": "function hidden() {}\n",
	130	"web/app.js": "function shown() {}\n",
	131	})
	132	x := f.sweep(false)
	133	if x.State != "ok" \|\| x.Commit != head {
	134	t.Fatalf("index = %+v, want ok at %s", x, head)
	135	}
	136	got := strings.Join(f.names(x), " ")
	137	for _, want := range []string{"main.go:Hello", "docs/guide.md:Guide", "web/app.js:shown"} {
	138	if !strings.Contains(got, want) {
	139	t.Errorf("missing %s in %s", want, got)
	140	}
	141	}
	142	for _, skip := range []string{"Vendored", "Generated", "Big", "hidden"} {
	143	if strings.Contains(got, skip) {
	144	t.Errorf("%s indexed: %s", skip, got)
	145	}
	146	}
	147	if x.Files != 3 \|\| x.Symbols != 3 {
	148	t.Errorf("files %d symbols %d, want 3 and 3", x.Files, x.Symbols)
	149	}
	150	}
	151
	152	// A new commit whose tree is the indexed one is not indexed again; a
	153	// changed tree replaces the index, and force rebuilds an unchanged one.
	154	func TestWorkerKeysOnTree(t *testing.T) {
	155	f := newFixture(t)
	156	f.commit(map[string]string{"a.go": "package a\n\nfunc One() {}\n"})
	157	first := f.sweep(false)
	158
	159	f.commit(nil) // same tree, new commit
	160	same := f.sweep(false)
	161	if same.ID != first.ID \|\| same.Commit != first.Commit {
	162	t.Fatalf("unchanged tree was reindexed: %+v then %+v", first, same)
	163	}
	164
	165	f.commit(map[string]string{"a.go": "package a\n\nfunc Two() {}\n"})
	166	changed := f.sweep(false)
	167	if changed.ID == first.ID \|\| changed.Tree == first.Tree {
	168	t.Fatalf("changed tree kept the old index: %+v", changed)
	169	}
	170	if got := f.names(changed); len(got) != 1 \|\| got[0] != "a.go:Two" {
	171	t.Fatalf("symbols = %v, want only the new tree's", got)
	172	}
	173	var n int
	174	f.st.DB.QueryRow("SELECT COUNT(*) FROM symbol_indexes WHERE repo_id = ?", f.repo.ID).Scan(&n)
	175	if n != 1 {
	176	t.Fatalf("%d indexes for one repository, want 1", n)
	177	}
	178
	179	forced := f.sweep(true)
	180	if forced.ID == changed.ID \|\| forced.Tree != changed.Tree {
	181	t.Fatalf("force did not rebuild: %+v then %+v", changed, forced)
	182	}
	183	}
	184
	185	func TestWorkerSymbolCap(t *testing.T) {
	186	f := newFixture(t)
	187	f.commit(map[string]string{"a.go": "package a\n\nfunc A() {}\nfunc B() {}\nfunc C() {}\n"})
	188	f.w.MaxSymbols = 2
	189	x := f.sweep(false)
	190	if x.State != "partial" \|\| x.Symbols != 2 \|\| !strings.Contains(x.Note, "2 symbols") {
	191	t.Fatalf("index = %+v, want partial with 2 symbols", x)
	192	}
	193	}
	194
	195	func TestWorkerTimeBound(t *testing.T) {
	196	f := newFixture(t)
	197	f.commit(map[string]string{"a.go": "package a\n\nfunc A() {}\n"})
	198	f.w.MaxTime = time.Nanosecond
	199	x := f.sweep(false)
	200	if x.State != "partial" \|\| !strings.Contains(x.Note, "stopped after") {
	201	t.Fatalf("index = %+v, want partial on the time bound", x)
	202	}
	203	}
	204
	205	// breakBlob deletes the loose object of path at the default branch, so
	206	// the tree cannot be read.
	207	func (f *fixture) breakBlob(path string) {
	208	f.t.Helper()
	209	blob := f.git(f.bare, "rev-parse", "refs/heads/"+f.repo.DefaultBranch+":"+path)
	210	if err := os.Remove(filepath.Join(f.bare, "objects", blob[:2], blob[2:])); err != nil {
	211	f.t.Fatal(err)
	212	}
	213	}
	214
	215	// A tree that cannot be read is recorded as a failure beside the current
	216	// index, which stays current. The request is kept for one retry after the
	217	// backoff rather than tried again at once; a new request for the same
	218	// tree inside the backoff is not retried either.
	219	func TestWorkerFailureKeepsCurrentIndex(t *testing.T) {
	220	f := newFixture(t)
	221	f.commit(map[string]string{"a.go": "package a\n\nfunc A() {}\n"})
	222	good := f.sweep(false)
	223
	224	f.commit(map[string]string{"b.go": "package a\n\nfunc B() {}\n"})
	225	f.breakBlob("b.go")
	226	f.st.RequestSymbolIndex(f.repo.ID, false)
	227	f.w.Sweep(context.Background())
	228	cur, err := f.st.SymbolIndexFor(f.repo.ID)
	229	if err != nil \|\| cur.ID != good.ID {
	230	t.Fatalf("current index after a failure = %+v, %v; want %d", cur, err, good.ID)
	231	}
	232	fail, err := f.st.SymbolFailureFor(f.repo.ID)
	233	if err != nil \|\| !strings.Contains(fail.Note, "b.go") {
	234	t.Fatalf("failure = %+v, %v", fail, err)
	235	}
	236	if reqs, _ := f.st.SymbolRequests(); len(reqs) != 0 {
	237	t.Fatalf("a failed request is due again at once: %+v", reqs)
	238	}
	239	var attempts int
	240	f.st.DB.QueryRow("SELECT attempts FROM symbol_requests WHERE repo_id = ?", f.repo.ID).Scan(&attempts)
	241	if attempts != 1 {
	242	t.Fatalf("attempts = %d, want the request kept for one retry", attempts)
	243	}
	244
	245	// Asked again inside the backoff: the failed tree is left alone.
	246	f.st.RequestSymbolIndex(f.repo.ID, false)
	247	failed, err := f.w.Index(context.Background(), f.repo.ID, false)
	248	if failed \|\| err != nil {
	249	t.Fatalf("a recently failed tree was retried: %v, %v", failed, err)
	250	}
	251
	252	// After the backoff it is tried again, and still fails.
	253	f.w.Backoff = 0
	254	if failed, _ := f.w.Index(context.Background(), f.repo.ID, false); !failed {
	255	t.Fatal("a failed tree was not retried after the backoff")
	256	}
	257
	258	// A second failure of a retry is the end of it.
	259	f.st.DB.Exec("DELETE FROM symbol_requests")
	260	f.st.DB.Exec("INSERT INTO symbol_requests (repo_id, attempts) VALUES (?, 1)", f.repo.ID)
	261	f.w.Sweep(context.Background())
	262	var n int
	263	f.st.DB.QueryRow("SELECT COUNT(*) FROM symbol_requests").Scan(&n)
	264	if n != 0 {
	265	t.Fatal("a failed retry was kept for another")
	266	}
	267
	268	// A push that changes the tree builds, and clears the failure.
	269	f.git(f.src, "rm", "-q", "b.go")
	270	f.commit(map[string]string{"c.go": "package a\n\nfunc C() {}\n"})
	271	next := f.sweep(false)
	272	if next.ID == good.ID {
	273	t.Fatal("a new tree after a failure was not indexed")
	274	}
	275	if _, err := f.st.SymbolFailureFor(f.repo.ID); err != store.ErrNotFound {
	276	t.Fatalf("failure kept after a good index: %v", err)
	277	}
	278	}
	279
	280	// A new index is written in chunks no read sees: throughout the build the
	281	// old index is current, and the flip replaces it whole.
	282	func TestWorkerBuildIsInvisibleUntilPublished(t *testing.T) {
	283	f := newFixture(t)
	284	f.commit(map[string]string{"a.go": "package a\n\nfunc Old() {}\n"})
	285	old := f.sweep(false)
	286
	287	f.commit(map[string]string{"a.go": "package a\n\nfunc New1() {}\nfunc New2() {}\nfunc New3() {}\n"})
	288	f.w.ChunkRows = 1
	289	chunks := 0
	290	f.w.chunkHook = func(building int64) {
	291	chunks++
	292	cur, err := f.st.SymbolIndexFor(f.repo.ID)
	293	if err != nil \|\| cur.ID != old.ID {
	294	t.Errorf("mid-build current index = %+v, %v; want the old one", cur, err)
	295	}
	296	if got := f.names(cur); len(got) != 1 \|\| got[0] != "a.go:Old" {
	297	t.Errorf("mid-build reads %v", got)
	298	}
	299	if b, _ := f.st.SymbolIndexByID(building); b.State != "building" {
	300	t.Errorf("index being built is %q", b.State)
	301	}
	302	}
	303	next := f.sweep(false)
	304	if chunks != 3 {
	305	t.Fatalf("%d chunks, want one per row", chunks)
	306	}
	307	if got := f.names(next); len(got) != 3 {
	308	t.Fatalf("after the flip: %v", got)
	309	}
	310	var indexes, rows int
	311	f.st.DB.QueryRow("SELECT COUNT(*) FROM symbol_indexes").Scan(&indexes)
	312	f.st.DB.QueryRow("SELECT COUNT(*) FROM symbols").Scan(&rows)
	313	if indexes != 1 \|\| rows != 3 {
	314	t.Fatalf("%d indexes and %d rows left, want the new index alone", indexes, rows)
	315	}
	316	}
	317
	318	// A run that stopped mid-build leaves the old index current, and the next
	319	// run deletes what it wrote.
	320	func TestWorkerCleansUpInterruptedBuild(t *testing.T) {
	321	f := newFixture(t)
	322	f.commit(map[string]string{"a.go": "package a\n\nfunc Old() {}\n"})
	323	old := f.sweep(false)
	324
	325	// What a crash between chunks leaves.
	326	orphan, err := f.st.BeginSymbolIndex(f.repo.ID, "c", "t")
	327	if err != nil {
	328	t.Fatal(err)
	329	}
	330	if err := f.st.AddSymbols(orphan, []store.SymbolRow{{Name: "Half", Key: "Half", Kind: "function", Path: "x.go", Line: 1}}); err != nil {
	331	t.Fatal(err)
	332	}
	333	if cur, _ := f.st.SymbolIndexFor(f.repo.ID); cur.ID != old.ID {
	334	t.Fatalf("an unfinished build became current: %+v", cur)
	335	}
	336
	337	f.commit(nil) // same tree: nothing to build, but the leftovers go
	338	f.sweep(false)
	339	if _, err := f.st.SymbolIndexByID(orphan); err != store.ErrNotFound {
	340	t.Fatalf("interrupted build still there: %v", err)
	341	}
	342	var rows int
	343	f.st.DB.QueryRow("SELECT COUNT(*) FROM symbols WHERE index_id = ?", orphan).Scan(&rows)
	344	if rows != 0 {
	345	t.Fatalf("%d rows of the interrupted build left", rows)
	346	}
	347	}
	348
	349	// A hostile tree: names past the length cap are dropped, and long
	350	// headings stop at the byte budget with a partial index saying so.
	351	func TestWorkerHostileNames(t *testing.T) {
	352	f := newFixture(t)
	353	files := map[string]string{}
	354	huge := strings.Repeat("x", MaxNameBytes+1)
	355	for i := 0; i < 20; i++ {
	356	files[fmt.Sprintf("huge%02d.go", i)] = "package a\n\nfunc " + huge + "() {}\nvar " + huge + " int\n"
	357	}
	358	var md strings.Builder
	359	for i := 0; i < 500; i++ {
	360	fmt.Fprintf(&md, "# %03d %s\n", i, strings.Repeat("h", 200))
	361	}
	362	files["notes.md"] = md.String()
	363	f.commit(files)
	364	f.w.MaxBytes = 20_000
	365	x := f.sweep(false)
	366	if x.State != "partial" \|\| !strings.Contains(x.Note, "20000 bytes") {
	367	t.Fatalf("index = %+v, want partial at the byte budget", x)
	368	}
	369	var longest, total int
	370	f.st.DB.QueryRow("SELECT COALESCE(MAX(length(name)), 0), COALESCE(SUM(length(name) + length(key) + length(path)), 0) FROM symbols").
	371	Scan(&longest, &total)
	372	if longest > MaxNameBytes {
	373	t.Errorf("a %d-byte name was kept", longest)
	374	}
	375	if total > 20_000 \|\| x.Symbols == 0 {
	376	t.Errorf("%d symbols holding %d bytes, want some within the budget", x.Symbols, total)
	377	}
	378	}
	379
	380	func TestWorkerIgnoresEmptyRepository(t *testing.T) {
	381	f := newFixture(t)
	382	f.st.RequestSymbolIndex(f.repo.ID, false)
	383	f.w.Sweep(context.Background())
	384	if _, err := f.st.SymbolIndexFor(f.repo.ID); err != store.ErrNotFound {
	385	t.Fatalf("an empty repository has an index: %v", err)
	386	}
	387	if reqs, _ := f.st.SymbolRequests(); len(reqs) != 0 {
	388	t.Fatalf("request left: %+v", reqs)
	389	}
	390	}

internal/web/static/style.css +8

		@@ -1101,6 +1101,14 @@ article.release { padding: var(--sp-4); margin: var(--sp-4) 0; }
1101	1101	ul.matchlist { list-style: none; margin: var(--sp-3) 0; padding: 0; }
1102	1102	ul.matchlist li { margin-bottom: var(--sp-3); }
1103	1103	.matchpath { font-family: var(--mono); font-size: var(--fs-1); }
	1104	/* a name in highlighted source that links to its definition keeps its
	1105	token colour; the dotted underline is what marks it as a link */
	1106	.chroma a.sym { color: inherit; text-decoration: underline dotted; text-underline-offset: 2px; }
	1107	.chroma a.sym:hover { text-decoration-style: solid; }
	1108	details.symlist { margin: 0 0 var(--sp-3); font-size: var(--fs-1); }
	1109	details.symlist summary { cursor: pointer; color: var(--muted); }
	1110	details.symlist ul { list-style: none; margin: var(--sp-2) 0 0; padding: 0; columns: 16rem; }
	1111	details.symlist li { break-inside: avoid; padding: 2px 0; }
1104	1112	pre.matchline {
1105	1113	background: var(--inset);
1106	1114	border: 1px solid var(--line);

internal/web/templates/blob.html +4

		@@ -12,6 +12,10 @@
12	12	<span class="actions">{{if .Renderable}}{{if .Rendered}}<strong>rendered</strong> · <a href="?view=source">source</a>{{else}}<a href="?">rendered</a> · <strong>source</strong>{{end}} · {{end}}<a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/log/{{.Ref}}?path={{.Path}}">history</a> · {{if not .Binary}}<a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/blame/{{.Ref}}/{{.Path}}">blame</a> · {{end}}<a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/raw/{{.Ref}}/{{.Path}}">raw</a>{{if .Viewer}} · <a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/edit/{{.Ref}}/{{.Path}}">edit</a>{{end}}</span>
13	13	</div>
14	14	<p class="filefacts">{{if .Lines}}{{.Lines}} lines · {{end}}{{.Size}} bytes{{if .Exec}} · executable{{end}}{{if .Symlink}} · symlink{{end}}</p>
	15	{{if .Symbols}}<details class="symlist"><summary>{{len .Symbols}} symbol{{if ne (len .Symbols) 1}}s{{end}} in this file</summary>
	16	<ul>{{range .Symbols}}<li><a href="{{if $.Rendered}}?view=source{{end}}#L{{.Line}}"><code>{{.Name}}</code></a> <span class="meta">{{.Kind}}, line {{.Line}}</span></li>
	17	{{end}}</ul>
	18	</details>{{end}}
15	19	{{if .Image}}<div class="blobimage"><a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/raw/{{.Ref}}/{{.Path}}"><img src="/{{.Repo.OwnerName}}/{{.Repo.Name}}/raw/{{.Ref}}/{{.Path}}" alt="{{.Base}}"></a><p class="meta">{{.Size}} bytes</p></div>
16	20	{{else if .Binary}}<p class="empty-note">binary file, {{.Size}} bytes — <a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/raw/{{.Ref}}/{{.Path}}">download</a></p>
17	21	{{else if .Rendered}}<section class="readme"><div class="rendered">{{.RenderedHTML}}</div></section>

internal/web/templates/symbols.html added +32

		@@ -0,0 +1,32 @@
	1	{{define "width"}}wide{{end}}
	2	{{define "title"}}symbols · {{.Repo.OwnerName}}/{{.Repo.Name}}{{end}}
	3	{{define "content"}}
	4	<h1>Symbols</h1>
	5	<form method="get" action="/{{.Repo.OwnerName}}/{{.Repo.Name}}/symbols" class="searchform">
	6	<input type="search" name="q" aria-label="Find a definition" value="{{.Query}}" placeholder="a name or its start, on {{.Repo.DefaultBranch}}">
	7	<select name="kind" aria-label="Kind">
	8	<option value="">any kind</option>
	9	{{range .Kinds}}<option value="{{.}}"{{if eq . $.Kind}} selected{{end}}>{{.}}</option>
	10	{{end}}
	11	</select>
	12	<button type="submit" class="btn">Find</button>
	13	</form>
	14	{{if .Problem}}<p class="empty-note">{{.Problem}}</p>
	15	{{else if .Query}}
	16	{{if .Note}}<p class="meta">{{.Note}}</p>{{end}}
	17	{{if .Rows}}
	18	<div class="tablewrap"><table class="refs">
	19	<thead><tr><th scope="col">Name</th><th scope="col">Kind</th><th scope="col">Defined at</th></tr></thead>
	20	<tbody>
	21	{{range .Rows}}<tr>
	22	<td><a href="/{{$.Repo.OwnerName}}/{{$.Repo.Name}}/blob/{{$.Commit}}/{{.Path}}#L{{.Line}}"><code>{{.Name}}</code></a></td>
	23	<td>{{.Kind}}</td>
	24	<td class="sha">{{.Path}}:{{.Line}}</td>
	25	</tr>
	26	{{end}}
	27	</tbody>
	28	</table></div>
	29	{{if .Next}}<p><a href="{{.Next}}">next page</a></p>{{end}}
	30	{{else}}<p class="empty-note">no definitions matching “{{.Query}}”</p>{{end}}
	31	{{end}}
	32	{{end}}

internal/web/web_test.go +1 −1

		@@ -89,7 +89,7 @@ func TestWhenNamesTheZone(t *testing.T) {
89	89	// none. The merge request page picks wide for its diff view, so it gets
90	90	// a per-view define instead of a fixed one.
91	91	func TestMainWidthClass(t *testing.T) {
92		wide := map[string]bool{"tree.html": true, "blob.html": true, "blame.html": true, "log.html": true, "commit.html": true, "compare.html": true, "builds.html": true, "build.html": true, "search.html": true, "globalsearch.html": true, "edit.html": true, "dashboard.html": true, "issues.html": true, "mrs.html": true, "mrrangediff.html": true, "explore.html": true, "notifications.html": true, "settings.html": true, "account.html": true, "admin.html": true, "adminusers.html": true, "queries.html": true}
	92	wide := map[string]bool{"tree.html": true, "blob.html": true, "blame.html": true, "log.html": true, "commit.html": true, "compare.html": true, "builds.html": true, "build.html": true, "search.html": true, "symbols.html": true, "globalsearch.html": true, "edit.html": true, "dashboard.html": true, "issues.html": true, "mrs.html": true, "mrrangediff.html": true, "explore.html": true, "notifications.html": true, "settings.html": true, "account.html": true, "admin.html": true, "adminusers.html": true, "queries.html": true}
93	93	bounded := map[string]bool{"landing.html": true, "fork.html": true, "login.html": true, "logout.html": true, "register.html": true, "registered.html": true, "new.html": true, "issuenew.html": true, "mrnew.html": true, "snippetnew.html": true, "privacy.html": true, "404.html": true}
94	94	perView := map[string]string{"mr.html": `{{define "width"}}{{if eq .View "diff"}}wide{{else}}reading{{end}}{{end}}`}
95	95	for _, name := range Pages() {

Code navigation: symbol index and jump to definition !533