feat(catalog): add scheduled Forgejo discovery #51
No reviewers
Labels
No labels
automerge
bug
dependencies
duplicate
help wanted
invalid
lang
rust
migrations
opencode
question
security
type
documentation
type
enhancement
type
maintenance
type
refactor
vulnerability
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
Personal/backstage!51
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/forgejo-discovery-24"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
catalog-info.yamlas the sole source of its catalog metadata; repositories without one are omitted.docs/forgejo-discovery.mdanddocs/decisions/001-forgejo-catalog-source-of-truth.md.Refs #24
Validation
bun install --frozen-lockfilebun run tscbun run lint:allbun run prettier:checkbun run build:backendCI=1 bun run test:ci --runInBand(10 tests)E2E was not run locally because a pre-existing backend process occupies the configured localhost port; it was left untouched. Woodpecker PR pipeline #137 is running on the current head.
Objective
feat(catalog): add scheduled Forgejo discovery) inPersonal/backstagefor bugs, security, performance, code quality, and best practices.summary,verdict,comments[]), no markdown fences, no extra text; language: English.Important Details
approve/changes/comment; severity choices:bug/security/performance/style; comments needfile,line,side(new/old, omit for new files),severity,message; emptycomments: []if no new issues./woodpecker/src/code.martials.no/Personal/backstage; diff at/woodpecker/src/code.martials.no/Personal/backstage/.tmp5UbAab/pr.diff(50KB cap per read; read withoffset/limit; earlier reads: lines 1–542, 543–1743 region, 1743+ region — diff lines ~1743–2443+ covered, but lines ~1200–1743 gap exists and may not have been read).bashtool broken (NotFound: ChildProcess.spawnfor any command incl.echo ok);glob/rg(ripgrep) failing;node_modulesabsent (cannot inspect installed packages); directory reads viareadtool work; webfetch toraw.githubusercontent.comandapi.github.comworks (404s = wrong paths,search/codeAPI = 401).@backstage/configConfigReader.getOptionalBooleancoerces strings"true"/"false"/"y"/"yes"/"1"/"on"→enabled: ${FORGEJO_DISCOVERY_ENABLED:-true}works (NOT a bug).--config) values win; objects deep-merge, arrays from later config fully REPLACE earlier arrays → E2E overlay'scatalog.locationsreplaces base's,enabled: falsemerges correctly.packages/app/e2e-tests/fixtures/catalog/containsforgejo-repos.yaml,org.yaml,services.yaml,systems.yaml.GiteaUrlReaderexists in@backstage/backend-defaultsentrypoints/urlReader/lib/GiteaUrlReader.ts, included inUrlReaders.default; predicate =url.host === integration.config.host; converts.../src/branch/<ref>/<path>→/api/v1/repos/<owner>/<name>/contents/<path>?ref=<ref>with integration credentials → Location target format is correct.readGiteaConfigtrims trailing slashes frombaseUrl;parseGiteaUrlusesurl.replace(baseUrl, '')— breaks ifhtml_urlhost ≠ configured integration host (FORGEJO_URL vs server ROOT_URL mismatch); edge case, likely not worth flagging.catalog.rules=allow: [Component, System, API, Resource, Location]inapp-config.yaml(per-location rules on org.yaml for User/Group).SchedulerServiceTaskConfigsupports optionalinitialDelay; whether task runs immediately at startup (withoutinitialDelay) is still unverified — README/AGENTS claim "at startup and every 15 minutes".listRepositoriesbreaks onbatch.length < pageSize; silently truncates if server capslimitbelow 50 (old sync script had same pattern; Gitea honors limit ≤ 50) — possible low-severity robustness note; alternative termination: break on empty batch.html_urlwithout validating host matches integration host (SSRF/hardening suggestion; mirrors built-in providers).refwhendefault_branchempty, Location target hardcodes'main'fallback (minor).hasCatalogDescriptortreats any 200 as present (no content check) — minor.readDuration:frequencyCronsilently takes precedence overminutes/secondsif both present — minor.webServer.commandpaths (--config ../../app-config.yaml,--config e2e-tests/app-config.yaml,--config ../app/e2e-tests/app-config.yaml) assumebackstage-cli repo startsets child cwd to package dir — consistent with Backstage docs (--config ../../app-config.yamlpattern) but not 100% verified; CI E2E (Woodpecker pipeline #137) will exercise it since CI setswebServer: []only whenCIenv set — actually CI path setswebServer: [], so verify how CI starts servers before flagging.readfor files/dirs, webfetch/websearch for upstream source.Work State
Completed
ForgejoEntityProvider.ts,module.ts, tests,app-config.yamlprovider block, removal ofexamples/forgejo-repos.yaml,scripts/sync-forgejo-catalog.mjs,sync:forgejoscript,@backstage/plugin-catalog-backend-module-giteadep removal, octokit lockfile churn, new docsdocs/forgejo-discovery.md,docs/decisions/001-forgejo-catalog-source-of-truth.md.packages/backend/package.jsondependency additions (@backstage/backend-plugin-api,@backstage/integration,@backstage/plugin-catalog-node,@backstage/types) via bun.lock diff.readGiteaConfigtrailing-slash trim,catalog.rulesinteraction with provider-emitted entities.playwright.config.ts,package.json,.woodpecker/application-checks.yaml,app-config.yaml(working tree = PR head).Active
initialDelay) — last fetch:packages/backend-defaults/src/entrypoints/scheduler/lib/TaskWorker.ts(retrieved but not yet analyzed for first-run timing).Blocked
docs/forgejo-discovery.mdtail,examples/forgejo-repos.yamldeletion,package.json,packages/app/e2e-tests/app-config.yaml,packages/backend/package.json, provider TS source start) not yet read — provider implementation details (listRepositories,hasCatalogDescriptor,mapConcurrent,locationSpecToLocationEntityusage,readProviderConfigs) need direct line numbers for comments.TaskWorkerfirst-run behavior unresolved; GitHub code search API unavailable (401).Next Move
offset~1200,limit~550) of/woodpecker/src/code.martials.no/Personal/backstage/.tmp5UbAab/pr.diffto capture exact provider source line numbers (needed forcomments[].lineonForgejoEntityProvider.ts).TaskWorker.tsexecuteOnce/startlogic or search docs), then decide final findings (pagination truncation robustness,html_urlhost validation,default_branch/'main'mismatch, cron-vs-duration precedence) and emit the single raw JSON object withverdict(likelyapproveorcommentgiven balanced style unless a real bug surfaces).Relevant Files
/woodpecker/src/code.martials.no/Personal/backstage/.tmp5UbAab/pr.diff: the PR diff under review; read in chunks withoffset/limit; gap at ~1200–1743.packages/backend/src/catalog/forgejo/ForgejoEntityProvider.ts: core new provider; primary target for correctness findings; need exact line numbers from diff.packages/backend/src/catalog/forgejo/ForgejoEntityProvider.test.ts: test coverage (timeout, reconciliation, disabled provider).packages/backend/src/catalog/forgejo/module.ts,packages/backend/src/index.ts: plugin registration viacatalogProcessingExtensionPoint.app-config.yaml:integrations.gitea.baseUrl: ${FORGEJO_URL:-https://code.martials.no},catalog.providers.forgejo.mainblock,catalog.rules: [Component, System, API, Resource, Location].packages/app/e2e-tests/app-config.yaml+playwright.config.ts: E2E overlay (enabled: false, replacementcatalog.locations) and webServer commands with relative--configpaths.docs/forgejo-discovery.md,docs/decisions/001-forgejo-catalog-source-of-truth.md: new docs referenced by README/AGENTS.bun.lock,package.json,.woodpecker/application-checks.yaml,.prettierignore: supporting changes (dep swap, lockfile octokit churn, CI path-exclude removal).Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed.
Now reading the unread middle portion of the PR diff to capture exact provider line numbers.
Now verifying the scheduler's first-run behavior (docs claim a refresh at startup).
Re-checking the new docs for accuracy against the verified implementation.
Verifying one critical detail: how
getGiteaRequestOptionstreats an emptypassword(app-config uses${FORGEJO_TOKEN:-}).Final verification of the exact line numbers in the new provider file before writing the review.
{"summary":"Replaces the static Forgejo catalog sync script with a scheduled backend entity provider: ForgejoEntityProvider lists configured user/organization repositories through the Gitea integration (paginated, bounded-concurrency descriptor checks), emits a full-snapshot Location set only for repositories containing catalog-info.yaml, and keeps the last successful catalog whenever a non-404 error, malformed response, or timeout occurs. The change also swaps the built-in gitea module for local wiring, adds thorough unit tests plus an E2E overlay that disables live discovery and loads fixture catalogs, and documents the design in docs/forgejo-discovery.md and an ADR. I verified the env-string boolean coercion of FORGEJO_DISCOVERY_ENABLED, multi-config merge semantics (overlay arrays replace, booleans merge), scheduler startup behavior (duration cadence without initialDelay runs immediately, so the docs' "refreshes at startup" claim holds), Playwright --config path resolution against upstream cli-module-build sources (config paths resolve relative to each package dir), GiteaUrlReader compatibility of the src/branch/ target format, empty-password behavior in getGiteaRequestOptions (public requests stay unauthenticated as documented), and the accuracy of the docs' claims about the built-in Gitea provider. Remaining findings are non-blocking robustness and hardening suggestions.","verdict":"approve","comments":[{"file":"packages/backend/src/catalog/forgejo/ForgejoEntityProvider.ts","line":341,"severity":"bug","message":"Pagination ends whenever a page returns fewer than pageSize items, but a Forgejo instance that caps limit below 50 (e.g. a lowered API.MAX_RESULT_LIMIT) yields a short first page, silently dropping every subsequent repository - and because refresh then submits a full mutation, those repositories' locations are removed from the catalog. Conversely, a server that ignores the page parameter never terminates this loop, since the scheduler's task AbortSignal is not propagated to fetch. Break on an empty batch with a generous page cap instead; that costs one extra request per owner per refresh but handles both cases."},{"file":"packages/backend/src/catalog/forgejo/ForgejoEntityProvider.ts","line":255,"severity":"security","message":"The Location target is built from the API-supplied html_url without checking that its host matches the configured integration. If Forgejo's ROOT_URL differs from FORGEJO_URL, the target host fails GiteaUrlReader's predicate (url.host === integration host), so it falls through to the generic unauthenticated fetcher; the same applies to an off-host html_url returned by a non-conforming server. Build the target from integration.config.baseUrl plus owner/name (as apiBaseUrl already does) and reject repositories whose html_url host does not match the provider host."},{"file":"packages/backend/src/catalog/forgejo/ForgejoEntityProvider.ts","line":149,"severity":"performance","message":"When one descriptor check rejects, Promise.all rejects and refresh fails, but the other workers keep draining the remaining queue and issuing requests whose results are discarded. On a large owner set a failed refresh can therefore continue hitting requestTimeoutSeconds long after the error was recorded; a shared stop flag (or the task AbortSignal) checked in the while loop would fail fast."},{"file":"packages/backend/src/catalog/forgejo/ForgejoEntityProvider.ts","line":74,"severity":"style","message":"frequencyCron silently takes precedence: a config like frequency: { cron: '...', minutes: 5 } drops the duration keys with no warning, and readProviderConfigs accepts either form. Throw when both cron and duration keys are present so the misconfiguration surfaces at startup instead of being silently ignored."}]}
*🤖 Review by opencode (opencode/mimo-v2.6-flash-free) — verdict: comment