Metadata Fetching¶

When a bookmark is saved, tldrq fetches its title, description, thumbnail and, for repositories, stats. Everything lives in src/lib/utils/metadata.ts, with image URL handling in src/lib/utils/ui.ts.

Where it runs¶

fetchOGMetadata(url) runs on the server, once per bookmark:

• from POST /api/bookmarks, before the insert;
• from GET /api/metadata?url=…, which nothing in the UI calls but which is handy for trying the fetcher on a URL (see the API Reference).

Metadata is never fetched again for a saved bookmark.

The result¶

export interface OGMetadata {
  title: string | null;
  description: string | null;
  image_url: string | null;
  site_name: string | null;
  repo_stars?: number | null;
  repo_forks?: number | null;
  repo_language?: string | null;
}

The fields are stored on the bookmark row as they are (see Database).

Dispatch¶

fetchOGMetadata tries the platform detectors in order and uses the first that matches:

flowchart TD
    U[URL] --> Y{YouTube video ID?}
    Y -- yes --> YH[YouTube oEmbed]
    Y -- no --> X{X or Twitter status?}
    X -- yes --> XH[X oEmbed and page image]
    X -- no --> V{Vimeo?}
    V -- yes --> VH[Vimeo oEmbed]
    V -- no --> TT{TikTok?}
    TT -- yes --> TH[TikTok oEmbed]
    TT -- no --> SP{Spotify?}
    SP -- yes --> SPH[Spotify oEmbed]
    SP -- no --> R{Reddit?}
    R -- yes --> RH[Reddit oEmbed and scraping]
    R -- no --> SC{SoundCloud?}
    SC -- yes --> SCH[SoundCloud oEmbed]
    SC -- no --> F{Flickr?}
    F -- yes --> FH[Flickr oEmbed]
    F -- no --> SS{SlideShare?}
    SS -- yes --> SSH[SlideShare oEmbed]
    SS -- no --> GH{github.com?}
    GH -- yes --> GHH[GitHub REST API]
    GH -- no --> GL{gitlab.com?}
    GL -- yes --> GLH[GitLab REST API]
    GL -- no --> I{Instagram?}
    I -- yes --> IH[Scraping, site name Instagram]
    I -- no --> G[Generic scraping]

Platform handlers¶

Most oEmbed handlers use fetchOEmbedMetadata(oembedUrl, siteName), the standard mapping: title from the oEmbed title, description as By plus author_name, image_url from thumbnail_url, and a fixed site_name.

GitHub and GitLab match on the host alone. Their handlers then need at least two path segments (owner/repo); with fewer, a non-OK response, or an error, they fall back to generic scraping. GitLab projects in subgroups (group/subgroup/project) are looked up by their first two segments, so they fall back too.

Generic scraping¶

fetchGenericOGMetadata(url) fetches the page with User-Agent: tldrqBot/1.0. A non-OK response gives all nulls. Otherwise Cheerio parses the HTML:

Values are trimmed, and empty strings become null.

Timeouts and failures¶

fetchWithTimeout aborts a request whose response hasn't started arriving within 5000 ms (5 seconds). The timer is cleared as soon as fetch resolves with the headers, so reading the body has no limit. The limit is per request: a GitHub lookup that fails and falls back to scraping can wait about 10 seconds, while X and Reddit make their two requests in parallel.

tldrqBot/1.0 is sent for generic scraping, X's page fetch and the GitHub and GitLab APIs; oEmbed requests use the default user agent.

Every handler returns nulls when something fails, and fetchOGMetadata wraps the whole dispatch in a try/catch that returns all nulls, so saving a bookmark never fails because of metadata. The GitHub API is called without authentication, so GitHub rate-limits it per IP address; a rate-limited request isn't OK, and the bookmark gets scraped metadata without stats.

Images¶

resolveImageUrl(src, pageUrl) in src/lib/utils/ui.ts trims the value, resolves a relative path against the page URL, and returns null unless the result is an http: or https: URL. The scrapers pass response.url, the final URL after redirects, as the page URL. Both bookmark cards call it again when rendering, which fixes rows saved before it existed.

Adding a platform¶

1. Write an isXUrl(url) detector: parse with new URL(url) inside a try/catch, and check the host and path.
2. Write fetchXMetadata(url). If the site has oEmbed, fetchOEmbedMetadata(oembedUrl, "Name") may be all it needs.
3. Add it to the chain in fetchOGMetadata, before any check that would also match its URLs. The GitHub and GitLab checks match by host only.
4. If its links belong in Videos, Tweets or Repos, export the detector and add it to the auto-categorization in src/app/api/bookmarks/route.ts. Otherwise they go to Articles.
5. Update the table on this page and the one in the User Guide's Saving Bookmarks.

Try it with GET /api/metadata?url=… before saving real bookmarks.