URL Indexing

Abstract#

This NIP defines kind:31263 (parameterized replaceable event) to publish metadata about URLs. Each event represents the indexing result of a URL and contains normalized identifiers and extracted metadata.

These events can be used to build decentralized search indexes, discovery systems, and content catalogs.

Event Structure#

{
  "kind": 31263,
  "tags": [
    ["d", "<normalized-url>"],
    ["title", "<title>"],
    ["description", "<meta-description>"],
    ["l", "<language-code>"],
    ["keywords", "<comma-separated-keywords>"],
    ["x", "<sha256-page-hash>"],
    ["m", "<mime-type>"]
  ],
  "content": "<raw-text-snippet>"
}

Tags#

• d: Normalized URL identifier (see normalization rules below). (Required)
• title: Title of the page (from <title> tag). (Optional)
• description: Meta description (from <meta name="description">). (Optional)
• l: Language of the content (ISO 639-1 code, e.g., en, fr). (Optional)
• keywords: Comma-separated keywords (from <meta name="keywords">). (Optional)
• x: SHA-256 hash of the full page content (for deduplication). (Optional)
• m: MIME type of the content (e.g., text/html, application/pdf). (Optional)

Content#

A plain-text snippet extracted from the main readable content.

Normalized URL#

1. Convert the scheme and host to lowercase:

• HTTP://Example.COM → http://example.com.

2. Remove default ports:

• https://example.com:443 → https://example.com.

3. Remove the www. prefix:

• https://www.example.com → https://example.com.

4. Strip non-essential query parameters:

• Remove tracking parameters (e.g., ?utm_source=..., ?ref=...).
• Keep essential parameters (e.g., ?lang=fr, ?id=123).
• Example: https://example.com/page?lang=fr&ref=abc → https://example.com/page?lang=fr.

5. Remove fragments (#section):

• https://example.com/page#intro → https://example.com/page.

6. Resolve relative paths:

• https://example.com/a/../b → https://example.com/b.

7. Remove trailing slashes (except for root /):

• https://example.com/path/ → https://example.com/path.

8. Percent-encode special characters:

• https://example.com/my page → https://example.com/my%20page.

9. Sort remaining query parameters alphabetically:

• https://example.com/page?b=2&a=1 → https://example.com/page?a=1&b=2.

10. Normalize Internationalized Domain Names (IDNs):

• Convert IDNs to Punycode (e.g., https://例子.测试 → https://xn--fsqu00a.xn--0zwm56d).

Conflicting Metadata#

In cases where multiple events with the same normalized URL have conflicting metadata, clients can prioritize events based on the following criteria:

• Hash verification: Events with a verified hash match are prioritized.
• Trusted sources: Events from trusted indexers or contacts are prioritized.
• Timestamp: More recent events may be prioritized if other criteria are equal.

Example#

{
  "id": "...",
  "pubkey": "...",
  "sig": "...",
  "created_at": 1769366915,
  "kind": 31263,
  "tags": [
    ["d", "https://example.com/articles/ostriches?lang=en"],
    ["title", "The Social Life of Ostriches"],
    ["description", "Analysis of ostrich social behaviors."],
    ["l", "en"],
    ["keywords", "ostriches,animals,science"],
    ["x", "6f5635035f36ad500b4fc4bb7816bb72ef5594e1bcae44fa074c5e988fc4c0fe"],
    ["m", "text/html"]
  ],
  "content": "Ostriches live in groups called troops. They display complex social behaviors, including cooperative breeding and communal nesting. These behaviors allow them to gain benefits from social interactions, such as increased protection from predators and more effective rearing of their young."
}