Deduplicating Queued Shares Before Sync

This guide is part of IndexedDB Persistence for Offline Share Queues, which itself is part of Offline-First PWA Patterns for Web Share API.

Offline queues attract duplicates: a user double-taps the share button when the first tap seems to do nothing, a retry loop re-enqueues instead of updating an existing record, or two service worker instances flush the same queue at once. Without deduplication the recipient gets the same link twice. The reliable fix lives in the storage layer — compute a stable content hash of the normalised payload, put a unique index on it, and let the database reject the second write instead of trying to detect duplicates by scanning.

Feature Detection Gate

Hashing depends on the Web Crypto SubtleCrypto interface, which is only exposed in secure contexts. Gate on it before you rely on hash-based dedup, with a non-cryptographic fallback for the rare environment that lacks it.

// dedup-gate.js
export function canHashPayload() {
  return (
    window.isSecureContext === true &&
    typeof crypto !== 'undefined' &&
    typeof crypto.subtle?.digest === 'function'
  );
}

If canHashPayload() is false, fall back to a cheap synchronous hash (below) so dedup still functions — just without cryptographic collision resistance, which the queue does not actually need.

Solution Walkthrough

Step 1 — Normalise, then hash the payload

Normalisation is the step that makes dedup work. Trim and collapse whitespace, and canonicalise the URL so https://ex.com/p?a=1 and https://ex.com/p?a=1# collapse to one hash. Hash the joined, normalised fields — never the raw input.

// content-hash.js
export function normalisePayload({ title = '', text = '', url = '' }) {
  const clean = (s) => String(s).trim().replace(/\s+/g, ' ');
  let canonicalUrl = clean(url);
  try {
    const u = new URL(canonicalUrl);
    u.hash = '';                 // fragment never changes the share target
    canonicalUrl = u.href;
  } catch {
    // not a valid absolute URL — keep the trimmed string
  }
  // Join with a delimiter that cannot appear in the fields.
  return [clean(title), clean(text), canonicalUrl].join('�');
}

export async function contentHash(payload) {
  const normalised = normalisePayload(payload);
  if (!window.isSecureContext || !crypto.subtle?.digest) {
    return fnv1a(normalised); // non-crypto fallback
  }
  const bytes = new TextEncoder().encode(normalised);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  return [...new Uint8Array(digest)]
    .map((b) => b.toString(16).padStart(2, '0'))
    .join('');
}

function fnv1a(str) {
  let h = 0x811c9dc5;
  for (let i = 0; i < str.length; i += 1) {
    h ^= str.charCodeAt(i);
    h = Math.imul(h, 0x01000193);
  }
  return (h >>> 0).toString(16);
}

Because crypto.subtle.digest is async, compute the hash before opening the IndexedDB transaction — awaiting it inside an open transaction would let the transaction auto-commit and throw TransactionInactiveError, the same rule the persistence guide enforces for every write.

Step 2 — Upsert-or-skip against a unique index

Store the hash on a unique contentHash index (created in the versioned upgrade). Then add() either succeeds — a genuinely new share — or throws ConstraintError, which is your signal that this exact intent is already queued.

// dedup-enqueue.js
import { openShareDB, STORE } from './share-db.js';
import { contentHash } from './content-hash.js';

export async function enqueueDeduplicated(payload, serialisedFiles = []) {
  const hash = await contentHash(payload); // async work BEFORE the transaction

  const record = {
    id: crypto.randomUUID(),
    createdAt: Date.now(),
    status: 'pending',
    retries: 0,
    contentHash: hash,
    payload: { ...normaliseFields(payload), files: serialisedFiles }
  };

  const db = await openShareDB();
  return new Promise((resolve, reject) => {
    const tx = db.transaction(STORE, 'readwrite');
    const request = tx.objectStore(STORE).add(record); // unique index enforces dedup

    request.onsuccess = () => resolve({ queued: true, id: record.id });
    request.onerror = (event) => {
      if (request.error?.name === 'ConstraintError') {
        event.preventDefault(); // swallow so the tx does not abort loudly
        resolve({ queued: false, duplicate: true });
      }
    };
    tx.oncomplete = () => db.close();
    tx.onerror = () => reject(tx.error);
  });
}

function normaliseFields(p) {
  const clean = (s) => String(s ?? '').trim().replace(/\s+/g, ' ');
  return { title: clean(p.title), text: clean(p.text), url: clean(p.url) };
}

Calling event.preventDefault() in the request’s onerror stops the ConstraintError from bubbling up and aborting the whole transaction, letting you resolve cleanly with duplicate: true. The caller treats a duplicate as success — the user’s intent is already captured — and shows the same “queued” confirmation.

Step 3 — Add a time window when re-sharing should be allowed

Sometimes an identical share should be allowed again after a while — a user reposting the same article the next day. Fold a coarse time bucket into the hash so identical content within the window collapses to one entry but a later attempt hashes differently.

// windowed-hash.js
import { normalisePayload } from './content-hash.js';

const WINDOW_MS = 10 * 60 * 1000; // 10-minute dedup window

export async function windowedHash(payload, now = Date.now()) {
  const bucket = Math.floor(now / WINDOW_MS); // same value for 10 minutes
  const material = `${normalisePayload(payload)}�${bucket}`;
  const bytes = new TextEncoder().encode(material);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  return [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, '0')).join('');
}

Choose the window to match intent: a few seconds is enough to absorb double-taps, ten minutes catches retry storms, and no time component at all makes the dedup permanent for as long as the record lives in the queue.

Failure Modes and Recovery

Browser and Platform Caveat

crypto.subtle.digest requires a secure context and is available in Chromium 37+, Safari 11+, and Firefox 34+, including inside service workers — so both the foreground enqueue and a worker-side flush can hash consistently. The unique-index behaviour is uniform across engines: add() against an existing unique key always throws ConstraintError and aborts unless you preventDefault(). The one real divergence is that a queue on iOS Safari may be evicted after seven idle days, which resets dedup state along with the records — acceptable, since an evicted queue has nothing left to duplicate against. Keep the non-cryptographic FNV fallback for locked-down WebViews where crypto.subtle is absent; dedup does not need collision resistance, only stability.

FAQ

Related

• IndexedDB Persistence for Offline Share Queues
• Background Sync for Deferred Shares
• Offline-First PWA Patterns for Web Share API