How we track website changes

Fetching a page and diffing it against yesterday's copy is a weekend project. Living with that diff is the actual job. At KeepTabz we track competitor websites for the edits that mean something: a price, a positioning line, a product that appeared, a terms change, a hiring push. Most of what moves on those pages is none of those things.

This is how we separate the two.

A page is a function, and functions return different values

Open a product site twice, an hour apart, and the HTML will disagree with itself. A chat widget has a new opener. The suggested prompts rotated. "We usually reply in a few minutes" became "A teammate is online." A carousel advanced, a testimonial swapped, a view counter ticked, a consent banner shifted, an experiment served a different headline.

Each of those is a real difference in the bytes. A person watching the company would not call any of them a change. The company did not decide anything. The page did what it was built to do, which is to be slightly different for every visit.

That gap is the whole problem. Change tracking is useful when it tells you the company moved. It is useless when it tells you the page rendered.

Why the noise is harder to remove now

A static brochure from ten years ago could be hashed. Ignore the footer, hash the main column, email someone when the hash flips. A lot of monitoring still works that way, and it falls over on the sites people actually care about.

The important edit and the disposable text look the same in a diff. Both show up as lines added or lines replaced. "Starting at $49" becoming "$39" is a few characters. A chatbot transcript that rewrites itself is a few hundred. Size is a bad proxy. The price change is the story. The transcript is the widget working. Any filter that keeps the short edit and drops the long one has to know what the words are doing, because the diff itself will not say.

Chat copy often talks about the real change. A widget that says "Ask me about our new team plan" on the day the pricing page gained a team plan looks, to a keyword check, like a second announcement. It is an echo. If you file the echo as its own event, the digest doubles and the source of the news gets blurry. We treat the dynamic message as something the page says in passing, and we keep the underlying page edit as the change.

You cannot cut the widget out and keep the page. The chat bubble sits on top of the pricing table, the docs, the homepage. Drop the region that contains the widget and you can drop the price with it. Keep the region and you ingest the greeting. The filter has to be local. Each difference is judged on its own, in the context of the page it sits on, which is a slower kind of work than a CSS selector.

Screenshots disagree with the text, in both directions. A chat panel, a skeleton loader, or a cookie bar moves pixels on a page whose offer is identical. A sentence in the terms can change the commercial relationship and barely move the layout. Pixel diffing and text diffing each miss a class of event the other one catches. We keep both, and we let the rating decide.

There is more disposable text than there used to be. Experiments, personalization, recommendations, countdowns, social proof, and embedded chat are default parts of a marketing site. Each one is a small random variable. Multiply that by every page you watch, across every competitor, every day, and a tracker that alerts on raw diffs becomes a channel people mute. Once they mute it, the quiet pricing edit that landed two weeks before the launch post is just another unread line.

Significance is also about durability. A line that is part of the page is still there the next time you look. A line that belongs to the session is gone, replaced by a cousin of itself. Waiting a week would make that obvious and would also throw away the lead time, which is the reason to watch the site in the first place. "Significant enough" is a judgment you make on this scrape, with the previous scrape in hand: is this difference big enough in meaning, and stable enough as part of the page, that someone should hear about it today?

That combination is why noise removal stays hard. The web optimized for a fresh visit. Tracking optimized for a stable document. Those two goals share a URL.

What we actually run

A cron job starts each check. Firecrawl crawls the page, the snapshot goes into the database, a diff finds what moved, and an AI pass decides what is worth keeping. The diagram is that path. The labels on the right are the moments a screenshot is involved, and the panel at the bottom is the rule for when the model actually opens the picture.

The cron job starts the crawler

A cron job starts one check per tracked page. Firecrawl fetches it. The content comes back as markdown: headings, paragraphs, prices, the words a person can read. The same fetch takes a screenshot.

Markdown is what the diff will compare. It is close to the meaning of the page, and far enough from the DOM that a reordered stylesheet does not look like a launch.

The snapshot is stored before anyone judges it

The crawl writes a snapshot row before there is a verdict: the URL, the time, the markdown, and the screenshot. An unchanged page still leaves that row. The history of a page is the sequence of snapshots, whether or not any of them became an alert.

The diff compares markdown

Each new snapshot is set against the previous one for that URL. Same words, and the check stops. The screenshot stays on the snapshot. It is not the input to the diff. Different words, and we get a set of concrete differences: this block was added, this sentence was replaced, this section disappeared.

That set is still a pile. It includes the price, the typo, the rotated quote, and the chat opener. "A paragraph changed" is too coarse to rate. "The hero line was replaced" is a change we can judge. "The chat greeting went from hello to a question about onboarding" is a different kind of change.

AI analysis rates every difference

For each difference we ask two questions, and we keep a short reason with the answer.

Is it important? Importance is what the edit would mean to someone watching the company. A new price, a rewritten homepage claim, a product added or removed, a docs or changelog page that reveals a feature, a terms edit, a wave of roles on the hiring page: those are decisions. A swapped illustration, a footer year, a typo, a reordered logo strip: those are maintenance. The first group scores as important. The second scores as noise.

Is it significant enough? A change can be real and still be too small to send. One adjective that leaves the offer where it was. A testimonial rotating through a set list. A timestamp. The difference has to be large enough in meaning that a reader would update what they believe about the company.

Dynamic chat is where this bar does its most visible work. Messages inside the chatbot are generated for the visit, so they stay out of the change log. The greeting, the suggested prompts, the "reply in a few minutes" line, the transcript that quotes a feature already on the page: all of that is left untracked. The page around the widget is still compared. A pricing table that moved underneath an open chat panel is still rated, and it can still be stored. The panel's own words stay out.

Both bars have to pass. An important edit that is a single inconsequential word stays quiet. A large edit that is a chatbot rewriting its welcome stays quiet. A headline that changes what the company says it is, or a price that changes the offer, is the thing we keep.

The screenshot is opened in this step in two cases. The edit is visual: a layout move, a new hero image, a block you have to see to judge. Or the words are ambiguous because a chatbot covers the same region as a real change. A headline rewrite is scored from the markdown. Either way the picture remains on the snapshot, with the check.

The change that passes is stored on its own

A difference that clears both bars is written as its own row. The row holds a score, tags for severity and category, the before and after, a title, and why it matters. The snapshot remains the evidence. The change row is the decision, and it is what the product renders.

What a tracked change looks like

This card is that row. It is a real one from KeepTabz: Notion's homepage, checked on 6 October 2026.

The pieces on the card are the fields we stored after the AI pass.

• Notion repositions its homepage around AI is the title written for the change.
• 82 is the score. Major and Messaging are the tags.
• The body is the diff that survived rating. The hero headline was replaced, from "Your connected workspace for wiki, docs & projects" to "The AI workspace that works for you." "AI Agents" was added to the main navigation.
• Why it matters: new positioning shift is the reason kept with the rating.
• notion.com is the page the snapshot was taken from.

The same check can produce lines that never earn a row, and so never show up on a card like this one.

• The chat opener changed from "Hi, how can I help?" to "Want a hand picking a plan?" The words are new. They belong to the widget. They stay off the card.
• The footer copyright rolled from 2025 to 2026. That edit is real, and it is too small. It stays in the snapshot history.
• A pricing row that moves, and moves enough that the offer is different, is the kind of line that sits next to a hero rewrite. Before, after, score, and the reason.

The standard we hold the system to

A tracker that reports every diff has perfect recall and no readers. A tracker that only pings on a hand-maintained list of selectors stays quiet and misses the edit that showed up in a block nobody labeled. The useful behavior is in the middle, and it has to be re-earned on pages we have not special-cased.

The chatbot is the example I keep coming back to because it is honest about the modern page. Some of the text is the company. Some of the text is a performance for the current visitor. They are delivered in the same response, often in the same visual region, sometimes using the same nouns. Removing the noise means reading the difference and being willing to call a brand-new sentence unchanged, when the sentence was always going to be new.

That is the work. A cron job starts the crawl. Firecrawl returns the markdown and the screenshot, and the database keeps both. The diff finds the movement. The rating decides which differences are allowed to become a card.