knotelResources
Sign in
Browse resources

Send data

Filtering noise

Keep scanner probes, crawlers and health checks out of your traces, cheaply.

Any public site gets a steady stream of bots probing for /wp-admin, /.env and phpinfo.php. Stored as traces, that noise buries real requests and costs database writes. knotel drops it before it's stored.

How other tools handle it

Tracing platforms all filter before storage rather than hiding noise afterwards. Honeycomb's Refinery drops or samples spans by rules before they reach Honeycomb, Datadog has ingestion and exclusion filters, Sentry has inbound filters for crawlers and known noise, and the OpenTelemetry Collector has a filter processor. knotel does the same inside its ingest endpoint, where it's free.

Where to filter, cheapest first

WhereSavesGood for
Cloudflare WAF, at your site's edgeYour Worker never runs, and no spans existBlocking scanners outright
Workers SDK ignoreThe ingest requestHealth checks and paths you never want traced
knotel ingest filtersStorage and database writesEvery sender (browser, SDK, OpenTelemetry), no redeploys

Block scanners at the edge

If a site is on Cloudflare, a WAF custom rule (Security → WAF → Custom rules, available on every plan) blocks probes before they reach your Worker, which also saves the Worker request. For example, with the action set to Block:

(http.request.uri.path contains ".php") or
(http.request.uri.path contains "/wp-") or
(http.request.uri.path contains "/.env") or
(http.request.uri.path contains "/.git")

Ingest filter presets

Open a project's Filters page. Presets are maintained with knotel; Vulnerability scanners is on for every project unless you turn it off.

PresetDropsMatches
Vulnerability scannersProbes for PHP, WordPress, dotfiles, admin tools and config files that bots send to every public site.*.php *.php/* *.asp *.aspx *.jsp *.cgi /cgi-bin/* */wp-admin* */wp-content* */wp-includes* */wp-login* */xmlrpc* /wordpress* */.env* */.git* */.svn* */.aws* */.ssh* */.docker* */.vscode* */.idea* */.ds_store *phpinfo* *phpmyadmin* */_profiler* */_environment* /server-status* /server-info* */vendor/phpunit* /actuator* /hnap1* /boaform* */etc/passwd* *../* *.sql *.bak *.ini *.cfg *.log
Crawler filesrobots.txt, sitemaps, favicons and other files fetched by crawlers and browsers rather than people./robots.txt /sitemap*.xml /favicon.ico /apple-touch-icon* /ads.txt /app-ads.txt /humans.txt /.well-known/security.txt
Bots and scriptsRequests whose user agent is a search crawler, link previewer, scanner or HTTP library. Also hides uptime checks.bot crawler spider slurp facebookexternalhit headlesschrome python-requests python-urllib curl/ wget go-http-client okhttp zgrab masscan nuclei

Custom rules

Add up to 50 rules. A request is dropped when any rule (or enabled preset) matches. Matching ignores case.

FieldsMatches
URL path, Span name, HTTP route, HTTP status, HTTP method, Service, User agentis, starts with, ends with, contains, matches pattern
  • URL path is /health: drop uptime checks.
  • HTTP method is OPTIONS: drop CORS preflights.
  • URL path matches pattern /api/internal/*: drop a whole area.
  • HTTP status matches pattern 404: drop not-found responses. Statuses are matched as text, so 4* means any 4xx.
  • User agent contains uptimerobot: drop one monitor.

How matching works

  • Rules look at requests into a service: server spans and trace roots. A browser page load, a fetch, and a Worker request all qualify.
  • When a request is dropped, spans under it in the same upload are dropped too. The SDKs send a request's spans together, so a scanner request disappears completely. Very large traces (over 50 spans from the browser) can arrive in pieces, leaving a few child spans behind.
  • Filtering happens in memory before anything is written, so dropped spans cost no D1 writes or storage. They're counted as spans_filtered on the Usage page and the Filters page.
  • Saved changes reach every Cloudflare location within about a minute.

Test and clean up

Test on the last 24 hours runs the rules on the page, including unsaved changes, against recent requests and shows what would be dropped and which rule matched, so you can check a rule before it hides anything. Delete matching spans removes stored requests from the retention window that the rules match, with the spans under them.

Filtering is permanent
Dropped spans are never stored, so they can't be recovered. Test broad rules (like every 404) before saving.

Skip tracing in a Worker

To avoid sending spans at all, give the Workers SDK an ignore function. The request is still served normally. See Cloudflare Workers.

export default instrument(handler, (env) => ({
  endpoint: "https://YOUR-INSTANCE",
  key: env.KNOTEL_KEY,
  service: "api",
  ignore: (request) => {
    const { pathname } = new URL(request.url);
    return pathname === "/health" || /\.(php|asp|env)$/i.test(pathname);
  },
}));