Web Automation Basics

Resources

Browser Automation

https://webscraping.fyi/overview/browser-automation/

https://github.com/angrykoala/awesome-browser-automation

Web Scraping Bot Detection/Blocking

https://scrapfly.io/academy/scraper-blocking

https://scrapfly.io/blog/posts/how-to-scrape-without-getting-blocked-tutorial/

https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-tls

https://scrapfly.io/blog/posts/how-to-avoid-web-scraping-blocking-javascript

https://docs.apify.com/academy/anti-scraping

https://docs.apify.com/academy/anti-scraping/techniques/fingerprinting

HTTP Header

To scrape data from a web page without getting detected, we have to carefully configure headers:

User Agent 和浏览器版本

IP Address

ipinfo

{
 ip:"104.234.157.142",
 city:"Jordan",
 region:"Yau Tsim Mong District",
 country:"HK",
 loc:"22.3054,114.1700",
 postal:"999077",
 timezone:"Asia/Hong_Kong",
 asn:{
  asn:"AS400618",
  name:"Prime Security Corp.",
  domain:"primesecuritycorp.com",
  route:"104.234.157.0/24",
  type:"hosting"
 },
 company:{
  name:"ONTAR-40 (Velcom INC)",
  domain:"velcom.com",
  type:"hosting"
 },
 privacy:{
  vpn:false,
  proxy:false,
  tor:false,
  relay:false,
  hosting:true,
  service:""
 },
 abuse:{
  address:"US, FL, Tampa, 8010 Woodland Center Blvd, Suite 700, 33614",
  country:"US",
  email:"abuse@hivelocity.net",
  name:"HIvelocity Network Abuse Administrator",
  network:"104.234.152.0/21",
  phone:"+1-888-869-4678"
 },
 domains:{
  page:0,
  total:0,
  domains:[]
 }
}

首先，在 ICANN 和 ARIN 上查 104.234.157.142，只能查到一家叫 Internet Utilities NA LLC 的公司。这个公司的 ip range 是 104.234.128.0/19。这只能说明 Internet Utilities 从 ARIN 那里获得了这些 ip，实际使用的人可能不是他们。

HIvelocity 和 Prime Security 这两家公司是怎么来的？是 ipinfo 通过查询公开的 BGP 信息获得的。Hivelocity (AS29802) 发送过 BGP 消息，说 104.234.152.0/21 是他们管的。Prime Security (AS400618) 也发送过 BGP 消息，说 104.234.157.0/24 是他们管的。

company 也是 ipinfo 自己分析出来的，ipinfo 通过某种方式知道 ONTAR-40 (Velcom INC) 是实际使用 104.234.157.142 这个 ip 的公司。

这个 ip 的层级关系是：Internet Utilities -> Hivelocity (AS29802) -> Prime Security Corp. (AS400618) → Velcom INC。

business ip 的问题

上面的信誉排名并不是广告商的视角。从广告商的视角出发，business ip 最好不要用。

第一，business 类型的 ip 可能实际上是 hosting。You are assuming "Business IP" means "IBM Headquarters." In the proxy world, "Business IP" usually means "Cheap VPS Hosting" or "School Library" or "Small Office."

Ad Tech Databases: Companies like MaxMind and Digital Element categorize IP ranges.

The Trap: Many "Business" ISPs (like Cogent or Level3) sell to both legitimate corporations AND massive server farms.

The Verdict: Because it is hard to tell the difference, Ad Networks often treat ALL "Business" ASNs as "Potential Datacenter/Bot" traffic by default. It is a "guilty until proven innocent" system.

第二，广告商认为 business 类型没有价值。Advertisers pay for clicks because they want to sell something. Ad algorithms have trillions of data points on conversion rates.

Residential/Mobile IP User: High probability of buying. They are at home, relaxed, often on a personal device. Value: $$$

Corporate/Business IP User: Low probability of buying.

The Result: Ad networks often pre-filter Business IPs (classifying them as "Non-Performing") to save money for their advertisers. If your bot forces them to serve ads to these IPs, you are destroying their conversion metrics.

第三，business 类型的流量会影响你帐号的质量。比如你有 1/3 流量来自 business，太高了，你的账号会被直接封掉。Including Business IPs in your rotation lowers your overall "Traffic Quality Score" and makes you look like a server farm.

Other

Passive TCP/IP Fingerprint

检测原理

Most proxies (SOCKS5, HTTP, HTTPS CONNECT) work at the Application or Session layer (Layer 5/7), not the Packet layer (Layer 3).

Your Chain: You -> Proxy -> Website

The Process:

Result: The website sees the Proxy Server's TCP/IP Fingerprint (TTL, Window Size, MSS, etc.).

Since most proxy servers run on Linux, the website usually sees a Linux TCP fingerprint.

检测工具和结果

例外：Firewall

Passive TCP fingerprinting has one major weakness: The Network "Mangles" Packets.

The Scenario: A legitimate user is on a real Windows PC at a University or Corporate Office.

The "Mangle": Their connection goes through a massive Corporate Firewall (e.g., Cisco/Palo Alto) before hitting the internet.

The Result: The Corporate Firewall rewrites the TCP headers for security.

It normalizes the TTL. It changes the Window Size.

The False Positive: To Cloudflare, this legitimate Windows user now looks like a "Cisco Router" or "Linux Firewall."

Because of this, Cloudflare cannot block solely on TCP fingerprints.

解决方法：Spoofing

Because the "Linux Proxy vs. Windows User-Agent" mismatch is such a common detection method, high-end proxy providers (specifically Mobile Proxy sellers) have started adding a feature called "TCP/IP Fingerprint Spoofing" (or "Passive OS Masquerading").

How it works: The proxy server is configured to intentionally "mangle" its outgoing packets to look like Windows or macOS.

What they change: They force the Linux server to send a TTL of 128 (Windows) instead of 64 (Linux) and change the TCP Window Size to mimic a desktop OS.

安全公司是否采用

https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler/

这种方式是最有效、最容易的检测方式。In fact, Cloudflare has publicly blogged about how they took the open-source p0f tool and compiled it into BPF (Berkeley Packet Filter) bytecode so it can run directly inside the Linux Kernel of their edge servers. This allows them to check the TCP fingerprint of every single packet that hits their network at lightning speed without slowing down traffic.

判断是否被检测

没办法，检测在服务器端进行。

TLS Fingerprint

检测原理

Transport Layer Security (TLS) is an end-to-end encryption protocol used in all HTTPS connections. HTTP clients perform the TLS handshake differently, leading to a unique fingerprint called JA3. If the generated fingerprint is different from the the regular browsers, it can lead to web scraping blocking.

检测工具和结果

解决方法

Only by using high-quality anti-detect browsers or specialized libraries (like curl_cffi or uTLS) that replicate the browser's TLS behavior byte-for-byte.

安全公司是否采用

It is a fundamental pillar of Cloudflare's security architecture. It is running on every request.

In fact, Cloudflare is one of the primary reasons TLS fingerprinting became an industry standard. Because they see roughly 20% of all internet traffic, they have the world's largest dataset of "Normal" vs. "Abnormal" TLS fingerprints.

Here is exactly how they and other major companies deploy it.

1 Cloudflare: The "Fingerprint Database"

Cloudflare doesn't just look at your fingerprint in isolation. They compare it against a massive global list.

2 AWS WAF: It's a Native Feature

As of late 2023, Amazon Web Services (AWS) added native JA3 fingerprinting to their Web Application Firewall (WAF).

This means any website hosted on AWS can click a single button to enable a rule that says:

"Block any request where the JA3 Fingerprint matches known bot libraries."

This is no longer "secret tech" used only by experts; it is a standard checkbox for millions of website administrators.

判断是否被检测

没办法，检测在服务器端进行。

Latency Detection

检测原理：TCP Timing

Step 1: The Request (No Timer Yet)

Step 2: The Challenge (Start Timer)

Step 3: The Reply (Stop Timer)

Step 4: The Calculation 20ms - 0ms = 20ms

The web server (e.g., NGINX or Cloudflare) can query its own Linux Kernel (using a command like tcp_info) to get this exact number: tcpi_rtt.

检测原理：Websocket Timing

Step 0: Setup

The website loads a JavaScript file into your browser. This script creates a WebSocket connection to the server.

Step 1: The Browser Starts the Stopwatch

Step 2: The Outward Journey

Step 3: The Server Acts as a Mirror

Step 4: The Return Journey

Step 5: The Browser Stops the Stopwatch

Step 6: The "Gotcha" (Reporting the Result)

检测原理：Final Comparison

The Server now has two numbers on its desk:

Server's Conclusion:

"The IP address is close (10ms), but the Browser is far away (150ms). This user is hiding behind a proxy."

检测原理：TCP Timing Jitter

While RTT is the "speed" (10ms vs 150ms), Jitter is the "variance" of that speed over time.

Datacenter TCP RTT: If your TCP RTT is exactly 10.00ms, 10.01ms, 10.00ms... that is the "Flat Line" (Datacenter Jitter) we discussed.

Residential TCP RTT: If it is 35ms, 42ms, 29ms, 150ms... that is the "Messy Line" (Residential Jitter).

So, security systems look at both:

检测原理：具体数值

Scenario Physical Distance Typical RTT Range
Same City < 50 miles 5ms – 20ms
Same State/Region < 300 miles 15ms – 40ms
Cross-Country (US) NY ↔ LA 60ms – 90ms
Trans-Atlantic NY ↔ London 80ms – 130ms
Trans-Pacific LA ↔ Tokyo 120ms – 200ms
Global / Remote Brazil ↔ Japan 250ms – 400ms+

1 The "Perfect Match" (Same City / State)

2 The "Regional" Variance (Same Continent)

3 The "Cross-Ocean" Giveaways (The Danger Zone)

4 The "Cross-World" Suicide (Asia to US)

The "Jitter" Factor (How to look real)

It's not just about being fast; it's about being consistently inconsistent.

解决方法

安全公司是否采用

The "Big 3" (Cloudflare, Akamai, DataDome): YES. They have the scale to handle the data and the sophisticated ML models to filter out false positives.

Mid-Tier WAFs (Generic WordPress Plugins, Basic Firewalls): NO. They usually stick to simpler checks (IP reputation lists, basic User-Agent blocking). Implementing a WebSocket handshake for every visitor is too heavy for them.

Custom Enterprise Security: YES. Banks and airlines often implement this specifically to catch account-takeover bots that use residential proxies.

This detection is used because:

This detection is not used because:

The Solution: Companies like Cloudflare have to maintain massive "Allowlists" of Starlink IP ranges and Corporate VPNs to prevent banning millions of real users. Smaller security companies cannot maintain these lists accurately, so they turn this feature OFF by default to avoid complaints.

如何判断是否被检测

If you want to know if a site is running this on you, look at your Developer Tools -> Network Tab.

JavaScript Fingerprint

JavaScript-based fingerprinting applies when web scraping with a headless browser automation tool, such as Selenium, Playwright, and Puppeteer. Since these web scraping tools are JavaScript-enabled, the target website can execute remote code on the client's machine. This remote code execution can reveal several details about the client, such as the:

Browser Automation Tool Leaks

Many web-browser automation tools leak information about themselves to javascript execution context - meaning javascript can easily tell the browser is being controlled by a program rather than a human being.

How is fingerprint collected

Sites employ multiple levels and different approaches to collect browser fingerprints. However, they all have one thing in common: they are using a script written in JavaScript to evaluate the target browser's context and collect information about it (oftentimes also storing it in their database, or in a cookie).

To add, detailed fingerprinting is a resource intensive operation and most websites cannot afford to have their users wait there 2 seconds for the content to load.

What makes a fingerprint

How does website server handle fingerprint data

Detect when fingerprint is collected

Because of how common it has become to obfuscate fingerprinting scripts, there are many extensions that help identify fingerprinting scripts due to the fact that browser fingerprinting is such a big privacy question.

DNS 泄漏

Commercial Security Network

网站一般不会自己开发代码检查 bot，一般是通过商用解决方案比如 cloudflare 来检查 bot。

这些商用解决方案的检查手段一定非常成熟，以 cloudflare 为例：

业内做的比较好的安全公司有：

流量伪装

看另一篇文章。

Web Scraping Tools

Web Scraping Mitigations

https://github.com/TheGP/untidetect-tools

https://github.com/niespodd/browser-fingerprinting

Ad Alliance Automation Implementation