网页源代码获取方法及特殊场景处理

0 次阅读

网页源代码是分析页面结构、排查前端问题、学习网页实现方式以及进行数据处理时经常需要查看的内容。根据使用环境不同,获取网页源代码的方法也有所区别。普通静态页面可以直接通过浏览器查看,而遇到 JavaScript 动态渲染、接口异步加载、登录验证等情况时,仅仅查看初始 HTML 往往无法得到完整内容。

下面按照常见使用场景介绍网页源代码获取方法,并重点说明动态网页、中文编码、请求头、反爬限制等特殊情况的处理思路。

一、什么是网页源代码

网页源代码通常指服务器返回给浏览器的 HTML 文档,其中可能包含页面结构、文本内容、CSS 引用、JavaScript 引用、图片地址以及各种元数据。

例如,一个简单网页可能返回:

HTML



     charset="UTF-8">
    </span><span>示例页面</span><span class="ͼo">


    

Hello World

需要注意的是,“网页源代码”和“浏览器开发者工具中看到的 DOM”并不完全相同。

网页源代码通常对应服务器最初返回的 HTML,而开发者工具 Elements 面板展示的是浏览器解析并执行 JavaScript 后形成的当前 DOM。

因此,当网页使用 Vue、React、Angular 等前端框架动态生成内容时,两者可能存在明显差异。

二、直接使用浏览器查看网页源代码

最简单的方法是不需要编写任何程序,直接使用浏览器查看。

以 Chrome、Edge 等浏览器为例,可以在网页空白位置点击鼠标右键,选择“查看网页源代码”。

也可以直接使用快捷键:

Ctrl + U

浏览器通常会打开一个新的页面,并显示服务器返回的 HTML 源代码。

还可以在地址栏中使用:

view-source:https://example.com

这种方式适合快速检查:

  • HTML 标签结构

  • </code> 标题</p> </li> <li data-section-id="14d478a" data-start="911" data-end="925"> <p data-start="913" data-end="925"><code data-start="913" data-end="921"> 元信息</p> </li> <li data-section-id="wqatc7" data-start="926" data-end="936"> <p data-start="928" data-end="936">CSS 文件引用</p> </li> <li data-section-id="55iswn" data-start="937" data-end="954"> <p data-start="939" data-end="954">JavaScript 文件引用</p> </li> <li data-section-id="3a5zj5" data-start="955" data-end="966"> <p data-start="957" data-end="966">图片和静态资源地址</p> </li> <li data-section-id="1c8uvux" data-start="967" data-end="978"> <p data-start="969" data-end="978">页面中的结构化数据</p> </li> <li data-section-id="i64usa" data-start="979" data-end="991"> <p data-start="981" data-end="991">页面是否包含目标文本</p> </li> </ul> <p data-start="993" data-end="998">例如搜索:</p> <pre class="overflow-visible! px-0!" data-start="1000" data-end="1052"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#fa6359" fill="currentColor"></use>HTML</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><div role="group" aria-label="代码块视图切换" class="group/code-block-preview-toggle relative flex items-center gap-0.5"><div aria-hidden="true" class="bg-token-interactive-bg-secondary-press pointer-events-none absolute start-0 top-0 size-9 rounded-full motion-safe:transition-transform motion-safe:duration-200 motion-safe:ease-in-out translate-x-0 group-has-[[data-code-block-preview-toggle-preview]:hover]/code-block-preview-toggle:translate-x-[calc(100%+2px)] group-has-[[data-code-block-preview-toggle-preview]:focus-visible]/code-block-preview-toggle:translate-x-[calc(100%+2px)]"></div><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use></span><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use></span></div><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div data-custom-highlighting-behavior="boundary" class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative [container-type:inline-size]"><div class="w-full overflow-x-hidden overflow-y-auto" style="max-height: 56.25cqw; transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼo"><meta</span><span> </span><span class="ͼn">name</span><span class="ͼg">=</span><span class="ͼk">"description"</span><span> </span><span class="ͼn">content</span><span class="ͼg">=</span><span class="ͼk">"网页描述"</span><span class="ͼo">></span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="1054" data-end="1072">可以快速确认页面 SEO 描述信息。</p> <h2 data-section-id="2h0p2k" data-start="1074" data-end="1093">三、使用浏览器开发者工具查看源码</h2> <p data-start="1095" data-end="1134">如果需要分析网页实际运行后的结构,开发者工具通常比“查看网页源代码”更加实用。</p> <p data-start="1136" data-end="1156">Chrome、Edge 等浏览器可以按:</p> <pre class="overflow-visible! px-0!" data-start="1158" data-end="1173"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>F12</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="1175" data-end="1178">或者:</p> <pre class="overflow-visible! px-0!" data-start="1180" data-end="1208"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>Ctrl + Shift + I</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="1210" data-end="1218">打开开发者工具。</p> <h3 data-section-id="1o2qfbd" data-start="1220" data-end="1238">1. Elements 面板</h3> <p data-start="1240" data-end="1264">Elements 面板用于查看当前页面 DOM。</p> <p data-start="1266" data-end="1281">例如原始 HTML 可能只有:</p> <pre class="overflow-visible! px-0!" data-start="1283" data-end="1315"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#fa6359" fill="currentColor"></use>HTML</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><div role="group" aria-label="代码块视图切换" class="group/code-block-preview-toggle relative flex items-center gap-0.5"><div aria-hidden="true" class="bg-token-interactive-bg-secondary-press pointer-events-none absolute start-0 top-0 size-9 rounded-full motion-safe:transition-transform motion-safe:duration-200 motion-safe:ease-in-out translate-x-0 group-has-[[data-code-block-preview-toggle-preview]:hover]/code-block-preview-toggle:translate-x-[calc(100%+2px)] group-has-[[data-code-block-preview-toggle-preview]:focus-visible]/code-block-preview-toggle:translate-x-[calc(100%+2px)]"></div><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use></span><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use></span></div><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div data-custom-highlighting-behavior="boundary" class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative [container-type:inline-size]"><div class="w-full overflow-x-hidden overflow-y-auto" style="max-height: 56.25cqw; transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼo"><div</span><span> </span><span class="ͼn">id</span><span class="ͼg">=</span><span class="ͼk">"app"</span><span class="ͼo">></div></span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="1317" data-end="1336">JavaScript 执行之后却变成:</p> <pre class="overflow-visible! px-0!" data-start="1338" data-end="1408"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#fa6359" fill="currentColor"></use>HTML</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><div role="group" aria-label="代码块视图切换" class="group/code-block-preview-toggle relative flex items-center gap-0.5"><div aria-hidden="true" class="bg-token-interactive-bg-secondary-press pointer-events-none absolute start-0 top-0 size-9 rounded-full motion-safe:transition-transform motion-safe:duration-200 motion-safe:ease-in-out translate-x-0 group-has-[[data-code-block-preview-toggle-preview]:hover]/code-block-preview-toggle:translate-x-[calc(100%+2px)] group-has-[[data-code-block-preview-toggle-preview]:focus-visible]/code-block-preview-toggle:translate-x-[calc(100%+2px)]"></div><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use></span><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use></span></div><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div data-custom-highlighting-behavior="boundary" class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative [container-type:inline-size]"><div class="w-full overflow-x-hidden overflow-y-auto" style="max-height: 56.25cqw; transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼo"><div</span><span> </span><span class="ͼn">id</span><span class="ͼg">=</span><span class="ͼk">"app"</span><span class="ͼo">></span><span> </span><span class="ͼo"><h1></span><span>网页标题</span><span class="ͼo"></h1></span><span> </span><span class="ͼo"><p></span><span>动态加载的内容</span><span class="ͼo"></p></span><span> </span><span class="ͼo"></div></span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="1410" data-end="1446">这种情况下,Elements 中看到的内容明显比原始网页源代码更加完整。</p> <h3 data-section-id="1twwnmh" data-start="1448" data-end="1465">2. Network 面板</h3> <p data-start="1467" data-end="1493">Network 面板适合分析网页数据究竟从哪里获取。</p> <p data-start="1495" data-end="1508">刷新页面后,可以重点观察:</p> <pre class="overflow-visible! px-0!" data-start="1510" data-end="1546"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>Fetch/XHR Doc JS CSS Img</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="1548" data-end="1585">如果页面内容来自后端接口,通常可以在 Fetch/XHR 中找到对应请求。</p> <p data-start="1587" data-end="1602">查看请求详情时,可以重点关注:</p> <ul data-start="1604" data-end="1721"> <li data-section-id="12399wy" data-start="1604" data-end="1617"> <p data-start="1606" data-end="1617">Request URL</p> </li> <li data-section-id="pe6a06" data-start="1618" data-end="1634"> <p data-start="1620" data-end="1634">Request Method</p> </li> <li data-section-id="sr5c45" data-start="1635" data-end="1652"> <p data-start="1637" data-end="1652">Request Headers</p> </li> <li data-section-id="6aosbx" data-start="1653" data-end="1678"> <p data-start="1655" data-end="1678">Query String Parameters</p> </li> <li data-section-id="1wk0ldj" data-start="1679" data-end="1696"> <p data-start="1681" data-end="1696">Request Payload</p> </li> <li data-section-id="g8pbqj" data-start="1697" data-end="1707"> <p data-start="1699" data-end="1707">Response</p> </li> <li data-section-id="lxbh3l" data-start="1708" data-end="1721"> <p data-start="1710" data-end="1721">Status Code</p> </li> </ul> <p data-start="1723" data-end="1770">如果发现页面的数据来自 JSON 接口,那么直接分析接口响应通常比解析最终 HTML 更可靠。</p> <h2 data-section-id="1gd35cb" data-start="1772" data-end="1794">四、使用 Python 获取网页源代码</h2> <p data-start="1796" data-end="1817">对于批量处理网页,可以使用 Python。</p> <p data-start="1819" data-end="1857">最常见的方式是 <code data-start="1827" data-end="1837">requests</code> 配合 <code data-start="1841" data-end="1856">BeautifulSoup</code>。</p> <p data-start="1859" data-end="1866">首先安装依赖:</p> <pre class="overflow-visible! px-0!" data-start="1868" data-end="1915"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Bash</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>pip install requests beautifulsoup4</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="1917" data-end="1924">基本代码如下:</p> <pre class="overflow-visible! px-0!" data-start="1926" data-end="2089"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼm">url</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼk">"https://example.com"</span><span> </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>get(</span><span class="ͼm">url</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span>) </span><span class="ͼm">response</span><span class="ͼg">.</span><span>raise_for_status() </span><span class="ͼm">html</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">response</span><span class="ͼg">.</span><span>text </span><span class="ͼm">print</span><span>(</span><span class="ͼm">html</span><span>)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="2091" data-end="2126"><code data-start="2091" data-end="2106">response.text</code> 返回的是经过字符编码处理后的文本内容。</p> <p data-start="2128" data-end="2152">如果希望直接查看服务器返回的原始字节,可以使用:</p> <pre class="overflow-visible! px-0!" data-start="2154" data-end="2197"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">html_bytes</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">response</span><span class="ͼg">.</span><span>content</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="2199" data-end="2219">两者的区别在处理中文网页时尤其值得注意。</p> <h2 data-section-id="8t920" data-start="2221" data-end="2247">五、使用 BeautifulSoup 解析网页</h2> <p data-start="2249" data-end="2291">获取 HTML 后,如果只是打印全部源代码,实际价值有限。更多时候需要提取特定标签。</p> <p data-start="2293" data-end="2296">例如:</p> <pre class="overflow-visible! px-0!" data-start="2298" data-end="2548"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼg">from</span><span> </span><span class="ͼm">bs4</span><span> </span><span class="ͼg">import</span><span> </span><span class="ͼm">BeautifulSoup</span><span> </span><span class="ͼm">url</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼk">"https://example.com"</span><span> </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>get(</span><span class="ͼm">url</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span>) </span><span class="ͼm">response</span><span class="ͼg">.</span><span>raise_for_status() </span><span class="ͼm">soup</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">BeautifulSoup</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>text, </span><span class="ͼk">"html.parser"</span><span>) </span><span class="ͼm">print</span><span>(</span><span class="ͼm">soup</span><span class="ͼg">.</span><span>title</span><span class="ͼg">.</span><span>get_text(</span><span class="ͼm">strip</span><span class="ͼg">=</span><span class="ͼj">True</span><span>))</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="2550" data-end="2557">提取所有链接:</p> <pre class="overflow-visible! px-0!" data-start="2559" data-end="2632"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">for</span><span> </span><span class="ͼm">link</span><span> </span><span class="ͼg">in</span><span> </span><span class="ͼm">soup</span><span class="ͼg">.</span><span>find_all(</span><span class="ͼk">"a"</span><span>): </span><span class="ͼm">print</span><span>(</span><span class="ͼm">link</span><span class="ͼg">.</span><span>get(</span><span class="ͼk">"href"</span><span>))</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="2634" data-end="2641">提取所有标题:</p> <pre class="overflow-visible! px-0!" data-start="2643" data-end="2751"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">for</span><span> </span><span class="ͼm">heading</span><span> </span><span class="ͼg">in</span><span> </span><span class="ͼm">soup</span><span class="ͼg">.</span><span>find_all([</span><span class="ͼk">"h1"</span><span>, </span><span class="ͼk">"h2"</span><span>, </span><span class="ͼk">"h3"</span><span>]): </span><span class="ͼm">print</span><span>(</span><span class="ͼm">heading</span><span class="ͼg">.</span><span>get_text(</span><span class="ͼk">" "</span><span>, </span><span class="ͼm">strip</span><span class="ͼg">=</span><span class="ͼj">True</span><span>))</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="2753" data-end="2779">这种方式适合进行网页结构分析、内容提取和自动化处理。</p> <h2 data-section-id="1ox86cl" data-start="2781" data-end="2803">六、使用 urllib 获取网页源代码</h2> <p data-start="2805" data-end="2844">如果不希望安装第三方库,可以使用 Python 标准库中的 <code data-start="2835" data-end="2843">urllib</code>。</p> <pre class="overflow-visible! px-0!" data-start="2846" data-end="3023"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">from</span><span> </span><span class="ͼm">urllib</span><span class="ͼg">.</span><span class="ͼm">request</span><span> </span><span class="ͼg">import</span><span> </span><span class="ͼm">urlopen</span><span> </span><span class="ͼm">url</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼk">"https://example.com"</span><span> </span><span class="ͼg">with</span><span> </span><span class="ͼm">urlopen</span><span>(</span><span class="ͼm">url</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span>) </span><span class="ͼg">as</span><span> </span><span class="ͼm">response</span><span>: </span><span class="ͼm">html</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">response</span><span class="ͼg">.</span><span>read() </span><span class="ͼm">print</span><span>(</span><span class="ͼm">html</span><span class="ͼg">.</span><span>decode(</span><span class="ͼk">"utf-8"</span><span>))</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3025" data-end="3091">不过实际项目中,<code data-start="3033" data-end="3043">requests</code> 通常具有更加直观的 API,在请求头、参数、Cookie、Session 等场景下也更加方便。</p> <h2 data-section-id="149wh0t" data-start="3093" data-end="3108">七、处理中文网页乱码问题</h2> <p data-start="3110" data-end="3135">获取网页源代码时,一个非常常见的问题就是中文乱码。</p> <p data-start="3137" data-end="3147">例如网页实际内容是:</p> <pre class="overflow-visible! px-0!" data-start="3149" data-end="3169"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>这是一个测试页面</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="3171" data-end="3179">程序输出却变成:</p> <pre class="overflow-visible! px-0!" data-start="3181" data-end="3206"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>杩欐槸涓€涓祴璇曢〉闈�</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="3208" data-end="3224">这通常与字符编码识别不正确有关。</p> <p data-start="3226" data-end="3235">可以先查看响应头:</p> <pre class="overflow-visible! px-0!" data-start="3237" data-end="3294"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>headers</span><span class="ͼg">.</span><span>get(</span><span class="ͼk">"Content-Type"</span><span>))</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3296" data-end="3302">也可以检查:</p> <pre class="overflow-visible! px-0!" data-start="3304" data-end="3342"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>encoding)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3344" data-end="3352">必要时手动指定:</p> <pre class="overflow-visible! px-0!" data-start="3354" data-end="3416"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">response</span><span class="ͼg">.</span><span>encoding </span><span class="ͼg">=</span><span> </span><span class="ͼk">"utf-8"</span><span> </span><span class="ͼm">html</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">response</span><span class="ͼg">.</span><span>text</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3418" data-end="3453">如果网站实际使用其他编码,则应该根据页面声明和服务器返回信息进行设置。</p> <p data-start="3455" data-end="3470">HTML 中常见的编码声明是:</p> <pre class="overflow-visible! px-0!" data-start="3472" data-end="3506"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#fa6359" fill="currentColor"></use>HTML</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><div role="group" aria-label="代码块视图切换" class="group/code-block-preview-toggle relative flex items-center gap-0.5"><div aria-hidden="true" class="bg-token-interactive-bg-secondary-press pointer-events-none absolute start-0 top-0 size-9 rounded-full motion-safe:transition-transform motion-safe:duration-200 motion-safe:ease-in-out translate-x-0 group-has-[[data-code-block-preview-toggle-preview]:hover]/code-block-preview-toggle:translate-x-[calc(100%+2px)] group-has-[[data-code-block-preview-toggle-preview]:focus-visible]/code-block-preview-toggle:translate-x-[calc(100%+2px)]"></div><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use></span><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use></span></div><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div data-custom-highlighting-behavior="boundary" class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative [container-type:inline-size]"><div class="w-full overflow-x-hidden overflow-y-auto" style="max-height: 56.25cqw; transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼo"><meta</span><span> </span><span class="ͼn">charset</span><span class="ͼg">=</span><span class="ͼk">"UTF-8"</span><span class="ͼo">></span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3508" data-end="3571">需要注意,不能看到中文就直接固定使用 UTF-8。更稳妥的方式是结合 HTTP 响应头、HTML 元信息以及实际网页情况判断。</p> <h2 data-section-id="2pgz6w" data-start="3573" data-end="3590">八、网页需要请求头时如何处理</h2> <p data-start="3592" data-end="3625">有些网站对没有浏览器特征的 HTTP 请求处理方式不同,直接执行:</p> <pre class="overflow-visible! px-0!" data-start="3627" data-end="3658"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">requests</span><span class="ͼg">.</span><span>get(</span><span class="ͼm">url</span><span>)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="3660" data-end="3685">可能返回异常页面、403 状态码或者提示访问受限。</p> <p data-start="3687" data-end="3704">可以设置常见的 HTTP 请求头:</p> <pre class="overflow-visible! px-0!" data-start="3706" data-end="4061"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼm">headers</span><span> </span><span class="ͼg">=</span><span> { </span><span class="ͼk">"User-Agent"</span><span>: ( </span><span class="ͼk">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "</span><span> </span><span class="ͼk">"AppleWebKit/537.36 (KHTML, like Gecko) "</span><span> </span><span class="ͼk">"Chrome/128.0.0.0 Safari/537.36"</span><span> ) } </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>get( </span><span class="ͼk">"https://example.com"</span><span>, </span><span class="ͼm">headers</span><span class="ͼg">=</span><span class="ͼm">headers</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span> ) </span><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>status_code) </span><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>text)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="4063" data-end="4120">请求头并不是“绕过所有限制”的方法。网站可能还会检查 Cookie、Referer、访问频率、身份验证状态等信息。</p> <p data-start="4122" data-end="4164">因此遇到访问限制时,应优先遵守网站的服务条款、robots 规则以及适用的法律法规。</p> <h2 data-section-id="14f7wla" data-start="4166" data-end="4186">九、动态网页为什么获取不到完整内容</h2> <p data-start="4188" data-end="4211">这是获取网页源代码过程中最容易产生误解的地方。</p> <p data-start="4213" data-end="4228">例如浏览器打开页面后能够看到:</p> <pre class="overflow-visible! px-0!" data-start="4230" data-end="4255"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>商品名称 价格 库存 评论</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="4257" data-end="4286">但使用 <code data-start="4261" data-end="4271">requests</code> 获取 HTML 后,却只有:</p> <pre class="overflow-visible! px-0!" data-start="4288" data-end="4320"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#fa6359" fill="currentColor"></use>HTML</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><div role="group" aria-label="代码块视图切换" class="group/code-block-preview-toggle relative flex items-center gap-0.5"><div aria-hidden="true" class="bg-token-interactive-bg-secondary-press pointer-events-none absolute start-0 top-0 size-9 rounded-full motion-safe:transition-transform motion-safe:duration-200 motion-safe:ease-in-out translate-x-0 group-has-[[data-code-block-preview-toggle-preview]:hover]/code-block-preview-toggle:translate-x-[calc(100%+2px)] group-has-[[data-code-block-preview-toggle-preview]:focus-visible]/code-block-preview-toggle:translate-x-[calc(100%+2px)]"></div><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use></span><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use></span></div><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div data-custom-highlighting-behavior="boundary" class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative [container-type:inline-size]"><div class="w-full overflow-x-hidden overflow-y-auto" style="max-height: 56.25cqw; transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼo"><div</span><span> </span><span class="ͼn">id</span><span class="ͼg">=</span><span class="ͼk">"app"</span><span class="ͼo">></div></span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="4322" data-end="4335">这并不意味着程序获取失败。</p> <p data-start="4337" data-end="4360">很可能网页采用了 JavaScript 渲染。</p> <p data-start="4362" data-end="4369">典型过程如下:</p> <pre class="overflow-visible! px-0!" data-start="4371" data-end="4524"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>浏览器请求 HTML ↓ 服务器返回基础页面 ↓ JavaScript 加载 ↓ 请求后端 API ↓ 获取 JSON 数据 ↓ JavaScript 修改 DOM ↓ 用户看到完整页面</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="4526" data-end="4585"><code data-start="4526" data-end="4536">requests</code> 只负责发送 HTTP 请求和接收响应,并不会像浏览器一样自动执行网页中的 JavaScript。</p> <h2 data-section-id="er1xng" data-start="4587" data-end="4603">十、动态网页的三种处理方式</h2> <h3 data-section-id="1pyrov0" data-start="4605" data-end="4621">方法一:直接寻找数据接口</h3> <p data-start="4623" data-end="4633">这是比较推荐的方式。</p> <p data-start="4635" data-end="4684">打开浏览器开发者工具,进入 Network → Fetch/XHR,刷新页面,然后观察返回的数据。</p> <p data-start="4686" data-end="4691">例如发现:</p> <pre class="overflow-visible! px-0!" data-start="4693" data-end="4725"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>/api/products?page=1</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="4727" data-end="4730">返回:</p> <pre class="overflow-visible! px-0!" data-start="4732" data-end="4838"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>JSON</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>{ "data": [ { "name": </span><span class="ͼk">"产品A"</span><span>, "price": </span><span class="ͼj">99</span><span> } ] }</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="4840" data-end="4872">那么程序可以直接请求该接口,而不是解析浏览器渲染后的 HTML。</p> <p data-start="4874" data-end="4897">这种方式通常结构更加清晰,数据处理效率也更高。</p> <h3 data-section-id="lvw9zf" data-start="4899" data-end="4918">方法二:使用 Selenium</h3> <p data-start="4920" data-end="4965">如果网页必须依赖 JavaScript 执行,可以使用 Selenium 控制真实浏览器。</p> <p data-start="4967" data-end="4970">安装:</p> <pre class="overflow-visible! px-0!" data-start="4972" data-end="5004"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Bash</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>pip install selenium</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="5006" data-end="5011">简单示例:</p> <pre class="overflow-visible! px-0!" data-start="5013" data-end="5204"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">from</span><span> </span><span class="ͼm">selenium</span><span> </span><span class="ͼg">import</span><span> </span><span class="ͼm">webdriver</span><span> </span><span class="ͼm">driver</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">webdriver</span><span class="ͼg">.</span><span>Chrome() </span><span class="ͼg">try</span><span>: </span><span class="ͼm">driver</span><span class="ͼg">.</span><span>get(</span><span class="ͼk">"https://example.com"</span><span>) </span><span class="ͼm">html</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">driver</span><span class="ͼg">.</span><span>page_source </span><span class="ͼm">print</span><span>(</span><span class="ͼm">html</span><span>) </span><span class="ͼg">finally</span><span>: </span><span class="ͼm">driver</span><span class="ͼg">.</span><span>quit()</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="5206" data-end="5210">这里的:</p> <pre class="overflow-visible! px-0!" data-start="5212" data-end="5244"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">driver</span><span class="ͼg">.</span><span>page_source</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="5246" data-end="5295">获取的是浏览器当前页面的 HTML 内容,因此对于依赖 JavaScript 渲染的网站更加有效。</p> <p data-start="5297" data-end="5342">不过 Selenium 启动浏览器的资源消耗较大,大规模任务中需要合理控制并发和访问频率。</p> <h3 data-section-id="baf3el" data-start="5344" data-end="5365">方法三:使用 Playwright</h3> <p data-start="5367" data-end="5391">Playwright 同样适合处理现代动态网页。</p> <p data-start="5393" data-end="5396">安装:</p> <pre class="overflow-visible! px-0!" data-start="5398" data-end="5451"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Bash</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>pip install playwright playwright install</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="5453" data-end="5456">示例:</p> <pre class="overflow-visible! px-0!" data-start="5458" data-end="5715"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">from</span><span> </span><span class="ͼm">playwright</span><span class="ͼg">.</span><span class="ͼm">sync_api</span><span> </span><span class="ͼg">import</span><span> </span><span class="ͼm">sync_playwright</span><span> </span><span class="ͼg">with</span><span> </span><span class="ͼm">sync_playwright</span><span>() </span><span class="ͼg">as</span><span> </span><span class="ͼm">p</span><span>: </span><span class="ͼm">browser</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">p</span><span class="ͼg">.</span><span>chromium</span><span class="ͼg">.</span><span>launch() </span><span class="ͼm">page</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">browser</span><span class="ͼg">.</span><span>new_page() </span><span class="ͼm">page</span><span class="ͼg">.</span><span>goto(</span><span class="ͼk">"https://example.com"</span><span>) </span><span class="ͼm">html</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">page</span><span class="ͼg">.</span><span>content() </span><span class="ͼm">print</span><span>(</span><span class="ͼm">html</span><span>) </span><span class="ͼm">browser</span><span class="ͼg">.</span><span>close()</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="5717" data-end="5769">相比传统 HTTP 请求方式,浏览器自动化工具更适合处理 JavaScript、异步加载和复杂页面交互。</p> <h2 data-section-id="axs6j" data-start="5771" data-end="5787">十一、登录页面的源代码获取</h2> <p data-start="5789" data-end="5822">如果目标网页需要登录,仅仅访问 URL 往往无法获得登录后的内容。</p> <p data-start="5824" data-end="5850">这种情况下,需要区分“登录页面”和“登录后的页面”。</p> <p data-start="5852" data-end="5920">对于自己有权限访问的网站,可以通过浏览器登录后分析 Network 请求,确认实际使用的 Cookie、Session 或其他认证机制。</p> <p data-start="5922" data-end="5950">Python 中可以使用 <code data-start="5935" data-end="5944">Session</code> 保持会话:</p> <pre class="overflow-visible! px-0!" data-start="5952" data-end="6188"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼm">session</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>Session() </span><span class="ͼm">session</span><span class="ͼg">.</span><span>headers</span><span class="ͼg">.</span><span>update({ </span><span class="ͼk">"User-Agent"</span><span>: </span><span class="ͼk">"Mozilla/5.0"</span><span> }) </span><span class="ͼm">session</span><span class="ͼg">.</span><span>get(</span><span class="ͼk">"https://example.com/login"</span><span>) </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">session</span><span class="ͼg">.</span><span>get(</span><span class="ͼk">"https://example.com/user"</span><span>) </span><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>text)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="6190" data-end="6252">实际登录流程可能还包含 CSRF Token、验证码、双因素认证等机制,因此不能简单假设提交用户名和密码就一定能够完成登录。</p> <p data-start="6254" data-end="6294">涉及第三方网站时,也应确保自己拥有相应访问权限,不要尝试绕过身份认证或安全控制。</p> <h2 data-section-id="fv2vtz" data-start="6296" data-end="6314">十二、网页源代码保存到本地文件</h2> <p data-start="6316" data-end="6339">如果需要长期分析,可以将 HTML 保存下来。</p> <pre class="overflow-visible! px-0!" data-start="6341" data-end="6555"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼm">url</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼk">"https://example.com"</span><span> </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>get(</span><span class="ͼm">url</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span>) </span><span class="ͼm">response</span><span class="ͼg">.</span><span>raise_for_status() </span><span class="ͼg">with</span><span> </span><span class="ͼm">open</span><span>(</span><span class="ͼk">"page.html"</span><span>, </span><span class="ͼk">"w"</span><span>, </span><span class="ͼm">encoding</span><span class="ͼg">=</span><span class="ͼk">"utf-8"</span><span>) </span><span class="ͼg">as</span><span> </span><span class="ͼm">file</span><span>: </span><span class="ͼm">file</span><span class="ͼg">.</span><span>write(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>text)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="6557" data-end="6571">随后可以直接使用浏览器打开:</p> <pre class="overflow-visible! px-0!" data-start="6573" data-end="6594"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>page.html</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="6596" data-end="6617">如果担心字符编码转换,也可以保存原始字节:</p> <pre class="overflow-visible! px-0!" data-start="6619" data-end="6703"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">with</span><span> </span><span class="ͼm">open</span><span>(</span><span class="ͼk">"page.html"</span><span>, </span><span class="ͼk">"wb"</span><span>) </span><span class="ͼg">as</span><span> </span><span class="ͼm">file</span><span>: </span><span class="ͼm">file</span><span class="ͼg">.</span><span>write(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>content)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="6705" data-end="6722">这种方式更适合保留服务器原始响应。</p> <h2 data-section-id="ho024i" data-start="6724" data-end="6742">十三、获取网页源代码时常见问题</h2> <h3 data-section-id="uwugf8" data-start="6744" data-end="6757">1. 返回 403</h3> <p data-start="6759" data-end="6766">常见原因包括:</p> <ul data-start="6768" data-end="6827"> <li data-section-id="1lry5pw" data-start="6768" data-end="6776"> <p data-start="6770" data-end="6776">请求频率过高</p> </li> <li data-section-id="1ari263" data-start="6777" data-end="6792"> <p data-start="6779" data-end="6792">请求头与正常浏览器差异较大</p> </li> <li data-section-id="wjfkbx" data-start="6793" data-end="6801"> <p data-start="6795" data-end="6801">网站要求登录</p> </li> <li data-section-id="nqh6cw" data-start="6802" data-end="6812"> <p data-start="6804" data-end="6812">网站存在访问控制</p> </li> <li data-section-id="13ttlev" data-start="6813" data-end="6827"> <p data-start="6815" data-end="6827">IP 或网络环境受到限制</p> </li> </ul> <p data-start="6829" data-end="6856">应该先确认访问权限和网站规则,而不是不断提高请求频率。</p> <h3 data-section-id="k4upr4" data-start="6858" data-end="6871">2. 返回 404</h3> <p data-start="6873" data-end="6897">404 通常说明请求 URL 对应的资源不存在。</p> <p data-start="6899" data-end="6904">需要检查:</p> <pre class="overflow-visible! px-0!" data-start="6906" data-end="6967"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>url) </span><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>status_code)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="6969" data-end="7002">有些网站还会进行重定向,因此最终 URL 可能与最初请求地址不同。</p> <h3 data-section-id="lozh3n" data-start="7004" data-end="7027">3. 获取到的 HTML 与浏览器不同</h3> <p data-start="7029" data-end="7034">优先检查:</p> <ol data-start="7036" data-end="7123"> <li data-section-id="12wzhcu" data-start="7036" data-end="7057"> <p data-start="7039" data-end="7057">页面是否依赖 JavaScript。</p> </li> <li data-section-id="ph1wv7" data-start="7058" data-end="7076"> <p data-start="7061" data-end="7076">数据是否由 API 异步获取。</p> </li> <li data-section-id="kct5h6" data-start="7077" data-end="7089"> <p data-start="7080" data-end="7089">是否存在登录状态。</p> </li> <li data-section-id="1g0jwgh" data-start="7090" data-end="7105"> <p data-start="7093" data-end="7105">是否需要 Cookie。</p> </li> <li data-section-id="1mradh0" data-start="7106" data-end="7123"> <p data-start="7109" data-end="7123">是否根据请求头返回不同内容。</p> </li> </ol> <h3 data-section-id="1xcsfk8" data-start="7125" data-end="7142">4. HTML 内容不完整</h3> <p data-start="7144" data-end="7188">如果响应本身只有部分结构,需要查看页面是否通过 JavaScript 动态加载其他内容。</p> <p data-start="7190" data-end="7243">不要简单地认为 <code data-start="7198" data-end="7208">requests</code> “漏掉了网页内容”,因为它获取的是服务器实际返回的 HTTP 响应。</p> <h2 data-section-id="26jgx4" data-start="7245" data-end="7262">十四、网页源代码获取方法对比</h2> <div class="group TyagGW_tableContainer"><div tabindex="-1" class="TyagGW_tableWrapper flex flex-col-reverse w-fit"><table data-start="7264" data-end="7562" class="w-fit min-w-(--thread-content-width)"><thead data-start="7264" data-end="7297"><tr data-start="7264" data-end="7297"><th data-start="7264" data-end="7269" data-col-size="sm" class="last:pe-10">方法</th><th data-start="7269" data-end="7276" data-col-size="sm" class="last:pe-10">适用场景</th><th data-start="7276" data-end="7289" data-col-size="sm" class="last:pe-10">JavaScript</th><th data-start="7289" data-end="7297" data-col-size="sm" class="last:pe-10">使用难度</th></tr></thead><tbody data-start="7316" data-end="7562"><tr data-start="7316" data-end="7353"><td data-start="7316" data-end="7327" data-col-size="sm">Ctrl + U</td><td data-col-size="sm" data-start="7327" data-end="7341">快速查看原始 HTML</td><td data-col-size="sm" data-start="7341" data-end="7347">不执行</td><td data-col-size="sm" data-start="7347" data-end="7353">很低</td></tr><tr data-start="7354" data-end="7393"><td data-start="7354" data-end="7371" data-col-size="sm">开发者工具 Elements</td><td data-col-size="sm" data-start="7371" data-end="7382">分析当前 DOM</td><td data-col-size="sm" data-start="7382" data-end="7388">已执行</td><td data-col-size="sm" data-start="7388" data-end="7393">低</td></tr><tr data-start="7394" data-end="7425"><td data-start="7394" data-end="7404" data-col-size="sm">Network</td><td data-col-size="sm" data-start="7404" data-end="7414">分析接口和资源</td><td data-col-size="sm" data-start="7414" data-end="7420">可观察</td><td data-col-size="sm" data-start="7420" data-end="7425">中</td></tr><tr data-start="7426" data-end="7458"><td data-start="7426" data-end="7437" data-col-size="sm">requests</td><td data-col-size="sm" data-start="7437" data-end="7447">静态网页、接口</td><td data-col-size="sm" data-start="7447" data-end="7453">不执行</td><td data-col-size="sm" data-start="7453" data-end="7458">低</td></tr><tr data-start="7459" data-end="7498"><td data-start="7459" data-end="7475" data-col-size="sm">BeautifulSoup</td><td data-col-size="sm" data-start="7475" data-end="7487">HTML 内容解析</td><td data-col-size="sm" data-start="7487" data-end="7493">不执行</td><td data-col-size="sm" data-start="7493" data-end="7498">低</td></tr><tr data-start="7499" data-end="7529"><td data-start="7499" data-end="7510" data-col-size="sm">Selenium</td><td data-col-size="sm" data-start="7510" data-end="7519">浏览器自动化</td><td data-col-size="sm" data-start="7519" data-end="7524">支持</td><td data-col-size="sm" data-start="7524" data-end="7529">中</td></tr><tr data-start="7530" data-end="7562"><td data-start="7530" data-end="7543" data-col-size="sm">Playwright</td><td data-col-size="sm" data-start="7543" data-end="7552">现代动态网页</td><td data-col-size="sm" data-start="7552" data-end="7557">支持</td><td data-col-size="sm" data-start="7557" data-end="7562">中</td></tr></tbody></table><div class="relative h-0 self-end select-none"><div class="absolute end-0 flex items-end" style="height: 33px"><span class="" data-state="closed"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></span></div></div></div></div> <p data-start="7564" data-end="7670">选择方法时,不应该一味追求复杂工具。静态页面使用 <code data-start="7589" data-end="7599">requests</code> 已经足够,动态页面则应该优先判断是否存在可直接调用的数据接口;确实依赖浏览器执行环境时,再考虑 Selenium 或 Playwright。</p> <h2 data-section-id="tle7pl" data-start="7672" data-end="7692">十五、网页源代码获取的实用排查流程</h2> <p data-start="7694" data-end="7722">遇到一个网页无法正常获取时,可以按照下面的顺序进行排查:</p> <pre class="overflow-visible! px-0!" data-start="7724" data-end="7932"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="pointer-events-none absolute end-1.5 top-1 z-2 md:end-2 md:top-1"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use></div><div class="relative"><div class="pe-11 pt-3" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span>浏览器打开网页 ↓ 查看“网页源代码” ↓ 判断目标内容是否存在 ↓ 存在 → 直接使用 HTTP 请求获取 ↓ 不存在 ↓ 检查 Elements 和 Network ↓ 是否存在数据接口? ↓ ↓ 是 否 ↓ ↓ 调用接口 使用浏览器自动化</span></code></pre></div></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></pre> <p data-start="7934" data-end="7968">这种思路比一开始就使用 Selenium 更高效,也更容易定位问题。</p> <h2 data-section-id="jte8la" data-start="7970" data-end="7986">十六、实际开发中的注意事项</h2> <p data-start="7988" data-end="8051">获取网页源代码并不只是把 HTML 下载下来。真正稳定的程序还需要考虑网络超时、重定向、字符编码、异常响应以及资源释放等问题。</p> <p data-start="8053" data-end="8056">例如:</p> <pre class="overflow-visible! px-0!" data-start="8058" data-end="8424"><div class="relative w-full mt-4 mb-1"><div class=""><div class="contents"><div class="border border-token-border-light border-radius-3xl corner-superellipse/1.1 rounded-3xl"><div class="relative h-full w-full border-radius-3xl bg-(--code-block-surface) corner-superellipse/1.1 overflow-clip rounded-3xl [--code-block-surface:var(--bg-elevated-secondary)] dark:[--code-block-surface:var(--composer-surface-primary)] lxnfua_clipPathFallback"><div class="select-none sticky z-2 top-(--sticky-padding-top)"><div class="flex w-full items-center justify-between py-1.5 ps-4 pe-1.5 font-sans md:ps-5 bg-(--code-block-surface)"><div class="flex max-w-[75%] min-w-0 cursor-default items-center text-sm font-medium justify-self-start text-token-text-primary"><use href="/cdn/assets/sprites-core-fe545f09.svg#72c6a7" fill="currentColor"></use>Python</div><div class="flex flex-row items-center gap-0.5 justify-self-end"><use href="/cdn/assets/sprites-core-fe545f09.svg#7fccfb" fill="currentColor"></use><div class="flex w-full items-center justify-center gap-1.5"><use href="/cdn/assets/sprites-core-fe545f09.svg#6ef7a4" fill="currentColor"></use>运行</div></div></div></div><div class="pointer-events-none absolute inset-x-4 top-12 bottom-4"><div class="pointer-events-none sticky z-40 shrink-0 z-1!" style="top: calc(calc(var(--sticky-padding-top) + 48px) - 1px * 3); margin-bottom: calc(-4px); height: calc(4px); mask-image: linear-gradient(transparent 25%, white 75%)"><div class="sticky bg-token-border-light" style="top: calc(var(--sticky-padding-top) + 48px); height: 1px"></div></div></div><div class="relative"><div class="h-full min-h-0 min-w-0"><div class="h-full min-h-0 min-w-0" style="width: 100%; transform: none"><div class=""><div class="relative"><div class="" style="transform: none"><div class="relative z-0 flex max-w-full"><div id="code-block-viewer" dir="ltr" class="q9tKkq_viewer cm-editor z-10 light:cm-light dark:cm-light flex h-full w-full flex-col items-stretch ͼd ͼr"><div class="cm-scroller"><pre class="cm-content q9tKkq_readonly m-0"><code><span class="ͼg">import</span><span> </span><span class="ͼm">requests</span><span> </span><span class="ͼm">url</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼk">"https://example.com"</span><span> </span><span class="ͼg">try</span><span>: </span><span class="ͼm">response</span><span> </span><span class="ͼg">=</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>get( </span><span class="ͼm">url</span><span>, </span><span class="ͼm">timeout</span><span class="ͼg">=</span><span class="ͼj">10</span><span>, </span><span class="ͼm">headers</span><span class="ͼg">=</span><span>{ </span><span class="ͼk">"User-Agent"</span><span>: </span><span class="ͼk">"Mozilla/5.0"</span><span> } ) </span><span class="ͼm">response</span><span class="ͼg">.</span><span>raise_for_status() </span><span class="ͼm">print</span><span>(</span><span class="ͼm">response</span><span class="ͼg">.</span><span>text) </span><span class="ͼg">except</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>Timeout: </span><span class="ͼm">print</span><span>(</span><span class="ͼk">"请求超时"</span><span>) </span><span class="ͼg">except</span><span> </span><span class="ͼm">requests</span><span class="ͼg">.</span><span>RequestException </span><span class="ͼg">as</span><span> </span><span class="ͼm">e</span><span>: </span><span class="ͼm">print</span><span>(</span><span class="ͼk">f"请求失败:</span><span>{</span><span class="ͼm">e</span><span>}</span><span class="ͼk">"</span><span>)</span></code></pre></div></div></div></div></div></div></div></div><div class=""><div class="" style="width: 100%; transform: none"></div></div></div></div></div></div></div></div></pre> <p data-start="8426" data-end="8496">批量处理网页时,还应该设置合理的请求间隔,避免短时间产生大量请求。对于需要身份认证、验证码或其他访问控制的页面,应通过正常授权方式完成访问。</p> <h2 data-section-id="11496um" data-start="8498" data-end="8506">十七、总结</h2> <p data-start="8508" data-end="8652">网页源代码获取可以从浏览器操作和程序自动化两个方向入手。简单查看页面时,<code data-start="8544" data-end="8554">Ctrl + U</code> 是最快的方法;需要分析 JavaScript 执行后的页面结构,可以使用开发者工具 Elements;如果需要知道动态数据的来源,则应该重点查看 Network 中的 Fetch/XHR 请求。</p> <p data-start="8654" data-end="8792">Python 场景下,静态网页通常可以使用 <code data-start="8676" data-end="8686">requests</code> 获取 HTML,再结合 BeautifulSoup 完成解析。对于 JavaScript 动态渲染页面,应先寻找后端数据接口,只有在确实需要浏览器执行环境时,再使用 Selenium 或 Playwright。</p> <p data-start="8794" data-end="8901">尤其需要注意,<strong data-start="8801" data-end="8838">原始网页源代码、接口响应和浏览器最终 DOM 是三个不同层面的内容</strong>。理解它们之间的关系,才能针对不同网站选择合适的获取方式,也能避免“浏览器明明有内容,但 Python 获取不到”的常见问题。</p> </div> </div> </article> <div class="bg-white rounded-lg shadow-card border border-slate-100 p-5 mb-6"> <div class="flex items-center gap-2 mb-2.5"> <i class="fa-solid fa-tags text-slate-400 text-xs"></i> <span class="text-sm font-semibold text-slate-700">文章标签</span> </div> <div class="flex flex-wrap gap-2"> <a href="https://www.nzw6.cn/tag/网页源代码" class="bg-slate-50 hover:bg-primary hover:text-white text-slate-600 text-xs px-3 py-1.5 rounded-lg font-medium border border-slate-200 hover:border-primary transition-custom">网页源代码</a> <a href="https://www.nzw6.cn/tag/HTML" class="bg-slate-50 hover:bg-primary hover:text-white text-slate-600 text-xs px-3 py-1.5 rounded-lg font-medium border border-slate-200 hover:border-primary transition-custom">HTML</a> <a href="https://www.nzw6.cn/tag/Python" class="bg-slate-50 hover:bg-primary hover:text-white text-slate-600 text-xs px-3 py-1.5 rounded-lg font-medium border border-slate-200 hover:border-primary transition-custom">Python</a> <a href="https://www.nzw6.cn/tag/requests" class="bg-slate-50 hover:bg-primary hover:text-white text-slate-600 text-xs px-3 py-1.5 rounded-lg font-medium border border-slate-200 hover:border-primary transition-custom">requests</a> <a href="https://www.nzw6.cn/tag/Selenium" class="bg-slate-50 hover:bg-primary hover:text-white text-slate-600 text-xs px-3 py-1.5 rounded-lg font-medium border border-slate-200 hover:border-primary transition-custom">Selenium</a> </div> </div> <style> #prev-next-bar { margin-bottom: 24px; padding: 20px; background: #fff; border: 1px solid #f1f5f9; border-radius: 8px; box-shadow: 0 2px 8px rgba(0, 0, 0, .04); } #prev-next-bar .article-nav { display: flex; gap: 24px; } #prev-next-bar a.article-nav-link { display: flex; align-items: center; gap: 8px; flex: 1 1 0; min-width: 0; margin: 0; padding: 0; color: #475569; font-size: 14px; text-decoration: none; } #prev-next-bar .article-nav-next { justify-content: flex-end; } #prev-next-bar .article-nav-label, #prev-next-bar .article-nav-arrow { flex: 0 0 auto; color: #94a3b8; white-space: nowrap; } #prev-next-bar .article-nav-title { display: block; min-width: 0; overflow: hidden; white-space: nowrap; text-overflow: ellipsis; } #prev-next-bar a.article-nav-link:hover { color: #2563eb; } @media (max-width: 640px) { #prev-next-bar .article-nav { flex-direction: column; gap: 14px; } #prev-next-bar .article-nav-next { justify-content: flex-start; } } </style> <div id="prev-next-bar"> <div class="article-nav"> <a class="article-nav-link" href="/kaifa/1440.html"> <i class="article-nav-arrow fa-solid fa-chevron-left"></i> <span class="article-nav-label">上一篇:</span> <span class="article-nav-title">Windows 11系统定时关机实现方法详解</span> </a> <a class="article-nav-link article-nav-next" href="/kaifa/1442.html"> <span class="article-nav-label">下一篇:</span> <span class="article-nav-title">Excel VBA自动筛选:基于多列条件统计行数</span> <i class="article-nav-arrow fa-solid fa-chevron-right"></i> </a> </div> </div> <div class="related-recommend bg-white rounded-lg shadow-card border border-slate-100 p-6" style="display:none;"> <div class="flex items-center gap-3 mb-5"> <h3 class="text-base font-semibold text-slate-900">相关文章</h3> <div class="flex-1 h-px bg-slate-100"></div> </div> <div class="space-y-2"> <a href="/kaifa/1450.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">Python包管理:使用pip安装、升级与卸载指定版本包</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1449.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">在VS Code中配置Python开发环境的完整指南</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1441.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">网页源代码获取方法及特殊场景处理</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1438.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">Seaborn kdeplot中bw_adjust参数详解</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1422.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">Python爬虫实战:豆瓣电影Top250数据采集指南</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1409.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">解决ValueError: Columns must be same length as key错误</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> </div> </div> <div class="random-recommend bg-white rounded-lg shadow-card border border-slate-100 p-6" style="display:none;"> <div class="flex items-center gap-3 mb-5"> <h3 class="text-base font-semibold text-slate-900">推荐阅读</h3> <div class="flex-1 h-px bg-slate-100"></div> </div> <div class="space-y-2"> <a href="/kaifa/1419.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">C语言求绝对值的方法详解</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1390.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">带HSM的MCU在车载系统中的安全应用解析</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/845.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">基于Qt QGraphicsView框架的数据流可视化工具设计</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1381.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">ComfyUI中Qwen2-VL模型加载失败的解决方案</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/1382.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">网易云音乐IP属地显示规则与位置信息获取方案</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/kaifa/744.html" class="flex items-center justify-between p-3 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-3">微信小程序view元素自动换行设置</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> </div> </div> </div> <aside class="w-full space-y-5" aria-label="相关推荐"> <a href="/fuwuqi/" class="group relative block min-h-[250px] overflow-hidden rounded-lg border border-red-200/40 bg-[#b51f2c] p-5 text-white shadow-card transition-custom hover:-translate-y-0.5 hover:shadow-card-hover"> <img src="/static/images/server-promo-bg.svg" alt="" aria-hidden="true" loading="lazy" decoding="async" class="absolute inset-0 h-full w-full object-cover"> <div class="absolute inset-0 bg-gradient-to-r from-[#741126]/95 via-[#a71927]/82 to-[#d7352a]/20"></div> <div class="relative z-10 flex min-h-[210px] flex-col"> <div class="mb-3 text-sm font-semibold text-amber-100">阿里云 · 腾讯云</div> <h3 class="text-2xl font-bold leading-tight">云服务器特惠</h3> <div class="my-3 flex items-end gap-1 text-[#ffe179]"> <span class="pb-1 text-lg font-bold">¥</span> <strong class="text-[38px] leading-none tracking-tight">99</strong> <span class="pb-1 text-sm font-semibold text-white">/年起</span> </div> <p class="text-base font-medium text-red-50">国内、香港及海外机型可选</p> <span class="mt-auto inline-flex w-fit items-center gap-2 rounded-lg bg-white px-4 py-2.5 text-base font-bold text-[#a71927] shadow-md transition-custom group-hover:bg-[#fff4d6]"> 查看优惠 <i class="fa-solid fa-arrow-right text-xs transition-transform group-hover:translate-x-1"></i> </span> </div> </a> <div class="bg-white rounded-lg shadow-card border border-slate-100 p-5"> <div class="flex items-center gap-3 mb-4"> <h3 class="text-sm font-semibold text-slate-900">推荐模板</h3> <div class="flex-1 h-px bg-slate-100"></div> </div> <div class="space-y-2.5"> <a href="/templates/1103.html" class="flex items-center gap-3 p-2 rounded-lg hover:bg-slate-50 transition-custom group"> <div class="w-14 h-14 shrink-0 rounded-lg overflow-hidden"> <img src="/static/upload/image/20260902/s736.webp" alt="律师事务所网站模板整站源码" width="56" height="56" loading="lazy" decoding="async" class="w-full h-full object-cover"> </div> <div class="flex-1 min-w-0"> <div class="text-[14px] text-slate-700 group-hover:text-primary transition-custom line-clamp-2 font-medium leading-snug">律师事务所网站模板整站源码</div> <div class="text-xs text-red-500 font-bold mt-1">¥80</div> </div> </a> <a href="/templates/1102.html" class="flex items-center gap-3 p-2 rounded-lg hover:bg-slate-50 transition-custom group"> <div class="w-14 h-14 shrink-0 rounded-lg overflow-hidden"> <img src="/static/upload/image/20260902/s735.webp" alt="财务财税企业记账网站模板整站源码" width="56" height="56" loading="lazy" decoding="async" class="w-full h-full object-cover"> </div> <div class="flex-1 min-w-0"> <div class="text-[14px] text-slate-700 group-hover:text-primary transition-custom line-clamp-2 font-medium leading-snug">财务财税企业记账网站模板整站源码</div> <div class="text-xs text-red-500 font-bold mt-1">¥80</div> </div> </a> <a href="/templates/1101.html" class="flex items-center gap-3 p-2 rounded-lg hover:bg-slate-50 transition-custom group"> <div class="w-14 h-14 shrink-0 rounded-lg overflow-hidden"> <img src="/static/upload/image/20260902/s734.webp" alt="招标投标信息网站模板整站源码" width="56" height="56" loading="lazy" decoding="async" class="w-full h-full object-cover"> </div> <div class="flex-1 min-w-0"> <div class="text-[14px] text-slate-700 group-hover:text-primary transition-custom line-clamp-2 font-medium leading-snug">招标投标信息网站模板整站源码</div> <div class="text-xs text-red-500 font-bold mt-1">¥80</div> </div> </a> <a href="/templates/1100.html" class="flex items-center gap-3 p-2 rounded-lg hover:bg-slate-50 transition-custom group"> <div class="w-14 h-14 shrink-0 rounded-lg overflow-hidden"> <img src="/static/upload/image/20260902/s733.webp" alt="机电设备网站模板 - 产品带视频 带三级栏目" width="56" height="56" loading="lazy" decoding="async" class="w-full h-full object-cover"> </div> <div class="flex-1 min-w-0"> <div class="text-[14px] text-slate-700 group-hover:text-primary transition-custom line-clamp-2 font-medium leading-snug">机电设备网站模板 - 产品带视频 带三级栏目</div> <div class="text-xs text-red-500 font-bold mt-1">¥80</div> </div> </a> <a href="/templates/1099.html" class="flex items-center gap-3 p-2 rounded-lg hover:bg-slate-50 transition-custom group"> <div class="w-14 h-14 shrink-0 rounded-lg overflow-hidden"> <img src="/static/upload/image/20260821/1787310069259663.webp" alt="LED照明灯具网站模板整站源码 - 带三级栏目" width="56" height="56" loading="lazy" decoding="async" class="w-full h-full object-cover"> </div> <div class="flex-1 min-w-0"> <div class="text-[14px] text-slate-700 group-hover:text-primary transition-custom line-clamp-2 font-medium leading-snug">LED照明灯具网站模板整站源码 - 带三级栏目</div> <div class="text-xs text-red-500 font-bold mt-1">¥80</div> </div> </a> </div> </div> <div class="bg-white rounded-lg shadow-card border border-slate-100 p-5"> <div class="flex items-center gap-3 mb-4"> <h3 class="text-sm font-semibold text-slate-900">推荐服务</h3> <div class="flex-1 h-px bg-slate-100"></div> </div> <div class="space-y-2"> <a href="/service/33.html" class="flex items-center justify-between p-2.5 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-2 font-medium">PbootCMS被黑恢复与安全加固</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/service/1143.html" class="flex items-center justify-between p-2.5 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-2 font-medium">WordPress插件开发定制</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/service/1172.html" class="flex items-center justify-between p-2.5 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-2 font-medium">协会官网制作</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/service/1222.html" class="flex items-center justify-between p-2.5 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-2 font-medium">证书查询系统开发</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> <a href="/service/1231.html" class="flex items-center justify-between p-2.5 rounded-lg hover:bg-slate-50 transition-custom group"> <span class="text-sm text-slate-600 group-hover:text-primary transition-custom line-clamp-1 flex-1 mr-2 font-medium">酒店预订小程序开发</span> <i class="fa-solid fa-angle-right text-slate-300 group-hover:text-primary transition-custom text-xs"></i> </a> </div> </div> </aside> </div> </div> </section> </main> <footer class="bg-dark text-slate-400 py-8 lg:py-10"> <div class="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8"> <div class="grid grid-cols-2 md:grid-cols-4 gap-6 lg:gap-8 mb-6"> <div class="col-span-2 md:col-span-1"> <div class="mb-4"> <img src="/2022022108032469.png" alt="牛站云"/> </div> <p class="text-[15px] leading-relaxed text-slate-400">牛站云专注于互联网数字化服务,提供网站建设、APP开发、小程序开发、系统定制、功能插件及云服务器等产品与服务,助力企业和个人快速开展线上业务。</p> </div> <div> <h4 class="text-[15px] font-semibold text-white mb-4">技术方案</h4> <ul class="space-y-3 text-[15px]"> <li><a href="/wangzhanjianshe/" class="hover:text-white transition-colors duration-200 line-clamp-1">国内企业官网建设整体解决方案</a></li> <li><a href="/waimaodulizhan/" class="hover:text-white transition-colors duration-200 line-clamp-1">外贸独立站英文多语言营销型官网建设</a></li> <li><a href="/xitongdingzhi/" class="hover:text-white transition-colors duration-200 line-clamp-1">企业业务系统定制</a></li> <li><a href="/xiangmuweihu/" class="hover:text-white transition-colors duration-200 line-clamp-1">已有项目接手与升级</a></li> </ul> </div> <div> <h4 class="text-[15px] font-semibold text-white mb-4">推荐阅读</h4> <ul class="space-y-3 text-[15px]"> <li><a href="/kaifa/1382.html" class="hover:text-white transition-colors duration-200 line-clamp-1">网易云音乐IP属地显示规则与位置信息获取方案</a></li> <li><a href="/kaifa/894.html" class="hover:text-white transition-colors duration-200 line-clamp-1">JVM初始化错误:VM agent library加载失败解决方案</a></li> <li><a href="/kaifa/920.html" class="hover:text-white transition-colors duration-200 line-clamp-1">C++中实现打印输出的两种常用方法</a></li> <li><a href="/kaifa/734.html" class="hover:text-white transition-colors duration-200 line-clamp-1">C++中DLL导出/导入符号与类注册的宏定义解析</a></li> <li><a href="/kaifa/720.html" class="hover:text-white transition-colors duration-200 line-clamp-1">JavaScript设计个人网页:从结构到交互</a></li> </ul> </div> <div class="col-span-2 md:col-span-1"> <h4 class="text-[15px] font-semibold text-white mb-4">客服微信</h4> <div class="flex gap-4"> <div class="flex-1 text-center"> <div class="bg-white rounded-lg p-2 mb-2"> <img src="/static/upload/image/20260713/1783949484149422.png" alt="定制开发" class="w-full aspect-square object-contain"> </div> <span class="text-sm text-slate-500">定制开发</span> </div> <div class="flex-1 text-center"> <div class="bg-white rounded-lg p-2 mb-2"> <img src="/static/upload/image/20260713/1783949541877769.png" alt="业务咨询" class="w-full aspect-square object-contain"> </div> <span class="text-sm text-slate-500">业务咨询</span> </div> </div> </div> </div> <div class="border-t border-slate-700/60 pt-8 flex flex-col md:flex-row justify-between items-center gap-4"> <p class="text-sm text-slate-500">© 2026 牛站云. 保留所有权利。</p> <div class="flex flex-wrap items-center gap-x-6 gap-y-2 text-sm text-slate-500"> <a href="https://beian.miit.gov.cn/" rel="nofollow" target="_blank" class="hover:text-white transition-colors duration-200">闽ICP备18018384号-6</a> <span>闽公网安备35020602002865号</span> <a href="https://www.nzw6.cn/privacy.html" class="hover:text-white transition-colors duration-200">隐私政策</a> <a href="https://www.nzw6.cn/terms.html" class="hover:text-white transition-colors duration-200">服务条款</a> <a href="https://www.nzw6.cn/cookies.html" class="hover:text-white transition-colors duration-200">Cookie 政策</a> </div> </div> </div> </footer> <script> var _hmt = _hmt || []; (function() { var hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?46998839740903d8bc8aa57f51751056"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); </script> <script> document.addEventListener('DOMContentLoaded', function() { document.querySelectorAll('.article-content pre').forEach(function(pre) { if (pre.querySelector('.code-copy-btn')) return; // 已有则跳过 var btn = document.createElement('button'); btn.className = 'code-copy-btn'; btn.setAttribute('title', '复制代码'); btn.innerHTML = '<i class="fa-regular fa-copy"></i>'; btn.addEventListener('click', function() { var code = pre.querySelector('code') ? pre.querySelector('code') : pre; var text = code.textContent || code.innerText; var success = function() { btn.innerHTML = '<i class="fa-solid fa-check"></i>'; btn.classList.add('copied'); setTimeout(function() { btn.innerHTML = '<i class="fa-regular fa-copy"></i>'; btn.classList.remove('copied'); }, 2000); }; if (navigator.clipboard && navigator.clipboard.writeText) { navigator.clipboard.writeText(text).then(success); } else { var textarea = document.createElement('textarea'); textarea.value = text; textarea.style.cssText = 'position:fixed;left:-9999px'; document.body.appendChild(textarea); textarea.select(); document.execCommand('copy'); document.body.removeChild(textarea); success(); } }); pre.appendChild(btn); }); var prevLink = document.querySelector('#prev-next-bar .prev-link'); var prevEmpty = document.querySelector('#prev-next-bar .prev-empty'); if (prevLink) { var ptxt = prevLink.textContent.replace('上一篇', '').trim(); if (!ptxt || /[\[\{<]/.test(ptxt)) { prevLink.style.display = 'none'; if (prevEmpty) prevEmpty.style.display = ''; } } var nextLink = document.querySelector('#prev-next-bar .next-link'); var nextEmpty = document.querySelector('#prev-next-bar .next-empty'); if (nextLink) { var ntxt = nextLink.textContent.replace('下一篇', '').trim(); if (!ntxt || /[\[\{<]/.test(ntxt)) { nextLink.style.display = 'none'; if (nextEmpty) nextEmpty.style.display = ''; } } var relatedBox = document.querySelector('.related-recommend'); var randomBox = document.querySelector('.random-recommend'); if (relatedBox && randomBox) { var links = relatedBox.querySelectorAll('a[href]'); var validCount = 0; for (var i = 0; i < links.length; i++) { var linkText = links[i].textContent.trim(); if (linkText && linkText.indexOf('[list:title]') === -1 && linkText.indexOf('{') === -1) { validCount++; } } if (validCount > 1) { relatedBox.style.display = ''; } else { randomBox.style.display = ''; } } }); </script> <script src='/Spider/?url=%2Fkaifa%2F1441.html' async='async'></script> <script src='/?p=/Do/visits/id/1441' async='async'></script> </body> </html>