一、技术方案源于哪些需求
目标：给出一个网页 URL，最终得到正文 Markdown，其中不再包含侧边栏/广告/导航。
技术选型：
- readability（readability-lxml）做正文去噪
- markdownify 把清洗后的 HTML 转 Markdown
- 预清理由 BeautifulSoup 负责
- 客户端渲染站点交给 Playwright
Doubao-Seed-Evolving 通过一次对话生成了代码初版；经过提炼的关键实现和踩坑内容，将在下文展开。
二、初版实现
这是我输入提示词后AI给的回答：
以下为核心结构节选：
import requestsfrom bs4 import BeautifulSoupfrom readability import Documentimport markdownify as mdDEFAULT_UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ..."def fetch_url(url, timeout=15, verify_ssl=True):if not url.startswith(("http://", "https://")):url = "https://" + urlheaders = {"User-Agent": DEFAULT_UA}resp = requests.get(url, headers=headers, timeout=timeout, verify=verify_ssl)resp.raise_for_status()return resp.textdef extract_markdown(html, page_url=""):soup = BeautifulSoup(html, "lxml")for tag in soup.find_all(["script", "style", "nav", "aside", "footer"]):tag.decompose()doc = Document(str(soup), url=page_url)content_html = doc.summary()return md(content_html, heading_style="ATX", convert=["table", "pre"])
注意：第一版有个坑——markdownify 不能同传 convert 和 strip，否则运行时直接报错。预清理已处理 script/style，所以删掉 strip 即可。这个修复也是模型在看到报错后给出的。
三、三类网站的实测结果
测试对象涵盖 3 个不同类型的网页，结果在此按实际情况记录：
| 网站类型 | 结果 | 现象 |
|---|
| SSR 技术博客 | ✅ | 完整保留代码块、语言标注和表格 |
| 公众号 | ⚠️ | 标题显示 [no-title]，正文则为 OK |
| 头条（CSR） | ❌ | 正文没有内容，仅出现 # [no-title] |
测试的三个网页中，一个直接成功，一个缺失标题，另一个完全只剩空壳。第一版并没有框架预设中那么糟糕，但实际问题更有意思，因为不同网站分别暴露了不同短板。
四、标题提取退化的关键修复
og:title 不在 readability-lxml 的 Document.summary() 读取范围内，它只提取 </code>。解析 bug 会被微信文章的标题格式触发，结果便是 [no-title]。</p><p></p><p>修复：语义化标题提取应作为一层处理，置于 extract_markdown 之前。</p><p><code><span class="hljs-keyword">def</span> <span class="hljs-title function_">extract_title</span>(<span class="hljs-params">soup, doc</span>):og = soup.find(<span class="hljs-string">"meta"</span>, <span class="hljs-built_in">property</span>=<span class="hljs-string">"og:title"</span>)<span class="hljs-keyword">if</span> og <span class="hljs-keyword">and</span> og.get(<span class="hljs-string">"content"</span>):<span class="hljs-keyword">return</span> og[<span class="hljs-string">"content"</span>].strip()tw = soup.find(<span class="hljs-string">"meta"</span>, attrs={<span class="hljs-string">"name"</span>: <span class="hljs-string">"twitter:title"</span>})<span class="hljs-keyword">if</span> tw <span class="hljs-keyword">and</span> tw.get(<span class="hljs-string">"content"</span>):<span class="hljs-keyword">return</span> tw[<span class="hljs-string">"content"</span>].strip()h1 = soup.find(<span class="hljs-string">"h1"</span>)<span class="hljs-keyword">if</span> h1 <span class="hljs-keyword">and</span> h1.get_text(strip=<span class="hljs-literal">True</span>):<span class="hljs-keyword">return</span> h1.get_text(strip=<span class="hljs-literal">True</span>)<span class="hljs-keyword">return</span> doc.short_title()</code></pre><h2>五、关键修复二：客户端渲染站点的空壳</h2><p>头条这类站点正文由 JS 异步加载，requests 拿到的只是 <code><div id="root"></code> 空壳。</p><p></p><p>修复：通过 --render 参数启用 Playwright 无头浏览器，完成渲染等待后再开始提取。</p><p><code><span class="hljs-keyword">from</span> playwright.sync_api <span class="hljs-keyword">import</span> sync_playwright<span class="hljs-keyword">def</span> <span class="hljs-title function_">fetch_rendered</span>(<span class="hljs-params">url</span>):<span class="hljs-keyword">with</span> sync_playwright() <span class="hljs-keyword">as</span> p:browser = p.chromium.launch(headless=<span class="hljs-literal">True</span>)page = browser.new_page()page.goto(url, wait_until=<span class="hljs-string">"networkidle"</span>)html = page.content()browser.close()<span class="hljs-keyword">return</span> html</code></pre><p>另：微信/头条图片是懒加载，真实地址在 data-src，需补一段把 data-src 回填到 src，否则全文裂图。</p><h2>六、最终使用方法</h2><p><code>python extract_article.py https://juejin.cn/post/xxx -o article.mdpython extract_article.py https://www.toutiao.com/xxx --render -o toutiao.md</code></pre><p>静态站不需要额外依赖，动态站加 --render。</p>
</div>
<!-- -->
<div class="content_btn_item clearfix">
<span>喜欢(0)</span>
</div>
<!-- -->
<div class="content_page">
<a class="img_box" href="/news/3432295.html">
<div class="bg">
<p class="page">上一篇</p>
<p class="info text_overflow_2">盗墓笔记启程积尸地怎么过 积尸地副本打法流程攻略</p>
</div>
<img src="https://www.0405.net/uploads/20260729/logo_6a696f4f705431.jpg" alt="盗墓笔记启程积尸地怎么过 积尸地副本打法流程攻略">
</a>
<a class="img_box" href="/news/3432297.html">
<div class="bg">
<p class="page">下一篇</p>
<p class="info text_overflow_2">重生之我成为模型 第三篇 · 我只提供可能性，答案由你们翻译</p>
</div>
<img src="https://www.0405.net/uploads/20260729/logo_6a696f7b8a56f1.webp" alt="重生之我成为模型 第三篇 · 我只提供可能性，答案由你们翻译">
</a>
</div>
</div>
<!-- 列表 -->
<div style="padding-bottom: 10px;">
<div class="px15" style="font-weight: bold;font-size: .36rem;padding-top: .3rem;">猜你喜欢</div>
<ul class="px15 news_box">
<li>
<a class="clearfix" href="/news/3433228.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69b20d0c4be1.webp">
</span>
<div class="content">
<p class="title text_overflow_2">电影《亚瑟王与圆桌骑士》剧情介绍</p>
<p class="text"><span class="icon time">2026-07-29</span> <span class="icon heart">3433228</span></p>
</div>
</a>
</li>
<li>
<a class="clearfix" href="/news/3433227.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69b205e404f1.webp">
</span>
<div class="content">
<p class="title text_overflow_2">电视剧《实习医生风云第一季》剧情介绍</p>
<p class="text"><span class="icon time">2026-07-29</span> <span class="icon heart">3433227</span></p>
</div>
</a>
</li>
<li>
<a class="clearfix" href="/news/3433226.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69b1df45ea11.webp">
</span>
<div class="content">
<p class="title text_overflow_2">高效挑选pdf翻译成中文 免费在线工具的五个技巧</p>
<p class="text"><span class="icon time">2026-07-29</span> <span class="icon heart">3433226</span></p>
</div>
</a>
</li>
<li>
<a class="clearfix" href="/news/3433225.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69b1d85d5201.webp">
</span>
<div class="content">
<p class="title text_overflow_2">提升工作效率并满足市场需求的pdf翻译策略</p>
<p class="text"><span class="icon time">2026-07-29</span> <span class="icon heart">3433225</span></p>
</div>
</a>
</li>
<li>
<a class="clearfix" href="/news/3433224.html">
<span class="img_item">
<img lazy-src="https://www.0405.net/uploads/20260729/logo_6a69b1d06c5b81.webp">
</span>
<div class="content">
<p class="title text_overflow_2">高效PDF在线翻译与企业翻译质量和数据安全保障</p>
<p class="text"><span class="icon time">2026-07-29</span> <span class="icon heart">3433224</span></p>
</div>
</a>
</li>
</ul>
</div>
<!-- 底部 -->
<div class="footer_box">
<p>Copyright © 2022 漫资讯 · Powered By WordPress</p>
</div>
<div class="back_top" id="back_top"><a href="javascript:;"></a></div>
</body>
<script src="/mob/js/jquery.js" type="text/javascript" charset="utf-8"></script>
<script src="/mob/js/lazy.js" type="text/javascript" charset="utf-8"></script>
<script src="/mob/js/flexible.js" type="text/javascript" charset="utf-8"></script>
<script src="/mob/js/index.js" type="text/javascript" charset="utf-8"></script>
<script type="text/javascript">
var _paq = window._paq = window._paq || [];
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//tongji.zhangwan.net/";
_paq.push(['setTrackerUrl', u+'matomo.php']);
_paq.push(['setSiteId', '9']);
// Add this code below within the Matomo JavaScript tracker code
// Important: the tracker url includes the /matomo.php
var secondaryTrackerUrl = u+'matomo.php';
var secondaryWebsiteId = 27;
// Also send all of the tracking data to this other Matomo server, in website ID 77
_paq.push(['addTracker', secondaryTrackerUrl, secondaryWebsiteId]);
// That's it!
var d=document,
g=d.createElement('script'),
s=d.getElementsByTagName('script')[0];
g.type='text/javascript';
g.async=true;
g.src=u+'matomo.js';
s.parentNode.insertBefore(g,s);
})();
</script> <script>
var _hmt = _hmt || [];
(function() {
var hm = document.createElement("script");
hm.src = "https://hm.baidu.com/hm.js?e9f2069e6989ecb71b4b1c8018eaf5ed";
var s = document.getElementsByTagName("script")[0];
s.parentNode.insertBefore(hm, s);
})();
</script>
</html>
