node.js web框架_使用Node.js进行Web爬取的终极指南

node.js web框架

So what’s web scraping anyway? It involves automating away the laborious task of collecting information from websites.

那么，什么是网络抓取？它涉及自动化从网站收集信息的艰巨任务。

There are a lot of use cases for web scraping: you might want to collect prices from various e-commerce sites for a price comparison site. Or perhaps you need flight times and hotel/AirBNB listings for a travel site. Maybe you want to collect emails from various directories for sales leads, or use data from the internet to train machine learning/AI models. Or you could even be wanting to build a search engine like Google!

Web抓取有很多用例：您可能希望从各种电子商务网站收集价格以进行价格比较。或者，您可能需要旅行时间的航班和酒店/ AirBNB列表。也许您想从各个目录收集电子邮件以获取销售线索，或者使用互联网上的数据来训练机器学习/ AI模型。或者，您甚至可能想要构建像Google这样的搜索引擎！

Getting started with web scraping is easy, and the process can be broken down into two main parts:

网页抓取很容易上手，该过程可以分为两个主要部分：

acquiring the data using an HTML request library or a headless browser,
使用HTML请求库或无头浏览器获取数据，
and parsing the data to get the exact information you want.
并解析数据以获得所需的确切信息。

This guide will walk you through the process with the popular Node.js request-promise module, CheerioJS, and Puppeteer. Working through the examples in this guide, you will learn all the tips and tricks you need to become a pro at gathering any data you need with Node.js!

本指南将通过流行的Node.js 请求承诺模块CheerioJS和Puppeteer引导您完成该过程。通过阅读本指南中的示例，您将学到成为专业人士使用Node.js收集所需数据所需的所有提示和技巧！

We will be gathering a list of all the names and birthdays of U.S. presidents from Wikipedia and the titles of all the posts on the front page of Reddit.

我们将在Wikipedia上收集美国总统的所有姓名和生日的列表，以及Reddit主页上所有职位的标题。

First things first: Let’s install the libraries we’ll be using in this guide (Puppeteer will take a while to install as it needs to download Chromium as well).

首先，首先要安装我们将在本指南中使用的库(Puppeteer需要花一些时间才能安装，因为它也需要下载Chromium)。

发出第一个请求 (Making your first request)

Next, let’s open a new text file (name the file potusScraper.js), and write a quick function to get the HTML of the Wikipedia “List of Presidents” page.

接下来，让我们打开一个新的文本文件(将文件命名为potusScraper.js)，并编写一个快速函数以获取Wikipedia“总统名单”页面HTML。

Output:

输出：

使用Chrome DevTools (Using Chrome DevTools)

Cool, we got the raw HTML from the web page! But now we need to make sense of this giant blob of text. To do that, we’ll need to use Chrome DevTools to allow us to easily search through the HTML of a web page.

太酷了，我们从网页上获得了原始HTML！但是现在，我们需要弄清这一巨大的文本斑点。为此，我们需要使用Chrome DevTools来轻松搜索网页HTML。

Using Chrome DevTools is easy: simply open Google Chrome, and right click on the element you would like to scrape (in this case I am right clicking on George Washington, because we want to get links to all of the individual presidents’ Wikipedia pages):

使用Chrome DevTools很容易：只需打开Google Chrome，然后右键单击要剪贴的元素(在这种情况下，我右键单击George Washington，因为我们希望获得指向所有总统个人维基百科页面的链接) ：

Now, simply click inspect, and Chrome will bring up its DevTools pane, allowing you to easily inspect the page’s source HTML.

现在，只需单击检查，Chrome就会弹出其DevTools窗格，使您可以轻松检查页面的源HTML。

用Cheerio.js解析HTML (Parsing HTML with Cheerio.js)

Awesome, Chrome DevTools is now showing us the exact pattern we should be looking for in the code (a “big” tag with a hyperlink inside of it). Let’s use Cheerio.js to parse the HTML we received earlier to return a list of links to the individual Wikipedia pages of U.S. presidents.

太棒了，Chrome DevTools现在向我们展示了我们应该在代码中寻找的确切模式(“大”标签中带有超链接)。让我们使用Cheerio.js解析我们之前收到HTML，以返回指向美国总统个人Wikipedia页面的链接列表。

Output:

输出：

We check to make sure there are exactly 45 elements returned (the number of U.S. presidents), meaning there aren’t any extra hidden “big” tags elsewhere on the page. Now, we can go through and grab a list of links to all 45 presidential Wikipedia pages by getting them from the “attribs” section of each element.

我们检查以确保返回的确有45个元素(美国总统的数量)，这意味着页面上其他任何地方都没有多余的隐藏“大”标签。现在，我们可以通过从每个元素的“攻击者”部分获取所有45个总统维基百科页面的链接列表。

Output:

输出：

Now we have a list of all 45 presidential Wikipedia pages. Let’s create a new file (named potusParse.js), which will contain a function to take a presidential Wikipedia page and return the president’s name and birthday. First things first, let’s get the raw HTML from George Washington’s Wikipedia page.

现在，我们列出了所有45个总统维基百科页面。让我们创建一个新文件(名为potusParse.js)，该文件将包含一个获取总统Wikipedia页面并返回总统的姓名和生日的函数。首先，让我们从George Washington的Wikipedia页面获取原始HTML。

Output:

输出：

Let’s once again use Chrome DevTools to find the syntax of the code we want to parse, so that we can extract the name and birthday with Cheerio.js.

让我们再次使用Chrome DevTools查找我们要解析的代码的语法，以便我们可以使用Cheerio.js提取名称和生日。

So we see that the name is in a class called “firstHeading” and the birthday is in a class called “bday”. Let’s modify our code to use Cheerio.js to extract these two classes.

因此，我们看到该名称在一个名为“ firstHeading”的类中，而生日在一个名为“ bday”的类中。让我们修改代码以使用Cheerio.js提取这两个类。

Output:

输出：

全部放在一起 (Putting it all together)

Perfect! Now let’s wrap this up into a function and export it from this module.

完善！现在，让我们将其包装为一个函数，然后从该模块中将其导出。

Now let’s return to our original file potusScraper.js and require the potusParse.js module. We’ll then apply it to the list of wikiUrls we gathered earlier.

现在，让我们回到原始文件potusScraper.js，并需要potusParse.js模块。然后，将其应用于我们之前收集的WikiUrl列表。

Output:

输出：

渲染JavaScript页面 (Rendering JavaScript Pages)

Voilà! A list of the names and birthdays of all 45 U.S. presidents. Using just the request-promise module and Cheerio.js should allow you to scrape the vast majority of sites on the internet.

瞧！所有45位美国总统的姓名和生日的列表。仅使用request-promise模块和Cheerio.js应该可以让您抓取Internet上的绝大多数站点。

Recently, however, many sites have begun using JavaScript to generate dynamic content on their websites. This causes a problem for request-promise and other similar HTTP request libraries (such as axios and fetch), because they only get the response from the initial request, but they cannot execute the JavaScript the way a web browser can.

但是，最近，许多站点已开始使用JavaScript在其网站上生成动态内容。这对请求承诺和其他类似的HTTP请求库(例如axios和fetch)造成了问题，因为它们仅从初始请求中获取响应，但是无法像Web浏览器那样执行JavaScript。

Thus, to scrape sites that require JavaScript execution, we need another solution. In our next example, we will get the titles for all of the posts on the front page of Reddit. Let’s see what happens when we try to use request-promise as we did in the previous example.

因此，要抓取需要执行JavaScript的网站，我们需要另一个解决方案。在下一个示例中，我们将在Reddit的首页上获得所有帖子的标题。让我们看看当我们尝试使用上一个示例中的请求承诺时会发生什么。

Output:

输出：

Here’s what the output looks like:

输出如下所示：

Hmmm…not quite what we want. That’s because getting the actual content requires you to run the JavaScript on the page! With Puppeteer, that’s no problem.

嗯...不是我们想要的。那是因为获取实际内容需要您在页面上运行JavaScript！使用Puppeteer，这没问题。

Puppeteer is an extremely popular new module brought to you by the Google Chrome team that allows you to control a headless browser. This is perfect for programmatically scraping pages that require JavaScript execution. Let’s get the HTML from the front page of Reddit using Puppeteer instead of request-promise.

Puppeteer是Google Chrome小组为您带来的一种非常受欢迎的新模块，可让您控制无头浏览器。对于以编程方式抓取需要执行JavaScript的页面而言，这是完美的选择。让我们使用Puppeteer而不是request-promise从Reddit的首页获取HTML。

Output:

输出：

Nice! The page is filled with the correct content!

真好！该页面填充了正确的内容！

Now we can use Chrome DevTools like we did in the previous example.

现在，我们可以像上一个示例一样使用Chrome DevTools。

It looks like Reddit is putting the titles inside “h2” tags. Let’s use Cheerio.js to extract the h2 tags from the page.

看来Reddit会将标题放在“ h2”标签中。让我们使用Cheerio.js从页面中提取h2标签。

Output:

输出：

其他资源 (Additional Resources)

And there’s the list! At this point you should feel comfortable writing your first web scraper to gather data from any website. Here are a few additional resources that you may find helpful during your web scraping journey:

有清单！在这一点上，您应该编写第一个Web抓取工具以从任何网站收集数据都感到很舒服。以下是一些其他资源，在您的网络抓取过程中可能会有所帮助：