ai驱动数据安全治理_AI驱动的Web数据收集解决方案的新起点

ai驱动数据安全治理

Data gathering consists of many time-consuming and complex activities. These include proxy management, data parsing, infrastructure management, overcoming fingerprinting anti-measures, rendering JavaScript-heavy websites at scale, and much more. Is there a way to automate these processes? Absolutely.

数据收集包括许多耗时且复杂的活动。 这些措施包括代理管理,数据解析,基础结构管理,克服指纹防措施,大规模渲染JavaScript繁重的网站等。 有没有办法使这些过程自动化? 绝对。

Finding a more manageable solution for a large-scale data gathering has been on the minds of many in the web scraping community. Specialists saw a lot of potential in applying AI (Artificial Intelligence) and ML (Machine Learning) to web scraping. However, only recently, actions toward data gathering automation using AI applications have been taken. This is no wonder, as AI and ML algorithms became more robust at large-scale only in recent years together with advancement in computing solutions.

网络抓取社区中的许多人一直在寻找为大规模数据收集提供更易管理的解决方案。 专家们看到了将AI(人工智能)和ML(机器学习)应用于网页抓取的巨大潜力。 但是,直到最近,才采取行动使用AI应用程序进行数据收集自动化。 这也就不足为奇了,因为AI和ML算法直到最近几年才随着计算解决方案的进步而变得更加强大。

By applying AI-powered solutions in data gathering, we can help automate tedious manual work and ensure a much better quality of the collected data. To better grasp the struggles of web scraping, let’s look into the process of data gathering, its biggest challenges, and possible future solutions that might ease and potentially solve mentioned challenges.

通过在数据收集中应用基于AI的解决方案,我们可以帮助完成繁琐的手工工作,并确保所收集数据的质量更高。 为了更好地掌握Web抓取的工作,让我们研究数据收集的过程,最大的挑战以及将来可能缓解和潜在解决上述挑战的解决方案。

数据收集:逐步 (Data collection: step by step)

To better understand the web scraping process, it’s best to visualize it in a value chain:

为了更好地了解网络抓取过程,最好在价值链中对其进行可视化处理:

As you can see, web scraping takes up four distinct actions:

如您所见,Web抓取采取了四个不同的操作:

  1. Crawling path building and URL collection.

    搜寻路径建立和URL收集。
  2. Scraper development and its support.

    刮板的开发及其支持。
  3. Proxy acquisition and management.

    代理获取和管理。
  4. Data fetching and parsing.

    数据获取和解析。

Anything that goes beyond those terms is considered to be data engineering or part of data analysis.

超出这些术语的任何内容都被视为数据工程或数据分析的一部分。

By pinpointing which actions belong to the web scraping category, it becomes easier to find the most common data gathering challenges. It also allows us to see which parts can be automated and improved with the help of AI and ML powered solutions.

通过查明哪些动作属于Web抓取类别,可以更轻松地找到最常见的数据收集难题。 它还使我们能够看到哪些零件可以借助AI和ML支持的解决方案进行自动化和改进。

大规模刮刮挑战 (Large-scale scraping challenges)

Traditional data gathering from the web requires a lot of governance and quality assurance. Of course, the difficulties that come with data gathering increase together with the scale of the scraping project. Let’s dig a little deeper into the said challenges by going through our value chain’s actions and analyzing potential issues.

从网络收集传统数据需要大量的管理和质量保证。 当然,数据收集带来的困难随着抓取项目的规模而增加。 让我们通过价值链的行动并分析潜在问题,对上述挑战进行更深入的研究。

建立搜寻路径并收集URL (Building a crawling path and collecting URLs)

Building a crawling path is the first and essential part of data gathering. To put it simply, a crawling path is a library of URLs from which data will be extracted. The biggest challenge here is not the collection of the website URLs that you want to scrape, but obtaining all the necessary URLs of the initial targets. That could mean dozens, if not hundreds of URLs that will need to be scraped, parsed, and identified as important URLs for your case.

建立爬网路径是数据收集的首要且必不可少的部分。 简单来说,爬网路径是一个URL库,将从中提取数据。 这里最大的挑战不是您要抓取的网站URL的集合,而是获得初始目标的所有必需URL。 这可能意味着需要抓取,解析和标识数十个(如果不是数百个)URL,这对于您的案例而言是重要的URL。

刮板的开发及其维护 (Scraper development and its maintenance)

Building a scraper comes with a whole new set of issues. There are a lot of factors to look out for when doing so:

构建刮板会带来一系列全新问题。 这样做时要注意很多因素:

  • Choosing the language, APIs, frameworks, etc.

    选择语言,API,框架等。
  • Testing out what you’ve built.

    测试您的构建。
  • Infrastructure management and maintenance.

    基础架构管理和维护。
  • Overcoming fingerprinting anti-measures.

    克服指纹防措施。
  • Rendering JavaScript-heavy websites at scale.

    大规模渲染JavaScript繁重的网站。

These are just the tip of the iceberg that you will encounter when building a web scraper. There are plenty more smaller and time consuming things that will accumulate into larger issues.

这些只是构建网络刮板时遇到的冰山一角。 还有很多小而费时的事情会累积成更大的问题。

代理收购与管理 (Proxy acquisition and management)

Proxy management will be a challenge, especially to those new to scraping. There are so many little mistakes one can make to block batches of proxies until successfully scraping a site. Proxy rotation is a good practice, but it doesn’t illuminate all the issues and requires constant management and upkeep of the infrastructure. So if you are relying on a proxy vendor, a good and frequent communication will be necessary.

代理管理将是一个挑战,特别是对于那些刚开始使用的人。 在成功刮取站点之前,阻止批次代理存在很多小错误。 代理轮换是一种很好的做法,但是它不能说明所有问题,并且需要对基础架构进行持续的管理和维护。 因此,如果您依赖代理供应商,则需要进行良好且频繁的沟通。

数据获取和解析 (Data fetching and parsing)

Data parsing is the process of making the acquired data understandable and usable. While creating a parser might sound easy, its further maintenance will cause big problems. Adapting to different page formats and website changes will be a constant struggle and will require your developers teams’ attention more often than you can expect.

数据解析是使获取的数据易于理解和使用的过程。 尽管创建解析器听起来很容易,但对其进行进一步的维护将导致大问题。 适应不同的页面格式和网站更改将一直是一个难题,并且将需要您的开发团队更多的注意力。

As you can see, traditional web scraping comes with many challenges, requires a lot of manual labour, time, and resources. However, the brightside with computing is that almost all things can be automated. And as the development of AI and ML powered web scraping is emerging, creating a future-proof large-scale data gathering becomes a more realistic solution.

如您所见,传统的Web抓取面临许多挑战,需要大量的人工,时间和资源。 但是,计算的亮点是几乎所有事物都可以自动化。 随着AI和ML支持的Web抓取技术的发展不断涌现,创建面向未来的大规模数据收集已成为一种更为现实的解决方案。

使网页抓取永不过时 (Making web scraping future-proof)

In what way AI and ML can innovate and improve web scraping? According to Oxylabs Next-Gen Residential Proxy AI & ML advisory board member Jonas Kubilius, an AI researcher, Marie Sklodowska-Curie Alumnus, and Co-Founder of Three Thirds:

AI和ML以什么方式可以创新和改善网页抓取? 根据Oxylabs下一代住宅代理AI和ML顾问委员会成员Jonas Kubilius的说法,他是AI研究人员Marie Sklodowska-Curie Alumnus和“三分之三”的联合创始人:

“There are recurring patterns in web content that are typically scraped, such as how prices are encoded and displayed, so in principle, ML should be able to learn to spot these patterns and extract the relevant information. The research challenge here is to learn models that generalize well across various websites or that can learn from a few human-provided examples. The engineering challenge is to scale up these solutions to realistic web scraping loads and pipelines.

“网络内容中经常会出现重复出现的模式,例如价格的编码和显示方式,因此,原则上,机器学习应该能够发现这些模式并提取相关信息。 这里的研究挑战是学习在各种网站上都能很好地概括的模型,或者可以从一些人类提供的示例中学习模型。 工程上的挑战是将这些解决方案扩展到实际的Web抓取负载和管道。

Instead of manually developing and managing the scrapers code for each new website and URL, creating an AI and ML-powered solution will simplify the data gathering pipeline. This will take care of proxy pool management, data parsing maintenance, and other tedious work.

创建一个由AI和ML支持的解决方案将简化数据收集流程,而不是为每个新网站和URL手动开发和管理刮板代码。 这将负责代理池管理,数据解析维护以及其他繁琐的工作。

Not only does AI and ML-powered solutions enable developers to build highly scalable data extraction tools, but it also enables data science teams to prototype rapidly. It also stands as a backup to your existing custom-built code if it was ever to break.

由AI和ML支持的解决方案不仅使开发人员能够构建高度可扩展的数据提取工具,而且还使数据科学团队能够快速进行原型制作。 如果曾经破解过,它也可以作为现有定制代码的备份。

网页抓取的未来前景如何 (What the future holds for web scraping)

As we already established, creating fast data processing pipelines along with cutting edge ML techniques can offer an unparalleled competitive advantage in the web scraping community. And looking at today’s market, the implementation of AI and ML in data gathering has already started.

正如我们已经确定的那样,创建快速的数据处理管道以及最先进的ML技术可以在Web抓取社区中提供无与伦比的竞争优势。 纵观当今市场,已经开始在数据收集中实施AI和ML。

For this reason, Oxylabs is introducing Next-Gen Residential Proxies which are powered by the latest AI applications.

因此,Oxylabs推出了由最新的AI应用程序提供支持的下一代住宅代理 。

Next-Gen Residential Proxies were built with heavy-duty data retrieval operations in mind. They enable web data extraction without delays or errors. The product is as customizable as a regular proxy, but at the same time, it guarantees a much higher success rate and requires less maintenance. Custom headers and IP stickiness are both supported, alongside reusable cookies and POST requests. Its main benefits are:

下一代住宅代理的构建考虑了重型数据检索操作。 它们使Web数据提取没有延迟或错误。 该产品可以像常规代理一样进行自定义,但是同时,它可以确保更高的成功率并需要更少的维护。 支持自定义标头和IP粘性,以及可重用的cookie和POST请求。 它的主要优点是:

  • 100% success rate

    成功率100%
  • AI-Powered Dynamic Fingerprinting (CAPTCHA, block, and website change handling)

    AI驱动的动态指纹识别(CAPTCHA,阻止和网站更改处理)
  • Machine Learning based HTML parsing

    基于机器学习HTML解析
  • Easy integration (like any other proxy)

    易于集成(像其他代理一样)
  • Auto-Retry system

    自动重试系统
  • JavaScript rendering

    JavaScript渲染
  • Patented proxy rotation system

    专利代理旋转系统

Going back to our previous web scraping value chain, you can see which parts of web scraping can be automated and improved with AI and ML-powered Next-Gen Residential Proxies.

回到我们以前的网络抓取价值链,您可以看到可以使用AI和ML支持的下一代住宅代理来自动化和改进网络抓取的哪些部分。

Image for post
Source: Oxylabs’ design team
资料来源:Oxylabs的设计团队

The Next-Gen Residential Proxy solution automates almost the whole scraping process, making it a truly strong competitor for future-proof web scraping.

下一代住宅代理解决方案几乎可以自动化整个刮削过程,使其成为永不过时的网络刮削的真正强大竞争对手。

This project will be continuously developed and improved by Oxylabs in-house ML engineering team and a board of advisors, Jonas Kubilius, Adi Andrei, Pujaa Rajan, and Ali Chaudhry, specializing in the fields of Artificial Intelligence and ML engineering.

Oxylabs内部的ML工程团队和顾问委员会Jonas KubiliusAdi AndreiPujaa RajanAli Chaudhry将继续开发和改进此项目,该委员会专门研究人工智能和ML工程领域。

结语 (Wrapping up)

As the scale of web scraping projects increase, automating data gathering becomes a high priority for businesses that want to stay ahead of the competition. With the improvement of AI algorithms in recent years, along with the increase in compute power and the growth of the talent pool has made AI implementations possible in a number of industries, web scraping included.

随着网络抓取项目规模的扩大,对于希望保持竞争优势的企业而言,自动化数据收集已成为当务之急。 近年来,随着AI算法的改进,以及计算能力的提高和人才库的增长,使得许多行业都可以实施AI,其中包括Web抓取。

Establishing AI and ML-powered data gathering techniques offers a great competitive advantage in the industry, as well as save copious amounts of time and resources. It is the new future of large-scale web scraping, and a good head start of the development of future-proof solutions.

建立由AI和ML支持的数据收集技术在行业中提供了巨大的竞争优势,并且节省了大量的时间和资源。 这是大规模刮网的新未来,也是开发面向未来的解决方案的良好开端。

翻译自: https://towardsdatascience.com/the-new-beginnings-of-ai-powered-web-data-gathering-solutions-a8e95f5e1d3f

ai驱动数据安全治理

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/389361.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

铁拳nat映射_铁拳如何重塑我的数据可视化设计流程

铁拳nat映射It’s been a full year since I’ve become an independent data visualization designer. When I first started, projects that came to me didn’t relate to my interests or skills. Over the past eight months, it’s become very clear to me that when cl…

DengAI —如何应对数据科学竞赛? (EDA)

了解机器学习 (Understanding ML) This article is based on my entry into DengAI competition on the DrivenData platform. I’ve managed to score within 0.2% (14/9069 as on 02 Jun 2020). Some of the ideas presented here are strictly designed for competitions li…

java.net.SocketException: Software caused connection abort: socket write erro

场景:接口测试 编辑器:eclipse 版本:Version: 2018-09 (4.9.0) testng版本:TestNG version 6.14.0 执行testng.xml时报错信息: 出现此报错原因之一:网上有人说是testng版本与eclipse版本不一致造成的&#…

使用K-Means对美因河畔法兰克福的社区进行聚类

介绍 (Introduction) This blog post summarizes the results of the Capstone Project in the IBM Data Science Specialization on Coursera. Within the project, the districts of Frankfurt am Main in Germany shall be clustered according to their venue data using t…

样本均值的抽样分布_抽样分布样本均值

样本均值的抽样分布One of the most important concepts discussed in the context of inferential data analysis is the idea of sampling distributions. Understanding sampling distributions helps us better comprehend and interpret results from our descriptive as …

玩转ceph性能测试---对象存储(一)

笔者最近在工作中需要测试ceph的rgw,于是边测试边学习。首先工具采用的intel的一个开源工具cosbench,这也是业界主流的对象存储测试工具。 1、cosbench的安装,启动下载最新的cosbench包wget https://github.com/intel-cloud/cosbench/release…

因果关系和相关关系 大数据_数据科学中的相关性与因果关系

因果关系和相关关系 大数据Let’s jump into it right away.让我们马上进入。 相关性 (Correlation) Correlation means relationship and association to another variable. For example, a movement in one variable associates with the movement in another variable. For…

vue取数据第一个数据_我作为数据科学家的第一个月

vue取数据第一个数据A lot.很多。 I landed my first job as a Data Scientist at the beginning of August, and like any new job, there’s a lot of information to take in at once.我于8月初找到了数据科学家的第一份工作,并且像任何新工作一样,一…

STL-开篇

基本概念 STL: Standard Template Library,标准模板库 定义: c引入的一个标准类库 特点:1)数据结构和算法的 c实现( 采用模板类和模板函数)2)数据的存储和算法的分离3)高…

rcp rapido_为什么气流非常适合Rapido

rcp rapidoBack in 2019, when we were building our data platform, we started building the data platform with Hadoop 2.8 and Apache Hive, managing our own HDFS. The need for managing workflows whether it’s data pipelines, i.e. ETL’s, machine learning predi…

Mysql5.7开启远程

2019独角兽企业重金招聘Python工程师标准>>> 1.注掉bind-address #bind-address 127.0.0.1 2.开启远程访问权限 grant all privileges on *.* to root"xxx.xxx.xxx.xxx" identified by "密码"; 或 grant all privileges on *.* to root"%…

分类结果可视化python_可视化分类结果的另一种方法

分类结果可视化pythonI love good data visualizations. Back in the days when I did my PhD in particle physics, I was stunned by the histograms my colleagues built and how much information was accumulated in one single plot.我喜欢出色的数据可视化。 早在我获得…

算法组合 优化算法_算法交易简化了风险价值和投资组合优化

算法组合 优化算法Photo by Markus Spiske (left) and Jamie Street (right) on UnsplashMarkus Spiske (左)和Jamie Street(右)在Unsplash上的照片 In the last post, we saw how actual algorithms are developed and tested. In this post, we will figure out the level of…

PS抠发丝技巧 「选择并遮住…」

PS抠发丝技巧 「选择并遮住…」 现在的海报设计,大多数都有模特MM,然而MM的头发实用太多了,有的还飘起来…… 对于设计师(特别是淘宝美工)没有一个强大、快速、实用的抠发丝技巧真的混不去哦。而PS CC 2017版本开始,就有了一个强大…

covid 19如何重塑美国科技公司的工作文化

未来 , 技术 , 观点 (Future, Technology, Opinion) Who would have thought that a single virus would take down the whole world and make us stay inside our homes? A pandemic wave that has altered our lives in such a way that no human (bi…

python生日悖论分析_生日悖论

python生日悖论分析If you have a group of people in a room, how many do you need to for it to be more likely than not, that two or more will have the same birthday?如果您在一个房间里有一群人,那么您需要多少个才能使两个或两个以上的人有相同的生日&a…

rstudio 管道符号_R中的管道指南

rstudio 管道符号R基础知识 (R Fundamentals) Data analysis often involves many steps. A typical journey from raw data to results might involve filtering cases, transforming values, summarising data, and then running a statistical test. But how can we link al…

蒙特卡洛模拟预测股票_使用蒙特卡洛模拟来预测极端天气事件

蒙特卡洛模拟预测股票In a previous article, I outlined the limitations of conventional time series models such as ARIMA when it comes to forecasting extreme temperature values, which in and of themselves are outliers in the time series.在上一篇文章中 &#…

直方图绘制与直方图均衡化实现

一,直方图的绘制 1.直方图的概念: 在图像处理中,经常用到直方图,如颜色直方图、灰度直方图等。 图像的灰度直方图就描述了图像中灰度分布情况,能够很直观的展示出图像中各个灰度级所 占的多少。 图像的灰度直方图是灰…

时间序列因果关系_分析具有因果关系的时间序列干预:货币波动

时间序列因果关系When examining a time series, it is quite common to have an intervention influence that series at a particular point.在检查时间序列时,在特定时间点对该序列产生干预影响是很常见的。 Some examples of this could be:例如: …