Công cụ Web Scraping trong Python

Web scraping: trích xuất dữ liệu từ web, chuyển phi cấu trúc thành có cấu trúc. 6 thư viện phổ biến.

So sánh nhanh

1. urllib (stdlib)

from urllib.request import urlopen
html = urlopen("https://www.tutorialspoint.com/").read()
print(html[:500])

2. Requests (khuyên dùng cho HTTP)

import requests
r = requests.get('https://www.tutorialspoint.com/')
print(r.status_code)      # 200
print(r.encoding)         # UTF-8
print(r.headers['Content-Type'])
print(r.url)
print(r.elapsed)

3. BeautifulSoup (parse HTML)

from bs4 import BeautifulSoup
import requests

r = requests.get("https://www.tutorialspoint.com/")
soup = BeautifulSoup(r.text, 'html.parser')

for link in soup.find_all('a'):
    print(link.get('href'))

4. lxml (parser nhanh, XPath)

from lxml import etree
root = etree.Element('html')
etree.SubElement(root, 'head')
etree.SubElement(root, 'title')
etree.SubElement(root, 'body')
print(etree.tostring(root, pretty_print=True).decode())
#