Đọc nội dung trang web thành String trong Java (3 cách)

3 mức độ: JDK built-in, Apache HttpClient, Jsoup (HTML parser).

Cách 1: java.net.URL + openStream() – Zero dependency

import java.io.IOException;
import java.net.URL;
import java.util.Scanner;

public class ReadWebJDK {
    public static void main(String[] args) throws IOException {
        URL url = new URL("https://httpbin.org/html");
        try (Scanner sc = new Scanner(url.openStream(), "UTF-8")) {
            StringBuilder sb = new StringBuilder();
            while (sc.hasNextLine()) sb.append(sc.nextLine()).append("\n");
            String html = sb.toString();
            String text = html.replaceAll("<[^>]*>", "").replaceAll("\\s+", " ").trim();
            System.out.println(text);
        }
    }
}

Cách 2: Apache HttpClient 4.5+ – HTTP client chuẩn

    org.apache.httpcomponents
    httpclient
    4.5.14

import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import java.util.Scanner;

public class ReadWebHttpClient {
    public static void main(String[] args) throws Exception {
        try (CloseableHttpClient client = HttpClients.createDefault()) {
            HttpGet get = new HttpGet("https://httpbin.org/html");
            try (var response = client.execute(get);
                 Scanner sc = new Scanner(response.getEntity().getContent(), "UTF-8")) {
                StringBuilder sb = new StringBuilder();
                while (sc.hasNextLine()) sb.append(sc.nextLine()).append("\n");
                System.out.println(sb.toString().replaceAll("<[^>]*>", ""));
            }
        }
    }
}

Cách 3: Jsoup – Parse HTML đúng chuẩn (Khuyên dùng cho scraping)

    org.jsoup
    jsoup
    1.17.2

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class ReadWebJsoup {
    public static void main(String[] args) throws IOException {
        Document doc = Jsoup.connect("https://httpbin.org/html").get();
        // Lấy text thuần (bỏ tag, decode entity)
        String text = doc.body().text();
        System.out.println(text);
        
        // Hoặc selector CSS
        String h1 = doc.select("h1").text();
        System.out.println("H1: " + h1);
    }
}

So sánh

Java 11+: HttpClient chuẩn (java.net.http)

import java.net.URI;
import java.net.http.*;

HttpClient client = HttpClient.newHttpClient();
HttpRequest req = HttpRequest.newBuilder(URI.create("https://httpbin.org/html")).build();
String html = client.send(req, HttpResponse.BodyHandlers.ofString()).body();