注意： 本文内容于 2024-11-09 19:20:07 创建，可能不会在此平台上进行更新。如果您希望查看最新版本或更多相关内容，请访问原文地址：SSL证书以及实现HTTP反向代理。感谢您的关注与支持！
之前写的HTTP反向代理工具，在实际使用时，碰到反代失败的问题。跟踪了一下，才发现是由于对方使用了自签名SSL证书，导致发起HTTP请求时，验证失败。因此简单记录一下。
针对该问题的复现，从两个方面来展开
- 理解SSL
- 忽略SSL
一、理解SSL
1.1 HTTPS与SSL
SSL是用于加密传输的协议，也是最初的加密标准，目前已被TLS取代，但由于历史原因，大家还是会称为SSL。
HTTPS是HTTP上实现加密传输的协议，依赖SSL/TLS来确保安全性。
从不求甚解的角度来理解，HTTPS=HTTP+SSL/TLS
1.2 证书分类
SSL常见的证书分类有两种
- CA证书
- 自签名证书
这两者的区别如下
| 特性 | 自签名证书 | CA证书 |
|---|---|---|
| 签发机构 | 由证书持有者自己签发 | 由受信任的证书颁发机构（CA）签发 |
| 信任级别 | 默认不被浏览器或操作系统信任，需手动安装信任 | 被大多数浏览器和操作系统默认信任 |
| 身份验证 | 无身份验证，持有者自行生成证书 | CA会对证书持有者进行身份验证 |
| 安全性 | 安全性较低，可能被伪造或滥用 | 高安全性，通过身份验证保障证书真实性 |
| 应用场景 | 适用于开发、测试和内部网络 | 适用于生产环境和面向互联网的服务 |
| 成本 | 免费 | 需要付费，费用根据证书类型和CA机构不同而异 |
| 浏览器警告 | 会弹出“不安全连接”警告 | 不会弹出警告，用户信任度高 |
| 管理复杂度 | 管理简单，但不适合公开环境 | 管理较复杂，需要向CA申请和续期 |
互联网服务使用的一般都是CA证书，由于操作系统已经内置了一系列根证书，当访问一个使用CA签发证书的HTTPS网站时，就不会出现“不安全连接”的警告。
而自签名证书，由于操作系统缺少对其的信任，访问就会被拦截了。此时服务提供方，需要给调用方提供自签名证书，以便调用方可以信任该连接。
1.3 OpenSSL生成自签名证书
1.3.1 扩展名说明
像我购买的CA证书，部署到Nginx时，一般都是.pem和.key文件。但在自己生成证书的过程中，发现还有.crt文件。直观的感受是，这些扩展名特别的混乱。经过查阅资料，下面简单记录这些扩展名的区别。
- crt: 存储证书(公钥)。该证书可提供给第三方使用，比如HTTPS客户端
- key: 私钥。该私钥文件只应给服务提供者使用。
- csr: 向证书颁发机构申请签署密钥的请求，不包含密钥本身。
- pem: 基于Base64编码的文本格式。它可以是上述任何文件。
- der: 基于二进制编码的文本格式。它可以是上述任何文件。
参考
ssl - Difference between pem, crt, key files - Stack Overflow
Difference between .pem and .crt and how to use them - Help - Let’s Encrypt Community Support
1.3.2 自签名证书
下面使用OpenSSL生成自签名的公钥和私钥证书。
# 生成一个2048位的RSA私钥，并保存到private.key文件中
openssl genrsa -out private.key 2048# 根据私钥，生成一个证书签名请求
openssl req -new -key private.key -out request.csr# X.509是SSL/TLS中最常用的公钥证书标准
# 通过私钥和证书签名请求，生成一个时效为365天的证书
openssl x509 -req -days 365 -in request.csr -signkey private.key -out public.pem# 验证证书内容
openssl x509 -in public.pem -noout -text
二、忽略SSL
2.1 服务端部署证书
2.1.1 Nginx
以Nginx为例，部署证书
worker_processes 1;events {worker_connections 1024;
}http {include mime.types;default_type application/octet-stream;server {listen 443 ssl;server_name 10.0.0.1;ssl_certificate /usr/local/nginx/conf/cert/public.pem;ssl_certificate_key /usr/local/nginx/conf/cert/private.key;location / {root /usr/local/nginx/html;index index.html;}}
}
2.1.2 Vertx
使用Java中的Vertx 4.5.10版本开启HTTPServer
PemKeyCertOptions pemKeyCertOptions = new PemKeyCertOptions()//使用自签名证书开启ssl.addCertPath("/usr/local/nginx/conf/cert/public.pem").addKeyPath("/usr/local/nginx/conf/cert/private.key");
Future<HttpServer> serverFuture = vertx.createHttpServer(new HttpServerOptions().setSsl(true).setKeyCertOptions(pemKeyCertOptions))//注册路由.requestHandler(router).listen(port);
serverFuture.onComplete(re -> {if (re.succeeded()) {log.info("http server started on port {}", port);} else {log.error("http server failed to start", re.cause());}
});
2.2 客户端忽略校验
2.2.1 CURL
curl忽略ssl校验比较简单，添加-k参数即可。
curl -k "https://10.0.0.10:443"
2.2.2 Apache HttpPClient and OkHttpClient
设置忽略SSL的核心逻辑如下，具体的写法还需根据框架而定。
/*** 信任所有SSL证书，包括CA证书和自签名证书。实现效果类似于`curl -k`* @see <a href="https://blog.csdn.net/qq_20683411/article/details/142996223">Apache HttpClient 4.3.2 和 4.5.13 - 忽略证书问题_apache 4.3.5 忽略ssl-CSDN博客</a>*/
public SSLContext trustAllCerts() {try {TrustManager[] trustManagers = {new X509TrustManager() {@Overridepublic void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}@Overridepublic void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}@Overridepublic X509Certificate[] getAcceptedIssuers() {return new X509Certificate[0];}}};SSLContext sslContext = SSLContext.getInstance("SSL");sslContext.init(null, trustManagers, new SecureRandom());return sslContext;} catch (Exception ignore) {}return null;
}public HostnameVerifier getNoopHostnameVerifier() {return new HostnameVerifier() {@Overridepublic boolean verify(final String s, final SSLSession sslSession) {return true;}@Overridepublic final String toString() {return "NO_OP";}};
}
三、HTTP反向代理
这个主要是带着学习的目的实现的。
- java8
- springboot2.5.14
- okhttp3
直接上源码meethigher/http-proxy-boot: 使用SpringBoot实现的开箱即用的HTTP反向代理工具
import okhttp3.*;
import okio.BufferedSink;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import javax.net.ssl.*;
import javax.servlet.ServletException;
import javax.servlet.ServletInputStream;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.*;
import java.util.concurrent.TimeUnit;/*** 该内容主要是想着学习一下底层HTTP反向代理的实现** @author <a href="https://meethigher.top">chenchuancheng</a>* @see <a href="https://github.com/mitre/HTTP-Proxy-Servlet">mitre/HTTP-Proxy-Servlet: Smiley's HTTP Proxy implemented as a Java servlet</a>* @since 2024/11/09 22:43*/
public class ProxyServlet extends HttpServlet {protected static final Logger log = LoggerFactory.getLogger(ProxyServlet.class);protected final OkHttpClient client;protected final String targetUrl; // 目标服务器信息protected final boolean corsControl; // 跨域控制。当为true时，跨域信息都由自身服务管理protected final boolean allowCORS; // 是否允许跨域。当corsControl为true时，该参数方可生效。protected final boolean logEnable; // 启用日志protected final boolean forwardIp; // 遵循代理规范，将实际调用方的ip和protocol传给目标服务器protected final boolean preserveHost; // 保留原host，这个仅对请求头有效。protected final boolean preserveCookie; // 保留原cookie。这个对请求头和响应头均有效。protected final String logFormat; // 日志格式/*** 跨域相关的响应头*/protected final List<String> allowCORSHeaders = Arrays.asList("access-control-allow-origin",//指定哪些域可以访问资源。可以是特定域名，也可以是通配符 *，表示允许所有域访问。"access-control-allow-methods",//指定允许的HTTP方法，如 GET、POST、PUT、DELETE 等。"access-control-allow-headers",//指定允许的请求头。"access-control-allow-credentials",//指定是否允许发送凭据（如Cookies）。值为 true 表示允许，且不能使用通配符 *。"access-control-expose-headers",//指定哪些响应头可以被浏览器访问。"access-control-max-age",//指定预检请求的结果可以被缓存的时间（以秒为单位）。"access-control-request-method",//在预检请求中使用，指示实际请求将使用的方法。"access-control-request-headers"//在预检请求中使用，指示实际请求将使用的自定义头。);/*** 不应该被复制的逐跳标头*/protected final String[] hopByHopHeaders = new String[]{"Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization","TE", "Trailers", "Transfer-Encoding", "Upgrade"};/*** 默认的日志格式*/public static final String LOG_FORMAT_DEFAULT = "{method} -- {userAgent} -- {remoteAddr}:{remotePort} -- {source} --> {target} -- {statusCode} consumed {consumedMills} ms";protected void doLog(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, long startMills) {if (logEnable) {String queryString = httpServletRequest.getQueryString();String logInfo = logFormat.replace("{method}", httpServletRequest.getMethod()).replace("{userAgent}", httpServletRequest.getHeader("User-Agent")).replace("{remoteAddr}", httpServletRequest.getRemoteAddr()).replace("{remotePort}", String.valueOf(httpServletRequest.getRemotePort())).replace("{source}", queryString == null ? httpServletRequest.getRequestURL() : httpServletRequest.getRequestURL() + "?" + queryString).replace("{target}", rewriteUrlFromRequest(httpServletRequest)).replace("{statusCode}", String.valueOf(httpServletResponse.getStatus())).replace("{consumedMills}", String.valueOf(System.currentTimeMillis() - startMills));log.info("{}: {}", getServletName(), logInfo);}}/*** 是否包含逐跳标头*/protected boolean containsHopByHopHeader(String name) {for (String header : hopByHopHeaders) {if (header.equalsIgnoreCase(name)) {return true;}}return false;}/*** 不考虑contextPath* 获取代理请求url，不包含queryparams*/protected String getTargetUrl(HttpServletRequest request) {//request.getRequestURI();//包含contextPath的uriString uri = request.getPathInfo();//不包含contextPath的uriif (uri == null || uri.isEmpty()) {return targetUrl;} else {return targetUrl + uri;}}/*** 获取代理请求完整url，包含queryparams*/protected String rewriteUrlFromRequest(HttpServletRequest request) {String targetUrl = getTargetUrl(request);String queryString = request.getQueryString();return queryString == null ? targetUrl : targetUrl + "?" + queryString;}/*** 将重定向的url，重写为代理服务器的地址*/protected String rewriteUrlFromResponse(HttpServletRequest request, String locationUrl) {String targetUrl = getTargetUrl(request);if (locationUrl != null && locationUrl.startsWith(targetUrl)) {StringBuffer curUrl = request.getRequestURL();int pos;if ((pos = curUrl.indexOf("://")) >= 0) {if ((pos = curUrl.indexOf("/", pos + 3)) >= 0) {curUrl.setLength(pos);}}curUrl.append(request.getContextPath());curUrl.append(request.getServletPath());curUrl.append(locationUrl, targetUrl.length(), locationUrl.length());return curUrl.toString();}return locationUrl;}public ProxyServlet(OkHttpClient client, String targetUrl, boolean corsControl, boolean allowCORS, boolean logEnable, String logFormat, boolean forwardIp, boolean preserveHost, boolean preserveCookie) {this.client = client;this.targetUrl = targetUrl;this.corsControl = corsControl;this.allowCORS = allowCORS;this.logEnable = logEnable;this.forwardIp = forwardIp;this.preserveHost = preserveHost;this.preserveCookie = preserveCookie;this.logFormat = logFormat;}public ProxyServlet(OkHttpClient client, String targetUrl) {this(client, targetUrl, false, false, true, LOG_FORMAT_DEFAULT, false, false, false);}/*** 根据代理的规定，通过请求头进行真实信息的传递* X-Forwarded-For: 传输实际调用者ip* X-Forwarded-Proto: 传输实际调用者请求协议*/public void setXForwardedForHeader(HttpServletRequest request, Request.Builder requestBuilder) {if (forwardIp) {String forHeaderName = "X-Forwarded-For";String forHeader = request.getRemoteAddr();String existingForHeader = request.getHeader(forHeader);if (existingForHeader != null) {forHeader = existingForHeader + ", " + forHeader;}requestBuilder.header(forHeaderName, forHeader);String protoHeaderName = "X-Forwarded-Proto";String protoHeader = request.getScheme();requestBuilder.header(protoHeaderName, protoHeader);}}@Overrideprotected void service(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) throws ServletException, IOException {long start = System.currentTimeMillis();try {if (httpServletRequest.getMethod().equalsIgnoreCase("options") && corsControl && allowCORS) {httpServletResponse.setHeader("Access-Control-Allow-Origin", httpServletRequest.getHeader("origin"));httpServletResponse.setHeader("Access-Control-Allow-Methods", "*");httpServletResponse.setHeader("Access-Control-Allow-Headers", "*");httpServletResponse.setHeader("Access-Control-Allow-Credentials", "true");httpServletResponse.setHeader("Access-Control-Expose-Headers", "*");httpServletResponse.setStatus(HttpServletResponse.SC_OK);return;}String method = httpServletRequest.getMethod();String proxyRequestUrl = rewriteUrlFromRequest(httpServletRequest);Request.Builder requestBuilder = getInitRequestBuilder(httpServletRequest, httpServletResponse);requestBuilder.url(proxyRequestUrl);if ("get".equalsIgnoreCase(method)) {requestBuilder.method(method, null);} else {// 根据HTTP规定，复制请求体RequestBody requestBody;if (httpServletRequest.getHeader("Content-Length") != null || httpServletRequest.getHeader("Transfer-Encoding") != null) {try {ServletInputStream inputStream = httpServletRequest.getInputStream();requestBody = new StreamingRequestBody(MediaType.parse(httpServletRequest.getContentType()), inputStream);} catch (Exception e) {writeGatewayError(httpServletResponse, e.getMessage());return;}} else {requestBody = RequestBody.create(null, new byte[0]);}requestBuilder.method(method, requestBody);}copyRequestHeaders(httpServletRequest, requestBuilder);setXForwardedForHeader(httpServletRequest, requestBuilder);try (Response response = client.newCall(requestBuilder.build()).execute()) {httpServletResponse.setStatus(response.code());copyResponseHeaders(httpServletRequest, httpServletResponse, response);if (response.code() == 304) {// http状态码为304时，表示当客户端发起请求时，如果服务器发现请求的资源并没有自上次请求后发生任何更改，就会返回 304 状态码，同时不包含请求资源的实体内容。这意味着客户端可以继续使用缓存中的资源，从而避免不必要的数据传输，减少服务器负载和带宽消耗。httpServletResponse.setIntHeader("Content-Length", 0);} else {// 复制响应体ResponseBody responseBody = response.body();if (responseBody != null) {ServletOutputStream os = httpServletResponse.getOutputStream();InputStream is = responseBody.byteStream();int len;byte[] buffer = new byte[8192];while ((len = is.read(buffer)) != -1) {os.write(buffer, 0, len);}os.flush();}}} catch (Exception e) {writeGatewayError(httpServletResponse, e.getMessage());return;}} finally {doLog(httpServletRequest, httpServletResponse, start);}}protected Request.Builder getInitRequestBuilder(HttpServletRequest request, HttpServletResponse response) {return new Request.Builder();}/*** 复制响应头*/protected void copyResponseHeaders(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, Response response) {Set<String> names = response.headers().names();// httpServletResponse不支持移除请求头，因此按需添加头Map<String, String> needSetHeaderMap = new LinkedHashMap<>();for (String name : names) {if (containsHopByHopHeader(name)) {continue;}if ("Location".equalsIgnoreCase(name)) {// 重写重定向头needSetHeaderMap.put(name, rewriteUrlFromResponse(httpServletRequest, response.header(name)));} else if ("Set-Cookie".equalsIgnoreCase(name) || "Set-Cookie2".equalsIgnoreCase(name)) {// 保存Cookie信息if (preserveCookie) {needSetHeaderMap.put(name, response.header(name));}} else {needSetHeaderMap.put(name, response.header(name));}}/*** 跨域控制*/if (corsControl) {/*** 1. 清空所有与跨域相关的响应头* 2. 如果允许跨域，则添加跨域允许响应头*/Iterator<Map.Entry<String, String>> iterator = needSetHeaderMap.entrySet().iterator();while (iterator.hasNext()) {Map.Entry<String, String> entry = iterator.next();String keyIgnoreCase = entry.getKey().toLowerCase(Locale.ROOT);if (allowCORSHeaders.contains(keyIgnoreCase)) {iterator.remove();}}if (allowCORS) {needSetHeaderMap.put("Access-Control-Allow-Origin", httpServletRequest.getHeader("origin"));needSetHeaderMap.put("Access-Control-Allow-Methods", "*");needSetHeaderMap.put("Access-Control-Allow-Headers", "*");needSetHeaderMap.put("Access-Control-Allow-Credentials", "true");needSetHeaderMap.put("Access-Control-Expose-Headers", "*");}}for (String header : needSetHeaderMap.keySet()) {httpServletResponse.setHeader(header, needSetHeaderMap.get(header));}}/*** 复制请求头*/protected void copyRequestHeaders(HttpServletRequest httpServletRequest, Request.Builder requestBuilder) {Enumeration<String> headerNames = httpServletRequest.getHeaderNames();while (headerNames.hasMoreElements()) {String key = headerNames.nextElement();if ("host".equalsIgnoreCase(key)) {if (preserveHost) {requestBuilder.header(key, httpServletRequest.getHeader(key));}} else if ("cookie".equalsIgnoreCase(key)) {if (preserveCookie) {requestBuilder.header(key, httpServletRequest.getHeader(key));}} else {requestBuilder.header(key, httpServletRequest.getHeader(key));}}}protected void writeGatewayError(HttpServletResponse httpServletResponse, String msg) {httpServletResponse.setStatus(502);httpServletResponse.setContentType("text/html;charset=utf-8");try {httpServletResponse.getWriter().write(msg);} catch (Exception ignore) {}}/*** okhttpclient的流式请求体* 节省应用内存，通过流式传输数据，只会在网络传输过程中按需读取数据，而不会将整个请求体加载到内存。*/class StreamingRequestBody extends RequestBody {private final MediaType contentType;private final InputStream inputStream;public StreamingRequestBody(MediaType contentType, InputStream inputStream) {this.contentType = contentType;this.inputStream = inputStream;}@Overridepublic MediaType contentType() {return contentType;}@Overridepublic void writeTo(BufferedSink sink) throws IOException {// 每次读8KB的数据byte[] buffer = new byte[8192];int bytesRead;while ((bytesRead = inputStream.read(buffer)) != -1) {sink.write(buffer, 0, bytesRead);}}}private static OkHttpClient okHttpClient;public synchronized static OkHttpClient okHttpClient() {if (okHttpClient == null) {OkHttpClient.Builder builder = new OkHttpClient().newBuilder().connectionPool(new ConnectionPool(2, 60, TimeUnit.SECONDS)).connectTimeout(10, TimeUnit.SECONDS).readTimeout(30, TimeUnit.SECONDS).writeTimeout(30, TimeUnit.SECONDS);builder.followRedirects(false);builder.followSslRedirects(true);builder.hostnameVerifier(new HostnameVerifier() {@Overridepublic boolean verify(final String s, final SSLSession sslSession) {return true;}@Overridepublic final String toString() {return "NO_OP";}});try {X509TrustManager x509TrustManager = new X509TrustManager() {@Overridepublic void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}@Overridepublic void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}@Overridepublic X509Certificate[] getAcceptedIssuers() {return new X509Certificate[0];}};TrustManager[] trustManagers = {x509TrustManager};SSLContext sslContext = SSLContext.getInstance("SSL");sslContext.init(null, trustManagers, new SecureRandom());builder.sslSocketFactory(sslContext.getSocketFactory(), x509TrustManager);} catch (Exception ignore) {}okHttpClient = builder.build();}return okHttpClient;}
}
