diff --git a/Docs/net-http-over-tls.md b/Docs/net-http-over-tls.md index 498b6ba45..5ec9fc6b1 100644 --- a/Docs/net-http-over-tls.md +++ b/Docs/net-http-over-tls.md @@ -1,7 +1,10 @@ # net/http over TLS: diagnosis and proposed fix -Status: diagnosed, not fixed. Measured on IronRuby 4.0.1, Windows 11, .NET 10.0.12, -2026-10-23. +Status: both stages below are implemented; stage 2 replaced stage 1's net/protocol +overrides, so upstream's `net/protocol` runs unmodified. Measured on IronRuby 4.0.1, +Windows 11, .NET 10.0.12, 2026-10-23. The tables and "Why" describe the state before +stage 1; afterwards `Util/net-http-tls-check.rb` reports no failures for https either, +with or without `CA=1`. ## The suspicion @@ -90,6 +93,37 @@ gives real non-blocking I/O. `verify_callback`, when set, is called with the result and an `X509StoreContext` holding the chain, and it has the last word, as in MRI. +As built: + +- `Transport__#read_timeout=`/`#write_timeout=` (ms, -1 for none) and + `#connect(hostname, verify, timeout_ms)`, which runs `AuthenticateAsClientAsync` under a + `CancellationTokenSource`. A timeout raises `IO::TimeoutError` and marks the transport + broken; every later read or write raises `SSLError` ("SSL connection is unusable"). + Since stage 2 these bound the blocking calls only (`SSLSocket#connect`, `#sysread`, + `#syswrite`, through `#__connect`, `#__read_timeout=`, `#__write_timeout=`). +- Stage 1 also had `Src/StdLib/ironruby/net/protocol.rb` prepend overrides of + `ssl_socket_connect`, `rbuf_fill` and `write0` that handed net/protocol's timeouts to + those blocking calls, for `Net` and RubyGems' `Gem::Net`. Stage 2 made them unnecessary + and they are gone: see below. +- `Transport__#trust(certificates, system)`: the self-signed certificates from `ca_file`, + `ca_path` and `cert_store` are the roots, the others go in `ExtraStore`, as in + `X509::Store#verify`. `system` (a `cert_store` that had `set_default_paths` called) also + accepts a chain the system store trusts. + The chain has to be good for TLS server authentication (`ApplicationPolicy` serverAuth), as + SslStream's own check requires. +- `Transport__#verify_callback=` gets the chain and the `SslPolicyErrors`; `SSLSocket` calls + `verify_callback` for each certificate, root first, once per error (or once with + `preverify_ok` true), with an `X509::StoreContext`, following CRuby's `ossl_verify_cb_call`: + only `true` accepts and clears the error, the first rejection ends it, and an exception in + the callback is a rejection with a warning. It runs on the thread that finishes the handshake. +- `Thread#raise` or `Timeout.timeout` during the handshake cancels it, marks the transport + broken, and raises the interrupting exception rather than an `SSLError`. +- With no SNI name (net/http connecting to an IP address) the handshake ignores + `RemoteCertificateNameMismatch`, and `post_connection_check` checks the name against the + certificate (`X509Certificate2.MatchesHostname`, IP SANs included; without a subjectAltName + the common name, for an IP address too, as MRI's `verify_certificate_identity`). Before, it + checked nothing. + Risk: small. Blocking I/O stays as it is, only bounded. What it does not give is real non-blocking `read_nonblock`, so `IO.select([ssl_socket])` still says "readable" only when the TCP socket is. A reader that selects first and then reads can block in the @@ -114,6 +148,32 @@ Risk: medium. It touches `IO.select` for every caller, and the in-flight task ha cancelled cleanly on `close`. It is worth doing for servers and event loops (nio4r, puma over TLS), not for a client. +As built: + +- `IReadinessHook` (`Src/Libraries/Builtins/IoReadiness.cs`): `IsReady(kind)` answers true, + false, or null ("ask the socket"), and the hook signals the `ReadinessWaiter`s registered + with it when that answer may have changed. `IoReadiness.SetHook` keeps hooks per `RubyIO` + in a `ConditionalWeakTable`; with no hook set, the cost is one field read. `IO.select`, + `IO#wait*` (`IoOps.IsReady`, `WaitUntilReady`) and `NIO::Selector` ask the hook first and + register their waiter with it. nio4r masks off the kernel's answer for what a hook owns. +- `Transport__` is the hook of the TCP socket it runs on. Every read, write and handshake is + one of SslStream's asynchronous operations, at most one of each kind in flight: + `#read_nonblock`, `#write_nonblock` (at most 16 KB handed over at a time, so what it reports + written is about what a kernel buffer holds) and `#connect_nonblock` start one and answer + `:wait_readable`/`:wait_writable` while it runs; `#read`, `#write` and `#connect` start one + or take over the one in flight, and wait for it. +- Asking the hook whether a read would return starts that read. SslStream may hold bytes, + encrypted or decrypted, that the TCP socket no longer shows, and only a read finds them. What a + finished read brings beyond what was asked for waits in a read-ahead, which `#pending` counts. +- `SSLSocket#read_nonblock`, `#write_nonblock` and `#connect_nonblock` raise + `SSLErrorWaitReadable`/`SSLErrorWaitWritable` ("read would block", "write would block"), or + answer the symbol with `exception: false`, as on MRI. net/protocol's `rbuf_fill`, `write0` + and `ssl_socket_connect` then wait on `#to_io` with their own timeouts, so `open_timeout`, + `read_timeout` and `write_timeout` work without any IronRuby code in net/protocol, for + RubyGems' vendored `Gem::Net` as well. +- Not done: the server side (`SSLServer`, `#accept`) is still not implemented, so puma over + TLS still cannot run. + ## How to test it - `Util/net-http-tls-check.rb` (needs CRuby for the server): after stage 1 the @@ -125,8 +185,13 @@ puma over TLS), not for a client. exported certificate and `read_timeout: 1` (the test needs `[Options(Prism = true)]` to load `net/http`). -## Until then + That is `Src/IronRuby.Tests/Runtime/NetHttpTlsTests.cs`: `NetHttp_Tls1` (trust, the three + timeouts, the name check), `NetHttp_TlsVerifyCallback1`, `NetHttp_GemNetTls1`, + `NetHttp_PlainNoOpenSsl1` and `Ssl_NonBlocking1` (stage 2: `connect_nonblock`, + `read_nonblock`, `write_nonblock`, `IO.select` and `NIO::Selector` over TLS). + +## Before this -An application that needs timeouts or its own CA over HTTPS is better served by .NET's -`HttpClient` until stage 1 lands. The reason is not `IO.select` being `poll(2)`; it is -that `net/http` over TLS has no timeouts and ignores `ca_file`. +An application that needed timeouts or its own CA over HTTPS was better served by .NET's +`HttpClient`. The reason was never `IO.select` being `poll(2)`; it was that `net/http` over +TLS had no timeouts and ignored `ca_file`. diff --git a/Src/IronRuby.Tests/RubyTests.cs b/Src/IronRuby.Tests/RubyTests.cs index e59cc2bd5..ff919421d 100644 --- a/Src/IronRuby.Tests/RubyTests.cs +++ b/Src/IronRuby.Tests/RubyTests.cs @@ -708,6 +708,11 @@ public Tests(Driver/*!*/ driver) { Zlib_Streams1, Process_SpawnKill1, Json_ControlCharacters1, + NetHttp_Tls1, + NetHttp_TlsVerifyCallback1, + NetHttp_PlainNoOpenSsl1, + NetHttp_GemNetTls1, + Ssl_NonBlocking1, Sprintf_MriTieRounding1, Float_NegativeZero1, Math_CLibrary1, diff --git a/Src/IronRuby.Tests/Runtime/NetHttpTlsTests.cs b/Src/IronRuby.Tests/Runtime/NetHttpTlsTests.cs new file mode 100644 index 000000000..2cee7d923 --- /dev/null +++ b/Src/IronRuby.Tests/Runtime/NetHttpTlsTests.cs @@ -0,0 +1,699 @@ +/* **************************************************************************** + * + * Copyright (c) Microsoft Corporation. + * + * This source code is subject to terms and conditions of the Apache License, Version 2.0. A + * copy of the license can be found in the License.html file at the root of this distribution. If + * you cannot locate the Apache License, Version 2.0, please send an email to + * ironruby@microsoft.com. By using this source code in any fashion, you are agreeing to be bound + * by the terms of the Apache License, Version 2.0. + * + * You must not remove this notice, or any other, from this software. + * + * + * ***************************************************************************/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Net; +using System.Net.Security; +using System.Net.Sockets; +using System.Security.Cryptography; +using System.Security.Cryptography.X509Certificates; +using System.Text; +using System.Threading; + +namespace IronRuby.Tests { + + public partial class Tests { + // What every script below starts with: net/http (and openssl), which load without a + // warning, and the server's ports and CA files. + private static string NetHttpPrelude(TlsTestServer/*!*/ server, bool openssl) { + return @" +require 'ruby4' +require 'net/http' +" + (openssl ? "require 'openssl'\n" : "") + @" +PORTS = " + server.RubyPorts + @" +CA = " + server.RubyCertificatePaths + @" + +def http(**opts) + h = Net::HTTP.new('127.0.0.1', PORTS[opts.delete(:at) || :self]) + h.use_ssl = true + h.max_retries = 0 + opts.each { |name, value| h.__send__(""#{name}="", value) } + h +end +"; + } + + /// + /// Net::HTTP over TLS against a server in this process (TlsTestServer below), so that no + /// CRuby is needed: ca_file and cert_store decide what is trusted, and the read, write and + /// open (handshake) timeouts fire against a server that stops answering. IronRuby's + /// SSLSocket is a blocking SslStream; before Docs/net-http-over-tls.md stage 1 none of + /// those timeouts fired and only the system trust store counted. net/http needs prism. + /// + [Options(Prism = true)] + public void NetHttp_Tls1() { + using (var server = new TlsTestServer()) { + TestOutput(NetHttpPrelude(server, true) + @" +# ca_file: keep-alive requests both ways +http(ca_file: CA[:self]).start do |h| + p h.get('/data?size=100000').body.bytesize + body = 'y' * 300_000 + p h.post('/echo', body).body == body +end + +# cert_store; a leaf issued by a CA in ca_file +store = OpenSSL::X509::Store.new +store.add_file(CA[:self]) +p http(cert_store: store).start { |h| h.get('/data?size=3').body } +p http(at: :issued, ca_file: CA[:ca]).start { |h| h.get('/data?size=2').body } + +# the system store knows neither; a certificate for client authentication is no server's +[[:self, nil], [:client_auth, CA[:ca]]].each do |at, ca| + begin + http(at: at, ca_file: ca).start { |h| h.get('/data?size=3') } + rescue OpenSSL::SSL::SSLError + p [at, :refused] + end +end + +# read_timeout: the server never answers +t = Time.now +begin + http(ca_file: CA[:self], read_timeout: 1).start { |h| h.get('/stall') } +rescue Net::ReadTimeout + p [:read_timeout, Time.now - t < 5] +end + +# write_timeout: the server never reads the body +t = Time.now +begin + http(ca_file: CA[:self], write_timeout: 1).start { |h| h.post('/sink', 'z' * 64_000_000) } +rescue Net::WriteTimeout + p [:write_timeout, Time.now - t < 10] +end + +# open_timeout: the server accepts the connection but never answers the handshake +t = Time.now +begin + http(at: :stall, ca_file: CA[:self], open_timeout: 1).start { |h| h.get('/') } +rescue Net::OpenTimeout + p [:open_timeout, Time.now - t < 5] +end + +# Timeout.timeout around the same handshake: its own exception, not an SSLError +t = Time.now +begin + Timeout.timeout(1) { http(at: :stall, ca_file: CA[:self]).start { |h| h.get('/') } } +rescue Exception => e + p [e.class, Time.now - t < 5] +end + +# a refused handshake stays refused: a second connect does not call the socket connected +ctx = OpenSSL::SSL::SSLContext.new +ctx.set_params({}) +s = OpenSSL::SSL::SSLSocket.new(TCPSocket.new('127.0.0.1', PORTS[:self]), ctx) +2.times do + begin + s.connect + p :connected + rescue OpenSSL::SSL::SSLError + p :refused + end +end +s.close + +# the socket itself: the name check, and no use after a timeout +ctx = OpenSSL::SSL::SSLContext.new +ctx.set_params(ca_file: CA[:self]) +s = OpenSSL::SSL::SSLSocket.new(TCPSocket.new('127.0.0.1', PORTS[:self]), ctx) +s.sync_close = true +s.connect +p s.post_connection_check('127.0.0.1') +['example.com', ''].each do |name| + begin + s.post_connection_check(name) + rescue OpenSSL::SSL::SSLError + p [:name_mismatch, name] + end +end +s.write ""GET /stall HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n"" +s.__read_timeout = 0.5 +begin + s.sysread(10) +rescue IO::TimeoutError + p :io_timeout +end +begin + s.sysread(10) +rescue OpenSSL::SSL::SSLError => e + p e.message.include?('unusable') +end +s.close + +# a certificate with no subjectAltName is checked on its common name, an IP address too +ctx = OpenSSL::SSL::SSLContext.new +ctx.set_params(ca_file: CA[:common_name]) +s = OpenSSL::SSL::SSLSocket.new(TCPSocket.new('127.0.0.1', PORTS[:common_name]), ctx) +s.connect +p s.post_connection_check('10.0.0.5') +begin + s.post_connection_check('10.0.0.6') +rescue OpenSSL::SSL::SSLError + p :cn_mismatch +end +s.close +", @" +100000 +true +""xxx"" +""xx"" +[:self, :refused] +[:client_auth, :refused] +[:read_timeout, true] +[:write_timeout, true] +[:open_timeout, true] +[Timeout::Error, true] +:refused +:refused +true +[:name_mismatch, ""example.com""] +[:name_mismatch, """"] +:io_timeout +true +true +:cn_mismatch +"); + } + } + + /// + /// SSLContext#verify_callback over TLS, as CRuby's ossl_verify_cb_call has it: one call per + /// certificate and error, root first; only true accepts (and clears the error, so + /// verify_result is V_OK); the first rejection ends the check; an exception in the callback + /// is a rejection with a warning. The error codes are OpenSSL's: 18 for a lone untrusted + /// self-signed certificate, 10 expired, 9 not yet valid, 26 the wrong purpose. + /// + [Options(Prism = true)] + public void NetHttp_TlsVerifyCallback1() { + using (var server = new TlsTestServer()) { + TestOutput(NetHttpPrelude(server, true) + @" +require 'stringio' + +def verify(at, ca = nil, &callback) + seen = [] + cb = lambda { |ok, ctx| seen << [ok, ctx.error_depth, ctx.error]; callback.call(ok, ctx) } + result = begin + http(at: at, ca_file: ca, verify_callback: cb).start { |h| h.get('/data?size=1').body } + rescue OpenSSL::SSL::SSLError + :refused + end + [result, seen] +end + +# the callback overrides the system store, and sees the certificate's subject +subjects = [] +result = verify(:self) { |ok, ctx| subjects << ctx.current_cert.subject.to_s; true } +p [result, subjects] +# a chain, root first; true for both +p verify(:issued, CA[:ca]) { |ok, ctx| ok } +# every error is reported: accepting 18 does not accept the validity period as well +[:expired, :future].each do |at| + result, seen = verify(at) { |ok, ctx| ok || ctx.error == 18 } + p [result, seen.map(&:last)] +end +result, seen = verify(:client_auth, CA[:ca]) { |ok, ctx| ok } +p [result, seen.map(&:last)] +# only true accepts; the first rejection ends it +p verify(:issued, CA[:ca]) { |ok, ctx| 1 } +p verify(:issued, CA[:ca]) { |ok, ctx| false } +# an exception is a rejection, with a warning +err, $stderr = $stderr, StringIO.new +p verify(:issued, CA[:ca]) { |ok, ctx| raise 'boom' } +warning, $stderr = $stderr.string, err +p warning.include?('exception in verify_callback is ignored') + +# accepting an error clears it +ctx = OpenSSL::SSL::SSLContext.new +ctx.set_params(verify_callback: lambda { |ok, c| true }) +s = OpenSSL::SSL::SSLSocket.new(TCPSocket.new('127.0.0.1', PORTS[:self]), ctx) +s.connect +p s.verify_result +s.close +", @" +[[""x"", [[false, 0, 18]]], [""/CN=127.0.0.1""]] +[""x"", [[true, 1, 0], [true, 0, 0]]] +[:refused, [18, 10]] +[:refused, [18, 9]] +[:refused, [0, 26]] +[:refused, [[true, 1, 0]]] +[:refused, [[true, 1, 0]]] +[:refused, [[true, 1, 0]]] +true +0 +"); + } + } + + /// + /// Non-blocking TLS, the way an event loop drives it: connect_nonblock, read_nonblock and + /// write_nonblock answer :wait_readable / :wait_writable while SslStream's operation runs, + /// and IO.select on #to_io and NIO::Selector see it finish. Bytes SslStream has already + /// decrypted make the socket readable although the TCP socket under it is empty. + /// + [Options(Prism = true)] + public void Ssl_NonBlocking1() { + using (var server = new TlsTestServer()) { + TestOutput(@" +require 'ruby4' +require 'socket' +require 'openssl' +# nio.rb asks RubyGems what platform this is; the test runtime has no RubyGems +module Gem + def self.win_platform? + File::ALT_SEPARATOR == '\\' + end +end unless defined?(Gem) +require 'nio' +PORTS = " + server.RubyPorts + @" +CA = " + server.RubyCertificatePaths + @" + +def tls(at = :self) + ctx = OpenSSL::SSL::SSLContext.new + ctx.set_params(ca_file: CA[:self]) + OpenSSL::SSL::SSLSocket.new(TCPSocket.new('127.0.0.1', PORTS[at]), ctx) +end + +def read_response(s) + buf = ''.b + until (head = buf.index(""\r\n\r\n"")) && buf.bytesize >= head + 4 + buf[/Content-Length: (\d+)/, 1].to_i + case data = s.read_nonblock(4096, exception: false) + when :wait_readable then IO.select([s.to_io], nil, nil, 5) or raise 'read stalled' + else buf << data + end + end + buf[(head + 4)..-1] +end + +s = tls +until (r = s.connect_nonblock(exception: false)) == s + raise r.inspect unless r == :wait_readable + IO.select([s.to_io], nil, nil, 5) or raise 'handshake stalled' +end +p [s.state, s.post_connection_check('127.0.0.1')] + +# nothing to read yet +p s.read_nonblock(10, exception: false) +begin + s.read_nonblock(10) +rescue OpenSSL::SSL::SSLErrorWaitReadable => e + p [e.is_a?(IO::WaitReadable), e.message] +end +p IO.select([s.to_io], nil, nil, 0.2) + +s.write ""GET /data?size=100000 HTTP/1.1\r\nHost: x\r\n\r\n"" +p read_response(s).bytesize + +# one byte of a response: the rest is decrypted and waiting, and select says so +s.write ""GET /data?size=10 HTTP/1.1\r\nHost: x\r\n\r\n"" +IO.select([s.to_io], nil, nil, 5) +first = s.read_nonblock(1) +p [first, s.pending > 0, IO.select([s.to_io], nil, nil, 0) ? :readable : :not_readable] +read_response(s) rescue nil +s.read_nonblock(4096, exception: false) while s.pending > 0 + +# NIO::Selector over the TLS socket +selector = NIO::Selector.new +monitor = selector.register(s.to_io, :r) +p selector.select(0.2) +s.write_nonblock(""GET /data?size=7 HTTP/1.1\r\nHost: x\r\n\r\n"") +ready = selector.select(5) +p [ready.size, monitor.readable?] +selector.deregister(s.to_io) +s.close + +# a server that reads nothing: write_nonblock runs into :wait_writable, and select agrees +s = tls +s.connect +s.write ""POST /sink HTTP/1.1\r\nHost: x\r\nContent-Length: 100000000\r\n\r\n"" +chunk = 'z' * 65536 +written = 0 +result = nil +2000.times do + result = s.write_nonblock(chunk, exception: false) + if result == :wait_writable + # A write in flight is not a full buffer (on Windows one is in flight almost always): + # only a wait that runs out means the peer stopped reading. + break unless IO.select(nil, [s.to_io], nil, 0.5) + next + end + written += result +end +p [result, written > 0, written <= 16384 * 2000] +p IO.select(nil, [s.to_io], nil, 0.2) +begin + s.write_nonblock(chunk) +rescue OpenSSL::SSL::SSLErrorWaitWritable => e + p e.is_a?(IO::WaitWritable) +end +s.close +", @" +[""SSLOK "", true] +:wait_readable +[true, ""read would block""] +nil +100000 +[""H"", true, :readable] +nil +[1, true] +[:wait_writable, true, true] +nil +true +"); + } + } + + /// + /// RubyGems' vendored copy of net-http and net-protocol (Gem::Net, what `gem install` + /// fetches with) gets the same timeouts over TLS as the standard library's, and raises + /// its own Gem::Net errors. Nothing of it is patched: SSLSocket's non-blocking calls are + /// what upstream's timeouts wait on. + /// + [Options(Prism = true)] + public void NetHttp_GemNetTls1() { + using (var server = new TlsTestServer()) { + TestOutput(@" +require 'ruby4' +module Gem; end unless defined?(Gem) +require 'rubygems/vendored_net_http' +require 'openssl' +PORTS = " + server.RubyPorts + @" +CA = " + server.RubyCertificatePaths + @" + +def gem_http(at, **opts) + h = Gem::Net::HTTP.new('127.0.0.1', PORTS[at]) + h.use_ssl = true + h.ca_file = CA[:self] + h.max_retries = 0 + opts.each { |name, value| h.__send__(""#{name}="", value) } + h +end + +p gem_http(:self).start { |h| h.get('/data?size=5').body } +t = Time.now +begin + gem_http(:self, read_timeout: 1).start { |h| h.get('/stall') } +rescue Gem::Net::ReadTimeout + p [:read_timeout, Time.now - t < 5] +end +t = Time.now +begin + gem_http(:stall, open_timeout: 1).start { |h| h.get('/') } +rescue Gem::Net::OpenTimeout + p [:open_timeout, Time.now - t < 5] +end +p defined?(::Net::HTTP) +", @" +""xxxxx"" +[:read_timeout, true] +[:open_timeout, true] +nil +"); + } + } + + /// + /// A plain HTTP request does not load openssl: IronRuby's net/protocol checks for its own + /// SSLSocket without naming OpenSSL, which would run net/http's autoload. + /// + [Options(Prism = true)] + public void NetHttp_PlainNoOpenSsl1() { + using (var server = new TlsTestServer()) { + TestOutput(NetHttpPrelude(server, false) + @" +h = Net::HTTP.new('127.0.0.1', PORTS[:plain]) +p h.start { |c| [c.get('/data?size=4').body, c.post('/echo', 'abc').body] } +p Object.autoload?(:OpenSSL) +", @" +[""xxxx"", ""abc""] +""openssl"" +"); + } + } + + /// + /// HTTPS on loopback ports, one per certificate, with throwaway certificates whose PEM is + /// written to temporary files: + /// self self-signed for 127.0.0.1 (subjectAltName IP), its own CA + /// issued issued for 127.0.0.1 by the CA "ca", for server authentication + /// client_auth issued for 127.0.0.1 by "ca", for client authentication only + /// expired self-signed for 127.0.0.1, expired yesterday + /// future self-signed for 127.0.0.1, valid from tomorrow + /// common_name self-signed, CN=10.0.0.5 and no subjectAltName, its own CA + /// plain is the same without TLS, and stall accepts connections and never says a word. + /// GET /data?size=N answers N bytes, POST /echo sends the body back, /stall and /sink + /// answer nothing and read nothing more. + /// + private sealed class TlsTestServer : IDisposable { + private static readonly Oid ServerAuthentication = new Oid("1.3.6.1.5.5.7.3.1"); + private static readonly Oid ClientAuthentication = new Oid("1.3.6.1.5.5.7.3.2"); + + private readonly Dictionary _listeners = new Dictionary(); + private readonly Dictionary _pems = new Dictionary(); + private readonly List _certificates = new List(); + private readonly List _clients = new List(); + private readonly ManualResetEvent _stop = new ManualResetEvent(false); + private volatile bool _disposed; + + public TlsTestServer() { + var now = DateTimeOffset.UtcNow; + using (var ca = SelfSigned("CN=IronRuby Test CA", true, null, now.AddMinutes(-5), now.AddDays(1), true)) { + WritePem("ca", ca); + Listen("self", Keep(SelfSigned("CN=127.0.0.1", true, null, now.AddMinutes(-5), now.AddDays(1), false)), true); + Listen("issued", Keep(Issued(ca, ServerAuthentication))); + Listen("client_auth", Keep(Issued(ca, ClientAuthentication))); + Listen("expired", Keep(SelfSigned("CN=127.0.0.1", true, null, now.AddDays(-2), now.AddDays(-1), false))); + Listen("future", Keep(SelfSigned("CN=127.0.0.1", true, null, now.AddDays(1), now.AddDays(2), false))); + Listen("common_name", Keep(SelfSigned("CN=10.0.0.5", false, null, now.AddMinutes(-5), now.AddDays(1), false)), true); + } + Listen("plain", null); + Listen("stall", null); + } + + // { self: 1234, ... } for the Ruby side. + public string/*!*/ RubyPorts { + get { + var result = new StringBuilder("{ "); + foreach (var entry in _listeners) { + result.Append(entry.Key).Append(": ").Append(((IPEndPoint)entry.Value.LocalEndpoint).Port).Append(", "); + } + return result.Append("}").ToString(); + } + } + + public string/*!*/ RubyCertificatePaths { + get { + var result = new StringBuilder("{ "); + foreach (var entry in _pems) { + result.Append(entry.Key).Append(": '").Append(entry.Value.Replace("\\", "/")).Append("', "); + } + return result.Append("}").ToString(); + } + } + + private static CertificateRequest/*!*/ Request(string/*!*/ subject, RSA/*!*/ key, bool ipAddress, Oid usage, bool ca) { + var request = new CertificateRequest(subject, key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); + if (ipAddress) { + var names = new SubjectAlternativeNameBuilder(); + names.AddIpAddress(IPAddress.Loopback); + request.CertificateExtensions.Add(names.Build()); + } + if (usage != null) { + request.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension(new OidCollection { usage }, false)); + } + if (ca) { + request.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true)); + request.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.KeyCertSign | X509KeyUsageFlags.CrlSign, true)); + } + return request; + } + + private static X509Certificate2/*!*/ SelfSigned(string/*!*/ subject, bool ipAddress, Oid usage, + DateTimeOffset notBefore, DateTimeOffset notAfter, bool ca) { + using (var key = RSA.Create(2048)) { + return Request(subject, key, ipAddress, usage, ca).CreateSelfSigned(notBefore, notAfter); + } + } + + private static X509Certificate2/*!*/ Issued(X509Certificate2/*!*/ ca, Oid usage) { + using (var key = RSA.Create(2048)) { + var serial = new byte[8]; + RandomNumberGenerator.Fill(serial); + serial[0] &= 0x7f; + using (var issued = Request("CN=127.0.0.1", key, true, usage, false).Create(ca, DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddHours(12), serial)) { + return issued.CopyWithPrivateKey(key); + } + } + } + + // SChannel cannot serve with an ephemeral key: round-trip the certificate through PFX. + private X509Certificate2/*!*/ Keep(X509Certificate2/*!*/ created) { + using (created) { +#if NET9_0_OR_GREATER + var result = X509CertificateLoader.LoadPkcs12(created.Export(X509ContentType.Pfx), null); +#else + var result = new X509Certificate2(created.Export(X509ContentType.Pfx)); +#endif + _certificates.Add(result); + return result; + } + } + + private void WritePem(string/*!*/ name, X509Certificate2/*!*/ certificate) { + string path = Path.Combine(Path.GetTempPath(), "ironruby-tls-test-" + Guid.NewGuid().ToString("N") + ".pem"); + File.WriteAllText(path, certificate.ExportCertificatePem() + "\n"); + _pems[name] = path; + } + + private void Listen(string/*!*/ name, X509Certificate2 certificate, bool writePem = false) { + if (writePem) { + WritePem(name, certificate); + } + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + _listeners[name] = listener; + Action serve; + if (name == "stall") { + serve = client => _stop.WaitOne(); + } else if (certificate == null) { + serve = client => Serve(client.GetStream()); + } else { + serve = client => { + using (var ssl = new SslStream(client.GetStream(), false)) { + ssl.AuthenticateAsServer(certificate); + Serve(ssl); + } + }; + } + Start(() => Accept(listener, serve)); + } + + private static void Start(ThreadStart body) { + new Thread(body) { IsBackground = true }.Start(); + } + + private void Accept(TcpListener/*!*/ listener, Action/*!*/ serve) { + while (!_disposed) { + TcpClient client; + try { + client = listener.AcceptTcpClient(); + } catch (Exception) { + return; + } + lock (_clients) { + _clients.Add(client); + } + Start(() => { + try { + serve(client); + } catch (Exception) { + // the client went away, refused the certificate, or the test is over + } finally { + client.Dispose(); + } + }); + } + } + + private void Serve(Stream/*!*/ stream) { + while (true) { + string head = ReadHead(stream); + if (head == null) { + return; + } + string path = head.Split(' ')[1]; + if (path == "/stall" || path == "/sink") { + // answer nothing and read nothing until the client gives up + _stop.WaitOne(); + return; + } + + byte[] body; + if (path == "/echo") { + body = ReadExactly(stream, ContentLength(head)); + } else { + int size = Int32.Parse(path.Substring(path.IndexOf("size=") + 5)); + body = new byte[size]; + for (int i = 0; i < size; i++) { + body[i] = (byte)'x'; + } + } + byte[] header = Encoding.ASCII.GetBytes( + "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: " + body.Length + "\r\n\r\n"); + stream.Write(header, 0, header.Length); + stream.Write(body, 0, body.Length); + stream.Flush(); + } + } + + // The request line and headers, or null when the client closed the connection. + private static string ReadHead(Stream/*!*/ stream) { + var head = new StringBuilder(); + var one = new byte[1]; + while (!head.ToString().EndsWith("\r\n\r\n", StringComparison.Ordinal)) { + if (stream.Read(one, 0, 1) == 0) { + return null; + } + head.Append((char)one[0]); + } + return head.ToString(); + } + + private static int ContentLength(string/*!*/ head) { + foreach (string line in head.Split(new[] { "\r\n" }, StringSplitOptions.None)) { + if (line.StartsWith("content-length:", StringComparison.OrdinalIgnoreCase)) { + return Int32.Parse(line.Substring(15).Trim()); + } + } + return 0; + } + + private static byte[]/*!*/ ReadExactly(Stream/*!*/ stream, int length) { + var result = new byte[length]; + int offset = 0; + while (offset < length) { + int read = stream.Read(result, offset, length - offset); + if (read == 0) { + throw new EndOfStreamException(); + } + offset += read; + } + return result; + } + + public void Dispose() { + _disposed = true; + _stop.Set(); + foreach (var listener in _listeners.Values) { + listener.Stop(); + } + lock (_clients) { + foreach (var client in _clients) { + client.Dispose(); + } + } + foreach (var certificate in _certificates) { + certificate.Dispose(); + } + foreach (var path in _pems.Values) { + try { + File.Delete(path); + } catch (IOException) { + } + } + } + } + } +} diff --git a/Src/Libraries/Builtins/IoOps.cs b/Src/Libraries/Builtins/IoOps.cs index 4080f6c53..260230e74 100644 --- a/Src/Libraries/Builtins/IoOps.cs +++ b/Src/Libraries/Builtins/IoOps.cs @@ -806,16 +806,14 @@ internal static T WaitUntilReady(IList/*!*/ watched, IList/*!*/ var info = ThreadOps.RubyThreadInfo.FromThread(Thread.CurrentThread); bool wasBlocked = info.Blocked; ReadinessWaiter waiter = null; - List pipes = null; + List pipes = null; try { info.Blocked = true; - // In-process pipes have no descriptor to poll; they signal the waiter instead. - for (int i = 0; i < watched.Count; i++) { - if (!watched[i].Closed && IoReadiness.GetPipe(watched[i]) != null) { - waiter = ReadinessWaiter.Current; - break; - } + // In-process pipes have no descriptor to poll, and an IO with a readiness hook + // (a TLS socket) may be ready when its descriptor is not; they signal the waiter. + if (IoReadiness.NeedsWaiter(watched)) { + waiter = ReadinessWaiter.Current; } if (waiter != null) { pipes = IoReadiness.Register(waiter, watched); @@ -884,6 +882,14 @@ private static bool IsReady(RubyIO/*!*/ io, Readiness kind) { return true; } + var hook = IoReadiness.GetHook(io); + if (hook != null && kind != Readiness.Error) { + bool? ready = hook.IsReady(kind == Readiness.Read ? IoReadiness.Read : IoReadiness.Write); + if (ready.HasValue) { + return ready.Value; + } + } + var pipe = stream.BaseStream as RubyPipe; if (pipe != null) { // The mode settles the two ends of an in-process pipe: they share one queue, so diff --git a/Src/Libraries/Builtins/IoReadiness.cs b/Src/Libraries/Builtins/IoReadiness.cs index 3061f62be..b12ce7928 100644 --- a/Src/Libraries/Builtins/IoReadiness.cs +++ b/Src/Libraries/Builtins/IoReadiness.cs @@ -119,6 +119,23 @@ internal void Reset() { } } + /// + /// A layer that reads and writes an IO's stream itself, so that the kernel no longer knows + /// whether a read or a write of that layer would block: OpenSSL's TLS transport, whose + /// SslStream may already have drained the socket into a read that is still in flight, or + /// hold decrypted bytes the socket never shows. IO.select, IO#wait and NIO::Selector ask it + /// first, and it wakes their waiter when its answer changes. + /// + internal interface IReadinessHook { + /// + /// For IoReadiness.Read or Write: true or false when the hook knows, null to ask the + /// socket. Asking may start the I/O that answers it. + /// + bool? IsReady(int kind); + void AddWaiter(ReadinessWaiter/*!*/ waiter); + void RemoveWaiter(ReadinessWaiter/*!*/ waiter); + } + /// /// What IO.select, IO#wait and NIO::Selector#select have in common: watching a set of IOs and /// blocking until one of them might have become ready. Readiness itself is decided by the @@ -151,6 +168,22 @@ internal static RubyPipe GetPipe(RubyIO/*!*/ io) { return io.GetStream().BaseStream as RubyPipe; } + private static readonly System.Runtime.CompilerServices.ConditionalWeakTable/*!*/ _hooks = + new System.Runtime.CompilerServices.ConditionalWeakTable(); + + // No hook was ever set: the common case pays one read of a field. + private static volatile bool _anyHooks; + + internal static void SetHook(RubyIO/*!*/ io, IReadinessHook/*!*/ hook) { + _hooks.AddOrUpdate(io, hook); + _anyHooks = true; + } + + internal static IReadinessHook GetHook(RubyIO/*!*/ io) { + IReadinessHook hook; + return _anyHooks && _hooks.TryGetValue(io, out hook) ? hook : null; + } + #region Windows anonymous pipes [System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)] @@ -199,11 +232,25 @@ internal static bool IsWindowsPipeReadable(System.IO.FileStream/*!*/ pipe) { #endregion /// - /// Registers the waiter with every in-process pipe among , and - /// answers those pipes - null if there were none - for Unregister to take it off again. + /// Whether any of can only tell the waiter about a change - an + /// in-process pipe, or an IO with a readiness hook - so that a wait needs one. /// - internal static List Register(ReadinessWaiter/*!*/ waiter, IList/*!*/ ios) { - List pipes = null; + internal static bool NeedsWaiter(IList/*!*/ ios) { + for (int i = 0; i < ios.Count; i++) { + if (!ios[i].Closed && (GetPipe(ios[i]) != null || GetHook(ios[i]) != null)) { + return true; + } + } + return false; + } + + /// + /// Registers the waiter with every in-process pipe and readiness hook among + /// , and answers them - null if there were none - for Unregister to + /// take it off again. + /// + internal static List Register(ReadinessWaiter/*!*/ waiter, IList/*!*/ ios) { + List registered = null; for (int i = 0; i < ios.Count; i++) { if (ios[i].Closed) { continue; @@ -211,16 +258,26 @@ internal static List Register(ReadinessWaiter/*!*/ waiter, IList())).Add(pipe); + (registered ?? (registered = new List())).Add(pipe); + } + var hook = GetHook(ios[i]); + if (hook != null) { + hook.AddWaiter(waiter); + (registered ?? (registered = new List())).Add(hook); } } - return pipes; + return registered; } - internal static void Unregister(ReadinessWaiter/*!*/ waiter, List pipes) { - if (pipes != null) { - foreach (var pipe in pipes) { - pipe.RemoveWaiter(waiter); + internal static void Unregister(ReadinessWaiter/*!*/ waiter, List registered) { + if (registered != null) { + foreach (var item in registered) { + var pipe = item as RubyPipe; + if (pipe != null) { + pipe.RemoveWaiter(waiter); + } else { + ((IReadinessHook)item).RemoveWaiter(waiter); + } } } } @@ -259,6 +316,9 @@ private static void BlockUnix(IList/*!*/ ios, IList/*!*/ kinds, Rea return; } + // A hook nobody can be woken by is only seen by asking it again, and soon. + unwatchedPipe |= (waiter == null && GetHook(io) != null); + int descriptor; var socket = GetSocket(io); if (socket != null) { @@ -300,6 +360,7 @@ private static void BlockWindows(IList/*!*/ ios, IList/*!*/ kinds, if (ios[i].Closed) { return; } + unwatchedPipe |= (waiter == null && GetHook(ios[i]) != null); var socket = GetSocket(ios[i]); if (socket == null) { // A Windows pipe cannot sit in Socket.Select either: it is asked again soon. diff --git a/Src/Libraries/Initializers.Generated.cs b/Src/Libraries/Initializers.Generated.cs index 6c86dfd84..6e48b3c75 100644 --- a/Src/Libraries/Initializers.Generated.cs +++ b/Src/Libraries/Initializers.Generated.cs @@ -12277,8 +12277,18 @@ private static void LoadOpenSSL__SSL__Transport___Instance(IronRuby.Builtins.Rub ); DefineLibraryMethod(module, "connect", 0x11, + 0x00050000U, + new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.Connect) + ); + + DefineLibraryMethod(module, "connect_nonblock", 0x11, 0x00010000U, - new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.Connect) + new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.ConnectNonBlocking) + ); + + DefineLibraryMethod(module, "matches_hostname?", 0x11, + 0x00010002U, + new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.MatchesHostname) ); DefineLibraryMethod(module, "peer_cert", 0x11, @@ -12301,11 +12311,41 @@ private static void LoadOpenSSL__SSL__Transport___Instance(IronRuby.Builtins.Rub new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.Read) ); + DefineLibraryMethod(module, "read_nonblock", 0x11, + 0x00010000U, + new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.ReadNonBlocking) + ); + + DefineLibraryMethod(module, "read_timeout=", 0x11, + 0x00010000U, + new Action(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.SetReadTimeout) + ); + + DefineLibraryMethod(module, "trust", 0x11, + 0x00000002U, + new Action(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.Trust) + ); + + DefineLibraryMethod(module, "verify_callback=", 0x11, + 0x00000000U, + new Action(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.SetVerifyCallback) + ); + DefineLibraryMethod(module, "write", 0x11, 0x00010002U, new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.Write) ); + DefineLibraryMethod(module, "write_nonblock", 0x11, + 0x00010002U, + new Func(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.WriteNonBlocking) + ); + + DefineLibraryMethod(module, "write_timeout=", 0x11, + 0x00010000U, + new Action(IronRuby.StandardLibrary.OpenSsl.OpenSsl.SSL.Transport.SetWriteTimeout) + ); + } #endif diff --git a/Src/Libraries/Nio4r/Nio4r.cs b/Src/Libraries/Nio4r/Nio4r.cs index e0a7cd7e4..c89ada08c 100644 --- a/Src/Libraries/Nio4r/Nio4r.cs +++ b/Src/Libraries/Nio4r/Nio4r.cs @@ -345,6 +345,11 @@ private struct Probe { public RubyPipe Pipe; public int Descriptor; // Unix: what poll(2) is handed, or -1 public int Ready; + // A TLS socket's transport (IoReadiness.IReadinessHook): the interests it answers + // for itself, whatever the kernel says about the socket, and its answer. + public IReadinessHook Hook; + public int HookOwned; + public int HookReady; } private object SelectLocked(BlockParam block, int milliseconds) { @@ -353,6 +358,8 @@ private object SelectLocked(BlockParam block, int milliseconds) { var info = ThreadOps.RubyThreadInfo.FromThread(Thread.CurrentThread); bool wasBlocked = info.Blocked; List pipes = null; + List hooks = null; + bool hooked = false; Probe[] probes; try { info.Blocked = true; @@ -376,6 +383,14 @@ private object SelectLocked(BlockParam block, int milliseconds) { probe.Pipe.AddWaiter(_waiter); (pipes ?? (pipes = new List())).Add(probe.Pipe); } + probe.Hook = IoReadiness.GetHook(probe.IO); + if (probe.Hook != null) { + hooked = true; + if (_waiter != null) { + probe.Hook.AddWaiter(_waiter); + (hooks ?? (hooks = new List())).Add(probe.Hook); + } + } } monitor._probeIndex = count; probes[count++] = probe; @@ -399,10 +414,24 @@ private object SelectLocked(BlockParam block, int milliseconds) { long remaining = deadline - Environment.TickCount64; int wait = (readyCount > 0 || woken || remaining <= 0) ? 0 - : (int)Math.Min(remaining, (pipes != null && _waiter == null) ? 1 : IoReadiness.SliceMilliseconds); + : (int)Math.Min(remaining, ((pipes != null || hooked) && _waiter == null) ? 1 : IoReadiness.SliceMilliseconds); readyCount += AskKernel(probes, count, wait); + // What a hook answers for, it answers alone: the kernel may call its socket + // readable while the bytes are still on their way through the hook's read. + if (hooked) { + readyCount = 0; + for (int i = 0; i < count; i++) { + if (probes[i].Hook != null) { + probes[i].Ready = (probes[i].Ready & ~probes[i].HookOwned) | probes[i].HookReady; + } + if (probes[i].Ready != 0) { + readyCount++; + } + } + } + if (readyCount > 0 || woken || Volatile.Read(ref _wakeupRequested) != 0) { Interlocked.Exchange(ref _wakeupRequested, 0); break; @@ -439,6 +468,11 @@ private object SelectLocked(BlockParam block, int milliseconds) { pipe.RemoveWaiter(_waiter); } } + if (hooks != null) { + foreach (var hook in hooks) { + hook.RemoveWaiter(_waiter); + } + } info.Blocked = wasBlocked; } } @@ -461,6 +495,23 @@ private static int ProbeWithoutKernel(ref Probe probe) { ready |= InterestRead; } + if (probe.Hook != null) { + probe.HookOwned = probe.HookReady = 0; + foreach (int interest in new[] { InterestRead, InterestWrite }) { + if ((probe.Interests & interest) == 0) { + continue; + } + bool? answer = probe.Hook.IsReady(interest == InterestRead ? IoReadiness.Read : IoReadiness.Write); + if (answer.HasValue) { + probe.HookOwned |= interest; + if (answer.Value) { + probe.HookReady |= interest; + } + } + } + ready |= probe.HookReady; + } + if (probe.Pipe != null) { if ((probe.Interests & InterestRead) != 0 && io.Mode.CanRead() && probe.Pipe.CanReadWithoutBlocking) { ready |= InterestRead; diff --git a/Src/Libraries/OpenSSL/OpenSSL.cs b/Src/Libraries/OpenSSL/OpenSSL.cs index 55e9af5aa..022703dbe 100644 --- a/Src/Libraries/OpenSSL/OpenSSL.cs +++ b/Src/Libraries/OpenSSL/OpenSSL.cs @@ -15,14 +15,19 @@ #if FEATURE_CRYPTOGRAPHY using System; +using System.Collections; +using System.Collections.Generic; using System.Globalization; using System.Runtime.InteropServices; using System.Security.Cryptography; using System.Security.Cryptography.X509Certificates; using System.IO; +using System.Net; using System.Net.Security; using System.Net.Sockets; using System.Security.Authentication; +using System.Threading; +using System.Threading.Tasks; using IronRuby.StandardLibrary.Sockets; using System.Text; using IronRuby.Builtins; @@ -553,13 +558,56 @@ public static class SSL { /// One TLS connection. Not an MRI class: OpenSSL::SSL::SSLSocket is the /// Ruby class in openssl/ssl.rb and this is what it does its reading and /// writing through. + /// + /// Every read, write and handshake is one of SslStream's asynchronous operations, + /// at most one of each kind in flight. The *_nonblock methods start one and answer + /// :wait_readable or :wait_writable while it runs; the blocking ones start one, or + /// take over the one in flight, and wait for it, at most read_timeout or + /// write_timeout. The transport is the TCP socket's readiness hook + /// (IReadinessHook): IO.select and NIO::Selector ask it, not the socket, whether a + /// read would return, since an operation in flight may already have taken the + /// socket's bytes, and SslStream may hold bytes the socket never shows. Asking it + /// whether a read would return starts that read. /// [RubyClass("Transport__")] - public class Transport { + public class Transport : IReadinessHook { + private RubyContext _context; private Socket _socket; private SslStream _ssl; private bool _verify; private string _hostname; + private int _readTimeout = Timeout.Infinite; + private int _writeTimeout = Timeout.Infinite; + + // Guards what follows; never held while waiting. + private readonly object _sync = new object(); + private Task _pendingConnect; + private Task _pendingRead; + private byte[] _pendingReadBuffer; + private Task _pendingWrite; + // What a finished read brought that nobody has taken yet; _eof once it brought nothing. + private byte[] _readahead; + private int _readaheadOffset, _readaheadCount; + private bool _eof; + private bool _closed; + private readonly List/*!*/ _waiters = new List(); + + // A TLS record carries at most 16 KB; a non-blocking write hands over no more than + // that at a time, so that what it reports as written is about what a kernel buffer holds. + private const int ChunkSize = 16 * 1024; + + // The context's own CAs (ca_file, ca_path, cert_store): the self-signed ones + // are the roots, the others may complete the chain. Null means the system + // store's; _systemTrust says whether the system's count as well. + private X509Certificate2Collection _roots; + private X509Certificate2Collection _intermediates; + private bool _systemTrust = true; + private Proc _verifyCallback; + private Exception _callbackError; + + // Why the connection cannot be used any more, once a read, a write or the + // handshake timed out: SslStream is in an undefined state after that. + private string _broken; [RubyConstructor] public static Transport/*!*/ Create(RubyClass/*!*/ self, [NotNull]object/*!*/ io) { @@ -568,36 +616,340 @@ public class Transport { throw RubyExceptions.CreateTypeError("SSLSocket needs a socket to run on"); } var result = new Transport(); + result._context = self.Context; result._socket = socket.Socket; // The Ruby socket layer leaves Socket.Blocking false and does its own // polling; SslStream drives the socket itself and needs it blocking. result._socket.Blocking = true; + IoReadiness.SetHook(socket, result); return result; } + /// + /// The CA certificates to check the server's chain against, as X509Certificate2s + /// (OpenSSL::X509::Certificate#__clr). As in OpenSSL::X509::Store#verify, the + /// self-signed ones are trusted and the others only help build the chain. + /// system says whether the system's trust store counts as well, which it does + /// for a cert_store that had set_default_paths called on it. + /// + [RubyMethod("trust")] + public static void Trust(Transport/*!*/ self, [NotNull]IList/*!*/ certificates, bool system) { + var roots = new X509Certificate2Collection(); + var intermediates = new X509Certificate2Collection(); + foreach (object item in certificates) { + var certificate = item as X509Certificate2; + if (certificate == null) { + throw RubyExceptions.CreateTypeError("trust anchors must be X509Certificate2s"); + } + intermediates.Add(certificate); + if (certificate.SubjectName.RawData.AsSpan().SequenceEqual(certificate.IssuerName.RawData)) { + roots.Add(certificate); + } + } + self._roots = roots; + self._intermediates = intermediates; + self._systemTrust = system; + } + + /// + /// Called once with the chain and the SslPolicyErrors as an Integer when the + /// server's certificate has been checked; what it returns decides. + /// OpenSSL::SSL::SSLSocket runs SSLContext#verify_callback from it. + /// + [RubyMethod("verify_callback=")] + public static void SetVerifyCallback(Transport/*!*/ self, Proc callback) { + self._verifyCallback = callback; + } + + // What a server certificate has to be good for: TLS server authentication. + private static readonly Oid ServerAuthentication = new Oid("1.3.6.1.5.5.7.3.1"); + private bool Validate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors errors) { - return !_verify || errors == SslPolicyErrors.None; + if (!_verify) { + return true; + } + // #connect gave up on this handshake (Thread#raise, Timeout), or the socket was + // closed under it (an open_timeout); nothing waits for an answer. + if (_broken != null || _closed) { + return false; + } + + // No SNI name means net/http connected to an IP address: it checks the + // name itself afterwards (SSLSocket#post_connection_check), as on MRI. + if (_hostname.Length == 0) { + errors &= ~SslPolicyErrors.RemoteCertificateNameMismatch; + } + + X509Chain custom = null; + X509Certificate2 leaf = null; + try { + if (_roots != null && certificate != null) { + custom = new X509Chain(); + var policy = custom.ChainPolicy; + policy.RevocationMode = X509RevocationMode.NoCheck; + policy.TrustMode = X509ChainTrustMode.CustomRootTrust; + policy.ApplicationPolicy.Add(ServerAuthentication); + policy.CustomTrustStore.AddRange(_roots); + policy.ExtraStore.AddRange(_intermediates); + // The intermediates the server sent. + if (chain != null) { + policy.ExtraStore.AddRange(chain.ChainPolicy.ExtraStore); + } + leaf = new X509Certificate2(certificate); + bool trusted = custom.Build(leaf); + if (_systemTrust && (errors & SslPolicyErrors.RemoteCertificateChainErrors) == 0) { + trusted = true; + } else { + chain = custom; + } + errors = trusted + ? errors & ~SslPolicyErrors.RemoteCertificateChainErrors + : errors | SslPolicyErrors.RemoteCertificateChainErrors; + } + + if (_verifyCallback == null) { + return errors == SslPolicyErrors.None; + } + + // SslStream runs this on whichever thread finished reading the server's + // certificate. SSLSocket's callback handles the exceptions of + // SSLContext#verify_callback itself; anything else is carried back to #connect. + try { + return _verifyCallback.Call(null, chain, (int)errors) is bool accepted && accepted; + } catch (Exception error) { + _callbackError = error; + return false; + } + } finally { + custom?.Dispose(); + leaf?.Dispose(); + } } /// /// The client handshake. hostname is the SNI name and the name the /// certificate is checked against; verify false accepts any certificate, - /// which is what OpenSSL::SSL::VERIFY_NONE means. + /// which is what OpenSSL::SSL::VERIFY_NONE means. timeout is in + /// milliseconds, -1 for none; a handshake that takes longer raises + /// IO::TimeoutError. /// [RubyMethod("connect")] public static Transport/*!*/ Connect(Transport/*!*/ self, - [DefaultProtocol]MutableString hostname, bool verify) { + [DefaultProtocol]MutableString hostname, bool verify, [DefaultProtocol, DefaultParameterValue(-1)]int timeout) { + + // The token bounds the handshake and cancels the read it is waiting in. + using (var cancel = timeout < 0 ? new CancellationTokenSource() : new CancellationTokenSource(timeout)) { + Task task = self.StartConnect(hostname, verify, cancel.Token); + try { + task.Wait(); + } catch (AggregateException) { + // FinishConnect reports it + } catch (ThreadInterruptedException) { + // Thread#raise or Thread#kill (Timeout.timeout among them) woke this + // thread up. The handshake goes on without it: stop it, make sure a + // late certificate check does not call into Ruby, and let the + // exception that was raised into this thread through. + self._broken = "the TLS handshake was interrupted"; + cancel.Cancel(); +#if FEATURE_THREAD && FEATURE_EXCEPTION_STATE + RubyUtils.TranslateThreadInterrupt(); +#endif + throw; + } + if (cancel.IsCancellationRequested && !task.IsCompletedSuccessfully) { + // An OperationCanceledException, or an IOException from the cancelled read. + lock (self._sync) { + self._pendingConnect = null; + } + throw self.TimedOut("TLS handshake timed out after " + timeout + " ms"); + } + } + self.FinishConnect(); + return self; + } + + /// + /// The handshake without waiting for it: the transport once it is done, else + /// :wait_readable, and SSLSocket#connect_nonblock asks again after IO.select. + /// + [RubyMethod("connect_nonblock")] + public static object/*!*/ ConnectNonBlocking(Transport/*!*/ self, [DefaultProtocol]MutableString hostname, bool verify) { + Task task = self.StartConnect(hostname, verify, CancellationToken.None); + if (!task.IsCompleted) { + return self._context.CreateAsciiSymbol("wait_readable"); + } + self.FinishConnect(); + return self; + } + + // The handshake in flight, started if it is not. + private Task/*!*/ StartConnect(MutableString hostname, bool verify, CancellationToken cancel) { + lock (_sync) { + CheckUsable(); + if (_pendingConnect != null) { + return _pendingConnect; + } + if (_ssl != null) { + return Task.CompletedTask; + } + _verify = verify; + _hostname = hostname == null ? "" : hostname.ConvertToString(); + _callbackError = null; + try { + _ssl = new SslStream(new NetworkStream(_socket, false), false, + new RemoteCertificateValidationCallback(Validate)); + var options = new SslClientAuthenticationOptions { TargetHost = _hostname }; + _pendingConnect = _ssl.AuthenticateAsClientAsync(options, cancel); + } catch (Exception error) { + throw new SSLError(Unwrap(error).Message); + } + NotifyOnCompletion(_pendingConnect); + return _pendingConnect; + } + } - self._verify = verify; - self._hostname = hostname == null ? "" : hostname.ConvertToString(); + private void FinishConnect() { + Task task; + lock (_sync) { + task = _pendingConnect; + _pendingConnect = null; + } + if (task == null) { + return; + } try { - self._ssl = new SslStream(new NetworkStream(self._socket, false), false, - new RemoteCertificateValidationCallback(self.Validate)); - self._ssl.AuthenticateAsClient(self._hostname); + task.GetAwaiter().GetResult(); } catch (Exception error) { + // The stream never authenticated: a second #connect must not find it and + // call the connection made. + lock (_sync) { + _broken = "the TLS handshake failed"; + } + if (_callbackError != null) { + throw _callbackError; + } throw new SSLError(Unwrap(error).Message); } - return self; + } + + /// Milliseconds a blocking read may wait, -1 for ever. + [RubyMethod("read_timeout=")] + public static void SetReadTimeout(Transport/*!*/ self, [DefaultProtocol]int milliseconds) { + self._readTimeout = milliseconds < 0 ? Timeout.Infinite : Math.Max(milliseconds, 1); + } + + /// Milliseconds a blocking write may wait, -1 for ever. + [RubyMethod("write_timeout=")] + public static void SetWriteTimeout(Transport/*!*/ self, [DefaultProtocol]int milliseconds) { + self._writeTimeout = milliseconds < 0 ? Timeout.Infinite : Math.Max(milliseconds, 1); + } + + // Waits for an operation in flight, as a blocking call does; false when the time + // ran out first. A Thread#raise or Thread#kill gets through, and the operation + // stays in flight for whoever asks next. + private static bool WaitFor(Task/*!*/ task, int milliseconds) { + try { + return task.Wait(milliseconds); + } catch (AggregateException) { + // done, with an error the caller reports + return true; + } catch (ThreadInterruptedException) { +#if FEATURE_THREAD && FEATURE_EXCEPTION_STATE + RubyUtils.TranslateThreadInterrupt(); +#endif + throw; + } + } + + private void NotifyOnCompletion(Task/*!*/ task) { + if (!task.IsCompleted) { + task.ContinueWith(_ => NotifyWaiters(), CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + } + } + + private void NotifyWaiters() { + ReadinessWaiter[] waiters; + lock (_waiters) { + waiters = _waiters.ToArray(); + } + foreach (var waiter in waiters) { + waiter.Signal(); + } + } + + void IReadinessHook.AddWaiter(ReadinessWaiter/*!*/ waiter) { + lock (_waiters) { + _waiters.Add(waiter); + } + } + + void IReadinessHook.RemoveWaiter(ReadinessWaiter/*!*/ waiter) { + lock (_waiters) { + _waiters.Remove(waiter); + } + } + + /// + /// What IO.select and NIO::Selector are told about the TCP socket under this + /// transport. Before the handshake and after #close the socket answers for + /// itself; a broken connection is ready, so that the next call says so. + /// + bool? IReadinessHook.IsReady(int kind) { + lock (_sync) { + if (_ssl == null || _closed) { + return null; + } + if (_broken != null) { + return true; + } + if (_pendingConnect != null) { + return _pendingConnect.IsCompleted; + } + if (kind == IoReadiness.Read) { + if (_readaheadCount > 0 || _eof) { + return true; + } + if (_pendingRead == null) { + try { + StartRead(ChunkSize); + } catch (Exception) { + // the read reports it + return true; + } + } + return _pendingRead.IsCompleted; + } + return _pendingWrite == null ? (bool?)null : _pendingWrite.IsCompleted; + } + } + + /// + /// Whether the server's certificate is valid for name, a host name or an IP + /// address: SSLSocket#post_connection_check. As MRI's + /// verify_certificate_identity, a certificate without a subjectAltName is + /// matched on its common name, for an IP address too. + /// + [RubyMethod("matches_hostname?")] + public static bool MatchesHostname(Transport/*!*/ self, [DefaultProtocol, NotNull]MutableString/*!*/ name) { + var certificate = self._ssl?.RemoteCertificate; + string host = name.ConvertToString(); + if (certificate == null || host.Length == 0) { + return false; + } + using (var certificate2 = new X509Certificate2(certificate)) { + IPAddress address, commonNameAddress; + if (certificate2.Extensions["2.5.29.17"] == null && IPAddress.TryParse(host, out address)) { + string commonName = certificate2.GetNameInfo(X509NameType.SimpleName, false); + return IPAddress.TryParse(commonName, out commonNameAddress) && commonNameAddress.Equals(address); + } + try { + return certificate2.MatchesHostname(host, true, true); + } catch (ArgumentException) { + // not a host name at all + return false; + } + } } // An SslStream failure arrives wrapped in an AuthenticationException whose @@ -607,54 +959,238 @@ private bool Validate(object sender, X509Certificate certificate, X509Chain chai ? error.InnerException : error; } + // Marks the connection unusable and builds the IO::TimeoutError to raise. + private Exception/*!*/ TimedOut(string/*!*/ message) { + _broken = message; + object ioClass; + if (_context.ObjectClass.TryGetConstant(null, "IO", out ioClass) && ioClass is RubyModule) { + var error = _context.CreateLibraryException((RubyModule)ioClass, "TimeoutError", message); + if (error != null) { + return error; + } + } + return new SSLError(message); + } + + private void CheckUsable() { + if (_broken != null) { + throw new SSLError("SSL connection is unusable: " + _broken); + } + } + + // The stream, once the handshake is over; call with _sync held. private SslStream/*!*/ Stream { get { - if (_ssl == null) { + CheckUsable(); + if (_closed) { + throw RubyExceptions.CreateIOError("closed stream"); + } + if (_ssl == null || _pendingConnect != null) { throw new SSLError("SSL session is not started yet"); } return _ssl; } } - /// Up to length bytes, or nil at end of stream. - [RubyMethod("read")] - public static MutableString Read(Transport/*!*/ self, [DefaultProtocol]int length) { - byte[] buffer = new byte[length]; - int read; + // Errors of SslStream's operations, as Ruby sees them. + private Exception/*!*/ Translate(Exception/*!*/ error) { + if (error is ObjectDisposedException || _closed) { + return RubyExceptions.CreateIOError("closed stream"); + } + return new SSLError(Unwrap(error).Message); + } + + // Starts a read into a buffer of its own; call with _sync held and none in flight. + private void StartRead(int length) { + var stream = Stream; + _pendingReadBuffer = new byte[Math.Max(Math.Min(length, 1 << 20), ChunkSize)]; + _pendingRead = stream.ReadAsync(_pendingReadBuffer, 0, _pendingReadBuffer.Length); + NotifyOnCompletion(_pendingRead); + } + + // Moves what the finished read brought to the read-ahead; call with _sync held. + private void FinishRead() { + var task = _pendingRead; + _pendingRead = null; + int count; try { - read = self.Stream.Read(buffer, 0, length); - } catch (IOException error) { - throw new SSLError(Unwrap(error).Message); + count = task.GetAwaiter().GetResult(); + } catch (Exception error) { + throw Translate(error); } - if (read == 0) { + if (count == 0) { + _eof = true; + } else { + _readahead = _pendingReadBuffer; + _readaheadOffset = 0; + _readaheadCount = count; + } + _pendingReadBuffer = null; + } + + // Up to length bytes of the read-ahead, or null if it is empty; call with _sync held. + private MutableString TakeReadahead(int length) { + if (_readaheadCount == 0) { return null; } - Array.Resize(ref buffer, read); - return MutableString.CreateBinary(buffer); + int count = Math.Min(length, _readaheadCount); + var result = new byte[count]; + Buffer.BlockCopy(_readahead, _readaheadOffset, result, 0, count); + _readaheadOffset += count; + _readaheadCount -= count; + if (_readaheadCount == 0) { + _readahead = null; + } + return MutableString.CreateBinary(result); + } + + /// Up to length bytes, or nil at end of stream; waits at most read_timeout. + [RubyMethod("read")] + public static MutableString Read(Transport/*!*/ self, [DefaultProtocol]int length) { + Task task; + lock (self._sync) { + _ = self.Stream; + var data = self.TakeReadahead(length); + if (data != null || self._eof) { + return data; + } + if (self._pendingRead == null) { + self.StartRead(length); + } + task = self._pendingRead; + } + if (!WaitFor(task, self._readTimeout)) { + throw self.TimedOut("read timed out after " + self._readTimeout + " ms"); + } + lock (self._sync) { + if (self._pendingRead == task) { + self.FinishRead(); + } + return self.TakeReadahead(length); + } + } + + /// + /// Up to length bytes if a read has brought them, nil at end of stream, else + /// :wait_readable, with a read left in flight that IO.select sees finish. + /// + [RubyMethod("read_nonblock")] + public static object ReadNonBlocking(Transport/*!*/ self, [DefaultProtocol]int length) { + lock (self._sync) { + _ = self.Stream; + var data = self.TakeReadahead(length); + if (data != null || self._eof) { + return data; + } + if (self._pendingRead == null) { + self.StartRead(length); + } + if (!self._pendingRead.IsCompleted) { + return self._context.CreateAsciiSymbol("wait_readable"); + } + self.FinishRead(); + return self.TakeReadahead(length); + } } + // Takes over a write in flight before another; call without _sync held. + private void FinishWrite(Task/*!*/ task) { + lock (_sync) { + if (_pendingWrite == task) { + _pendingWrite = null; + } + } + try { + task.GetAwaiter().GetResult(); + } catch (Exception error) { + throw Translate(error); + } + } + + /// All of data; waits at most write_timeout. [RubyMethod("write")] public static int Write(Transport/*!*/ self, [DefaultProtocol, NotNull]MutableString/*!*/ data) { byte[] bytes = data.ConvertToBytes(); - try { - self.Stream.Write(bytes, 0, bytes.Length); - self.Stream.Flush(); - } catch (IOException error) { - throw new SSLError(Unwrap(error).Message); + Task pending, task; + lock (self._sync) { + self.CheckUsable(); + pending = self._pendingWrite; + } + if (pending != null) { + if (!WaitFor(pending, self._writeTimeout)) { + throw self.TimedOut("write timed out after " + self._writeTimeout + " ms"); + } + self.FinishWrite(pending); + } + lock (self._sync) { + var stream = self.Stream; + try { + task = stream.WriteAsync(bytes, 0, bytes.Length); + } catch (Exception error) { + throw self.Translate(error); + } + self._pendingWrite = task; + self.NotifyOnCompletion(task); + } + if (!WaitFor(task, self._writeTimeout)) { + throw self.TimedOut("write timed out after " + self._writeTimeout + " ms"); } + self.FinishWrite(task); return bytes.Length; } + /// + /// How much of data was handed over - at most 16 KB, which may still be on its way, + /// as bytes in a kernel buffer are - or :wait_writable while the last one is. + /// + [RubyMethod("write_nonblock")] + public static object WriteNonBlocking(Transport/*!*/ self, [DefaultProtocol, NotNull]MutableString/*!*/ data) { + Task pending; + lock (self._sync) { + self.CheckUsable(); + pending = self._pendingWrite; + if (pending != null && !pending.IsCompleted) { + return self._context.CreateAsciiSymbol("wait_writable"); + } + } + if (pending != null) { + self.FinishWrite(pending); + } + byte[] bytes = data.ConvertToBytes(); + int count = Math.Min(bytes.Length, ChunkSize); + Task task; + lock (self._sync) { + var stream = self.Stream; + try { + task = stream.WriteAsync(bytes, 0, count); + } catch (Exception error) { + throw self.Translate(error); + } + if (!task.IsCompleted) { + self._pendingWrite = task; + self.NotifyOnCompletion(task); + return count; + } + } + self.FinishWrite(task); + return count; + } + [RubyMethod("close")] public static void Close(Transport/*!*/ self) { - if (self._ssl != null) { + SslStream ssl; + lock (self._sync) { + ssl = self._ssl; + self._closed = true; + } + if (ssl != null) { try { - self._ssl.Dispose(); + ssl.Dispose(); } catch (Exception) { // closing a connection the peer already dropped is not an error } - self._ssl = null; } + self.NotifyWaiters(); } /// The peer's certificate as DER, or nil before the handshake. @@ -692,10 +1228,16 @@ public static MutableString Cipher(Transport/*!*/ self) { return MutableString.CreateAscii(self._ssl.NegotiatedCipherSuite.ToString()); } - /// Bytes already decrypted and waiting, which IO.select cannot see. + /// Bytes already decrypted and waiting to be read. [RubyMethod("pending")] public static int Pending(Transport/*!*/ self) { - return self._ssl == null ? 0 : (self._ssl.CanRead && self._socket.Available > 0 ? self._socket.Available : 0); + lock (self._sync) { + // A failed read is left for the next read to report. + if (self._pendingRead != null && self._pendingRead.IsCompletedSuccessfully) { + self.FinishRead(); + } + return self._readaheadCount; + } } } #endif diff --git a/Src/StdLib/ironruby/openssl.rb b/Src/StdLib/ironruby/openssl.rb index 47d9a8c35..8ebe9f645 100644 --- a/Src/StdLib/ironruby/openssl.rb +++ b/Src/StdLib/ironruby/openssl.rb @@ -22,10 +22,10 @@ module SSL VERIFY_FAIL_IF_NO_PEER_CERT = 2 VERIFY_CLIENT_ONCE = 4 - # The settings an SSLSocket is made with. Only #verify_mode is acted on - # (see openssl/ssl.rb): the TLS behind SSLSocket is .NET's SslStream, which - # takes its trust anchors, protocol versions and cipher list from the - # platform rather than from a context object. + # The settings an SSLSocket is made with. #verify_mode, #ca_file, #ca_path, + # #cert_store and #verify_callback are acted on (see openssl/ssl.rb); the + # TLS behind SSLSocket is .NET's SslStream, which takes protocol versions + # and the cipher list from the platform rather than from a context object. class SSLContext SESSION_CACHE_OFF = 0x0000 SESSION_CACHE_CLIENT = 0x0001 @@ -591,6 +591,35 @@ module X509 class ExtensionError < OpenSSLError; end unless const_defined?(:ExtensionError, false) class StoreError < OpenSSLError; end unless const_defined?(:StoreError, false) + # OpenSSL's X509_V_* verification results (x509_vfy.h), which StoreContext#error, + # Store#error and SSLSocket#verify_result answer with. + { + OK: 0, ERR_UNSPECIFIED: 1, ERR_UNABLE_TO_GET_ISSUER_CERT: 2, ERR_UNABLE_TO_GET_CRL: 3, + ERR_UNABLE_TO_DECRYPT_CERT_SIGNATURE: 4, ERR_UNABLE_TO_DECRYPT_CRL_SIGNATURE: 5, + ERR_UNABLE_TO_DECODE_ISSUER_PUBLIC_KEY: 6, ERR_CERT_SIGNATURE_FAILURE: 7, + ERR_CRL_SIGNATURE_FAILURE: 8, ERR_CERT_NOT_YET_VALID: 9, ERR_CERT_HAS_EXPIRED: 10, + ERR_CRL_NOT_YET_VALID: 11, ERR_CRL_HAS_EXPIRED: 12, ERR_ERROR_IN_CERT_NOT_BEFORE_FIELD: 13, + ERR_ERROR_IN_CERT_NOT_AFTER_FIELD: 14, ERR_ERROR_IN_CRL_LAST_UPDATE_FIELD: 15, + ERR_ERROR_IN_CRL_NEXT_UPDATE_FIELD: 16, ERR_OUT_OF_MEM: 17, ERR_DEPTH_ZERO_SELF_SIGNED_CERT: 18, + ERR_SELF_SIGNED_CERT_IN_CHAIN: 19, ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY: 20, + ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE: 21, ERR_CERT_CHAIN_TOO_LONG: 22, ERR_CERT_REVOKED: 23, + ERR_INVALID_CA: 24, ERR_PATH_LENGTH_EXCEEDED: 25, ERR_INVALID_PURPOSE: 26, + ERR_CERT_UNTRUSTED: 27, ERR_CERT_REJECTED: 28, ERR_SUBJECT_ISSUER_MISMATCH: 29, + ERR_AKID_SKID_MISMATCH: 30, ERR_AKID_ISSUER_SERIAL_MISMATCH: 31, ERR_KEYUSAGE_NO_CERTSIGN: 32, + ERR_UNABLE_TO_GET_CRL_ISSUER: 33, ERR_UNHANDLED_CRITICAL_EXTENSION: 34, + ERR_KEYUSAGE_NO_CRL_SIGN: 35, ERR_UNHANDLED_CRITICAL_CRL_EXTENSION: 36, ERR_INVALID_NON_CA: 37, + ERR_PROXY_PATH_LENGTH_EXCEEDED: 38, ERR_KEYUSAGE_NO_DIGITAL_SIGNATURE: 39, + ERR_PROXY_CERTIFICATES_NOT_ALLOWED: 40, ERR_INVALID_EXTENSION: 41, + ERR_INVALID_POLICY_EXTENSION: 42, ERR_NO_EXPLICIT_POLICY: 43, ERR_DIFFERENT_CRL_SCOPE: 44, + ERR_UNSUPPORTED_EXTENSION_FEATURE: 45, ERR_UNNESTED_RESOURCE: 46, ERR_PERMITTED_VIOLATION: 47, + ERR_EXCLUDED_VIOLATION: 48, ERR_SUBTREE_MINMAX: 49, ERR_APPLICATION_VERIFICATION: 50, + ERR_UNSUPPORTED_CONSTRAINT_TYPE: 51, ERR_UNSUPPORTED_CONSTRAINT_SYNTAX: 52, + ERR_UNSUPPORTED_NAME_SYNTAX: 53, ERR_CRL_PATH_VALIDATION_ERROR: 54, + ERR_HOSTNAME_MISMATCH: 62, ERR_EMAIL_MISMATCH: 63, ERR_IP_ADDRESS_MISMATCH: 64, + }.each do |name, code| + const_set("V_#{name}", code) unless const_defined?("V_#{name}", false) + end + # A certificate extension. The value is kept as the OpenSSL configuration # string it was created from; the DER lives in the CLR extension object. class Extension @@ -855,6 +884,7 @@ def initialize @error = nil @error_string = nil @chain = nil + @default_paths = false end def add_cert(certificate) @@ -877,7 +907,10 @@ def add_path(path) self end + # Store#verify still trusts only the certificates added; an SSLSocket whose + # context has this store trusts the system's certificates as well. def set_default_paths + @default_paths = true self end @@ -907,32 +940,124 @@ def verify(certificate, chain = nil) @error = 0 @error_string = "ok" else - @error, @error_string = __first_error(builder) + @error, @error_string = IronRubyOpenSSL__.x509_error(builder.ChainStatus, true) end result end - private + # What SSLSocket builds its trust anchors from. + def __certificates # :nodoc: + @certificates.dup + end - # X509Chain reports a set of flags; MRI reports the X509_V_ERR_* code of the - # first problem OpenSSL hit. Only the codes these flags correspond to are - # mapped, in roughly the order OpenSSL would notice them. - def __first_error(builder) - builder.ChainStatus.to_a.each do |status| - case status.Status.to_s - when "NotTimeValid", "CtlNotTimeValid" - return [10, "certificate has expired"] - when "NotSignatureValid" - return [7, "certificate signature failure"] - when "UntrustedRoot", "ExplicitDistrust" - return [19, "self signed certificate in certificate chain"] - when "InvalidBasicConstraints" - return [24, "invalid CA certificate"] - end + def __default_paths? # :nodoc: + @default_paths + end + end + + # What a verify_callback is handed: the chain, the certificate being looked at, + # and the error it has, if any. Only SSLSocket makes these here; MRI's + # StoreContext.new(store, cert, chain).verify is not implemented. + class StoreContext + attr_reader :chain, :current_cert, :error_depth + attr_accessor :error, :error_string + + def self.__for_callback(chain, depth, error, error_string) # :nodoc: + context = allocate + context.instance_variable_set(:@chain, chain) + context.instance_variable_set(:@current_cert, chain[depth]) + context.instance_variable_set(:@error_depth, depth) + context.error = error + context.error_string = error_string + context + end + + def initialize(*) + raise NotImplementedError, "OpenSSL::X509::StoreContext.new is not implemented; use Store#verify" + end + end + end +end + +module IronRubyOpenSSL__ # :nodoc: all + # X509Chain reports a set of flags; OpenSSL reports an X509_V_ERR_* code for each + # problem it hits. +statuses+ is an X509ChainStatus[] (a chain's or one + # element's), and the result is every [code, message] the flags correspond to. + # +certificate+ (an OpenSSL::X509::Certificate), +depth+ and +chain_size+ say + # which certificate the flags are about, which some codes depend on: an untrusted + # self-signed certificate on its own is 18, inside a chain 19; a certificate + # outside its validity period is 9 before it and 10 after. + def self.x509_errors(statuses, certificate = nil, depth = nil, chain_size = nil) + x509 = ::OpenSSL::X509 + errors = statuses.to_a.map do |status| + case status.Status.to_s + when "NoError", "RevocationStatusUnknown", "OfflineRevocation" + nil + when "NotTimeValid", "CtlNotTimeValid" + if certificate && certificate.not_before && certificate.not_before > ::Time.now + [x509::V_ERR_CERT_NOT_YET_VALID, "certificate is not yet valid"] + else + [x509::V_ERR_CERT_HAS_EXPIRED, "certificate has expired"] + end + when "NotSignatureValid", "CtlNotSignatureValid" + [x509::V_ERR_CERT_SIGNATURE_FAILURE, "certificate signature failure"] + when "UntrustedRoot" + if depth == 0 && chain_size == 1 + [x509::V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT, "self signed certificate"] + else + [x509::V_ERR_SELF_SIGNED_CERT_IN_CHAIN, "self signed certificate in certificate chain"] end - [20, "unable to get local issuer certificate"] + when "PartialChain" + [x509::V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY, "unable to get local issuer certificate"] + when "ExplicitDistrust" + [x509::V_ERR_CERT_REJECTED, "certificate rejected"] + when "Revoked" + [x509::V_ERR_CERT_REVOKED, "certificate revoked"] + when "InvalidBasicConstraints" + [x509::V_ERR_INVALID_CA, "invalid CA certificate"] + when "NotValidForUsage", "CtlNotValidForUsage" + # Linux's .NET flags every certificate of a chain whose leaf has the wrong + # purpose; OpenSSL reports the one that does not allow server authentication. + next nil if certificate && server_authentication?(certificate) + [x509::V_ERR_INVALID_PURPOSE, "unsupported certificate purpose"] + when "HasNotPermittedNameConstraint" + [x509::V_ERR_PERMITTED_VIOLATION, "permitted subtree violation"] + when "HasExcludedNameConstraint" + [x509::V_ERR_EXCLUDED_VIOLATION, "excluded subtree violation"] + when "InvalidExtension" + [x509::V_ERR_INVALID_EXTENSION, "invalid or inconsistent certificate extension"] + else + [x509::V_ERR_UNSPECIFIED, "unspecified certificate verification error"] end end + # In the order OpenSSL's verify_chain meets them: building the chain, then the + # extensions and purpose, then signatures and validity periods. .NET's order is + # its own, and a verify_callback that rejects stops at the first. + errors.compact.uniq.sort_by { |code, _| X509_ERROR_ORDER.index(code) || X509_ERROR_ORDER.size } + end + + X509_ERROR_ORDER = [2, 18, 19, 20, 21, 23, 28, 24, 26, 41, 47, 48, 7, 9, 10].freeze + + SERVER_AUTHENTICATION = ["1.3.6.1.5.5.7.3.1", "2.5.29.37.0"].freeze # serverAuth, anyExtendedKeyUsage + + # Whether the certificate's own extendedKeyUsage allows TLS server + # authentication; a certificate without one allows every purpose. + def self.server_authentication?(certificate) + clr = certificate.__clr + return true if clr.nil? + usage = clr.Extensions.to_a.find { |extension| extension.Oid.Value.to_s == "2.5.29.37" } + return true if usage.nil? + usage.EnhancedKeyUsages.to_a.any? { |oid| SERVER_AUTHENTICATION.include?(oid.Value.to_s) } + end + + # The first of those, for Store#verify; with +failed+ the chain is known to be + # bad, so no flag still means "no trusted root". A flag without an OpenSSL + # equivalent also counts as that, as Store#verify has always reported it. + def self.x509_error(statuses, failed = false) + errors = x509_errors(statuses) + error = errors.find { |code, _| code != ::OpenSSL::X509::V_ERR_UNSPECIFIED } + return error if error + failed || !errors.empty? ? [20, "unable to get local issuer certificate"] : [0, "ok"] end end diff --git a/Src/StdLib/ironruby/openssl/ssl.rb b/Src/StdLib/ironruby/openssl/ssl.rb index 06a4afcdc..1e2528df0 100644 --- a/Src/StdLib/ironruby/openssl/ssl.rb +++ b/Src/StdLib/ironruby/openssl/ssl.rb @@ -3,15 +3,25 @@ # # What that costs, compared with MRI: # -# * there is no non-blocking handshake or non-blocking IO. SslStream drives -# the socket itself and has no "would block" answer, so #connect_nonblock -# does the whole handshake and #read_nonblock / #write_nonblock block. -# They never return :wait_readable, which the callers (net/protocol's -# BufferedIO, net/http) treat as "the data is here", so they work. -# * the trust decisions are .NET's: the system trust store, and the -# certificate's own name check against the SNI hostname. An -# SSLContext#cert_store, #ca_file or #verify_callback is not consulted -- -# only #verify_mode, and only for VERIFY_NONE against everything else. +# * SslStream drives the socket itself, so "would block" is about SslStream's +# operations, not the socket: #connect_nonblock, #read_nonblock and +# #write_nonblock start one and answer :wait_readable / :wait_writable (or +# raise SSLErrorWaitReadable / SSLErrorWaitWritable) while it runs. +# IO.select, IO#wait_readable on #to_io and NIO::Selector ask the transport, +# which is the TCP socket's readiness hook, so they see it finish - which is +# how net/protocol applies open_timeout, read_timeout and write_timeout, for +# the standard library's Net and RubyGems' Gem::Net alike. +# * the blocking calls (#connect, #sysread, #syswrite) wait for such an +# operation; #__connect(timeout), #__read_timeout= and #__write_timeout= +# bound the wait, and running out raises IO::TimeoutError and leaves the +# connection unusable. +# * the chain is built by .NET: against the system trust store, or, when the +# context has #ca_file, #ca_path or #cert_store, against the self-signed +# certificates in those (and the system store as well for a cert_store that +# had set_default_paths called on it); a certificate also has to be good for +# TLS server authentication. #verify_callback is called for each +# certificate of the chain, root first, once per error (or once with +# preverify_ok true), and has the last word. # * server sockets (SSLServer, #accept) are not implemented. module OpenSSL @@ -98,21 +108,46 @@ def initialize(io, context = nil) @connected = false @eof = false @sync = true + @verify_result = OpenSSL::X509::V_OK + @read_timeout = @write_timeout = -1 end def connect + __connect(nil) + end + + # The handshake, given at most +timeout+ seconds (nil: no limit); running out + # raises IO::TimeoutError. + def __connect(timeout) # :nodoc: return self if @connected - verify = @context.respond_to?(:verify_mode) && - @context.verify_mode != OpenSSL::SSL::VERIFY_NONE - @transport.connect(@hostname.to_s, verify) + @transport.connect(@hostname.to_s, __verification, __milliseconds(timeout)) @connected = true self end - # There is no partial handshake here, so this is #connect; the argument is - # accepted because net/protocol's ssl_socket_connect passes exception: false. - def connect_nonblock(options = nil) - connect + # Starts the handshake, or looks at the one in flight: self once it is done, + # else SSLErrorWaitReadable, or :wait_readable with exception: false. + # IO.select on #to_io says when to ask again. + def connect_nonblock(exception: true) + return self if @connected + if @transport.connect_nonblock(@hostname.to_s, __verification) == :wait_readable + raise SSLErrorWaitReadable, "read would block" if exception + return :wait_readable + end + @connected = true + self + end + + # How long one blocking read or write may wait, in seconds, nil for ever. + # Not MRI API; the non-blocking calls are bounded by whoever waits for them. + def __read_timeout=(seconds) # :nodoc: + milliseconds = __milliseconds(seconds) + @transport.read_timeout = @read_timeout = milliseconds unless milliseconds == @read_timeout + end + + def __write_timeout=(seconds) # :nodoc: + milliseconds = __milliseconds(seconds) + @transport.write_timeout = @write_timeout = milliseconds unless milliseconds == @write_timeout end def accept @@ -121,13 +156,15 @@ def accept end alias_method :accept_nonblock, :accept - # MRI checks the certificate against the hostname here; SslStream already - # did it during the handshake unless verification was off altogether. + # The certificate's names (subjectAltName DNS and IP entries, the common + # name failing those) against +hostname+, as MRI does. net/http calls this + # for an IP address too, which gets no SNI name and so no check during the + # handshake. def post_connection_check(hostname) return true if @context.respond_to?(:verify_mode) && @context.verify_mode == OpenSSL::SSL::VERIFY_NONE raise SSLError, "hostname \"#{hostname}\" does not match the server certificate" unless - hostname.to_s.downcase == @hostname.to_s.downcase || @hostname.to_s.empty? + @transport.matches_hostname?(hostname.to_s) true end @@ -161,8 +198,10 @@ def pending @transport.pending end + # V_OK for a connection that stands: a verify_callback that accepts a + # certificate with an error clears the error, as on MRI. def verify_result - OpenSSL::X509::V_OK + @verify_result end def sysread(length, buffer = nil) @@ -176,10 +215,18 @@ def sysread(length, buffer = nil) end def read_nonblock(length, buffer = nil, exception: true) - sysread(length, buffer) - rescue EOFError - raise if exception - nil + connect unless @connected + case data = @transport.read_nonblock(length) + when :wait_readable + raise SSLErrorWaitReadable, "read would block" if exception + :wait_readable + when nil + @eof = true + raise EOFError, "end of file reached" if exception + nil + else + buffer.nil? ? data : buffer.replace(data) + end end def readpartial(length, buffer = nil) @@ -230,8 +277,16 @@ def write(*data) data.inject(0) { |total, item| total + syswrite(item.to_s) } end + # How much of +data+ went out - at most 16 KB at a time - or SSLErrorWaitWritable + # (:wait_writable with exception: false) while the last part is still going. def write_nonblock(data, exception: true) - syswrite(data) + connect unless @connected + written = @transport.write_nonblock(data.to_s) + if written == :wait_writable + raise SSLErrorWaitWritable, "write would block" if exception + return :wait_writable + end + written end def <<(data) @@ -301,6 +356,103 @@ def alpn_protocol def npn_protocol nil end + + private + + # Whether the certificate is checked at all, with the transport told what to + # check it against. Once per socket: a handshake in flight reads it. + def __verification + return @verify unless @verify.nil? + @verify = @context.respond_to?(:verify_mode) && + @context.verify_mode != OpenSSL::SSL::VERIFY_NONE + if @verify + anchors, system = __trust_anchors + @transport.trust(anchors, system) unless anchors.nil? + callback = @context.respond_to?(:verify_callback) && @context.verify_callback + @transport.verify_callback = callback ? __verifier(callback) : nil + end + @verify + end + + # Seconds as the transport's milliseconds: -1 for nil (no limit), and at least 1, + # since 0 would mean "no limit" to the socket underneath. + def __milliseconds(seconds) + return -1 if seconds.nil? + seconds = Float(seconds) + return -1 if seconds.infinite? + [[(seconds * 1000).ceil, 1].max, 0x7fffffff].min + end + + # The certificates the context trusts in place of the system store, and whether + # the system store counts as well; nil when the context names no CAs. As on + # MRI, ca_file and ca_path add to cert_store. + def __trust_anchors + ca_file = @context.respond_to?(:ca_file) ? @context.ca_file : nil + ca_path = @context.respond_to?(:ca_path) ? @context.ca_path : nil + store = @context.respond_to?(:cert_store) ? @context.cert_store : nil + return nil if ca_file.nil? && ca_path.nil? && store.nil? + anchors = OpenSSL::X509::Store.new + anchors.add_file(ca_file) unless ca_file.nil? + anchors.add_path(ca_path) unless ca_path.nil? + certificates = anchors.__certificates + system = false + if store.respond_to?(:__certificates) + certificates = store.__certificates + certificates + system = store.__default_paths? + end + [certificates.map(&:__clr).compact, system] + end + + # The transport's callback for SSLContext#verify_callback. OpenSSL calls it for + # each certificate, from the root down to the server's own: once with + # preverify_ok true if the certificate passed, otherwise once per error, false + # and an X509::StoreContext saying which. .NET has checked the whole chain by + # the time this runs, so the errors come from its X509ChainElement statuses. It + # runs on the thread that finished the handshake. + def __verifier(callback) + lambda do |chain, policy_errors| + elements = chain.nil? ? [] : chain.ChainElements.to_a + certificates = elements.map do |element| + OpenSSL::X509::Certificate.new(IronRubyOpenSSL__.bytes(element.Certificate.RawData)) + end + errors = elements.each_with_index.map do |element, depth| + IronRubyOpenSSL__.x509_errors(element.ChainElementStatus, certificates[depth], depth, certificates.size) + end + next false if certificates.empty? + # A chain that does not reach a trusted root says so on the chain, not on an element. + if policy_errors & 4 != 0 && errors.all?(&:empty?) + errors[-1] = [IronRubyOpenSSL__.x509_error(chain.ChainStatus, true)] + end + if policy_errors & 2 != 0 + errors[0] += [[OpenSSL::X509::V_ERR_HOSTNAME_MISMATCH, "hostname mismatch"]] + end + (certificates.size - 1).downto(0).all? do |depth| + (errors[depth].empty? ? [[OpenSSL::X509::V_OK, "ok"]] : errors[depth]).all? do |code, message| + __verify_call(callback, OpenSSL::X509::StoreContext.__for_callback(certificates, depth, code, message)) + end + end + end + end + + # One call of the verify_callback, as CRuby's ossl_verify_cb_call: only true + # accepts, and clears the error; anything else rejects, with + # V_ERR_CERT_REJECTED if there was no error. An exception in the callback is a + # rejection too, not an exception out of #connect. + def __verify_call(callback, context) + accepted = begin + callback.call(context.error == OpenSSL::X509::V_OK, context) == true + rescue Exception + warn "exception in verify_callback is ignored" + false + end + if accepted + context.error = OpenSSL::X509::V_OK + else + context.error = OpenSSL::X509::V_ERR_CERT_REJECTED if context.error == OpenSSL::X509::V_OK + @verify_result = context.error + end + accepted + end end # MRI's SSLServer wraps a TCPServer; .NET's server-side SslStream is not @@ -312,8 +464,4 @@ def initialize(*) end end end - - module X509 - V_OK = 0 unless const_defined?(:V_OK, false) - end end diff --git a/Src/StdLib/ironruby/socket.rb b/Src/StdLib/ironruby/socket.rb index 02ed5d1cb..6a336bbe9 100644 --- a/Src/StdLib/ironruby/socket.rb +++ b/Src/StdLib/ironruby/socket.rb @@ -1693,7 +1693,7 @@ def new(remote_host, remote_port, local_host = nil, local_port = nil, else __ir_raw_new(remote_host, remote_port, local_host, local_port || 0) end - rescue Errno::ETIMEDOUT => e + rescue Errno::ETIMEDOUT raise IO::TimeoutError, "Connection timed out" if timeout raise end diff --git a/Src/StdLib/ironruby/win32/resolv.rb b/Src/StdLib/ironruby/win32/resolv.rb index c049c8471..8e509f9c2 100644 --- a/Src/StdLib/ironruby/win32/resolv.rb +++ b/Src/StdLib/ironruby/win32/resolv.rb @@ -6,6 +6,7 @@ load_assembly 'System.Net.NetworkInformation' load_assembly 'System.Net.Primitives' +load_assembly 'Microsoft.Win32.Registry' module Win32 module Resolv @@ -24,19 +25,25 @@ def self.get_hosts_path ::File.exist?(path) ? path : nil end - # [search domains, name servers], each nil when there are none. + # [search domains, name servers], each nil when there are none. A configured + # domain suffix search list is used alone, in its order, as Windows and CRuby do; + # otherwise the machine's domain and each interface's connection-specific suffix. def self.get_resolv_info search = [] nameserver = [] + configured = search_list + search.concat(configured) if configured information = ::System::Net::NetworkInformation - domain = information::IPGlobalProperties.GetIPGlobalProperties.DomainName.to_s - search << domain unless domain.empty? + if configured.nil? + domain = information::IPGlobalProperties.GetIPGlobalProperties.DomainName.to_s + search << domain unless domain.empty? + end information::NetworkInterface.GetAllNetworkInterfaces.each do |interface| next unless interface.OperationalStatus == information::OperationalStatus.Up next if interface.NetworkInterfaceType == information::NetworkInterfaceType.Loopback properties = interface.GetIPProperties suffix = properties.DnsSuffix.to_s - search << suffix unless suffix.empty? + search << suffix unless suffix.empty? || configured properties.DnsAddresses.each do |address| # fec0:0:0:ffff::1-3 are Windows' placeholders for "no IPv6 DNS server configured" next if address.IsIPv6SiteLocal @@ -48,5 +55,21 @@ def self.get_resolv_info nameserver.uniq! [search.empty? ? nil : search, nameserver.empty? ? nil : nameserver] end + + # The domain suffix search list an administrator configured (the SearchList value + # CRuby reads too), or nil when there is none. + def self.search_list + key = ::Microsoft::Win32::Registry.LocalMachine.OpenSubKey('SYSTEM\CurrentControlSet\Services\Tcpip\Parameters') + return nil if key.nil? + begin + list = key.GetValue("SearchList").to_s.split(/,\s*/).reject(&:empty?) + list.empty? ? nil : list + ensure + key.Dispose + end + rescue ::StandardError + nil + end + private_class_method :search_list end end