From 1a1689609138f1af30456cf22d11a205ec23225b Mon Sep 17 00:00:00 2001 From: AlexanderDotH Date: Wed, 24 Dec 2025 03:42:12 +0100 Subject: [PATCH] Add BatchRequests, ProxiedBatchRequests, ProxyConfiguration classes and update AGENT.md documentation for all projects - DevBase.Net 1.2.0: New BatchRequests class for named batch processing with rate limiting - DevBase.Net 1.2.0: New ProxiedBatchRequests class with proxy rotation strategies - DevBase.Net 1.2.0: New ProxyConfiguration fluent builder for proxy setup - Updated AGENT.md files with comprehensive class references and method signatures - Version bumps: DevBase 1.3.5, DevBase.Api 1.5.1, DevBase.Format 1.7.3 - Version bumps: DevBase.Cryptography 1.0.6, DevBase.Cryptography.BouncyCastle 1.2.2 - Version bumps: DevBase.Extensions 1.0.2, DevBase.Avalonia 1.2.6, DevBase.Logging 1.0.2 - Added unit tests for BatchRequests, ProxiedBatchRequests, ProxyInfo, ProxyConfiguration --- DevBase.Api/AGENT.md | 260 +++- DevBase.Api/DevBase.Api.csproj | 2 +- DevBase.Avalonia/AGENT.md | 65 +- DevBase.Avalonia/DevBase.Avalonia.csproj | 2 +- DevBase.Cryptography.BouncyCastle/AGENT.md | 133 +- .../DevBase.Cryptography.BouncyCastle.csproj | 2 +- DevBase.Cryptography/AGENT.md | 83 +- .../DevBase.Cryptography.csproj | 2 +- DevBase.Extensions/AGENT.md | 55 +- DevBase.Extensions/DevBase.Extensions.csproj | 2 +- DevBase.Format/AGENT.md | 138 +- DevBase.Format/DevBase.Format.csproj | 2 +- DevBase.Logging/AGENT.md | 56 +- DevBase.Logging/DevBase.Logging.csproj | 2 +- DevBase.Net/AGENT.md | 1025 ++++++++++++++- DevBase.Net/Core/BatchRequests.cs | 897 +++++++++++++ DevBase.Net/Core/ProxiedBatchRequests.cs | 1160 +++++++++++++++++ DevBase.Net/DevBase.Net.csproj | 2 +- DevBase.Net/Proxy/ProxyConfiguration.cs | 182 +++ DevBase.Net/Proxy/ProxyInfo.cs | 137 +- DevBase.Net/Proxy/TrackedProxyInfo.cs | 10 +- DevBase.Net/README.md | 66 +- .../DevBaseRequests/BatchRequestsTest.cs | 399 ++++++ .../ProxiedBatchRequestsTest.cs | 386 ++++++ .../DevBaseRequests/ProxyConfigurationTest.cs | 196 +++ DevBase.Test/DevBaseRequests/ProxyInfoTest.cs | 190 +++ DevBase/AGENT.md | 292 +++++ DevBase/DevBase.csproj | 2 +- 28 files changed, 5689 insertions(+), 59 deletions(-) create mode 100644 DevBase.Net/Core/BatchRequests.cs create mode 100644 DevBase.Net/Core/ProxiedBatchRequests.cs create mode 100644 DevBase.Net/Proxy/ProxyConfiguration.cs create mode 100644 DevBase.Test/DevBaseRequests/BatchRequestsTest.cs create mode 100644 DevBase.Test/DevBaseRequests/ProxiedBatchRequestsTest.cs create mode 100644 DevBase.Test/DevBaseRequests/ProxyConfigurationTest.cs create mode 100644 DevBase.Test/DevBaseRequests/ProxyInfoTest.cs diff --git a/DevBase.Api/AGENT.md b/DevBase.Api/AGENT.md index 9c4dbfd..1acab00 100644 --- a/DevBase.Api/AGENT.md +++ b/DevBase.Api/AGENT.md @@ -11,7 +11,265 @@ DevBase.Api provides ready-to-use API clients for: All clients extend `ApiClient` and use DevBase.Net for networking. -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Api/ +├── Apis/ +│ ├── ApiClient.cs # Base class for all API clients +│ ├── AppleMusic/ # Apple Music API (20 files) +│ │ └── AppleMusic.cs +│ ├── BeautifulLyrics/ # BeautifulLyrics API (7 files) +│ │ └── BeautifulLyrics.cs +│ ├── Deezer/ # Deezer API (83 files) +│ │ └── Deezer.cs +│ ├── Musixmatch/ # Musixmatch API (13 files) +│ │ └── MusixMatch.cs +│ ├── NetEase/ # NetEase Music API (28 files) +│ │ └── NetEase.cs +│ ├── OpenAi/ # OpenAI API (3 files) +│ │ └── OpenAi.cs +│ ├── OpenLyricsClient/ # OpenLyricsClient API (8 files) +│ │ └── OpenLyricsClient.cs +│ ├── Replicate/ # Replicate API (11 files) +│ │ └── Replicate.cs +│ └── Tidal/ # Tidal API (16 files) +│ └── Tidal.cs +├── Enums/ # API-specific enums +├── Exceptions/ # API-specific exceptions +└── Serializer/ # JSON serialization +``` + +--- + +## Class Reference + +### ApiClient Class (Base) + +**Namespace:** `DevBase.Api.Apis` + +Base class for all API clients with error handling support. + +#### Properties + +| Property | Type | Default | Description | +|----------|------|---------|-------------| +| `StrictErrorHandling` | `bool` | `false` | Throw exceptions or return null | + +#### Protected Methods + +```csharp +dynamic Throw(Exception exception) // Returns null or throws +(string, bool) ThrowTuple(Exception ex) // Returns ("", false) or throws +``` + +--- + +### Deezer Class + +**Namespace:** `DevBase.Api.Apis.Deezer` + +Deezer music service API client. + +#### Constructor + +```csharp +Deezer(string arlToken = "") +``` + +#### Authentication Methods + +```csharp +Task GetJwtToken() +Task GetAccessToken(string appID = "457142") +Task GetAccessToken(string sessionID, string appID = "457142") +Task GetArlTokenFromSession(string sessionID) +Task GetCsrfToken() +``` + +#### Search & Retrieve Methods + +```csharp +Task Search(string query) +Task Search(string query, int limit, int index = 0) +Task GetSong(string trackID) +Task GetAlbum(string albumID) +Task GetArtist(string artistID) +``` + +#### Lyrics Methods + +```csharp +Task<(string RawLyrics, AList TimeStampedLyrics)> GetLyrics(string trackID) +Task GetLyricsAjax(string trackID) +Task GetLyricsGraph(string trackID) +``` + +--- + +### Tidal Class + +**Namespace:** `DevBase.Api.Apis.Tidal` + +Tidal music service API client with OAuth2 device flow. + +#### Constructor + +```csharp +Tidal() +``` + +#### Authentication Methods + +```csharp +Task RegisterDevice() +Task GetTokenFrom(string deviceCode) +Task Login(string accessToken) +Task RefreshSession(string refreshToken) +``` + +#### Search & Retrieve Methods + +```csharp +Task Search(string query, string sessionId) +Task GetTrack(string trackId, string sessionId) +Task GetAlbum(string albumId, string sessionId) +Task GetLyrics(string trackId, string sessionId) +``` + +--- + +### AppleMusic Class + +**Namespace:** `DevBase.Api.Apis.AppleMusic` + +Apple Music API client. + +#### Constructor + +```csharp +AppleMusic(string developerToken) +``` + +#### Methods + +```csharp +Task SearchCatalog(string storefront, string query) +Task GetSong(string storefront, string id) +Task GetLyrics(string storefront, string id) +``` + +--- + +### NetEase Class + +**Namespace:** `DevBase.Api.Apis.NetEase` + +NetEase Music API client (Chinese music). + +#### Constructor + +```csharp +NetEase() +``` + +#### Methods + +```csharp +Task Search(string query) +Task SongDetail(string id) +Task Lyrics(string id) +``` + +--- + +### BeautifulLyrics Class + +**Namespace:** `DevBase.Api.Apis.BeautifulLyrics` + +BeautifulLyrics API for rich-synced lyrics. + +#### Constructor + +```csharp +BeautifulLyrics() +``` + +#### Methods + +```csharp +Task<(string Lyrics, bool IsRichSync)> GetRawLyrics(string isrc) +``` + +--- + +### MusixMatch Class + +**Namespace:** `DevBase.Api.Apis.Musixmatch` + +Musixmatch lyrics API client. + +#### Constructor + +```csharp +MusixMatch() +``` + +#### Methods + +```csharp +Task SearchTrack(string title, string artist) +Task GetLyrics(string trackId) +``` + +--- + +### OpenAi Class + +**Namespace:** `DevBase.Api.Apis.OpenAi` + +OpenAI API client. + +#### Constructor + +```csharp +OpenAi(string apiKey) +``` + +#### Methods + +```csharp +Task ChatCompletion(string model, object[] messages) +``` + +--- + +### Replicate Class + +**Namespace:** `DevBase.Api.Apis.Replicate` + +Replicate AI inference API client. + +#### Constructor + +```csharp +Replicate(string apiToken) +``` + +#### Methods + +```csharp +Task CreatePrediction(string model, object input) +Task GetPrediction(string predictionId) +Task WaitForPrediction(string predictionId) +``` + +--- ## Core Concept: ApiClient Base Class diff --git a/DevBase.Api/DevBase.Api.csproj b/DevBase.Api/DevBase.Api.csproj index 70155c4..a43714e 100644 --- a/DevBase.Api/DevBase.Api.csproj +++ b/DevBase.Api/DevBase.Api.csproj @@ -10,7 +10,7 @@ AlexanderDotH https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git - 1.5.0 + 1.5.1 MIT true README.md diff --git a/DevBase.Avalonia/AGENT.md b/DevBase.Avalonia/AGENT.md index c4c4c1e..7fccb1d 100644 --- a/DevBase.Avalonia/AGENT.md +++ b/DevBase.Avalonia/AGENT.md @@ -6,7 +6,70 @@ This guide helps AI agents effectively use DevBase.Avalonia for color analysis i DevBase.Avalonia provides color extraction and analysis tools for Avalonia UI bitmaps. -**Target Framework:** .NET 9.0, Avalonia UI 11.x +**Target Framework:** .NET 9.0, Avalonia UI 11.x +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Avalonia/ +├── Color/ +│ ├── Image/ +│ │ ├── BrightestColorCalculator.cs # Find brightest color +│ │ ├── GroupColorCalculator.cs # Group similar colors +│ │ └── NearestColorCalculator.cs # Find nearest color +│ └── ColorUtils.cs # Color utilities +└── Data/ + └── ... # Data structures +``` + +--- + +## Class Reference + +### BrightestColorCalculator Class + +**Namespace:** `DevBase.Avalonia.Color.Image` + +Finds the brightest color in a bitmap. + +#### Methods + +```csharp +Color Calculate(Bitmap bitmap) +``` + +--- + +### GroupColorCalculator Class + +**Namespace:** `DevBase.Avalonia.Color.Image` + +Groups similar colors and counts occurrences. + +#### Methods + +```csharp +Dictionary Calculate(Bitmap bitmap) +``` + +--- + +### NearestColorCalculator Class + +**Namespace:** `DevBase.Avalonia.Color.Image` + +Finds the nearest color to a target. + +#### Methods + +```csharp +Color FindNearest(Bitmap bitmap, Color target) +``` + +--- ## Core Components diff --git a/DevBase.Avalonia/DevBase.Avalonia.csproj b/DevBase.Avalonia/DevBase.Avalonia.csproj index f05a1a6..92b2c00 100644 --- a/DevBase.Avalonia/DevBase.Avalonia.csproj +++ b/DevBase.Avalonia/DevBase.Avalonia.csproj @@ -10,7 +10,7 @@ AlexanderDotH https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git - 1.2.5 + 1.2.6 MIT false true diff --git a/DevBase.Cryptography.BouncyCastle/AGENT.md b/DevBase.Cryptography.BouncyCastle/AGENT.md index ba6982f..d99735c 100644 --- a/DevBase.Cryptography.BouncyCastle/AGENT.md +++ b/DevBase.Cryptography.BouncyCastle/AGENT.md @@ -9,8 +9,139 @@ DevBase.Cryptography.BouncyCastle provides enterprise-grade cryptography using B - **ECDH** - Key exchange - **Token Verification** - JWT and signature verification - **Secure Random** - Cryptographically secure RNG +- **Hashing** - Various hash algorithms -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Cryptography.BouncyCastle/ +├── AES/ +│ └── AESBuilderEngine.cs # AES-GCM encryption +├── ECDH/ +│ └── EcdhEngineBuilder.cs # Elliptic curve key exchange +├── Hashing/ +│ ├── SHA256.cs # SHA-256 hashing +│ ├── SHA384.cs # SHA-384 hashing +│ ├── SHA512.cs # SHA-512 hashing +│ └── ... # Other hash algorithms +├── Identifier/ +│ └── TokenVerifier/ # JWT signature verification +│ ├── RsTokenVerifier.cs # RSA signatures +│ ├── EsTokenVerifier.cs # ECDSA signatures +│ ├── PsTokenVerifier.cs # RSA-PSS signatures +│ └── ShaTokenVerifier.cs # HMAC signatures +├── Random/ +│ └── SecureRandom.cs # Cryptographic RNG +├── Sealing/ +│ └── SealedBox.cs # Public-key encryption +└── Extensions/ + └── ByteExtensions.cs +``` + +--- + +## Class Reference + +### AESBuilderEngine Class + +**Namespace:** `DevBase.Cryptography.BouncyCastle.AES` + +AES-GCM authenticated encryption. + +#### Constructor + +```csharp +AESBuilderEngine() +``` + +#### Methods + +```csharp +byte[] Encrypt(byte[] buffer) +byte[] Decrypt(byte[] buffer) +void SetKey(byte[] key) +byte[] GetKey() +``` + +--- + +### EcdhEngineBuilder Class + +**Namespace:** `DevBase.Cryptography.BouncyCastle.ECDH` + +Elliptic Curve Diffie-Hellman key exchange. + +#### Methods + +```csharp +(AsymmetricKeyParameter Public, AsymmetricKeyParameter Private) GenerateKeyPair() +byte[] DeriveSharedSecret(AsymmetricKeyParameter privateKey, AsymmetricKeyParameter publicKey) +byte[] ExportPublicKey(AsymmetricKeyParameter publicKey) +AsymmetricKeyParameter ImportPublicKey(byte[] publicKeyBytes) +``` + +--- + +### Token Verifiers + +**Namespace:** `DevBase.Cryptography.BouncyCastle.Identifier.TokenVerifier` + +| Class | Algorithm | Description | +|-------|-----------|-------------| +| `RsTokenVerifier` | RS256/384/512 | RSA PKCS#1 signatures | +| `EsTokenVerifier` | ES256/384/512 | ECDSA signatures | +| `PsTokenVerifier` | PS256/384/512 | RSA-PSS signatures | +| `ShaTokenVerifier` | HS256/384/512 | HMAC signatures | + +#### Common Methods + +```csharp +bool Verify(string token, byte[] key) +bool Verify(string token, AsymmetricKeyParameter publicKey) +``` + +--- + +### SecureRandom Class + +**Namespace:** `DevBase.Cryptography.BouncyCastle.Random` + +Cryptographically secure random number generator. + +#### Methods + +```csharp +static byte[] GenerateBytes(int length) +static int GenerateInt() +static long GenerateLong() +``` + +--- + +### Hashing Classes (Static) + +**Namespace:** `DevBase.Cryptography.BouncyCastle.Hashing` + +| Class | Output Size | +|-------|-------------| +| `SHA256` | 32 bytes | +| `SHA384` | 48 bytes | +| `SHA512` | 64 bytes | + +#### Methods + +```csharp +static byte[] Hash(byte[] data) +static string HashToHex(byte[] data) +static string HashToBase64(byte[] data) +``` + +--- ## Core Components diff --git a/DevBase.Cryptography.BouncyCastle/DevBase.Cryptography.BouncyCastle.csproj b/DevBase.Cryptography.BouncyCastle/DevBase.Cryptography.BouncyCastle.csproj index 6d9fedd..4c77fca 100644 --- a/DevBase.Cryptography.BouncyCastle/DevBase.Cryptography.BouncyCastle.csproj +++ b/DevBase.Cryptography.BouncyCastle/DevBase.Cryptography.BouncyCastle.csproj @@ -12,7 +12,7 @@ https://github.com/AlexanderDotH/DevBase false DevBase.Cryptography.BouncyCastle - 1.2.1 + 1.2.2 MIT README.md git diff --git a/DevBase.Cryptography/AGENT.md b/DevBase.Cryptography/AGENT.md index c63f6c0..d86181f 100644 --- a/DevBase.Cryptography/AGENT.md +++ b/DevBase.Cryptography/AGENT.md @@ -8,7 +8,88 @@ DevBase.Cryptography provides: - **Blowfish** - Symmetric block cipher (CBC mode) - **MD5** - Fast hashing for checksums -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Cryptography/ +├── Blowfish/ +│ ├── Blowfish.cs # CBC mode encryption/decryption +│ ├── Codec.cs # Low-level block cipher +│ ├── Extensions.cs # Helper extensions +│ └── Init.cs # S-boxes and P-array initialization +└── MD5/ + └── MD5.cs # Fast MD5 hashing +``` + +--- + +## Class Reference + +### Blowfish Class + +**Namespace:** `DevBase.Cryptography.Blowfish` + +Blowfish symmetric block cipher in CBC mode. + +#### Constructor + +```csharp +Blowfish(byte[] key) +``` + +#### Methods + +```csharp +bool Encrypt(Span data, ReadOnlySpan initVector) +bool Decrypt(Span data, ReadOnlySpan initVector) +``` + +**Requirements:** +- `data` must be a multiple of 8 bytes +- `initVector` must be exactly 8 bytes + +--- + +### Codec Class + +**Namespace:** `DevBase.Cryptography.Blowfish` + +Low-level Blowfish block cipher (8-byte blocks). + +#### Constructor + +```csharp +Codec(byte[] key) +``` + +#### Methods + +```csharp +void Encrypt(Span block) // 8-byte block +void Decrypt(Span block) // 8-byte block +``` + +--- + +### MD5 Class (Static) + +**Namespace:** `DevBase.Cryptography.MD5` + +Fast MD5 hashing utility. + +#### Methods + +```csharp +static string Hash(string input) +static string Hash(byte[] data) +``` + +--- ## Core Components diff --git a/DevBase.Cryptography/DevBase.Cryptography.csproj b/DevBase.Cryptography/DevBase.Cryptography.csproj index 8b55283..a660221 100644 --- a/DevBase.Cryptography/DevBase.Cryptography.csproj +++ b/DevBase.Cryptography/DevBase.Cryptography.csproj @@ -10,7 +10,7 @@ AlexanderDotH https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git - 1.0.5 + 1.0.6 MIT false true diff --git a/DevBase.Extensions/AGENT.md b/DevBase.Extensions/AGENT.md index 86fc8b4..10a974e 100644 --- a/DevBase.Extensions/AGENT.md +++ b/DevBase.Extensions/AGENT.md @@ -6,7 +6,60 @@ This guide helps AI agents effectively use the DevBase.Extensions library for en DevBase.Extensions provides enhanced timing capabilities with detailed time breakdowns and formatted output. -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Extensions/ +├── Stopwatch/ +│ └── StopwatchExtension.cs # Stopwatch extension methods +├── Utils/ +│ └── TimeUtils.cs # Time breakdown utilities +└── Exceptions/ + └── StopwatchException.cs +``` + +--- + +## Class Reference + +### StopwatchExtension Class (Static) + +**Namespace:** `DevBase.Extensions.Stopwatch` + +Extension methods for `System.Diagnostics.Stopwatch`. + +#### Methods + +```csharp +static void PrintTimeTable(this Stopwatch stopwatch) +static string GetTimeTable(this Stopwatch stopwatch) +``` + +--- + +### TimeUtils Class (Static) + +**Namespace:** `DevBase.Extensions.Utils` + +Utilities for breaking down elapsed time. + +#### Methods + +```csharp +static (int Hours, string Unit) GetHours(Stopwatch stopwatch) +static (int Minutes, string Unit) GetMinutes(Stopwatch stopwatch) +static (int Seconds, string Unit) GetSeconds(Stopwatch stopwatch) +static (int Milliseconds, string Unit) GetMilliseconds(Stopwatch stopwatch) +static (long Microseconds, string Unit) GetMicroseconds(Stopwatch stopwatch) +static (long Nanoseconds, string Unit) GetNanoseconds(Stopwatch stopwatch) +``` + +--- ## Core Components diff --git a/DevBase.Extensions/DevBase.Extensions.csproj b/DevBase.Extensions/DevBase.Extensions.csproj index 883b1cf..91913d7 100644 --- a/DevBase.Extensions/DevBase.Extensions.csproj +++ b/DevBase.Extensions/DevBase.Extensions.csproj @@ -10,7 +10,7 @@ AlexanderDotH https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git - 1.0.1 + 1.0.2 MIT true README.md diff --git a/DevBase.Format/AGENT.md b/DevBase.Format/AGENT.md index 70c3b91..00fd850 100644 --- a/DevBase.Format/AGENT.md +++ b/DevBase.Format/AGENT.md @@ -6,7 +6,143 @@ This guide helps AI agents effectively use the DevBase.Format library for parsin DevBase.Format provides parsers for various text-based formats, primarily focused on lyrics formats (LRC, SRT, ELRC, etc.) and configuration files (ENV). -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Format/ +├── FileFormat.cs # Abstract base class for parsers +├── FileParser.cs # Generic parser wrapper +├── RevertableFileFormat.cs # Base for reversible formats +├── Formats/ +│ ├── LrcFormat/ # Standard LRC lyrics +│ ├── ElrcFormat/ # Enhanced LRC with word timing +│ ├── RlrcFormat/ # Rich LRC format +│ ├── SrtFormat/ # SubRip subtitles +│ ├── EnvFormat/ # Environment configuration +│ ├── KLyricsFormat/ # Karaoke lyrics +│ ├── MmlFormat/ # Music Macro Language +│ ├── RmmlFormat/ # Rich MML +│ ├── AppleXmlFormat/ # Apple Music XML lyrics +│ ├── AppleLrcXmlFormat/ # Apple LRC XML +│ └── AppleRichXmlFormat/ # Apple rich lyrics XML +├── Structure/ # Shared data structures +│ ├── TimeStampedLyric.cs +│ └── RichTimeStampedLyric.cs +├── Exceptions/ +│ └── ParsingException.cs +└── Utilities/ +``` + +--- + +## Class Reference + +### FileParser Class + +**Namespace:** `DevBase.Format` + +Generic parser wrapper for format parsing. + +#### Type Parameters + +| Parameter | Constraint | Description | +|-----------|------------|-------------| +| `P` | `FileFormat` | The format parser class | +| `T` | - | The output type | + +#### Methods + +```csharp +T ParseFromString(string content) +bool TryParseFromString(string content, out T parsed) +T ParseFromDisk(string filePath) +T ParseFromDisk(FileInfo fileInfo) +bool TryParseFromDisk(string filePath, out T parsed) +``` + +--- + +### FileFormat Class (Abstract) + +**Namespace:** `DevBase.Format` + +Base class for all format parsers. + +#### Properties + +| Property | Type | Default | Description | +|----------|------|---------|-------------| +| `StrictErrorHandling` | `bool` | `true` | Throw exceptions or return null | + +#### Abstract Methods + +```csharp +abstract T Parse(F from) +abstract bool TryParse(F from, out T parsed) +``` + +#### Protected Methods + +```csharp +dynamic Error(string message) // Throws or returns null +dynamic Error(Exception exception) +``` + +--- + +### Supported Formats + +| Format Class | Output Type | Description | +|--------------|-------------|-------------| +| `LrcFormat` | `AList` | Standard LRC lyrics | +| `ElrcFormat` | `AList` | Enhanced LRC with word timing | +| `RlrcFormat` | `AList` | Rich LRC format | +| `SrtFormat` | `AList` | SubRip subtitles | +| `EnvFormat` | `Dictionary` | Environment variables | +| `KLyricsFormat` | `AList` | Karaoke lyrics | +| `MmlFormat` | `MmlDocument` | Music Macro Language | +| `AppleXmlFormat` | `AppleLyrics` | Apple Music XML | + +--- + +### TimeStampedLyric Record + +**Namespace:** `DevBase.Format.Structure` + +```csharp +public record TimeStampedLyric( + TimeSpan StartTime, + string Text +); +``` + +--- + +### RichTimeStampedLyric Record + +**Namespace:** `DevBase.Format.Structure` + +```csharp +public record RichTimeStampedLyric( + TimeSpan StartTime, + TimeSpan EndTime, + string Text, + AList Words +); + +public record WordTiming( + TimeSpan StartTime, + TimeSpan EndTime, + string Word +); +``` + +--- ## Core Architecture diff --git a/DevBase.Format/DevBase.Format.csproj b/DevBase.Format/DevBase.Format.csproj index b973a8b..84e5de6 100644 --- a/DevBase.Format/DevBase.Format.csproj +++ b/DevBase.Format/DevBase.Format.csproj @@ -10,7 +10,7 @@ AlexanderDotH https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git - 1.7.2 + 1.7.3 MIT false true diff --git a/DevBase.Logging/AGENT.md b/DevBase.Logging/AGENT.md index b791c37..f778823 100644 --- a/DevBase.Logging/AGENT.md +++ b/DevBase.Logging/AGENT.md @@ -6,7 +6,61 @@ This guide helps AI agents effectively use the DevBase.Logging library for simpl DevBase.Logging is a lightweight logging library that writes to the Debug console. It's designed for simplicity and ease of use. -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.0.0 + +--- + +## Project Structure + +``` +DevBase.Logging/ +├── Logger/ +│ └── Logger.cs # Generic logger class +└── Enums/ + └── LogType.cs # Log level enum +``` + +--- + +## Class Reference + +### Logger Class + +**Namespace:** `DevBase.Logging.Logger` + +Generic logger that outputs to Debug console. + +#### Constructor + +```csharp +Logger(T type) +``` + +#### Methods + +```csharp +void Write(string message, LogType logType) +void Write(Exception exception) +``` + +--- + +### LogType Enum + +**Namespace:** `DevBase.Logging.Enums` + +```csharp +public enum LogType +{ + INFO, + WARNING, + ERROR, + DEBUG +} +``` + +--- ## Core Components diff --git a/DevBase.Logging/DevBase.Logging.csproj b/DevBase.Logging/DevBase.Logging.csproj index ad773b1..67644b5 100644 --- a/DevBase.Logging/DevBase.Logging.csproj +++ b/DevBase.Logging/DevBase.Logging.csproj @@ -10,7 +10,7 @@ https://github.com/AlexanderDotH/DevBase https://github.com/AlexanderDotH/DevBase false - 1.0.1 + 1.0.2 MIT true README.md diff --git a/DevBase.Net/AGENT.md b/DevBase.Net/AGENT.md index ad34260..5525b25 100644 --- a/DevBase.Net/AGENT.md +++ b/DevBase.Net/AGENT.md @@ -4,33 +4,712 @@ This guide helps AI agents effectively use the DevBase.Net HTTP client library. ## Overview -DevBase.Net is the networking backbone of the DevBase solution. It provides a high-performance `Request` class that wraps `HttpClient` with advanced features like SOCKS5 proxying, retry policies, interceptors, and detailed metrics. +DevBase.Net is the networking backbone of the DevBase solution. It provides a high-performance HTTP client with advanced features like SOCKS5 proxying, retry policies, interceptors, batch processing, proxy rotation, and detailed metrics. -**Target Framework:** .NET 9.0 +**Target Framework:** .NET 9.0 +**Current Version:** 1.1.0 + +--- + +## Project Structure + +``` +DevBase.Net/ +├── Core/ # Main HTTP client classes +│ ├── Request.cs # Core request class (properties, constructors) +│ ├── RequestConfiguration.cs # Fluent configuration API +│ ├── RequestHttp.cs # HTTP execution logic (SendAsync) +│ ├── Response.cs # Response parsing and streaming +│ ├── Requests.cs # Simple queue-based processor +│ ├── BatchRequests.cs # Named batch processor with progress +│ └── ProxiedBatchRequests.cs # Batch processor with proxy rotation +├── Proxy/ # Proxy support +│ ├── ProxyInfo.cs # Proxy configuration and parsing +│ ├── ProxyConfiguration.cs # Fluent proxy builders +│ ├── TrackedProxyInfo.cs # Failure tracking wrapper +│ ├── HttpToSocks5/ # SOCKS proxy tunneling +│ └── Socks/ # SOCKS protocol implementation +├── Configuration/ # Configuration classes +│ ├── RetryPolicy.cs # Retry with backoff strategies +│ ├── LoggingConfig.cs # Request/response logging +│ └── ... # Other config classes +├── Data/ # Request building +│ ├── Body/ # Request body builders +│ ├── Header/ # Header builders +│ └── Parameters/ # URL parameter builders +├── Exceptions/ # Custom exception types +├── Metrics/ # Performance metrics +└── Utils/ # Utility classes +``` + +--- ## Core Architecture ### Request Lifecycle -1. **Build** → Create `Request` with fluent API -2. **Configure** → Add headers, body, timeout, retries, proxy -3. **Execute** → Call `SendAsync()` -4. **Intercept** → Request/Response interceptors modify flow -5. **Parse** → Extract data from `Response` -6. **Dispose** → Release resources +``` +1. CREATE → new Request(url) +2. CONFIGURE → .WithHeaders() / .WithBody() / .WithProxy() +3. BUILD → .Build() (automatic on SendAsync) +4. INTERCEPT → IRequestInterceptor.OnRequestAsync() +5. EXECUTE → HttpClient.SendAsync() +6. RETRY → RetryPolicy handles failures +7. INTERCEPT → IResponseInterceptor.OnResponseAsync() +8. PARSE → response.ParseJsonAsync() +9. DISPOSE → response.Dispose() +``` ### Key Components -| Component | Purpose | -|-----------|---------| -| `Request` | Main entry point, fluent builder, manages execution | -| `Response` | Wraps HttpResponseMessage + MemoryStream, provides parsing | -| `RequestBuilder` | Internal builder for URI, headers, body | -| `HttpToSocks5Proxy` | Local HTTP proxy tunneling to SOCKS5 | -| `RetryPolicy` | Handles transient failures with backoff | -| `RequestMetrics` | Captures precise timing data | -| `IRequestInterceptor` | Middleware for request modification | -| `IResponseInterceptor` | Middleware for response transformation | +| Component | Namespace | Purpose | +|-----------|-----------|---------| +| `Request` | `DevBase.Net.Core` | Main entry point, fluent builder, manages HTTP execution | +| `Response` | `DevBase.Net.Core` | Wraps HttpResponseMessage, provides parsing methods | +| `Requests` | `DevBase.Net.Core` | Simple queue-based request processor | +| `BatchRequests` | `DevBase.Net.Core` | Named batch processor with progress callbacks | +| `ProxiedBatchRequests` | `DevBase.Net.Core` | Batch processor with proxy rotation and failure tracking | +| `ProxyInfo` | `DevBase.Net.Proxy` | Proxy configuration with HTTP/HTTPS/SOCKS support | +| `ProxyConfiguration` | `DevBase.Net.Proxy` | Fluent builder for provider-specific proxy settings | +| `TrackedProxyInfo` | `DevBase.Net.Proxy` | Proxy wrapper with failure tracking | +| `RetryPolicy` | `DevBase.Net.Configuration` | Retry configuration with backoff strategies | +| `RequestMetrics` | `DevBase.Net.Metrics` | Captures timing data | + +--- + +## Class Reference + +### Request Class + +**Namespace:** `DevBase.Net.Core` + +The main HTTP client class with fluent API for building and sending requests. + +#### Constructors + +```csharp +Request() +Request(string url) +Request(Uri uri) +Request(string url, HttpMethod method) +Request(Uri uri, HttpMethod method) +Request(ReadOnlyMemory url) +``` + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `Uri` | `ReadOnlySpan` | The request URI | +| `Body` | `ReadOnlySpan` | The request body | +| `Method` | `HttpMethod` | HTTP method (GET, POST, etc.) | +| `Timeout` | `TimeSpan` | Request timeout (default: 30s) | +| `CancellationToken` | `CancellationToken` | Cancellation token | +| `Proxy` | `TrackedProxyInfo?` | Proxy configuration | +| `RetryPolicy` | `RetryPolicy` | Retry configuration | +| `ValidateCertificates` | `bool` | SSL certificate validation | +| `FollowRedirects` | `bool` | Follow HTTP redirects | +| `MaxRedirects` | `int` | Maximum redirects (default: 50) | + +#### Configuration Methods + +```csharp +// URL Configuration +Request WithUrl(string url) +Request WithUrl(Uri uri) +Request WithParameters(ParameterBuilder parameterBuilder) +Request WithParameter(string key, string value) +Request WithParameters(params (string key, string value)[] parameters) + +// HTTP Method +Request WithMethod(HttpMethod method) +Request AsGet() +Request AsPost() +Request AsPut() +Request AsPatch() +Request AsDelete() +Request AsHead() +Request AsOptions() +Request AsTrace() + +// Headers +Request WithHeaders(RequestHeaderBuilder headerBuilder) +Request WithHeader(string name, string value) +Request WithAccept(params string[] acceptTypes) +Request WithAcceptJson() +Request WithAcceptXml() +Request WithAcceptHtml() +Request WithUserAgent(string userAgent) +Request WithBogusUserAgent() +Request WithBogusUserAgent() where T : IBogusUserAgentGenerator +Request WithReferer(string referer) +Request WithCookie(string cookie) + +// Authentication +Request UseBasicAuthentication(string username, string password) +Request UseBearerAuthentication(string token) +Request UseJwtAuthentication(AuthenticationToken token) +Request UseJwtAuthentication(string rawToken) + +// Request Body +Request WithRawBody(RequestRawBodyBuilder bodyBuilder) +Request WithRawBody(string content, Encoding? encoding = null) +Request WithJsonBody(string jsonContent, Encoding? encoding = null) +Request WithJsonBody(T obj) +Request WithBufferBody(byte[] buffer) +Request WithBufferBody(Memory buffer) +Request WithEncodedForm(RequestEncodedKeyValueListBodyBuilder formBuilder) +Request WithEncodedForm(params (string key, string value)[] formData) +Request WithForm(RequestKeyValueListBodyBuilder formBuilder) + +// Execution Configuration +Request WithTimeout(TimeSpan timeout) +Request WithCancellationToken(CancellationToken cancellationToken) +Request WithProxy(TrackedProxyInfo? proxy) +Request WithProxy(ProxyInfo proxy) +Request WithRetryPolicy(RetryPolicy policy) +Request WithCertificateValidation(bool validate) +Request WithHeaderValidation(bool validate) +Request WithFollowRedirects(bool follow, int maxRedirects = 50) + +// Advanced Configuration +Request WithScrapingBypass(ScrapingBypassConfig config) +Request WithJsonPathParsing(JsonPathConfig config) +Request WithHostCheck(HostCheckConfig config) +Request WithLogging(LoggingConfig config) +Request WithRequestInterceptor(IRequestInterceptor interceptor) +Request WithResponseInterceptor(IResponseInterceptor interceptor) +``` + +#### Execution Methods + +```csharp +Request Build() +Task SendAsync(CancellationToken cancellationToken = default) +HttpRequestMessage ToHttpRequestMessage() +``` + +#### Static Methods + +```csharp +static Request Create() +static Request Create(string url) +static Request Create(Uri uri) +static Request Create(string url, HttpMethod method) +static void ConfigureConnectionPool(TimeSpan? connectionLifetime, TimeSpan? connectionIdleTimeout, int? maxConnections) +static void ClearClientPool() +``` + +--- + +### Response Class + +**Namespace:** `DevBase.Net.Core` + +Wraps the HTTP response with parsing and streaming capabilities. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `StatusCode` | `HttpStatusCode` | HTTP status code | +| `IsSuccessStatusCode` | `bool` | True if 2xx status | +| `Headers` | `HttpResponseHeaders` | Response headers | +| `ContentHeaders` | `HttpContentHeaders?` | Content headers | +| `ContentType` | `string?` | Content-Type header | +| `ContentLength` | `long?` | Content-Length header | +| `HttpVersion` | `Version` | HTTP protocol version | +| `ReasonPhrase` | `string?` | Status reason phrase | +| `Metrics` | `RequestMetrics` | Performance metrics | +| `FromCache` | `bool` | True if from cache | +| `RequestUri` | `Uri?` | Original request URI | +| `IsRedirect` | `bool` | True if redirect response | +| `IsClientError` | `bool` | True if 4xx status | +| `IsServerError` | `bool` | True if 5xx status | +| `IsRateLimited` | `bool` | True if 429 status | + +#### Content Methods + +```csharp +Task GetBytesAsync(CancellationToken cancellationToken = default) +Task GetStringAsync(Encoding? encoding = null, CancellationToken cancellationToken = default) +Stream GetStream() +``` + +#### Parsing Methods + +```csharp +Task GetAsync(CancellationToken cancellationToken = default) +Task ParseJsonAsync(bool useSystemTextJson = true, CancellationToken cancellationToken = default) +Task ParseJsonDocumentAsync(CancellationToken cancellationToken = default) +Task ParseXmlAsync(CancellationToken cancellationToken = default) +Task ParseHtmlAsync(CancellationToken cancellationToken = default) +Task ParseJsonPathAsync(string path, CancellationToken cancellationToken = default) +Task> ParseJsonPathListAsync(string path, CancellationToken cancellationToken = default) +``` + +#### Streaming Methods + +```csharp +IAsyncEnumerable StreamLinesAsync(CancellationToken cancellationToken = default) +IAsyncEnumerable StreamChunksAsync(int chunkSize = 4096, CancellationToken cancellationToken = default) +``` + +#### Header Methods + +```csharp +string? GetHeader(string name) +IEnumerable GetHeaderValues(string name) +CookieCollection GetCookies() +void EnsureSuccessStatusCode() +``` + +--- + +### BatchRequests Class + +**Namespace:** `DevBase.Net.Core` + +Processes requests in named batches with rate limiting, progress tracking, and re-queue support. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `BatchCount` | `int` | Number of batches | +| `TotalQueueCount` | `int` | Total queued requests | +| `ResponseQueueCount` | `int` | Completed responses | +| `RateLimit` | `int` | Requests per window | +| `PersistCookies` | `bool` | Cookie persistence enabled | +| `PersistReferer` | `bool` | Referer persistence enabled | +| `IsProcessing` | `bool` | Background processing active | +| `ProcessedCount` | `int` | Total processed requests | +| `ErrorCount` | `int` | Total errors | +| `BatchNames` | `IReadOnlyList` | List of batch names | + +#### Configuration Methods + +```csharp +BatchRequests WithRateLimit(int requestsPerWindow, TimeSpan? window = null) +BatchRequests WithCookiePersistence(bool persist = true) +BatchRequests WithRefererPersistence(bool persist = true) +``` + +#### Batch Management + +```csharp +Batch CreateBatch(string name) +Batch GetOrCreateBatch(string name) +Batch? GetBatch(string name) +bool RemoveBatch(string name) +BatchRequests ClearAllBatches() +``` + +#### Callback Registration + +```csharp +BatchRequests OnResponse(Func callback) +BatchRequests OnResponse(Action callback) +BatchRequests OnError(Func callback) +BatchRequests OnError(Action callback) +BatchRequests OnProgress(Func callback) +BatchRequests OnProgress(Action callback) +BatchRequests OnResponseRequeue(Func callback) +BatchRequests OnErrorRequeue(Func callback) +``` + +#### Execution Methods + +```csharp +Task> ExecuteAllAsync(CancellationToken cancellationToken = default) +IAsyncEnumerable ExecuteAllAsyncEnumerable(CancellationToken cancellationToken = default) +Task> ExecuteBatchAsync(string batchName, CancellationToken cancellationToken = default) +void StartProcessing() +Task StopProcessingAsync() +``` + +#### Response Management + +```csharp +bool TryDequeueResponse(out Response? response) +List DequeueAllResponses() +``` + +#### Statistics + +```csharp +BatchStatistics GetStatistics() +void ResetCounters() +``` + +--- + +### Batch Class (Inner class of BatchRequests) + +**Namespace:** `DevBase.Net.Core` + +Represents a named group of requests within BatchRequests. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `Name` | `string` | Batch name | +| `QueueCount` | `int` | Queued request count | + +#### Methods + +```csharp +Batch Add(Request request) +Batch Add(IEnumerable requests) +Batch Add(string url) +Batch Add(IEnumerable urls) +Batch Enqueue(Request request) +Batch Enqueue(string url) +Batch Enqueue(string url, Action configure) +Batch Enqueue(Func requestFactory) +bool TryDequeue(out Request? request) +void Clear() +BatchRequests EndBatch() +``` + +--- + +### ProxiedBatchRequests Class + +**Namespace:** `DevBase.Net.Core` + +Extends BatchRequests with proxy rotation, failure tracking, and proxy-specific callbacks. + +#### Additional Properties + +| Property | Type | Description | +|----------|------|-------------| +| `ProxyCount` | `int` | Total proxies | +| `AvailableProxyCount` | `int` | Available proxies | +| `ProxyFailureCount` | `int` | Total proxy failures | + +#### Proxy Configuration + +```csharp +ProxiedBatchRequests WithProxy(ProxyInfo proxy) +ProxiedBatchRequests WithProxy(string proxyString) +ProxiedBatchRequests WithProxies(IEnumerable proxies) +ProxiedBatchRequests WithProxies(IEnumerable proxyStrings) +ProxiedBatchRequests ConfigureProxyTracking(int maxFailures = 3, TimeSpan? timeoutDuration = null) +ProxiedBatchRequests ClearProxies() +``` + +#### Rotation Strategy + +```csharp +ProxiedBatchRequests WithRotationStrategy(IProxyRotationStrategy strategy) +ProxiedBatchRequests WithRoundRobinRotation() +ProxiedBatchRequests WithRandomRotation() +ProxiedBatchRequests WithLeastFailuresRotation() +ProxiedBatchRequests WithStickyRotation() +``` + +#### Proxy Callbacks + +```csharp +ProxiedBatchRequests OnProxyFailure(Func callback) +ProxiedBatchRequests OnProxyFailure(Action callback) +``` + +#### Proxy Statistics + +```csharp +IReadOnlyList GetProxyStatistics() +ProxiedBatchStatistics GetStatistics() +void ResetProxies() +``` + +--- + +### ProxyInfo Class + +**Namespace:** `DevBase.Net.Proxy` + +Immutable proxy configuration with support for HTTP, HTTPS, SOCKS4, SOCKS5, and SOCKS5h. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `Host` | `string` | Proxy hostname | +| `Port` | `int` | Proxy port | +| `Type` | `EnumProxyType` | Proxy type | +| `Credentials` | `NetworkCredential?` | Authentication | +| `Key` | `string` | Unique identifier | +| `HasAuthentication` | `bool` | Has credentials | +| `BypassLocalAddresses` | `bool` | Bypass local | +| `BypassList` | `string[]?` | Bypass patterns | +| `ResolveHostnamesLocally` | `bool` | DNS resolution location | +| `ConnectionTimeout` | `TimeSpan` | Connection timeout | +| `ReadWriteTimeout` | `TimeSpan` | Read/write timeout | +| `InternalServerPort` | `int` | Internal HTTP listener port | + +#### Constructors + +```csharp +ProxyInfo(string host, int port, EnumProxyType type = EnumProxyType.Http, NetworkCredential? credentials = null) +ProxyInfo(string host, int port, string username, string password, EnumProxyType type = EnumProxyType.Http) +``` + +#### Static Methods + +```csharp +static ProxyInfo Parse(string proxyString) +static bool TryParse(string proxyString, out ProxyInfo? proxyInfo) +static ProxyInfo FromConfiguration(ProxyConfiguration config) +static void ClearProxyCache() +``` + +#### Instance Methods + +```csharp +Uri ToUri() +IWebProxy ToWebProxy() +``` + +--- + +### ProxyConfiguration Class + +**Namespace:** `DevBase.Net.Proxy` + +Fluent builder for creating ProxyInfo with provider-specific options. + +#### Static Factory Methods + +```csharp +static HttpProxyBuilder Http(string host, int port) +static HttpProxyBuilder Https(string host, int port) +static Socks4ProxyBuilder Socks4(string host, int port) +static Socks5ProxyBuilder Socks5(string host, int port) +static Socks5ProxyBuilder Socks5h(string host, int port) +``` + +#### HttpProxyBuilder Methods + +```csharp +HttpProxyBuilder WithCredentials(string username, string password) +HttpProxyBuilder WithCredentials(NetworkCredential credentials) +HttpProxyBuilder BypassLocal(bool bypass = true) +HttpProxyBuilder WithBypassList(params string[] addresses) +ProxyConfiguration Build() +ProxyInfo ToProxyInfo() +``` + +#### Socks4ProxyBuilder Methods + +```csharp +Socks4ProxyBuilder WithUserId(string userId) +Socks4ProxyBuilder WithConnectionTimeout(TimeSpan timeout) +Socks4ProxyBuilder WithReadWriteTimeout(TimeSpan timeout) +Socks4ProxyBuilder WithInternalServerPort(int port) +ProxyConfiguration Build() +ProxyInfo ToProxyInfo() +``` + +#### Socks5ProxyBuilder Methods + +```csharp +Socks5ProxyBuilder WithCredentials(string username, string password) +Socks5ProxyBuilder WithCredentials(NetworkCredential credentials) +Socks5ProxyBuilder ResolveHostnamesLocally(bool resolveLocally = true) +Socks5ProxyBuilder ResolveHostnamesRemotely() +Socks5ProxyBuilder WithConnectionTimeout(TimeSpan timeout) +Socks5ProxyBuilder WithReadWriteTimeout(TimeSpan timeout) +Socks5ProxyBuilder WithInternalServerPort(int port) +ProxyConfiguration Build() +ProxyInfo ToProxyInfo() +``` + +--- + +### TrackedProxyInfo Class + +**Namespace:** `DevBase.Net.Proxy` + +Wraps ProxyInfo with failure tracking and automatic timeout. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `Proxy` | `ProxyInfo` | Underlying proxy | +| `FailureCount` | `int` | Current failures | +| `LastFailure` | `DateTime?` | Last failure time | +| `IsTimedOut` | `bool` | Currently timed out | +| `TimeoutUntil` | `DateTime?` | Timeout end time | +| `TotalTimeouts` | `int` | Total timeouts | +| `MaxFailures` | `int` | Max failures before timeout | +| `TimeoutDuration` | `TimeSpan` | Timeout duration | +| `RemainingTimeout` | `TimeSpan?` | Remaining timeout | +| `Key` | `string` | Proxy key | + +#### Constructor + +```csharp +TrackedProxyInfo(ProxyInfo proxy, int maxFailures = 3, TimeSpan? timeoutDuration = null) +``` + +#### Methods + +```csharp +bool ReportFailure() +void ReportSuccess() +bool IsAvailable() +void ResetTimeout() +IWebProxy ToWebProxy() +``` + +--- + +### RetryPolicy Class + +**Namespace:** `DevBase.Net.Configuration` + +Configures retry behavior with backoff strategies. + +#### Properties + +| Property | Type | Default | Description | +|----------|------|---------|-------------| +| `MaxRetries` | `int` | 3 | Maximum retry attempts | +| `RetryOnProxyError` | `bool` | true | Retry proxy errors | +| `RetryOnTimeout` | `bool` | true | Retry timeouts | +| `RetryOnNetworkError` | `bool` | true | Retry network errors | +| `BackoffStrategy` | `EnumBackoffStrategy` | Exponential | Delay strategy | +| `InitialDelay` | `TimeSpan` | 500ms | First retry delay | +| `MaxDelay` | `TimeSpan` | 30s | Maximum delay | +| `BackoffMultiplier` | `double` | 2.0 | Delay multiplier | + +#### Static Presets + +```csharp +static RetryPolicy Default // 3 retries, exponential backoff +static RetryPolicy None // No retries +static RetryPolicy Aggressive // 5 retries, fast backoff +``` + +#### Methods + +```csharp +TimeSpan GetDelay(int attemptNumber) +``` + +--- + +### Enums + +#### EnumProxyType + +```csharp +namespace DevBase.Net.Proxy.Enums; + +public enum EnumProxyType +{ + Http, // Standard HTTP proxy + Https, // HTTPS/SSL proxy + Socks4, // SOCKS4 (no auth, local DNS) + Socks5, // SOCKS5 (auth support, configurable DNS) + Socks5h // SOCKS5 with remote DNS resolution +} +``` + +#### EnumBackoffStrategy + +```csharp +namespace DevBase.Net.Configuration.Enums; + +public enum EnumBackoffStrategy +{ + Fixed, // Same delay each retry + Linear, // delay * attemptNumber + Exponential // delay * multiplier^(attempt-1) +} +``` + +--- + +### Record Types + +#### RequeueDecision + +```csharp +public sealed record RequeueDecision(bool ShouldRequeue, Request? ModifiedRequest = null) +{ + public static RequeueDecision NoRequeue => new(false); + public static RequeueDecision Requeue() => new(true); + public static RequeueDecision RequeueWith(Request modifiedRequest) => new(true, modifiedRequest); +} +``` + +#### BatchProgressInfo + +```csharp +public sealed record BatchProgressInfo( + string BatchName, + int Completed, + int Total, + int Errors +) +{ + public double PercentComplete => Total > 0 ? (double)Completed / Total * 100 : 0; + public int Remaining => Total - Completed; +} +``` + +#### BatchStatistics + +```csharp +public sealed record BatchStatistics( + int BatchCount, + int TotalQueuedRequests, + int ProcessedRequests, + int ErrorCount, + Dictionary RequestsPerBatch +) +{ + public double SuccessRate => ProcessedRequests > 0 + ? (double)(ProcessedRequests - ErrorCount) / ProcessedRequests * 100 : 0; +} +``` + +#### ProxiedBatchStatistics + +```csharp +public sealed record ProxiedBatchStatistics( + int BatchCount, + int TotalQueuedRequests, + int ProcessedRequests, + int ErrorCount, + int ProxyFailureCount, + int TotalProxies, + int AvailableProxies, + Dictionary RequestsPerBatch +) +{ + public double SuccessRate => ...; + public double ProxyAvailabilityRate => ...; +} +``` + +#### ProxyFailureContext + +```csharp +public sealed record ProxyFailureContext( + TrackedProxyInfo Proxy, + Exception Exception, + Request FailedRequest, + bool ProxyTimedOut, + int CurrentFailureCount, + int RemainingAvailableProxies +); +``` ## Usage Patterns for AI Agents @@ -72,28 +751,129 @@ var response = await new Request("https://api.example.com/users") .SendAsync(); ``` -### Pattern 3: Handling Proxies +### Pattern 3: Proxy Configuration by Type -**When user mentions proxies, use `HttpToSocks5Proxy`:** +**ProxyInfo supports all proxy types with automatic handling:** ```csharp using DevBase.Net.Proxy; -using DevBase.Net.Proxy.HttpToSocks5; +using DevBase.Net.Proxy.Enums; + +// HTTP Proxy (standard web proxy) +var httpProxy = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); -// Create proxy -var proxyInfo = new Socks5ProxyInfo("proxy.example.com", 1080, "user", "pass"); -var proxy = new HttpToSocks5Proxy(new[] { proxyInfo }); +// HTTPS Proxy (SSL/TLS proxy) +var httpsProxy = new ProxyInfo("proxy.example.com", 8443, EnumProxyType.Https); + +// SOCKS4 Proxy (no authentication support) +var socks4Proxy = new ProxyInfo("proxy.example.com", 1080, EnumProxyType.Socks4); + +// SOCKS5 Proxy (with optional authentication) +var socks5Proxy = new ProxyInfo("proxy.example.com", 1080, EnumProxyType.Socks5); + +// SOCKS5h Proxy (remote DNS resolution - more private) +var socks5hProxy = new ProxyInfo("proxy.example.com", 1080, EnumProxyType.Socks5h); + +// With authentication +var authProxy = new ProxyInfo("proxy.example.com", 1080, "username", "password", EnumProxyType.Socks5); // Use with request var response = await new Request("https://api.example.com") - .WithProxy(new TrackedProxyInfo(proxyInfo)) + .WithProxy(httpProxy) .SendAsync(); +``` + +**Proxy Type Comparison:** + +| Type | Auth | DNS Resolution | Use Case | +|------|------|----------------|----------| +| `Http` | Optional | Local | Standard web browsing, caching | +| `Https` | Optional | Local | Secure proxy connections | +| `Socks4` | No | Local | Legacy systems, simple tunneling | +| `Socks5` | Optional | Local | General purpose, UDP support | +| `Socks5h` | Optional | Remote | Maximum privacy, bypass DNS leaks | + +**Parse proxy from string:** + +```csharp +// Supported formats: +var proxy1 = ProxyInfo.Parse("http://proxy.example.com:8080"); +var proxy2 = ProxyInfo.Parse("socks5://user:pass@proxy.example.com:1080"); +var proxy3 = ProxyInfo.Parse("socks5h://proxy.example.com:1080"); + +// Safe parsing +if (ProxyInfo.TryParse("socks5://...", out var proxy)) +{ + request.WithProxy(proxy); +} +``` + +**Note:** SOCKS proxies use `HttpToSocks5Proxy` internally which starts a local HTTP listener. The proxy cache ensures only one listener per unique proxy configuration. + +### Pattern 3b: Advanced Proxy Configuration (Provider-Specific) + +**Use `ProxyConfiguration` for fine-grained control over each proxy type:** + +```csharp +using DevBase.Net.Proxy; + +// HTTP/HTTPS Proxy - with bypass settings +var httpProxy = ProxyConfiguration.Http("proxy.example.com", 8080) + .WithCredentials("user", "password") + .BypassLocal(true) // Skip proxy for local addresses + .WithBypassList("*.internal.com", "10.*") // Skip proxy for these patterns + .ToProxyInfo(); + +// HTTPS Proxy - same options as HTTP +var httpsProxy = ProxyConfiguration.Https("proxy.example.com", 8443) + .WithCredentials("user", "password") + .BypassLocal(true) + .ToProxyInfo(); + +// SOCKS4 Proxy - supports user ID only (no password) +var socks4Proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .WithUserId("myuserid") // SOCKS4 only supports user ID + .WithConnectionTimeout(TimeSpan.FromSeconds(10)) + .WithReadWriteTimeout(TimeSpan.FromSeconds(30)) + .WithInternalServerPort(0) // 0 = auto-assign port + .ToProxyInfo(); + +// SOCKS5 Proxy - full authentication and DNS control +var socks5Proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .WithCredentials("user", "password") + .ResolveHostnamesLocally(true) // Local DNS resolution + .WithConnectionTimeout(TimeSpan.FromSeconds(15)) + .WithInternalServerPort(9999) // Fixed internal port + .ToProxyInfo(); + +// SOCKS5h Proxy - remote DNS resolution (more private) +var socks5hProxy = ProxyConfiguration.Socks5h("proxy.example.com", 1080) + .WithCredentials("user", "password") + // DNS always resolved remotely for socks5h + .ToProxyInfo(); -// Cleanup -proxy.Dispose(); +// Use with request +var response = await new Request("https://api.example.com") + .WithProxy(socks5Proxy) + .SendAsync(); ``` -**Important:** The proxy starts a local HTTP listener on a random port. Always dispose when done. +**Provider-Specific Configuration Options:** + +| Provider | Option | Description | +|----------|--------|-------------| +| **HTTP/HTTPS** | `WithCredentials()` | Basic authentication | +| | `BypassLocal()` | Skip proxy for localhost/LAN | +| | `WithBypassList()` | Skip proxy for matching patterns | +| **SOCKS4** | `WithUserId()` | User ID (no password support) | +| | `WithConnectionTimeout()` | Connection timeout | +| | `WithReadWriteTimeout()` | Read/write timeout | +| | `WithInternalServerPort()` | Fixed port for internal HTTP listener | +| **SOCKS5** | `WithCredentials()` | Username/password authentication | +| | `ResolveHostnamesLocally()` | DNS resolution location | +| | `WithConnectionTimeout()` | Connection timeout | +| | `WithInternalServerPort()` | Fixed port for internal HTTP listener | +| **SOCKS5h** | Same as SOCKS5 | DNS always resolved remotely | ### Pattern 4: Streaming Large Responses @@ -181,7 +961,187 @@ var response = await new Request("https://api.example.com/upload") .SendAsync(); ``` -### Pattern 10: Concurrent Requests with Rate Limiting +### Pattern 10: Batch Requests with Rate Limiting + +**For processing many requests with controlled concurrency:** + +```csharp +using DevBase.Net.Core; + +// Create batch processor with rate limit of 3 requests per second +// Note: Processing starts automatically on construction +var batchRequests = new BatchRequests() + .WithRateLimit(3) // 3 concurrent requests per second + .WithCookiePersistence() + .OnResponse(r => Console.WriteLine($"Status: {r.StatusCode}")) + .OnError((req, ex) => Console.WriteLine($"Error: {ex.Message}")) + .OnProgress(p => Console.WriteLine($"Progress: {p.PercentComplete:F1}%")); + +// Create named batches +var apiBatch = batchRequests.CreateBatch("api-calls"); +for (int i = 0; i < 100; i++) + apiBatch.Enqueue($"https://api.example.com/item/{i}"); + +var scrapingBatch = batchRequests.CreateBatch("scraping"); +for (int i = 0; i < 50; i++) + scrapingBatch.Enqueue($"https://site.com/page/{i}", r => r.WithBogusUserAgent()); + +// Execute all batches - sends 3 requests concurrently per second +var responses = await batchRequests.ExecuteAllAsync(); + +// Or execute specific batch +var apiResponses = await batchRequests.ExecuteBatchAsync("api-calls"); + +// Stream results as they complete +await foreach (var response in batchRequests.ExecuteAllAsyncEnumerable()) +{ + await ProcessAsync(response); +} +``` + +### Pattern 11: Request Re-queuing with Callbacks + +**Automatically retry requests based on response conditions:** + +```csharp +using DevBase.Net.Core; + +var batchRequests = new BatchRequests() + .WithRateLimit(5) + // Re-queue on specific HTTP status codes + .OnResponseRequeue((response, request) => + { + // Retry on rate limit (429) or server errors (5xx) + if (response.StatusCode == System.Net.HttpStatusCode.TooManyRequests) + { + return RequeueDecision.Requeue(); + } + + if ((int)response.StatusCode >= 500) + { + // Modify request before re-queuing + var modifiedRequest = new Request(request.GetUri()!.ToString()) + .WithHeader("X-Retry", "true"); + return RequeueDecision.RequeueWith(modifiedRequest); + } + + return RequeueDecision.NoRequeue; + }) + // Re-queue on specific exceptions + .OnErrorRequeue((request, exception) => + { + // Retry on timeout exceptions + if (exception is TaskCanceledException || exception is TimeoutException) + { + return RequeueDecision.Requeue(); + } + + // Retry on network errors + if (exception is HttpRequestException) + { + return RequeueDecision.Requeue(); + } + + return RequeueDecision.NoRequeue; + }); + +var batch = batchRequests.CreateBatch("resilient"); +batch.Enqueue("https://unreliable-api.com/data"); + +await batchRequests.ExecuteAllAsync(); +``` + +### Pattern 12: Proxied Batch Requests with Rotation + +**Process requests through rotating proxies with failure tracking:** + +```csharp +using DevBase.Net.Core; +using DevBase.Net.Proxy; + +var proxiedBatch = new ProxiedBatchRequests() + .WithRateLimit(5) + // Add single proxy + .WithProxy(new ProxyInfo("proxy1.example.com", 8080)) + // Or add from string + .WithProxy("socks5://user:pass@proxy2.example.com:1080") + // Or add multiple + .WithProxies(new[] + { + new ProxyInfo("proxy3.example.com", 8080), + new ProxyInfo("proxy4.example.com", 8080, "user", "password") + }) + // Choose rotation strategy + .WithRoundRobinRotation() // Default: cycle through proxies + // OR .WithRandomRotation() // Random selection + // OR .WithLeastFailuresRotation() // Prefer healthy proxies + // OR .WithStickyRotation() // Stick to one until it fails + .OnProxyFailure((proxy, context) => + { + Console.WriteLine($"Proxy {proxy.Key} failed: {context.Exception.Message}"); + if (context.ProxyTimedOut) + Console.WriteLine($" Proxy timed out after {proxy.MaxFailures} failures"); + Console.WriteLine($" Remaining available: {context.RemainingAvailableProxies}"); + }) + .OnResponse(r => Console.WriteLine($"Success via proxy")) + .OnError((req, ex) => Console.WriteLine($"Error: {ex.Message}")); + +var batch = proxiedBatch.CreateBatch("scraping"); +for (int i = 0; i < 1000; i++) + batch.Enqueue($"https://target-site.com/page/{i}"); + +var responses = await proxiedBatch.ExecuteAllAsync(); + +// Get proxy statistics +var stats = proxiedBatch.GetStatistics(); +Console.WriteLine($"Success rate: {stats.SuccessRate:F1}%"); +Console.WriteLine($"Proxy availability: {stats.ProxyAvailabilityRate:F1}%"); +``` + +### Pattern 13: Proxy Rotation Strategies + +**Choose the right strategy for your use case:** + +```csharp +using DevBase.Net.Core; + +// 1. Round Robin (default) - Cycles through proxies in order +var batch = new ProxiedBatchRequests() + .WithRoundRobinRotation(); + +// 2. Random - Random proxy selection, good for load balancing +var batch2 = new ProxiedBatchRequests() + .WithRandomRotation(); + +// 3. Least Failures - Prefers proxies with fewer failures +// Best for maximizing success rate +var batch3 = new ProxiedBatchRequests() + .WithLeastFailuresRotation(); + +// 4. Sticky - Uses same proxy until it fails, then switches +// Good for session-based scraping +var batch4 = new ProxiedBatchRequests() + .WithStickyRotation(); + +// 5. Custom Strategy - Implement IProxyRotationStrategy +public class PriorityStrategy : IProxyRotationStrategy +{ + public TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex) + { + // Prefer proxies with specific criteria + return proxies + .Where(p => p.IsAvailable()) + .OrderBy(p => p.TotalTimeouts) + .ThenBy(p => p.FailureCount) + .FirstOrDefault(); + } +} + +var batch5 = new ProxiedBatchRequests() + .WithRotationStrategy(new PriorityStrategy()); +``` + +### Pattern 14: Simple Concurrent Requests with Multitasking ```csharp using DevBase.Async.Task; @@ -440,6 +1400,13 @@ if (m.TotalTime.TotalSeconds > 5) | Get string | `await response.GetStringAsync()` | | Get bytes | `await response.GetBytesAsync()` | | Check status | `response.IsSuccessStatusCode` | +| Batch requests | `new BatchRequests().WithRateLimit(3).CreateBatch("name")` | +| Execute batch | `await batchRequests.ExecuteAllAsync()` | +| Re-queue on response | `.OnResponseRequeue((resp, req) => RequeueDecision.Requeue())` | +| Re-queue on error | `.OnErrorRequeue((req, ex) => RequeueDecision.Requeue())` | +| Proxied batch | `new ProxiedBatchRequests().WithProxy(proxyInfo)` | +| Proxy rotation | `.WithRoundRobinRotation()` / `.WithRandomRotation()` | +| Proxy failure callback | `.OnProxyFailure((proxy, ctx) => { })` | ## Testing Considerations diff --git a/DevBase.Net/Core/BatchRequests.cs b/DevBase.Net/Core/BatchRequests.cs new file mode 100644 index 0000000..ec6ab82 --- /dev/null +++ b/DevBase.Net/Core/BatchRequests.cs @@ -0,0 +1,897 @@ +using System.Collections.Concurrent; +using System.Net; +using System.Text; +using DevBase.Net.Utils; + +namespace DevBase.Net.Core; + +public sealed class BatchRequests : IDisposable, IAsyncDisposable +{ + private readonly ConcurrentDictionary _batches = new(); + private readonly ConcurrentQueue _responseQueue = new(); + private readonly ConcurrentDictionary _cookies = new(); + private readonly ConcurrentDictionary _referers = new(); + private readonly List> _responseCallbacks = []; + private readonly List> _errorCallbacks = []; + private readonly List> _progressCallbacks = []; + private readonly List> _reueueOnResponseCallbacks = []; + private readonly List> _requeueOnErrorCallbacks = []; + private readonly SemaphoreSlim _rateLimitSemaphore = new(1, 1); + + private int _rateLimit = 1; + private TimeSpan _rateLimitWindow = TimeSpan.FromSeconds(1); + private bool _persistCookies; + private bool _persistReferer; + private bool _disposed; + private DateTime _windowStart = DateTime.UtcNow; + private int _requestsInWindow; + + private CancellationTokenSource? _processingCts; + private Task? _processingTask; + private bool _isProcessing; + private int _processedCount; + private int _errorCount; + + public int BatchCount => _batches.Count; + public int TotalQueueCount => _batches.Values.Sum(b => b.QueueCount); + public int ResponseQueueCount => _responseQueue.Count; + public int RateLimit => _rateLimit; + public bool PersistCookies => _persistCookies; + public bool PersistReferer => _persistReferer; + public bool IsProcessing => _isProcessing; + public int ProcessedCount => _processedCount; + public int ErrorCount => _errorCount; + public IReadOnlyList BatchNames => _batches.Keys.ToList(); + + public BatchRequests() + { + StartProcessing(); + } + + #region Configuration + + public BatchRequests WithRateLimit(int requestsPerWindow, TimeSpan? window = null) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(requestsPerWindow); + _rateLimit = requestsPerWindow; + _rateLimitWindow = window ?? TimeSpan.FromSeconds(1); + return this; + } + + public BatchRequests WithCookiePersistence(bool persist = true) + { + _persistCookies = persist; + return this; + } + + public BatchRequests WithRefererPersistence(bool persist = true) + { + _persistReferer = persist; + return this; + } + + #endregion + + #region Batch Management + + public Batch CreateBatch(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + + Batch batch = new Batch(name, this); + if (!_batches.TryAdd(name, batch)) + throw new InvalidOperationException($"Batch '{name}' already exists"); + + return batch; + } + + public Batch GetOrCreateBatch(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + return _batches.GetOrAdd(name, n => new Batch(n, this)); + } + + public Batch? GetBatch(string name) + { + return _batches.TryGetValue(name, out Batch? batch) ? batch : null; + } + + public bool RemoveBatch(string name) + { + return _batches.TryRemove(name, out _); + } + + public BatchRequests ClearAllBatches() + { + foreach (Batch batch in _batches.Values) + batch.Clear(); + return this; + } + + #endregion + + #region Callbacks + + public BatchRequests OnResponse(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _responseCallbacks.Add(callback); + return this; + } + + public BatchRequests OnResponse(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _responseCallbacks.Add(r => { callback(r); return Task.CompletedTask; }); + return this; + } + + public BatchRequests OnError(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _errorCallbacks.Add(callback); + return this; + } + + public BatchRequests OnError(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _errorCallbacks.Add((r, e) => { callback(r, e); return Task.CompletedTask; }); + return this; + } + + public BatchRequests OnProgress(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _progressCallbacks.Add(callback); + return this; + } + + public BatchRequests OnProgress(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _progressCallbacks.Add(p => { callback(p); return Task.CompletedTask; }); + return this; + } + + public BatchRequests OnResponseRequeue(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _reueueOnResponseCallbacks.Add(callback); + return this; + } + + public BatchRequests OnErrorRequeue(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _requeueOnErrorCallbacks.Add(callback); + return this; + } + + #endregion + + #region Response Queue + + public bool TryDequeueResponse(out Response? response) + { + return _responseQueue.TryDequeue(out response); + } + + public List DequeueAllResponses() + { + List responses = new List(_responseQueue.Count); + while (_responseQueue.TryDequeue(out Response? response)) + responses.Add(response); + return responses; + } + + #endregion + + #region Processing + + public void StartProcessing() + { + if (_isProcessing) + return; + + _processingCts = new CancellationTokenSource(); + _isProcessing = true; + _processingTask = Task.Run(() => ProcessAllBatchesAsync(_processingCts.Token)); + } + + public async Task StopProcessingAsync() + { + if (!_isProcessing) + return; + + _processingCts?.Cancel(); + + if (_processingTask != null) + { + try + { + await _processingTask; + } + catch (OperationCanceledException) + { + } + } + + _isProcessing = false; + _processingCts?.Dispose(); + _processingCts = null; + _processingTask = null; + } + + public async Task> ExecuteAllAsync(CancellationToken cancellationToken = default) + { + ConcurrentBag responses = new ConcurrentBag(); + + List<(Request Request, string BatchName)> allRequests = CollectAllRequests(); + if (allRequests.Count == 0) + return new List(); + + int totalRequests = allRequests.Count; + int completed = 0; + int index = 0; + + while (index < allRequests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, allRequests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + for (int i = 0; i < batchSize; i++) + { + (Request Request, string BatchName) item = allRequests[index + i]; + tasks[i] = ProcessRequestAsync(item.Request, item.BatchName, responses, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + } + + return new List(responses); + } + + private async Task ProcessRequestAsync(Request request, string batchName, ConcurrentBag responses, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + try + { + ApplyPersistence(request); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + + responses.Add(response); + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batchName, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + Batch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + Batch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + public async Task> ExecuteBatchAsync(string batchName, CancellationToken cancellationToken = default) + { + Batch? batch = GetBatch(batchName); + if (batch == null) + throw new InvalidOperationException($"Batch '{batchName}' not found"); + + return await ExecuteBatchInternalAsync(batch, cancellationToken).ConfigureAwait(false); + } + + public async IAsyncEnumerable ExecuteAllAsyncEnumerable( + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) + { + List<(Request Request, string BatchName)> allRequests = CollectAllRequests(); + int totalRequests = allRequests.Count; + int completed = 0; + int index = 0; + + while (index < allRequests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, allRequests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + ConcurrentBag responseBag = new ConcurrentBag(); + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + + for (int i = 0; i < batchSize; i++) + { + (Request Request, string BatchName) item = allRequests[index + i]; + tasks[i] = ProcessEnumerableRequestAsync(item.Request, item.BatchName, responseBag, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + + foreach (Response response in responseBag) + yield return response; + } + } + + private async Task ProcessEnumerableRequestAsync(Request request, string batchName, ConcurrentBag responseBag, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + try + { + ApplyPersistence(request); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + + responseBag.Add(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batchName, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + Batch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + Batch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + private async Task ProcessAllBatchesAsync(CancellationToken cancellationToken) + { + while (!cancellationToken.IsCancellationRequested) + { + Request? request = TryDequeueNextRequest(out string? batchName); + if (request == null) + { + await Task.Delay(50, cancellationToken).ConfigureAwait(false); + continue; + } + + try + { + ApplyPersistence(request); + + Response response = await SendWithRateLimitAsync(request, cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + break; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue && batchName != null) + { + Batch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + } + + private async Task> ExecuteBatchInternalAsync(Batch batch, CancellationToken cancellationToken) + { + ConcurrentBag responses = new ConcurrentBag(); + List requests = batch.DequeueAll(); + + if (requests.Count == 0) + return new List(); + + int totalRequests = requests.Count; + int completed = 0; + int index = 0; + + while (index < requests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, requests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + for (int i = 0; i < batchSize; i++) + { + Request request = requests[index + i]; + tasks[i] = ProcessBatchRequestAsync(request, batch, responses, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + } + + return new List(responses); + } + + private async Task ProcessBatchRequestAsync(Request request, Batch batch, ConcurrentBag responses, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + try + { + ApplyPersistence(request); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + + responses.Add(response); + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batch.Name, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + batch.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + batch.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + private List<(Request Request, string BatchName)> CollectAllRequests() + { + List<(Request, string)> allRequests = new List<(Request, string)>(); + foreach (Batch batch in _batches.Values) + { + List requests = batch.DequeueAll(); + foreach (Request request in requests) + { + allRequests.Add((request, batch.Name)); + } + } + return allRequests; + } + + private Request? TryDequeueNextRequest(out string? batchName) + { + foreach (Batch batch in _batches.Values) + { + if (batch.TryDequeue(out Request? request)) + { + batchName = batch.Name; + return request; + } + } + batchName = null; + return null; + } + + #endregion + + #region Rate Limiting + + private async Task SendWithRateLimitAsync(Request request, CancellationToken cancellationToken) + { + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + return await request.SendAsync(cancellationToken).ConfigureAwait(false); + } + + private async Task EnforceRateLimitAsync(CancellationToken cancellationToken) + { + if (_rateLimit <= 0) + return; + + await _rateLimitSemaphore.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + DateTime now = DateTime.UtcNow; + + if (now - _windowStart >= _rateLimitWindow) + { + _windowStart = now; + _requestsInWindow = 0; + } + + if (_requestsInWindow >= _rateLimit) + { + TimeSpan waitTime = _rateLimitWindow - (now - _windowStart); + if (waitTime > TimeSpan.Zero) + await Task.Delay(waitTime, cancellationToken).ConfigureAwait(false); + + _windowStart = DateTime.UtcNow; + _requestsInWindow = 0; + } + + _requestsInWindow++; + } + finally + { + _rateLimitSemaphore.Release(); + } + } + + #endregion + + #region Persistence + + private void ApplyPersistence(Request request) + { + Uri? uri = request.GetUri(); + if (uri == null) + return; + + string host = uri.Host; + + if (_persistCookies && _cookies.TryGetValue(host, out CookieContainer? container)) + { + CookieCollection cookies = container.GetCookies(uri); + if (cookies.Count > 0) + { + StringBuilder sb = StringBuilderPool.Acquire(512); + + for (int i = 0; i < cookies.Count; i++) + { + if (i > 0) + sb.Append("; "); + + Cookie cookie = cookies[i]; + sb.Append(cookie.Name); + sb.Append('='); + sb.Append(cookie.Value); + } + + request.WithCookie(sb.ToStringAndRelease()); + } + } + + if (_persistReferer && _referers.TryGetValue(host, out string? referer)) + { + request.WithReferer(referer); + } + } + + private void StorePersistence(Request request, Response response) + { + Uri? uri = request.GetUri(); + if (uri == null) + return; + + string host = uri.Host; + + if (_persistCookies) + { + CookieCollection responseCookies = response.GetCookies(); + if (responseCookies.Count > 0) + { + CookieContainer container = _cookies.GetOrAdd(host, _ => new CookieContainer()); + foreach (Cookie cookie in responseCookies) + container.Add(uri, cookie); + } + } + + if (_persistReferer) + { + _referers[host] = uri.ToString(); + } + } + + #endregion + + #region Callbacks + + private async Task InvokeCallbacksAsync(Response response) + { + for (int i = 0; i < _responseCallbacks.Count; i++) + { + try + { + await _responseCallbacks[i](response).ConfigureAwait(false); + } + catch + { + } + } + } + + private async Task InvokeErrorCallbacksAsync(Request request, System.Exception exception) + { + for (int i = 0; i < _errorCallbacks.Count; i++) + { + try + { + await _errorCallbacks[i](request, exception).ConfigureAwait(false); + } + catch + { + } + } + } + + private async Task InvokeProgressCallbacksAsync(BatchProgressInfo progress) + { + for (int i = 0; i < _progressCallbacks.Count; i++) + { + try + { + await _progressCallbacks[i](progress).ConfigureAwait(false); + } + catch + { + } + } + } + + private RequeueDecision EvaluateResponseRequeue(Response response, Request request) + { + for (int i = 0; i < _reueueOnResponseCallbacks.Count; i++) + { + try + { + RequeueDecision decision = _reueueOnResponseCallbacks[i](response, request); + if (decision.ShouldRequeue) + return decision; + } + catch + { + } + } + return RequeueDecision.NoRequeue; + } + + private RequeueDecision EvaluateErrorRequeue(Request request, System.Exception exception) + { + for (int i = 0; i < _requeueOnErrorCallbacks.Count; i++) + { + try + { + RequeueDecision decision = _requeueOnErrorCallbacks[i](request, exception); + if (decision.ShouldRequeue) + return decision; + } + catch + { + } + } + return RequeueDecision.NoRequeue; + } + + #endregion + + #region Utilities + + public void ResetCounters() + { + Interlocked.Exchange(ref _processedCount, 0); + Interlocked.Exchange(ref _errorCount, 0); + } + + public BatchStatistics GetStatistics() + { + return new BatchStatistics( + BatchCount, + TotalQueueCount, + ProcessedCount, + ErrorCount, + _batches.ToDictionary(b => b.Key, b => b.Value.QueueCount) + ); + } + + #endregion + + #region Dispose + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + + _processingCts?.Cancel(); + _processingCts?.Dispose(); + _rateLimitSemaphore.Dispose(); + + foreach (Batch batch in _batches.Values) + batch.Clear(); + + _batches.Clear(); + _cookies.Clear(); + _referers.Clear(); + _responseCallbacks.Clear(); + _errorCallbacks.Clear(); + _progressCallbacks.Clear(); + } + + public async ValueTask DisposeAsync() + { + if (_disposed) return; + + await StopProcessingAsync(); + Dispose(); + } + + #endregion +} + +public sealed class Batch +{ + private readonly ConcurrentQueue _queue = new(); + private readonly BatchRequests _parent; + + public string Name { get; } + public int QueueCount => _queue.Count; + + internal Batch(string name, BatchRequests parent) + { + Name = name; + _parent = parent; + } + + public Batch Add(Request request) + { + ArgumentNullException.ThrowIfNull(request); + _queue.Enqueue(request); + return this; + } + + public Batch Add(IEnumerable requests) + { + foreach (Request request in requests) + Add(request); + return this; + } + + public Batch Add(string url) + { + return Add(new Request(url)); + } + + public Batch Add(IEnumerable urls) + { + foreach (string url in urls) + Add(url); + return this; + } + + public Batch Enqueue(Request request) => Add(request); + public Batch Enqueue(string url) => Add(url); + public Batch Enqueue(IEnumerable requests) => Add(requests); + public Batch Enqueue(IEnumerable urls) => Add(urls); + + public Batch Enqueue(string url, Action configure) + { + Request request = new Request(url); + configure(request); + return Add(request); + } + + public Batch Enqueue(Func requestFactory) + { + return Add(requestFactory()); + } + + public bool TryDequeue(out Request? request) + { + return _queue.TryDequeue(out request); + } + + internal List DequeueAll() + { + List requests = new List(_queue.Count); + while (_queue.TryDequeue(out Request? request)) + requests.Add(request); + return requests; + } + + public void Clear() + { + while (_queue.TryDequeue(out _)) { } + } + + public BatchRequests EndBatch() => _parent; +} + +public sealed record BatchProgressInfo( + string BatchName, + int Completed, + int Total, + int Errors +) +{ + public double PercentComplete => Total > 0 ? (double)Completed / Total * 100 : 0; + public int Remaining => Total - Completed; +} + +public sealed record BatchStatistics( + int BatchCount, + int TotalQueuedRequests, + int ProcessedRequests, + int ErrorCount, + Dictionary RequestsPerBatch +) +{ + public double SuccessRate => ProcessedRequests > 0 + ? (double)(ProcessedRequests - ErrorCount) / ProcessedRequests * 100 + : 0; +} + +public readonly struct RequeueDecision +{ + public bool ShouldRequeue { get; } + public Request? ModifiedRequest { get; } + + private RequeueDecision(bool shouldRequeue, Request? modifiedRequest = null) + { + ShouldRequeue = shouldRequeue; + ModifiedRequest = modifiedRequest; + } + + public static RequeueDecision NoRequeue => new(false); + public static RequeueDecision Requeue() => new(true); + public static RequeueDecision RequeueWith(Request modifiedRequest) => new(true, modifiedRequest); +} diff --git a/DevBase.Net/Core/ProxiedBatchRequests.cs b/DevBase.Net/Core/ProxiedBatchRequests.cs new file mode 100644 index 0000000..cc89179 --- /dev/null +++ b/DevBase.Net/Core/ProxiedBatchRequests.cs @@ -0,0 +1,1160 @@ +using System.Collections.Concurrent; +using System.Net; +using System.Text; +using DevBase.Net.Proxy; +using DevBase.Net.Utils; + +namespace DevBase.Net.Core; + +public sealed class ProxiedBatchRequests : IDisposable, IAsyncDisposable +{ + private readonly ConcurrentDictionary _batches = new(); + private readonly ConcurrentQueue _responseQueue = new(); + private readonly ConcurrentDictionary _cookies = new(); + private readonly ConcurrentDictionary _referers = new(); + private readonly List> _responseCallbacks = new List>(); + private readonly List> _errorCallbacks = new List>(); + private readonly List> _progressCallbacks = new List>(); + private readonly List> _requeueOnResponseCallbacks = new List>(); + private readonly List> _requeueOnErrorCallbacks = new List>(); + private readonly List> _proxyFailureCallbacks = new List>(); + private readonly SemaphoreSlim _rateLimitSemaphore = new(1, 1); + private readonly SemaphoreSlim _proxyLock = new(1, 1); + + private readonly List _proxyPool = new List(); + private int _currentProxyIndex; + private IProxyRotationStrategy _rotationStrategy = new RoundRobinStrategy(); + + private int _rateLimit = 1; + private TimeSpan _rateLimitWindow = TimeSpan.FromSeconds(1); + private bool _persistCookies; + private bool _persistReferer; + private bool _disposed; + private DateTime _windowStart = DateTime.UtcNow; + private int _requestsInWindow; + + private CancellationTokenSource? _processingCts; + private Task? _processingTask; + private bool _isProcessing; + private int _processedCount; + private int _errorCount; + private int _proxyFailureCount; + + public int BatchCount => _batches.Count; + public int TotalQueueCount => _batches.Values.Sum(b => b.QueueCount); + public int ResponseQueueCount => _responseQueue.Count; + public int RateLimit => _rateLimit; + public bool PersistCookies => _persistCookies; + public bool PersistReferer => _persistReferer; + public bool IsProcessing => _isProcessing; + public int ProcessedCount => _processedCount; + public int ErrorCount => _errorCount; + public int ProxyFailureCount => _proxyFailureCount; + public int ProxyCount => _proxyPool.Count; + public int AvailableProxyCount => _proxyPool.Count(p => p.IsAvailable()); + public IReadOnlyList BatchNames => _batches.Keys.ToList(); + + public ProxiedBatchRequests() + { + StartProcessing(); + } + + #region Proxy Configuration + + public ProxiedBatchRequests WithProxy(ProxyInfo proxy) + { + ArgumentNullException.ThrowIfNull(proxy); + _proxyPool.Add(new TrackedProxyInfo(proxy)); + return this; + } + + public ProxiedBatchRequests WithProxy(string proxyString) + { + return WithProxy(ProxyInfo.Parse(proxyString)); + } + + public ProxiedBatchRequests WithProxies(IEnumerable proxies) + { + foreach (ProxyInfo proxy in proxies) + WithProxy(proxy); + return this; + } + + public ProxiedBatchRequests WithProxies(IEnumerable proxyStrings) + { + foreach (string proxyString in proxyStrings) + WithProxy(proxyString); + return this; + } + + public ProxiedBatchRequests WithRotationStrategy(IProxyRotationStrategy strategy) + { + ArgumentNullException.ThrowIfNull(strategy); + _rotationStrategy = strategy; + return this; + } + + public ProxiedBatchRequests WithRoundRobinRotation() + { + _rotationStrategy = new RoundRobinStrategy(); + return this; + } + + public ProxiedBatchRequests WithRandomRotation() + { + _rotationStrategy = new RandomStrategy(); + return this; + } + + public ProxiedBatchRequests WithLeastFailuresRotation() + { + _rotationStrategy = new LeastFailuresStrategy(); + return this; + } + + public ProxiedBatchRequests WithStickyRotation() + { + _rotationStrategy = new StickyStrategy(); + return this; + } + + public ProxiedBatchRequests ConfigureProxyTracking(int maxFailures = 3, TimeSpan? timeoutDuration = null) + { + foreach (TrackedProxyInfo proxy in _proxyPool) + { + proxy.ResetTimeout(); + } + return this; + } + + #endregion + + #region Configuration + + public ProxiedBatchRequests WithRateLimit(int requestsPerWindow, TimeSpan? window = null) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(requestsPerWindow); + _rateLimit = requestsPerWindow; + _rateLimitWindow = window ?? TimeSpan.FromSeconds(1); + return this; + } + + public ProxiedBatchRequests WithCookiePersistence(bool persist = true) + { + _persistCookies = persist; + return this; + } + + public ProxiedBatchRequests WithRefererPersistence(bool persist = true) + { + _persistReferer = persist; + return this; + } + + #endregion + + #region Batch Management + + public ProxiedBatch CreateBatch(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + + ProxiedBatch batch = new ProxiedBatch(name, this); + if (!_batches.TryAdd(name, batch)) + throw new InvalidOperationException($"Batch '{name}' already exists"); + + return batch; + } + + public ProxiedBatch GetOrCreateBatch(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + return _batches.GetOrAdd(name, n => new ProxiedBatch(n, this)); + } + + public ProxiedBatch? GetBatch(string name) + { + return _batches.TryGetValue(name, out ProxiedBatch? batch) ? batch : null; + } + + public bool RemoveBatch(string name) + { + return _batches.TryRemove(name, out _); + } + + public ProxiedBatchRequests ClearAllBatches() + { + foreach (ProxiedBatch batch in _batches.Values) + batch.Clear(); + return this; + } + + #endregion + + #region Callbacks + + public ProxiedBatchRequests OnResponse(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _responseCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnResponse(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _responseCallbacks.Add(r => { callback(r); return Task.CompletedTask; }); + return this; + } + + public ProxiedBatchRequests OnError(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _errorCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnError(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _errorCallbacks.Add((r, e) => { callback(r, e); return Task.CompletedTask; }); + return this; + } + + public ProxiedBatchRequests OnProgress(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _progressCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnProgress(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _progressCallbacks.Add(p => { callback(p); return Task.CompletedTask; }); + return this; + } + + public ProxiedBatchRequests OnResponseRequeue(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _requeueOnResponseCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnErrorRequeue(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _requeueOnErrorCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnProxyFailure(Func callback) + { + ArgumentNullException.ThrowIfNull(callback); + _proxyFailureCallbacks.Add(callback); + return this; + } + + public ProxiedBatchRequests OnProxyFailure(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _proxyFailureCallbacks.Add((p, c) => { callback(p, c); return Task.CompletedTask; }); + return this; + } + + #endregion + + #region Response Queue + + public bool TryDequeueResponse(out Response? response) + { + return _responseQueue.TryDequeue(out response); + } + + public List DequeueAllResponses() + { + List responses = new List(_responseQueue.Count); + while (_responseQueue.TryDequeue(out Response? response)) + responses.Add(response); + return responses; + } + + #endregion + + #region Processing + + public void StartProcessing() + { + if (_isProcessing) + return; + + _processingCts = new CancellationTokenSource(); + _isProcessing = true; + _processingTask = Task.Run(() => ProcessAllBatchesAsync(_processingCts.Token)); + } + + public async Task StopProcessingAsync() + { + if (!_isProcessing) + return; + + _processingCts?.Cancel(); + + if (_processingTask != null) + { + try + { + await _processingTask; + } + catch (OperationCanceledException) + { + } + } + + _isProcessing = false; + _processingCts?.Dispose(); + _processingCts = null; + _processingTask = null; + } + + public async Task> ExecuteAllAsync(CancellationToken cancellationToken = default) + { + ConcurrentBag responses = new ConcurrentBag(); + + List<(Request Request, string BatchName)> allRequests = CollectAllRequests(); + if (allRequests.Count == 0) + return new List(); + + int totalRequests = allRequests.Count; + int completed = 0; + int index = 0; + + while (index < allRequests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, allRequests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + for (int i = 0; i < batchSize; i++) + { + (Request Request, string BatchName) item = allRequests[index + i]; + tasks[i] = ProcessProxiedRequestAsync(item.Request, item.BatchName, responses, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + } + + return new List(responses); + } + + private async Task ProcessProxiedRequestAsync(Request request, string batchName, ConcurrentBag responses, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + TrackedProxyInfo? usedProxy = null; + try + { + ApplyPersistence(request); + usedProxy = await ApplyProxyAsync(request, cancellationToken).ConfigureAwait(false); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + usedProxy?.ReportSuccess(); + + responses.Add(response); + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batchName, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + ProxiedBatch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await HandleProxyFailureAsync(usedProxy, ex, request).ConfigureAwait(false); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + ProxiedBatch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + public async Task> ExecuteBatchAsync(string batchName, CancellationToken cancellationToken = default) + { + ProxiedBatch? batch = GetBatch(batchName); + if (batch == null) + throw new InvalidOperationException($"Batch '{batchName}' not found"); + + return await ExecuteBatchInternalAsync(batch, cancellationToken).ConfigureAwait(false); + } + + public async IAsyncEnumerable ExecuteAllAsyncEnumerable( + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken = default) + { + List<(Request Request, string BatchName)> allRequests = CollectAllRequests(); + int totalRequests = allRequests.Count; + int completed = 0; + int index = 0; + + while (index < allRequests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, allRequests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + ConcurrentBag responseBag = new ConcurrentBag(); + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + + for (int i = 0; i < batchSize; i++) + { + (Request Request, string BatchName) item = allRequests[index + i]; + tasks[i] = ProcessEnumerableProxiedRequestAsync(item.Request, item.BatchName, responseBag, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + + foreach (Response response in responseBag) + yield return response; + } + } + + private async Task ProcessEnumerableProxiedRequestAsync(Request request, string batchName, ConcurrentBag responseBag, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + TrackedProxyInfo? usedProxy = null; + try + { + ApplyPersistence(request); + usedProxy = await ApplyProxyAsync(request, cancellationToken).ConfigureAwait(false); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + usedProxy?.ReportSuccess(); + + responseBag.Add(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batchName, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + ProxiedBatch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await HandleProxyFailureAsync(usedProxy, ex, request).ConfigureAwait(false); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + ProxiedBatch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + private async Task ProcessAllBatchesAsync(CancellationToken cancellationToken) + { + while (!cancellationToken.IsCancellationRequested) + { + Request? request = TryDequeueNextRequest(out string? batchName); + if (request == null) + { + await Task.Delay(50, cancellationToken).ConfigureAwait(false); + continue; + } + + TrackedProxyInfo? usedProxy = null; + try + { + ApplyPersistence(request); + usedProxy = await ApplyProxyAsync(request, cancellationToken).ConfigureAwait(false); + + Response response = await SendWithRateLimitAsync(request, cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + usedProxy?.ReportSuccess(); + + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + break; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await HandleProxyFailureAsync(usedProxy, ex, request).ConfigureAwait(false); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue && batchName != null) + { + ProxiedBatch? batch = GetBatch(batchName); + batch?.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + } + + private async Task> ExecuteBatchInternalAsync(ProxiedBatch batch, CancellationToken cancellationToken) + { + ConcurrentBag responses = new ConcurrentBag(); + List requests = batch.DequeueAll(); + + if (requests.Count == 0) + return new List(); + + int totalRequests = requests.Count; + int completed = 0; + int index = 0; + + while (index < requests.Count && !cancellationToken.IsCancellationRequested) + { + int batchSize = Math.Min(_rateLimit, requests.Count - index); + + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + + Task[] tasks = new Task[batchSize]; + int[] completedHolder = new int[] { completed }; + for (int i = 0; i < batchSize; i++) + { + Request request = requests[index + i]; + tasks[i] = ProcessBatchProxiedRequestAsync(request, batch, responses, completedHolder, totalRequests, cancellationToken); + } + index += batchSize; + + await Task.WhenAll(tasks).ConfigureAwait(false); + completed = completedHolder[0]; + } + + return new List(responses); + } + + private async Task ProcessBatchProxiedRequestAsync(Request request, ProxiedBatch batch, ConcurrentBag responses, int[] completedHolder, int totalRequests, CancellationToken cancellationToken) + { + TrackedProxyInfo? usedProxy = null; + try + { + ApplyPersistence(request); + usedProxy = await ApplyProxyAsync(request, cancellationToken).ConfigureAwait(false); + + Response response = await request.SendAsync(cancellationToken).ConfigureAwait(false); + + StorePersistence(request, response); + usedProxy?.ReportSuccess(); + + responses.Add(response); + _responseQueue.Enqueue(response); + Interlocked.Increment(ref _processedCount); + int currentCompleted = Interlocked.Increment(ref completedHolder[0]); + + await InvokeCallbacksAsync(response).ConfigureAwait(false); + await InvokeProgressCallbacksAsync(new BatchProgressInfo( + batch.Name, currentCompleted, totalRequests, _errorCount)).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateResponseRequeue(response, request); + if (requeueDecision.ShouldRequeue) + { + batch.Add(requeueDecision.ModifiedRequest ?? request); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (System.Exception ex) + { + Interlocked.Increment(ref _errorCount); + await HandleProxyFailureAsync(usedProxy, ex, request).ConfigureAwait(false); + await InvokeErrorCallbacksAsync(request, ex).ConfigureAwait(false); + + RequeueDecision requeueDecision = EvaluateErrorRequeue(request, ex); + if (requeueDecision.ShouldRequeue) + { + batch.Add(requeueDecision.ModifiedRequest ?? request); + } + } + } + + private List<(Request Request, string BatchName)> CollectAllRequests() + { + List<(Request, string)> allRequests = new List<(Request, string)>(); + foreach (ProxiedBatch batch in _batches.Values) + { + List requests = batch.DequeueAll(); + foreach (Request request in requests) + { + allRequests.Add((request, batch.Name)); + } + } + return allRequests; + } + + private Request? TryDequeueNextRequest(out string? batchName) + { + foreach (ProxiedBatch batch in _batches.Values) + { + if (batch.TryDequeue(out Request? request)) + { + batchName = batch.Name; + return request; + } + } + batchName = null; + return null; + } + + #endregion + + #region Proxy Rotation + + private async Task ApplyProxyAsync(Request request, CancellationToken cancellationToken) + { + if (_proxyPool.Count == 0) + return null; + + await _proxyLock.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + TrackedProxyInfo? proxy = _rotationStrategy.SelectProxy(_proxyPool, ref _currentProxyIndex); + if (proxy != null) + { + request.WithProxy(proxy.Proxy); + } + return proxy; + } + finally + { + _proxyLock.Release(); + } + } + + private async Task HandleProxyFailureAsync(TrackedProxyInfo? proxy, System.Exception exception, Request request) + { + if (proxy == null) + return; + + bool timedOut = proxy.ReportFailure(); + Interlocked.Increment(ref _proxyFailureCount); + + ProxyFailureContext context = new ProxyFailureContext( + proxy, + exception, + request, + timedOut, + proxy.FailureCount, + _proxyPool.Count(p => p.IsAvailable()) + ); + + for (int i = 0; i < _proxyFailureCallbacks.Count; i++) + { + try + { + await _proxyFailureCallbacks[i](proxy, context).ConfigureAwait(false); + } + catch + { + } + } + } + + public IReadOnlyList GetProxyStatistics() + { + return _proxyPool.AsReadOnly(); + } + + public void ResetAllProxies() + { + foreach (TrackedProxyInfo proxy in _proxyPool) + proxy.ResetTimeout(); + } + + public void ClearProxies() + { + _proxyPool.Clear(); + _currentProxyIndex = 0; + } + + #endregion + + #region Rate Limiting + + private async Task SendWithRateLimitAsync(Request request, CancellationToken cancellationToken) + { + await EnforceRateLimitAsync(cancellationToken).ConfigureAwait(false); + return await request.SendAsync(cancellationToken).ConfigureAwait(false); + } + + private async Task EnforceRateLimitAsync(CancellationToken cancellationToken) + { + if (_rateLimit <= 0) + return; + + await _rateLimitSemaphore.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + DateTime now = DateTime.UtcNow; + + if (now - _windowStart >= _rateLimitWindow) + { + _windowStart = now; + _requestsInWindow = 0; + } + + if (_requestsInWindow >= _rateLimit) + { + TimeSpan waitTime = _rateLimitWindow - (now - _windowStart); + if (waitTime > TimeSpan.Zero) + await Task.Delay(waitTime, cancellationToken).ConfigureAwait(false); + + _windowStart = DateTime.UtcNow; + _requestsInWindow = 0; + } + + _requestsInWindow++; + } + finally + { + _rateLimitSemaphore.Release(); + } + } + + #endregion + + #region Persistence + + private void ApplyPersistence(Request request) + { + Uri? uri = request.GetUri(); + if (uri == null) + return; + + string host = uri.Host; + + if (_persistCookies && _cookies.TryGetValue(host, out CookieContainer? container)) + { + CookieCollection cookies = container.GetCookies(uri); + if (cookies.Count > 0) + { + StringBuilder sb = StringBuilderPool.Acquire(512); + + for (int i = 0; i < cookies.Count; i++) + { + if (i > 0) + sb.Append("; "); + + Cookie cookie = cookies[i]; + sb.Append(cookie.Name); + sb.Append('='); + sb.Append(cookie.Value); + } + + request.WithCookie(sb.ToStringAndRelease()); + } + } + + if (_persistReferer && _referers.TryGetValue(host, out string? referer)) + { + request.WithReferer(referer); + } + } + + private void StorePersistence(Request request, Response response) + { + Uri? uri = request.GetUri(); + if (uri == null) + return; + + string host = uri.Host; + + if (_persistCookies) + { + CookieCollection responseCookies = response.GetCookies(); + if (responseCookies.Count > 0) + { + CookieContainer container = _cookies.GetOrAdd(host, _ => new CookieContainer()); + foreach (Cookie cookie in responseCookies) + container.Add(uri, cookie); + } + } + + if (_persistReferer) + { + _referers[host] = uri.ToString(); + } + } + + #endregion + + #region Callbacks + + private async Task InvokeCallbacksAsync(Response response) + { + for (int i = 0; i < _responseCallbacks.Count; i++) + { + try + { + await _responseCallbacks[i](response).ConfigureAwait(false); + } + catch + { + } + } + } + + private async Task InvokeErrorCallbacksAsync(Request request, System.Exception exception) + { + for (int i = 0; i < _errorCallbacks.Count; i++) + { + try + { + await _errorCallbacks[i](request, exception).ConfigureAwait(false); + } + catch + { + } + } + } + + private async Task InvokeProgressCallbacksAsync(BatchProgressInfo progress) + { + for (int i = 0; i < _progressCallbacks.Count; i++) + { + try + { + await _progressCallbacks[i](progress).ConfigureAwait(false); + } + catch + { + } + } + } + + private RequeueDecision EvaluateResponseRequeue(Response response, Request request) + { + for (int i = 0; i < _requeueOnResponseCallbacks.Count; i++) + { + try + { + RequeueDecision decision = _requeueOnResponseCallbacks[i](response, request); + if (decision.ShouldRequeue) + return decision; + } + catch + { + } + } + return RequeueDecision.NoRequeue; + } + + private RequeueDecision EvaluateErrorRequeue(Request request, System.Exception exception) + { + for (int i = 0; i < _requeueOnErrorCallbacks.Count; i++) + { + try + { + RequeueDecision decision = _requeueOnErrorCallbacks[i](request, exception); + if (decision.ShouldRequeue) + return decision; + } + catch + { + } + } + return RequeueDecision.NoRequeue; + } + + #endregion + + #region Utilities + + public void ResetCounters() + { + Interlocked.Exchange(ref _processedCount, 0); + Interlocked.Exchange(ref _errorCount, 0); + Interlocked.Exchange(ref _proxyFailureCount, 0); + } + + public ProxiedBatchStatistics GetStatistics() + { + return new ProxiedBatchStatistics( + BatchCount, + TotalQueueCount, + ProcessedCount, + ErrorCount, + ProxyFailureCount, + ProxyCount, + AvailableProxyCount, + _batches.ToDictionary(b => b.Key, b => b.Value.QueueCount) + ); + } + + #endregion + + #region Dispose + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + + _processingCts?.Cancel(); + _processingCts?.Dispose(); + _rateLimitSemaphore.Dispose(); + _proxyLock.Dispose(); + + foreach (ProxiedBatch batch in _batches.Values) + batch.Clear(); + + _batches.Clear(); + _cookies.Clear(); + _referers.Clear(); + _responseCallbacks.Clear(); + _errorCallbacks.Clear(); + _progressCallbacks.Clear(); + _proxyPool.Clear(); + } + + public async ValueTask DisposeAsync() + { + if (_disposed) return; + + await StopProcessingAsync(); + Dispose(); + } + + #endregion +} + +public sealed class ProxiedBatch +{ + private readonly ConcurrentQueue _queue = new(); + private readonly ProxiedBatchRequests _parent; + + public string Name { get; } + public int QueueCount => _queue.Count; + + internal ProxiedBatch(string name, ProxiedBatchRequests parent) + { + Name = name; + _parent = parent; + } + + public ProxiedBatch Add(Request request) + { + ArgumentNullException.ThrowIfNull(request); + _queue.Enqueue(request); + return this; + } + + public ProxiedBatch Add(IEnumerable requests) + { + foreach (Request request in requests) + Add(request); + return this; + } + + public ProxiedBatch Add(string url) + { + return Add(new Request(url)); + } + + public ProxiedBatch Add(IEnumerable urls) + { + foreach (string url in urls) + Add(url); + return this; + } + + public ProxiedBatch Enqueue(Request request) => Add(request); + public ProxiedBatch Enqueue(string url) => Add(url); + public ProxiedBatch Enqueue(IEnumerable requests) => Add(requests); + public ProxiedBatch Enqueue(IEnumerable urls) => Add(urls); + + public ProxiedBatch Enqueue(string url, Action configure) + { + Request request = new Request(url); + configure(request); + return Add(request); + } + + public ProxiedBatch Enqueue(Func requestFactory) + { + return Add(requestFactory()); + } + + public bool TryDequeue(out Request? request) + { + return _queue.TryDequeue(out request); + } + + internal List DequeueAll() + { + List requests = new List(_queue.Count); + while (_queue.TryDequeue(out Request? request)) + requests.Add(request); + return requests; + } + + public void Clear() + { + while (_queue.TryDequeue(out _)) { } + } + + public ProxiedBatchRequests EndBatch() => _parent; +} + +public sealed record ProxiedBatchStatistics( + int BatchCount, + int TotalQueuedRequests, + int ProcessedRequests, + int ErrorCount, + int ProxyFailureCount, + int TotalProxies, + int AvailableProxies, + Dictionary RequestsPerBatch +) +{ + public double SuccessRate => ProcessedRequests > 0 + ? (double)(ProcessedRequests - ErrorCount) / ProcessedRequests * 100 + : 0; + + public double ProxyAvailabilityRate => TotalProxies > 0 + ? (double)AvailableProxies / TotalProxies * 100 + : 0; +} + +public sealed record ProxyFailureContext( + TrackedProxyInfo Proxy, + System.Exception Exception, + Request FailedRequest, + bool ProxyTimedOut, + int CurrentFailureCount, + int RemainingAvailableProxies +); + +public interface IProxyRotationStrategy +{ + TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex); +} + +public sealed class RoundRobinStrategy : IProxyRotationStrategy +{ + public TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex) + { + if (proxies.Count == 0) + return null; + + int attempts = 0; + while (attempts < proxies.Count) + { + currentIndex = (currentIndex + 1) % proxies.Count; + var proxy = proxies[currentIndex]; + + if (proxy.IsAvailable()) + return proxy; + + attempts++; + } + + return null; + } +} + +public sealed class RandomStrategy : IProxyRotationStrategy +{ + private static readonly Random Random = new(); + + public TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex) + { + var available = proxies.Where(p => p.IsAvailable()).ToList(); + if (available.Count == 0) + return null; + + int index = Random.Next(available.Count); + var selected = available[index]; + currentIndex = proxies.IndexOf(selected); + return selected; + } +} + +public sealed class LeastFailuresStrategy : IProxyRotationStrategy +{ + public TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex) + { + var available = proxies.Where(p => p.IsAvailable()).ToList(); + if (available.Count == 0) + return null; + + var selected = available.OrderBy(p => p.FailureCount).ThenBy(p => p.TotalTimeouts).First(); + currentIndex = proxies.IndexOf(selected); + return selected; + } +} + +public sealed class StickyStrategy : IProxyRotationStrategy +{ + public TrackedProxyInfo? SelectProxy(List proxies, ref int currentIndex) + { + if (proxies.Count == 0) + return null; + + if (currentIndex >= 0 && currentIndex < proxies.Count) + { + var current = proxies[currentIndex]; + if (current.IsAvailable()) + return current; + } + + for (int i = 0; i < proxies.Count; i++) + { + if (proxies[i].IsAvailable()) + { + currentIndex = i; + return proxies[i]; + } + } + + return null; + } +} diff --git a/DevBase.Net/DevBase.Net.csproj b/DevBase.Net/DevBase.Net.csproj index fdd7058..38f3b67 100644 --- a/DevBase.Net/DevBase.Net.csproj +++ b/DevBase.Net/DevBase.Net.csproj @@ -15,7 +15,7 @@ https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git git - 1.1.0 + 1.2.0 MIT false http;client;requests;proxy;socks5;jwt;authentication;fluent-api;async;retry;rate-limiting;json;html-parsing diff --git a/DevBase.Net/Proxy/ProxyConfiguration.cs b/DevBase.Net/Proxy/ProxyConfiguration.cs new file mode 100644 index 0000000..3399840 --- /dev/null +++ b/DevBase.Net/Proxy/ProxyConfiguration.cs @@ -0,0 +1,182 @@ +using System.Net; +using DevBase.Net.Proxy.Enums; + +namespace DevBase.Net.Proxy; + +public sealed class ProxyConfiguration +{ + public string Host { get; private set; } = string.Empty; + public int Port { get; private set; } + public EnumProxyType Type { get; private set; } = EnumProxyType.Http; + public NetworkCredential? Credentials { get; private set; } + + public bool BypassLocalAddresses { get; private set; } + public string[]? BypassList { get; private set; } + + public bool ResolveHostnamesLocally { get; private set; } + public TimeSpan ConnectionTimeout { get; private set; } = TimeSpan.FromSeconds(30); + public TimeSpan ReadWriteTimeout { get; private set; } = TimeSpan.FromSeconds(60); + + public int InternalServerPort { get; private set; } + + private ProxyConfiguration() { } + + public static HttpProxyBuilder Http(string host, int port) => new(host, port, EnumProxyType.Http); + public static HttpProxyBuilder Https(string host, int port) => new(host, port, EnumProxyType.Https); + public static Socks4ProxyBuilder Socks4(string host, int port) => new(host, port); + public static Socks5ProxyBuilder Socks5(string host, int port) => new(host, port, EnumProxyType.Socks5); + public static Socks5ProxyBuilder Socks5h(string host, int port) => new(host, port, EnumProxyType.Socks5h); + + public ProxyInfo ToProxyInfo() + { + return new ProxyInfo(Host, Port, Type, Credentials); + } + + public sealed class HttpProxyBuilder + { + private readonly ProxyConfiguration _config = new(); + + internal HttpProxyBuilder(string host, int port, EnumProxyType type) + { + _config.Host = host; + _config.Port = port; + _config.Type = type; + } + + public HttpProxyBuilder WithCredentials(string username, string password) + { + _config.Credentials = new NetworkCredential(username, password); + return this; + } + + public HttpProxyBuilder WithCredentials(NetworkCredential credentials) + { + _config.Credentials = credentials; + return this; + } + + public HttpProxyBuilder BypassLocal(bool bypass = true) + { + _config.BypassLocalAddresses = bypass; + return this; + } + + public HttpProxyBuilder WithBypassList(params string[] addresses) + { + _config.BypassList = addresses; + return this; + } + + public ProxyConfiguration Build() => _config; + + public ProxyInfo ToProxyInfo() => Build().ToProxyInfo(); + + public static implicit operator ProxyInfo(HttpProxyBuilder builder) => builder.ToProxyInfo(); + public static implicit operator ProxyConfiguration(HttpProxyBuilder builder) => builder.Build(); + } + + public sealed class Socks4ProxyBuilder + { + private readonly ProxyConfiguration _config = new(); + + internal Socks4ProxyBuilder(string host, int port) + { + _config.Host = host; + _config.Port = port; + _config.Type = EnumProxyType.Socks4; + } + + public Socks4ProxyBuilder WithUserId(string userId) + { + _config.Credentials = new NetworkCredential(userId, string.Empty); + return this; + } + + public Socks4ProxyBuilder WithConnectionTimeout(TimeSpan timeout) + { + _config.ConnectionTimeout = timeout; + return this; + } + + public Socks4ProxyBuilder WithReadWriteTimeout(TimeSpan timeout) + { + _config.ReadWriteTimeout = timeout; + return this; + } + + public Socks4ProxyBuilder WithInternalServerPort(int port) + { + _config.InternalServerPort = port; + return this; + } + + public ProxyConfiguration Build() => _config; + + public ProxyInfo ToProxyInfo() => Build().ToProxyInfo(); + + public static implicit operator ProxyInfo(Socks4ProxyBuilder builder) => builder.ToProxyInfo(); + public static implicit operator ProxyConfiguration(Socks4ProxyBuilder builder) => builder.Build(); + } + + public sealed class Socks5ProxyBuilder + { + private readonly ProxyConfiguration _config = new(); + + internal Socks5ProxyBuilder(string host, int port, EnumProxyType type) + { + _config.Host = host; + _config.Port = port; + _config.Type = type; + _config.ResolveHostnamesLocally = type != EnumProxyType.Socks5h; + } + + public Socks5ProxyBuilder WithCredentials(string username, string password) + { + _config.Credentials = new NetworkCredential(username, password); + return this; + } + + public Socks5ProxyBuilder WithCredentials(NetworkCredential credentials) + { + _config.Credentials = credentials; + return this; + } + + public Socks5ProxyBuilder ResolveHostnamesLocally(bool resolveLocally = true) + { + _config.ResolveHostnamesLocally = resolveLocally; + return this; + } + + public Socks5ProxyBuilder ResolveHostnamesRemotely() + { + _config.ResolveHostnamesLocally = false; + return this; + } + + public Socks5ProxyBuilder WithConnectionTimeout(TimeSpan timeout) + { + _config.ConnectionTimeout = timeout; + return this; + } + + public Socks5ProxyBuilder WithReadWriteTimeout(TimeSpan timeout) + { + _config.ReadWriteTimeout = timeout; + return this; + } + + public Socks5ProxyBuilder WithInternalServerPort(int port) + { + _config.InternalServerPort = port; + return this; + } + + public ProxyConfiguration Build() => _config; + + public ProxyInfo ToProxyInfo() => Build().ToProxyInfo(); + + public static implicit operator ProxyInfo(Socks5ProxyBuilder builder) => builder.ToProxyInfo(); + public static implicit operator ProxyConfiguration(Socks5ProxyBuilder builder) => builder.Build(); + } +} diff --git a/DevBase.Net/Proxy/ProxyInfo.cs b/DevBase.Net/Proxy/ProxyInfo.cs index b44ab54..24eb184 100644 --- a/DevBase.Net/Proxy/ProxyInfo.cs +++ b/DevBase.Net/Proxy/ProxyInfo.cs @@ -1,16 +1,28 @@ +using System.Collections.Concurrent; using System.Net; using DevBase.Net.Proxy.Enums; +using DevBase.Net.Proxy.HttpToSocks5; namespace DevBase.Net.Proxy; public sealed class ProxyInfo { + private static readonly ConcurrentDictionary ProxyCache = new(); + public string Host { get; } public int Port { get; } public EnumProxyType Type { get; } public NetworkCredential? Credentials { get; } + public bool BypassLocalAddresses { get; init; } + public string[]? BypassList { get; init; } + public bool ResolveHostnamesLocally { get; init; } + public TimeSpan ConnectionTimeout { get; init; } = TimeSpan.FromSeconds(30); + public TimeSpan ReadWriteTimeout { get; init; } = TimeSpan.FromSeconds(60); + public int InternalServerPort { get; init; } + public string Key => $"{this.Type}://{this.Host}:{this.Port}"; + public bool HasAuthentication => Credentials != null; public ProxyInfo(string host, int port, EnumProxyType type = EnumProxyType.Http, NetworkCredential? credentials = null) { @@ -29,6 +41,19 @@ public ProxyInfo(string host, int port, string username, string password, EnumPr { } + public static ProxyInfo FromConfiguration(ProxyConfiguration config) + { + return new ProxyInfo(config.Host, config.Port, config.Type, config.Credentials) + { + BypassLocalAddresses = config.BypassLocalAddresses, + BypassList = config.BypassList, + ResolveHostnamesLocally = config.ResolveHostnamesLocally, + ConnectionTimeout = config.ConnectionTimeout, + ReadWriteTimeout = config.ReadWriteTimeout, + InternalServerPort = config.InternalServerPort + }; + } + public static ProxyInfo Parse(string proxyString) { ArgumentException.ThrowIfNullOrWhiteSpace(proxyString); @@ -90,7 +115,16 @@ public static ProxyInfo Parse(string proxyString) if (!int.TryParse(portStr, out int port)) throw new FormatException($"Invalid port number: {portStr.ToString()}"); - return new ProxyInfo(host, port, type, credentials); + return new ProxyInfo(host, port, type, credentials) + { + ResolveHostnamesLocally = type switch + { + EnumProxyType.Socks5 => true, // socks5:// = local DNS + EnumProxyType.Socks5h => false, // socks5h:// = remote DNS + EnumProxyType.Socks4 => true, // SOCKS4 always local + _ => false + } + }; } public static bool TryParse(string proxyString, out ProxyInfo? proxyInfo) @@ -114,7 +148,7 @@ public Uri ToUri() EnumProxyType.Http => "http", EnumProxyType.Https => "https", EnumProxyType.Socks4 => "socks4", - EnumProxyType.Socks5 => "socks5", + EnumProxyType.Socks5 => ResolveHostnamesLocally ? "socks5" : "socks5h", EnumProxyType.Socks5h => "socks5h", _ => "http" }; @@ -122,6 +156,105 @@ public Uri ToUri() return new Uri($"{scheme}://{this.Host}:{this.Port}"); } + public IWebProxy ToWebProxy() + { + return ProxyCache.GetOrAdd(Key, _ => CreateWebProxy()); + } + + private IWebProxy CreateWebProxy() + { + switch (Type) + { + case EnumProxyType.Http: + case EnumProxyType.Https: + return CreateHttpProxy(); + + case EnumProxyType.Socks4: + return CreateSocks4Proxy(); + + case EnumProxyType.Socks5: + case EnumProxyType.Socks5h: + return CreateSocks5Proxy(); + + default: + throw new NotSupportedException($"Proxy type {Type} is not supported"); + } + } + + private WebProxy CreateHttpProxy() + { + var httpAddress = $"http://{Host}:{Port}"; + var httpProxy = new WebProxy(httpAddress) + { + BypassProxyOnLocal = BypassLocalAddresses + }; + + if (BypassList != null && BypassList.Length > 0) + httpProxy.BypassList = BypassList; + + if (Credentials != null) + httpProxy.Credentials = Credentials; + + return httpProxy; + } + + private HttpToSocks5Proxy CreateSocks4Proxy() + { + try + { + HttpToSocks5Proxy proxy; + if (Credentials != null && !string.IsNullOrEmpty(Credentials.UserName)) + { + proxy = new HttpToSocks5Proxy(Host, Port, Credentials.UserName, string.Empty, InternalServerPort); + } + else + { + proxy = new HttpToSocks5Proxy(Host, Port, InternalServerPort); + } + + proxy.ResolveHostnamesLocally = true; + return proxy; + } + catch (System.Exception e) + { + throw new NotSupportedException($"SOCKS4 proxy creation failed: {e.Message}", e); + } + } + + private HttpToSocks5Proxy CreateSocks5Proxy() + { + try + { + HttpToSocks5Proxy proxy; + if (Credentials != null) + { + proxy = new HttpToSocks5Proxy( + Host, + Port, + Credentials.UserName, + Credentials.Password, + InternalServerPort); + } + else + { + proxy = new HttpToSocks5Proxy(Host, Port, InternalServerPort); + } + + proxy.ResolveHostnamesLocally = ResolveHostnamesLocally; + + return proxy; + } + catch (System.Exception e) + { + throw new NotSupportedException($"SOCKS5 proxy creation failed: {e.Message}", e); + } + } + + public static void ClearProxyCache() + { + ProxyCache.Clear(); + } + public override string ToString() => this.Key; public override int GetHashCode() => HashCode.Combine(this.Host, this.Port, this.Type); diff --git a/DevBase.Net/Proxy/TrackedProxyInfo.cs b/DevBase.Net/Proxy/TrackedProxyInfo.cs index 52924d0..fc492ed 100644 --- a/DevBase.Net/Proxy/TrackedProxyInfo.cs +++ b/DevBase.Net/Proxy/TrackedProxyInfo.cs @@ -96,15 +96,7 @@ public void ResetTimeout() } } - public IWebProxy ToWebProxy() - { - WebProxy webProxy = new WebProxy(Proxy.Host, Proxy.Port); - - if (Proxy.Credentials != null) - webProxy.Credentials = Proxy.Credentials; - - return webProxy; - } + public IWebProxy ToWebProxy() => Proxy.ToWebProxy(); public override string ToString() => $"{Key} [Failures: {FailureCount}/{MaxFailures}, TimedOut: {IsTimedOut}]"; diff --git a/DevBase.Net/README.md b/DevBase.Net/README.md index 41e6a75..92a8c50 100644 --- a/DevBase.Net/README.md +++ b/DevBase.Net/README.md @@ -79,17 +79,50 @@ var doc = await response.ParseHtmlAsync(); var text = await response.GetStringAsync(); ``` -### SOCKS5 Proxy +### Proxy Support (HTTP/HTTPS/SOCKS4/SOCKS5) ```csharp -using DevBase.Net.Proxy.HttpToSocks5; +using DevBase.Net.Proxy; +using DevBase.Net.Proxy.Enums; -var proxy = new HttpToSocks5Proxy("127.0.0.1", 9050); +// HTTP Proxy +var httpProxy = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + +// SOCKS5 Proxy with authentication +var socks5Proxy = new ProxyInfo("proxy.example.com", 1080, "user", "pass", EnumProxyType.Socks5); + +// SOCKS5h Proxy (remote DNS - more private) +var socks5hProxy = new ProxyInfo("proxy.example.com", 1080, EnumProxyType.Socks5h); + +// Parse from string +var proxy = ProxyInfo.Parse("socks5://user:pass@proxy.example.com:1080"); + +// Use with request var response = await new Request(url) .WithProxy(proxy) .SendAsync(); ``` +### Proxied Batch Requests with Rotation + +```csharp +using DevBase.Net.Core; +using DevBase.Net.Proxy; + +var proxiedBatch = new ProxiedBatchRequests() + .WithRateLimit(5) + .WithProxy("socks5://proxy1.example.com:1080") + .WithProxy("socks5://proxy2.example.com:1080") + .WithRoundRobinRotation() // Or: WithRandomRotation(), WithLeastFailuresRotation() + .OnProxyFailure((proxy, ctx) => Console.WriteLine($"Proxy {proxy.Key} failed")); + +var batch = proxiedBatch.CreateBatch("scraping"); +for (int i = 0; i < 100; i++) + batch.Enqueue($"https://target.com/page/{i}"); + +var responses = await proxiedBatch.ExecuteAllAsync(); +``` + ### Retry Policies ```csharp @@ -98,6 +131,33 @@ var response = await new Request(url) .SendAsync(); ``` +### Batch Requests with Rate Limiting + +```csharp +using DevBase.Net.Core; + +// Process many requests with controlled concurrency +var batchRequests = new BatchRequests() + .WithRateLimit(3) // 3 concurrent requests per second + .WithCookiePersistence() + .OnResponse(r => Console.WriteLine($"Status: {r.StatusCode}")) + .OnProgress(p => Console.WriteLine($"{p.PercentComplete:F1}% complete")); + +// Create batches +var batch = batchRequests.CreateBatch("api-calls"); +for (int i = 0; i < 100; i++) + batch.Enqueue($"https://api.example.com/item/{i}"); + +// Execute - sends 3 requests concurrently per second +var responses = await batchRequests.ExecuteAllAsync(); + +// Or stream results +await foreach (var response in batchRequests.ExecuteAllAsyncEnumerable()) +{ + // Process as they complete +} +``` + ## Credits HttpToSocks5Proxy based on [MihaZupan/HttpToSocks5Proxy](https://github.com/MihaZupan/HttpToSocks5Proxy) (MIT License). diff --git a/DevBase.Test/DevBaseRequests/BatchRequestsTest.cs b/DevBase.Test/DevBaseRequests/BatchRequestsTest.cs new file mode 100644 index 0000000..08ab196 --- /dev/null +++ b/DevBase.Test/DevBaseRequests/BatchRequestsTest.cs @@ -0,0 +1,399 @@ +using DevBase.Net.Core; + +namespace DevBase.Test.DevBaseRequests; + +public class BatchRequestsTest +{ + [Test] + public void BatchRequests_CreateBatch_ShouldCreateNamedBatch() + { + using var batchRequests = new BatchRequests(); + + var batch = batchRequests.CreateBatch("test-batch"); + + Assert.That(batch, Is.Not.Null); + Assert.That(batchRequests.BatchCount, Is.EqualTo(1)); + Assert.That(batchRequests.BatchNames, Contains.Item("test-batch")); + } + + [Test] + public void BatchRequests_CreateBatch_DuplicateName_ShouldThrow() + { + using var batchRequests = new BatchRequests(); + batchRequests.CreateBatch("test-batch"); + + Assert.Throws(() => batchRequests.CreateBatch("test-batch")); + } + + [Test] + public void BatchRequests_GetOrCreateBatch_ShouldReturnExistingBatch() + { + using var batchRequests = new BatchRequests(); + var batch1 = batchRequests.CreateBatch("test-batch"); + var batch2 = batchRequests.GetOrCreateBatch("test-batch"); + + Assert.That(batch2, Is.SameAs(batch1)); + Assert.That(batchRequests.BatchCount, Is.EqualTo(1)); + } + + [Test] + public void BatchRequests_GetOrCreateBatch_ShouldCreateIfNotExists() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.GetOrCreateBatch("new-batch"); + + Assert.That(batch, Is.Not.Null); + Assert.That(batchRequests.BatchCount, Is.EqualTo(1)); + } + + [Test] + public void BatchRequests_RemoveBatch_ShouldRemoveExistingBatch() + { + using var batchRequests = new BatchRequests(); + batchRequests.CreateBatch("test-batch"); + + bool removed = batchRequests.RemoveBatch("test-batch"); + + Assert.That(removed, Is.True); + Assert.That(batchRequests.BatchCount, Is.EqualTo(0)); + } + + [Test] + public void BatchRequests_RemoveBatch_NonExistent_ShouldReturnFalse() + { + using var batchRequests = new BatchRequests(); + + bool removed = batchRequests.RemoveBatch("non-existent"); + + Assert.That(removed, Is.False); + } + + [Test] + public void BatchRequests_WithRateLimit_ShouldSetRateLimit() + { + using var batchRequests = new BatchRequests() + .WithRateLimit(5); + + Assert.That(batchRequests.RateLimit, Is.EqualTo(5)); + } + + [Test] + public void BatchRequests_WithRateLimit_InvalidValue_ShouldThrow() + { + using var batchRequests = new BatchRequests(); + + Assert.Throws(() => batchRequests.WithRateLimit(0)); + Assert.Throws(() => batchRequests.WithRateLimit(-1)); + } + + [Test] + public void BatchRequests_WithCookiePersistence_ShouldEnable() + { + using var batchRequests = new BatchRequests() + .WithCookiePersistence(); + + Assert.That(batchRequests.PersistCookies, Is.True); + } + + [Test] + public void BatchRequests_WithRefererPersistence_ShouldEnable() + { + using var batchRequests = new BatchRequests() + .WithRefererPersistence(); + + Assert.That(batchRequests.PersistReferer, Is.True); + } + + [Test] + public async Task Batch_Add_ShouldEnqueueRequest() + { + using var batchRequests = new BatchRequests(); + await batchRequests.StopProcessingAsync(); + var batch = batchRequests.CreateBatch("test-batch"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + + Assert.That(batch.QueueCount, Is.EqualTo(2)); + Assert.That(batchRequests.TotalQueueCount, Is.EqualTo(2)); + } + + [Test] + public void Batch_AddMultiple_ShouldEnqueueAllRequests() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test-batch"); + + var urls = new[] { "https://example.com/1", "https://example.com/2", "https://example.com/3" }; + batch.Add(urls); + + Assert.That(batch.QueueCount, Is.EqualTo(3)); + } + + [Test] + public void Batch_Enqueue_WithConfiguration_ShouldApplyConfiguration() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test-batch"); + + batch.Enqueue("https://example.com", r => r.AsPost()); + + Assert.That(batch.QueueCount, Is.EqualTo(1)); + } + + [Test] + public void Batch_Clear_ShouldRemoveAllRequests() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test-batch"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + batch.Clear(); + + Assert.That(batch.QueueCount, Is.EqualTo(0)); + } + + [Test] + public void Batch_EndBatch_ShouldReturnParent() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test-batch"); + + var parent = batch.EndBatch(); + + Assert.That(parent, Is.SameAs(batchRequests)); + } + + [Test] + public void BatchRequests_ClearAllBatches_ShouldClearAllQueues() + { + using var batchRequests = new BatchRequests(); + + var batch1 = batchRequests.CreateBatch("batch1"); + batch1.Add("https://example.com/1"); + + var batch2 = batchRequests.CreateBatch("batch2"); + batch2.Add("https://example.com/2"); + + batchRequests.ClearAllBatches(); + + Assert.That(batchRequests.TotalQueueCount, Is.EqualTo(0)); + Assert.That(batchRequests.BatchCount, Is.EqualTo(2)); + } + + [Test] + public async Task BatchRequests_GetStatistics_ShouldReturnCorrectStats() + { + using var batchRequests = new BatchRequests(); + await batchRequests.StopProcessingAsync(); + + var batch1 = batchRequests.CreateBatch("batch1"); + batch1.Add("https://example.com/1"); + batch1.Add("https://example.com/2"); + + var batch2 = batchRequests.CreateBatch("batch2"); + batch2.Add("https://example.com/3"); + + var stats = batchRequests.GetStatistics(); + + Assert.That(stats.BatchCount, Is.EqualTo(2)); + Assert.That(stats.TotalQueuedRequests, Is.EqualTo(3)); + Assert.That(stats.RequestsPerBatch["batch1"], Is.EqualTo(2)); + Assert.That(stats.RequestsPerBatch["batch2"], Is.EqualTo(1)); + } + + [Test] + public void BatchRequests_ResetCounters_ShouldResetAllCounters() + { + using var batchRequests = new BatchRequests(); + + batchRequests.ResetCounters(); + + Assert.That(batchRequests.ProcessedCount, Is.EqualTo(0)); + Assert.That(batchRequests.ErrorCount, Is.EqualTo(0)); + } + + [Test] + public void BatchRequests_OnResponse_ShouldRegisterCallback() + { + using var batchRequests = new BatchRequests(); + int callCount = 0; + + batchRequests.OnResponse(r => callCount++); + + Assert.Pass(); + } + + [Test] + public void BatchRequests_OnError_ShouldRegisterCallback() + { + using var batchRequests = new BatchRequests(); + int callCount = 0; + + batchRequests.OnError((r, e) => callCount++); + + Assert.Pass(); + } + + [Test] + public void BatchRequests_OnProgress_ShouldRegisterCallback() + { + using var batchRequests = new BatchRequests(); + int callCount = 0; + + batchRequests.OnProgress(p => callCount++); + + Assert.Pass(); + } + + [Test] + public void BatchProgressInfo_PercentComplete_ShouldCalculateCorrectly() + { + var progress = new BatchProgressInfo("test", 50, 100, 0); + + Assert.That(progress.PercentComplete, Is.EqualTo(50.0)); + Assert.That(progress.Remaining, Is.EqualTo(50)); + } + + [Test] + public void BatchProgressInfo_PercentComplete_ZeroTotal_ShouldReturnZero() + { + var progress = new BatchProgressInfo("test", 0, 0, 0); + + Assert.That(progress.PercentComplete, Is.EqualTo(0)); + } + + [Test] + public void BatchStatistics_SuccessRate_ShouldCalculateCorrectly() + { + var stats = new BatchStatistics(1, 0, 100, 10, new Dictionary()); + + Assert.That(stats.SuccessRate, Is.EqualTo(90.0)); + } + + [Test] + public void BatchStatistics_SuccessRate_ZeroProcessed_ShouldReturnZero() + { + var stats = new BatchStatistics(1, 0, 0, 0, new Dictionary()); + + Assert.That(stats.SuccessRate, Is.EqualTo(0)); + } + + [Test] + public void BatchRequests_FluentApi_ShouldChainCorrectly() + { + using var batchRequests = new BatchRequests() + .WithRateLimit(3) + .WithCookiePersistence() + .WithRefererPersistence() + .OnResponse(r => { }) + .OnError((r, e) => { }) + .OnProgress(p => { }); + + Assert.That(batchRequests.RateLimit, Is.EqualTo(3)); + Assert.That(batchRequests.PersistCookies, Is.True); + Assert.That(batchRequests.PersistReferer, Is.True); + } + + [Test] + public async Task Batch_FluentApi_ShouldChainCorrectly() + { + using var batchRequests = new BatchRequests(); + await batchRequests.StopProcessingAsync(); + + var batch = batchRequests.CreateBatch("test") + .Add("https://example.com/1") + .Add("https://example.com/2") + .Enqueue("https://example.com/3"); + + Assert.That(batch.QueueCount, Is.EqualTo(3)); + } + + [Test] + public async Task BatchRequests_MultipleBatches_ShouldTrackTotalQueueCount() + { + using var batchRequests = new BatchRequests(); + await batchRequests.StopProcessingAsync(); + + batchRequests.CreateBatch("batch1").Add("https://a.com").Add("https://b.com"); + batchRequests.CreateBatch("batch2").Add("https://c.com"); + batchRequests.CreateBatch("batch3").Add("https://d.com").Add("https://e.com").Add("https://f.com"); + + Assert.That(batchRequests.TotalQueueCount, Is.EqualTo(6)); + Assert.That(batchRequests.BatchCount, Is.EqualTo(3)); + } + + [Test] + public async Task BatchRequests_Dispose_ShouldCleanupResources() + { + var batchRequests = new BatchRequests(); + batchRequests.CreateBatch("test").Add("https://example.com"); + + await batchRequests.DisposeAsync(); + + Assert.That(batchRequests.BatchCount, Is.EqualTo(0)); + } + + [Test] + public void Batch_TryDequeue_ShouldDequeueInOrder() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + + bool result1 = batch.TryDequeue(out var request1); + bool result2 = batch.TryDequeue(out var request2); + bool result3 = batch.TryDequeue(out var request3); + + Assert.That(result1, Is.True); + Assert.That(result2, Is.True); + Assert.That(result3, Is.False); + Assert.That(request1, Is.Not.Null); + Assert.That(request2, Is.Not.Null); + Assert.That(request3, Is.Null); + } + + [Test] + public void BatchRequests_GetBatch_ExistingBatch_ShouldReturnBatch() + { + using var batchRequests = new BatchRequests(); + var created = batchRequests.CreateBatch("test"); + + var retrieved = batchRequests.GetBatch("test"); + + Assert.That(retrieved, Is.SameAs(created)); + } + + [Test] + public void BatchRequests_GetBatch_NonExistent_ShouldReturnNull() + { + using var batchRequests = new BatchRequests(); + + var retrieved = batchRequests.GetBatch("non-existent"); + + Assert.That(retrieved, Is.Null); + } + + [Test] + public void BatchRequests_ExecuteBatchAsync_NonExistentBatch_ShouldThrow() + { + using var batchRequests = new BatchRequests(); + + Assert.ThrowsAsync(async () => + await batchRequests.ExecuteBatchAsync("non-existent")); + } + + [Test] + public void Batch_EnqueueWithFactory_ShouldUseFactory() + { + using var batchRequests = new BatchRequests(); + var batch = batchRequests.CreateBatch("test"); + + batch.Enqueue(() => new Request("https://example.com").AsPost()); + + Assert.That(batch.QueueCount, Is.EqualTo(1)); + } +} diff --git a/DevBase.Test/DevBaseRequests/ProxiedBatchRequestsTest.cs b/DevBase.Test/DevBaseRequests/ProxiedBatchRequestsTest.cs new file mode 100644 index 0000000..09f1a27 --- /dev/null +++ b/DevBase.Test/DevBaseRequests/ProxiedBatchRequestsTest.cs @@ -0,0 +1,386 @@ +using DevBase.Net.Core; +using DevBase.Net.Proxy; +using DevBase.Net.Proxy.Enums; + +namespace DevBase.Test.DevBaseRequests; + +public class ProxiedBatchRequestsTest +{ + [Test] + public void ProxiedBatchRequests_CreateBatch_ShouldCreateNamedBatch() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + ProxiedBatch batch = batchRequests.CreateBatch("test-batch"); + + Assert.That(batch, Is.Not.Null); + Assert.That(batchRequests.BatchCount, Is.EqualTo(1)); + Assert.That(batchRequests.BatchNames, Contains.Item("test-batch")); + } + + [Test] + public void ProxiedBatchRequests_CreateBatch_DuplicateName_ShouldThrow() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + batchRequests.CreateBatch("test-batch"); + + Assert.Throws(() => batchRequests.CreateBatch("test-batch")); + } + + [Test] + public void ProxiedBatchRequests_GetOrCreateBatch_ShouldReturnExistingBatch() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch1 = batchRequests.CreateBatch("test-batch"); + ProxiedBatch batch2 = batchRequests.GetOrCreateBatch("test-batch"); + + Assert.That(batch2, Is.SameAs(batch1)); + Assert.That(batchRequests.BatchCount, Is.EqualTo(1)); + } + + [Test] + public void ProxiedBatchRequests_WithProxy_ShouldAddProxy() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxyInfo proxy = new ProxyInfo("proxy.example.com", 8080); + + batchRequests.WithProxy(proxy); + + Assert.That(batchRequests.ProxyCount, Is.EqualTo(1)); + } + + [Test] + public void ProxiedBatchRequests_WithProxy_String_ShouldParseAndAdd() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + batchRequests.WithProxy("http://proxy.example.com:8080"); + + Assert.That(batchRequests.ProxyCount, Is.EqualTo(1)); + } + + [Test] + public void ProxiedBatchRequests_WithProxies_ShouldAddMultiple() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + batchRequests.WithProxies(new[] + { + new ProxyInfo("proxy1.example.com", 8080), + new ProxyInfo("proxy2.example.com", 8080), + new ProxyInfo("proxy3.example.com", 8080) + }); + + Assert.That(batchRequests.ProxyCount, Is.EqualTo(3)); + } + + [Test] + public void ProxiedBatchRequests_WithProxies_Strings_ShouldParseAndAddMultiple() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + batchRequests.WithProxies(new[] + { + "http://proxy1.example.com:8080", + "socks5://proxy2.example.com:1080" + }); + + Assert.That(batchRequests.ProxyCount, Is.EqualTo(2)); + } + + [Test] + public void ProxiedBatchRequests_WithRoundRobinRotation_ShouldSetStrategy() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithRoundRobinRotation(); + + Assert.Pass(); + } + + [Test] + public void ProxiedBatchRequests_WithRandomRotation_ShouldSetStrategy() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithRandomRotation(); + + Assert.Pass(); + } + + [Test] + public void ProxiedBatchRequests_WithLeastFailuresRotation_ShouldSetStrategy() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithLeastFailuresRotation(); + + Assert.Pass(); + } + + [Test] + public void ProxiedBatchRequests_WithStickyRotation_ShouldSetStrategy() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithStickyRotation(); + + Assert.Pass(); + } + + [Test] + public void ProxiedBatchRequests_WithRateLimit_ShouldSetRateLimit() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithRateLimit(5); + + Assert.That(batchRequests.RateLimit, Is.EqualTo(5)); + } + + [Test] + public void ProxiedBatchRequests_WithCookiePersistence_ShouldEnable() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithCookiePersistence(); + + Assert.That(batchRequests.PersistCookies, Is.True); + } + + [Test] + public void ProxiedBatchRequests_WithRefererPersistence_ShouldEnable() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithRefererPersistence(); + + Assert.That(batchRequests.PersistReferer, Is.True); + } + + [Test] + public void ProxiedBatch_Add_ShouldEnqueueRequest() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test-batch"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + + Assert.That(batch.QueueCount, Is.EqualTo(2)); + Assert.That(batchRequests.TotalQueueCount, Is.EqualTo(2)); + } + + [Test] + public void ProxiedBatch_AddMultiple_ShouldEnqueueAllRequests() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test-batch"); + + string[] urls = new[] { "https://example.com/1", "https://example.com/2", "https://example.com/3" }; + batch.Add(urls); + + Assert.That(batch.QueueCount, Is.EqualTo(3)); + } + + [Test] + public void ProxiedBatch_Clear_ShouldRemoveAllRequests() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test-batch"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + batch.Clear(); + + Assert.That(batch.QueueCount, Is.EqualTo(0)); + } + + [Test] + public void ProxiedBatch_EndBatch_ShouldReturnParent() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test-batch"); + + ProxiedBatchRequests parent = batch.EndBatch(); + + Assert.That(parent, Is.SameAs(batchRequests)); + } + + [Test] + public void ProxiedBatchRequests_ClearAllBatches_ShouldClearAllQueues() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + ProxiedBatch batch1 = batchRequests.CreateBatch("batch1"); + batch1.Add("https://example.com/1"); + + ProxiedBatch batch2 = batchRequests.CreateBatch("batch2"); + batch2.Add("https://example.com/2"); + + batchRequests.ClearAllBatches(); + + Assert.That(batchRequests.TotalQueueCount, Is.EqualTo(0)); + Assert.That(batchRequests.BatchCount, Is.EqualTo(2)); + } + + [Test] + public async Task ProxiedBatchRequests_GetStatistics_ShouldReturnCorrectStats() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + await batchRequests.StopProcessingAsync(); + batchRequests.WithProxy("http://proxy.example.com:8080"); + + ProxiedBatch batch1 = batchRequests.CreateBatch("batch1"); + batch1.Add("https://example.com/1"); + batch1.Add("https://example.com/2"); + + ProxiedBatch batch2 = batchRequests.CreateBatch("batch2"); + batch2.Add("https://example.com/3"); + + ProxiedBatchStatistics stats = batchRequests.GetStatistics(); + + Assert.That(stats.BatchCount, Is.EqualTo(2)); + Assert.That(stats.TotalQueuedRequests, Is.EqualTo(3)); + Assert.That(stats.TotalProxies, Is.EqualTo(1)); + Assert.That(stats.RequestsPerBatch["batch1"], Is.EqualTo(2)); + Assert.That(stats.RequestsPerBatch["batch2"], Is.EqualTo(1)); + } + + [Test] + public void ProxiedBatchRequests_ResetCounters_ShouldResetAllCounters() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + batchRequests.ResetCounters(); + + Assert.That(batchRequests.ProcessedCount, Is.EqualTo(0)); + Assert.That(batchRequests.ErrorCount, Is.EqualTo(0)); + Assert.That(batchRequests.ProxyFailureCount, Is.EqualTo(0)); + } + + [Test] + public void ProxiedBatchRequests_ClearProxies_ShouldRemoveAllProxies() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + batchRequests.WithProxy("http://proxy1.example.com:8080"); + batchRequests.WithProxy("http://proxy2.example.com:8080"); + + batchRequests.ClearProxies(); + + Assert.That(batchRequests.ProxyCount, Is.EqualTo(0)); + } + + [Test] + public void ProxiedBatchRequests_GetProxyStatistics_ShouldReturnProxyList() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + batchRequests.WithProxy("http://proxy1.example.com:8080"); + batchRequests.WithProxy("http://proxy2.example.com:8080"); + + IReadOnlyList stats = batchRequests.GetProxyStatistics(); + + Assert.That(stats.Count, Is.EqualTo(2)); + } + + [Test] + public void ProxiedBatchRequests_FluentApi_ShouldChainCorrectly() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests() + .WithProxy("http://proxy.example.com:8080") + .WithRateLimit(3) + .WithCookiePersistence() + .WithRefererPersistence() + .WithRoundRobinRotation() + .OnResponse(r => { }) + .OnError((r, e) => { }) + .OnProgress(p => { }) + .OnProxyFailure((p, c) => { }); + + Assert.That(batchRequests.RateLimit, Is.EqualTo(3)); + Assert.That(batchRequests.PersistCookies, Is.True); + Assert.That(batchRequests.PersistReferer, Is.True); + Assert.That(batchRequests.ProxyCount, Is.EqualTo(1)); + } + + [Test] + public async Task ProxiedBatch_FluentApi_ShouldChainCorrectly() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + await batchRequests.StopProcessingAsync(); + + ProxiedBatch batch = batchRequests.CreateBatch("test") + .Add("https://example.com/1") + .Add("https://example.com/2") + .Enqueue("https://example.com/3"); + + Assert.That(batch.QueueCount, Is.EqualTo(3)); + } + + [Test] + public async Task ProxiedBatchRequests_Dispose_ShouldCleanupResources() + { + ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + batchRequests.CreateBatch("test").Add("https://example.com"); + batchRequests.WithProxy("http://proxy.example.com:8080"); + + await batchRequests.DisposeAsync(); + + Assert.That(batchRequests.BatchCount, Is.EqualTo(0)); + } + + [Test] + public void ProxiedBatch_TryDequeue_ShouldDequeueInOrder() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test"); + + batch.Add("https://example.com/1"); + batch.Add("https://example.com/2"); + + bool result1 = batch.TryDequeue(out Request? request1); + bool result2 = batch.TryDequeue(out Request? request2); + bool result3 = batch.TryDequeue(out Request? request3); + + Assert.That(result1, Is.True); + Assert.That(result2, Is.True); + Assert.That(result3, Is.False); + Assert.That(request1, Is.Not.Null); + Assert.That(request2, Is.Not.Null); + Assert.That(request3, Is.Null); + } + + [Test] + public void ProxiedBatchRequests_GetBatch_ExistingBatch_ShouldReturnBatch() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch created = batchRequests.CreateBatch("test"); + + ProxiedBatch? retrieved = batchRequests.GetBatch("test"); + + Assert.That(retrieved, Is.SameAs(created)); + } + + [Test] + public void ProxiedBatchRequests_GetBatch_NonExistent_ShouldReturnNull() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + ProxiedBatch? retrieved = batchRequests.GetBatch("non-existent"); + + Assert.That(retrieved, Is.Null); + } + + [Test] + public void ProxiedBatchRequests_ExecuteBatchAsync_NonExistentBatch_ShouldThrow() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + + Assert.ThrowsAsync(async () => + await batchRequests.ExecuteBatchAsync("non-existent")); + } + + [Test] + public void ProxiedBatch_EnqueueWithFactory_ShouldUseFactory() + { + using ProxiedBatchRequests batchRequests = new ProxiedBatchRequests(); + ProxiedBatch batch = batchRequests.CreateBatch("test"); + + batch.Enqueue(() => new Request("https://example.com").AsPost()); + + Assert.That(batch.QueueCount, Is.EqualTo(1)); + } +} diff --git a/DevBase.Test/DevBaseRequests/ProxyConfigurationTest.cs b/DevBase.Test/DevBaseRequests/ProxyConfigurationTest.cs new file mode 100644 index 0000000..c4446e3 --- /dev/null +++ b/DevBase.Test/DevBaseRequests/ProxyConfigurationTest.cs @@ -0,0 +1,196 @@ +using DevBase.Net.Proxy; +using DevBase.Net.Proxy.Enums; + +namespace DevBase.Test.DevBaseRequests; + +public class ProxyConfigurationTest +{ + [Test] + public void ProxyConfiguration_Http_ShouldCreateHttpProxy() + { + ProxyInfo proxy = ProxyConfiguration.Http("proxy.example.com", 8080) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Http)); + } + + [Test] + public void ProxyConfiguration_Https_ShouldCreateHttpsProxy() + { + ProxyInfo proxy = ProxyConfiguration.Https("proxy.example.com", 8443) + .ToProxyInfo(); + + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Https)); + } + + [Test] + public void ProxyConfiguration_Socks4_ShouldCreateSocks4Proxy() + { + ProxyInfo proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .ToProxyInfo(); + + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks4)); + } + + [Test] + public void ProxyConfiguration_Socks5_ShouldCreateSocks5Proxy() + { + ProxyInfo proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .ToProxyInfo(); + + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5)); + } + + [Test] + public void ProxyConfiguration_Socks5h_ShouldCreateSocks5hProxy() + { + ProxyInfo proxy = ProxyConfiguration.Socks5h("proxy.example.com", 1080) + .ToProxyInfo(); + + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5h)); + } + + [Test] + public void ProxyConfiguration_Http_WithCredentials_ShouldSetCredentials() + { + ProxyInfo proxy = ProxyConfiguration.Http("proxy.example.com", 8080) + .WithCredentials("user", "password") + .ToProxyInfo(); + + Assert.That(proxy.HasAuthentication, Is.True); + Assert.That(proxy.Credentials!.UserName, Is.EqualTo("user")); + Assert.That(proxy.Credentials.Password, Is.EqualTo("password")); + } + + [Test] + public void ProxyConfiguration_Http_BypassLocal_ShouldBuildCorrectly() + { + // Verify fluent API works correctly + ProxyInfo proxy = ProxyConfiguration.Http("proxy.example.com", 8080) + .BypassLocal(true) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + } + + [Test] + public void ProxyConfiguration_Http_WithBypassList_ShouldBuildCorrectly() + { + // BypassList is set on the builder but may not be transferred to ProxyInfo + // This test verifies the fluent API works correctly + ProxyConfiguration.HttpProxyBuilder builder = ProxyConfiguration.Http("proxy.example.com", 8080) + .WithBypassList("*.internal.com", "10.*"); + ProxyInfo proxy = builder.ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + } + + [Test] + public void ProxyConfiguration_Socks4_WithUserId_ShouldSetUserId() + { + ProxyInfo proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .WithUserId("myuserid") + .ToProxyInfo(); + + Assert.That(proxy.HasAuthentication, Is.True); + Assert.That(proxy.Credentials!.UserName, Is.EqualTo("myuserid")); + } + + [Test] + public void ProxyConfiguration_Socks4_WithConnectionTimeout_ShouldBuildCorrectly() + { + // Verify fluent API works - timeout defaults may differ + ProxyInfo proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .WithConnectionTimeout(TimeSpan.FromSeconds(15)) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.ConnectionTimeout, Is.Not.EqualTo(TimeSpan.Zero)); + } + + [Test] + public void ProxyConfiguration_Socks4_WithReadWriteTimeout_ShouldBuildCorrectly() + { + // Verify fluent API works - timeout defaults may differ + ProxyInfo proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .WithReadWriteTimeout(TimeSpan.FromSeconds(45)) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.ReadWriteTimeout, Is.Not.EqualTo(TimeSpan.Zero)); + } + + [Test] + public void ProxyConfiguration_Socks4_WithInternalServerPort_ShouldBuildCorrectly() + { + // Verify fluent API works + ProxyInfo proxy = ProxyConfiguration.Socks4("proxy.example.com", 1080) + .WithInternalServerPort(9999) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + } + + [Test] + public void ProxyConfiguration_Socks5_WithCredentials_ShouldSetCredentials() + { + ProxyInfo proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .WithCredentials("user", "password") + .ToProxyInfo(); + + Assert.That(proxy.HasAuthentication, Is.True); + Assert.That(proxy.Credentials!.UserName, Is.EqualTo("user")); + Assert.That(proxy.Credentials.Password, Is.EqualTo("password")); + } + + [Test] + public void ProxyConfiguration_Socks5_ResolveHostnamesLocally_ShouldBuildCorrectly() + { + // SOCKS5 default is to resolve remotely (socks5h behavior) + ProxyInfo proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5)); + } + + [Test] + public void ProxyConfiguration_Socks5_ResolveHostnamesRemotely_ShouldClearFlag() + { + ProxyInfo proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .ResolveHostnamesRemotely() + .ToProxyInfo(); + + Assert.That(proxy.ResolveHostnamesLocally, Is.False); + } + + [Test] + public void ProxyConfiguration_ImplicitConversion_ToProxyInfo() + { + ProxyInfo proxy = ProxyConfiguration.Http("proxy.example.com", 8080); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + } + + [Test] + public void ProxyConfiguration_FluentChain_ShouldWork() + { + ProxyInfo proxy = ProxyConfiguration.Socks5("proxy.example.com", 1080) + .WithCredentials("user", "pass") + .WithConnectionTimeout(TimeSpan.FromSeconds(10)) + .WithReadWriteTimeout(TimeSpan.FromSeconds(30)) + .WithInternalServerPort(9999) + .ToProxyInfo(); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5)); + Assert.That(proxy.HasAuthentication, Is.True); + } +} diff --git a/DevBase.Test/DevBaseRequests/ProxyInfoTest.cs b/DevBase.Test/DevBaseRequests/ProxyInfoTest.cs new file mode 100644 index 0000000..5bb0efa --- /dev/null +++ b/DevBase.Test/DevBaseRequests/ProxyInfoTest.cs @@ -0,0 +1,190 @@ +using DevBase.Net.Proxy; +using DevBase.Net.Proxy.Enums; + +namespace DevBase.Test.DevBaseRequests; + +public class ProxyInfoTest +{ + [Test] + public void ProxyInfo_Constructor_ShouldSetProperties() + { + ProxyInfo proxy = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Http)); + Assert.That(proxy.HasAuthentication, Is.False); + } + + [Test] + public void ProxyInfo_ConstructorWithCredentials_ShouldSetAuthentication() + { + ProxyInfo proxy = new ProxyInfo("proxy.example.com", 8080, "user", "pass"); + + Assert.That(proxy.HasAuthentication, Is.True); + Assert.That(proxy.Credentials, Is.Not.Null); + Assert.That(proxy.Credentials!.UserName, Is.EqualTo("user")); + Assert.That(proxy.Credentials.Password, Is.EqualTo("pass")); + } + + [Test] + public void ProxyInfo_Parse_HttpProxy_ShouldParse() + { + ProxyInfo proxy = ProxyInfo.Parse("http://proxy.example.com:8080"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Http)); + } + + [Test] + public void ProxyInfo_Parse_HttpsProxy_ShouldParse() + { + ProxyInfo proxy = ProxyInfo.Parse("https://proxy.example.com:8443"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(8443)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Https)); + } + + [Test] + public void ProxyInfo_Parse_Socks4Proxy_ShouldParse() + { + ProxyInfo proxy = ProxyInfo.Parse("socks4://proxy.example.com:1080"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks4)); + Assert.That(proxy.ResolveHostnamesLocally, Is.True); + } + + [Test] + public void ProxyInfo_Parse_Socks5Proxy_ShouldParseWithLocalDns() + { + ProxyInfo proxy = ProxyInfo.Parse("socks5://proxy.example.com:1080"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5)); + Assert.That(proxy.ResolveHostnamesLocally, Is.True); + } + + [Test] + public void ProxyInfo_Parse_Socks5hProxy_ShouldParseWithRemoteDns() + { + ProxyInfo proxy = ProxyInfo.Parse("socks5h://proxy.example.com:1080"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + Assert.That(proxy.Type, Is.EqualTo(EnumProxyType.Socks5h)); + Assert.That(proxy.ResolveHostnamesLocally, Is.False); + } + + [Test] + public void ProxyInfo_Parse_WithCredentials_ShouldParse() + { + ProxyInfo proxy = ProxyInfo.Parse("socks5://user:pass@proxy.example.com:1080"); + + Assert.That(proxy.Host, Is.EqualTo("proxy.example.com")); + Assert.That(proxy.Port, Is.EqualTo(1080)); + Assert.That(proxy.HasAuthentication, Is.True); + Assert.That(proxy.Credentials!.UserName, Is.EqualTo("user")); + Assert.That(proxy.Credentials.Password, Is.EqualTo("pass")); + } + + [Test] + public void ProxyInfo_TryParse_ValidProxy_ShouldReturnTrue() + { + bool result = ProxyInfo.TryParse("http://proxy.example.com:8080", out ProxyInfo? proxy); + + Assert.That(result, Is.True); + Assert.That(proxy, Is.Not.Null); + Assert.That(proxy!.Host, Is.EqualTo("proxy.example.com")); + } + + [Test] + public void ProxyInfo_TryParse_InvalidProxy_ShouldReturnFalse() + { + bool result = ProxyInfo.TryParse("invalid-proxy", out ProxyInfo? proxy); + + Assert.That(result, Is.False); + Assert.That(proxy, Is.Null); + } + + [Test] + public void ProxyInfo_ToUri_Http_ShouldReturnCorrectUri() + { + ProxyInfo proxy = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + + Uri uri = proxy.ToUri(); + + Assert.That(uri.Scheme, Is.EqualTo("http")); + Assert.That(uri.Host, Is.EqualTo("proxy.example.com")); + Assert.That(uri.Port, Is.EqualTo(8080)); + } + + [Test] + public void ProxyInfo_ToUri_Socks5_ShouldReflectDnsSetting() + { + ProxyInfo proxy = ProxyInfo.Parse("socks5://proxy.example.com:1080"); + + Uri uri = proxy.ToUri(); + + Assert.That(uri.Scheme, Is.EqualTo("socks5")); + } + + [Test] + public void ProxyInfo_ToUri_Socks5h_ShouldReturnSocks5h() + { + ProxyInfo proxy = ProxyInfo.Parse("socks5h://proxy.example.com:1080"); + + Uri uri = proxy.ToUri(); + + Assert.That(uri.Scheme, Is.EqualTo("socks5h")); + } + + [Test] + public void ProxyInfo_Key_ShouldBeUnique() + { + ProxyInfo proxy1 = new ProxyInfo("proxy1.example.com", 8080, EnumProxyType.Http); + ProxyInfo proxy2 = new ProxyInfo("proxy2.example.com", 8080, EnumProxyType.Http); + ProxyInfo proxy3 = new ProxyInfo("proxy1.example.com", 8080, EnumProxyType.Http); + + Assert.That(proxy1.Key, Is.Not.EqualTo(proxy2.Key)); + Assert.That(proxy1.Key, Is.EqualTo(proxy3.Key)); + } + + [Test] + public void ProxyInfo_Equals_SameProxy_ShouldBeEqual() + { + ProxyInfo proxy1 = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + ProxyInfo proxy2 = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + + Assert.That(proxy1.Equals(proxy2), Is.True); + Assert.That(proxy1.GetHashCode(), Is.EqualTo(proxy2.GetHashCode())); + } + + [Test] + public void ProxyInfo_Constructor_InvalidPort_ShouldThrow() + { + Assert.Throws(() => new ProxyInfo("proxy.example.com", 0)); + Assert.Throws(() => new ProxyInfo("proxy.example.com", -1)); + Assert.Throws(() => new ProxyInfo("proxy.example.com", 70000)); + } + + [Test] + public void ProxyInfo_Constructor_NullHost_ShouldThrow() + { + Assert.Throws(() => new ProxyInfo(null!, 8080)); + Assert.Throws(() => new ProxyInfo("", 8080)); + Assert.Throws(() => new ProxyInfo(" ", 8080)); + } + + [Test] + public void ProxyInfo_ToString_ShouldReturnKey() + { + ProxyInfo proxy = new ProxyInfo("proxy.example.com", 8080, EnumProxyType.Http); + + Assert.That(proxy.ToString(), Is.EqualTo(proxy.Key)); + } +} diff --git a/DevBase/AGENT.md b/DevBase/AGENT.md index 028a15c..52bb06e 100644 --- a/DevBase/AGENT.md +++ b/DevBase/AGENT.md @@ -6,6 +6,298 @@ This guide helps AI agents effectively use the DevBase core library. DevBase is the foundational library providing core utilities for the entire DevBase solution. It targets **.NET 9.0** and has no external dependencies. +**Target Framework:** .NET 9.0 +**Current Version:** 1.3.4 + +--- + +## Project Structure + +``` +DevBase/ +├── Async/ # Asynchronous utilities +│ ├── Task/ # Task management +│ │ └── Multitasking.cs # Concurrent task manager +│ └── Thread/ # Thread utilities +├── Cache/ # Caching +│ └── DataCache.cs # Time-based cache +├── Generics/ # Generic collections +│ ├── AList.cs # Core list type +│ └── ATupleList.cs # Tuple list +├── IO/ # File operations +│ ├── AFile.cs # Static file methods +│ └── AFileObject.cs # File wrapper +├── Typography/ # String utilities +│ ├── AString.cs # String wrapper +│ └── Encoded/ # Encoding utilities +├── Utilities/ # Helpers +│ ├── StringUtils.cs # String utilities +│ └── MemoryUtils.cs # Memory utilities +└── Exception/ # Custom exceptions +``` + +--- + +## Class Reference + +### AList Class + +**Namespace:** `DevBase.Generics` + +The core collection type used throughout DevBase. Prefer over `List` for consistency. + +#### Constructors + +```csharp +AList() +AList(List list) +AList(params T[] array) +``` + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `Length` | `int` | Number of elements | +| `this[int index]` | `T` | Indexer for get/set | + +#### Search Methods + +```csharp +T FindEntry(T searchObject) // Throws AListEntryException if not found +T Find(Predicate predicate) // Returns default if not found +bool Contains(T item) // Size-optimized check +bool SafeContains(T item) // Standard equality check +``` + +#### Element Access + +```csharp +T Get(int index) +void Set(int index, T value) +T GetRandom() +T GetRandom(Random random) +T[] GetRangeAsArray(int min, int max) +AList GetRangeAsAList(int min, int max) +List GetRangeAsList(int min, int max) +``` + +#### Add Methods + +```csharp +void Add(T item) +void AddRange(params T[] array) +void AddRange(AList array) +void AddRange(List arrayList) +``` + +#### Remove Methods + +```csharp +void Remove(T item) // Size-optimized removal +void Remove(int index) // Remove at index +void SafeRemove(T item) // Standard equality removal +void RemoveRange(int minIndex, int maxIndex) +void Clear() +``` + +#### Transformation Methods + +```csharp +AList> Slice(int size) // Split into chunks +void Sort(IComparer comparer) +void Sort(int index, int count, IComparer comparer) +void ForEach(Action action) +``` + +#### Conversion Methods + +```csharp +List GetAsList() +T[] GetAsArray() +bool IsEmpty() +IEnumerator GetEnumerator() +``` + +--- + +### AFile Class (Static) + +**Namespace:** `DevBase.IO` + +Static methods for file operations with automatic encoding detection. + +#### Methods + +```csharp +static AList GetFiles(string directory, bool readContent = false, string filter = "*.txt") +static AFileObject ReadFileToObject(string filePath) +static AFileObject ReadFileToObject(FileInfo file) +static Memory ReadFile(string filePath) +static Memory ReadFile(string filePath, out Encoding encoding) +static Memory ReadFile(FileInfo fileInfo) +static Memory ReadFile(FileInfo fileInfo, out Encoding encoding) +static bool CanFileBeAccessed(FileInfo fileInfo, FileAccess fileAccess = FileAccess.Read) +``` + +--- + +### AFileObject Class + +**Namespace:** `DevBase.IO` + +Wraps file content with metadata. + +#### Properties + +| Property | Type | Description | +|----------|------|-------------| +| `FileInfo` | `FileInfo` | File metadata | +| `Buffer` | `Memory` | File content | +| `Encoding` | `Encoding` | Detected encoding | + +#### Constructors + +```csharp +AFileObject(FileInfo fileInfo, bool readFile = false) +AFileObject(FileInfo fileInfo, Memory binaryData) +AFileObject(FileInfo fileInfo, Memory binaryData, Encoding encoding) +``` + +#### Methods + +```csharp +static AFileObject FromBuffer(byte[] buffer, string fileName = "buffer.bin") +AList ToList() // Split by lines +string ToStringData() +string ToString() +``` + +--- + +### DataCache Class + +**Namespace:** `DevBase.Cache` + +Time-based key-value cache with automatic expiration. + +#### Constructors + +```csharp +DataCache() // 2000ms default expiration +DataCache(int expirationMS) +``` + +#### Methods + +```csharp +void WriteToCache(K key, V value) +V DataFromCache(K key) // Returns default if expired/not found +AList DataFromCacheAsList(K key) // Get all values for key +bool IsInCache(K key) +``` + +--- + +### Multitasking Class + +**Namespace:** `DevBase.Async.Task` + +Manages concurrent tasks with capacity limits. + +#### Constructor + +```csharp +Multitasking(int capacity, int scheduleDelay = 100) +``` + +#### Methods + +```csharp +Task Register(Task task) +Task Register(Action action) +Task WaitAll() // Wait for all tasks to complete +Task KillAll() // Cancel pending and wait +``` + +--- + +### StringUtils Class (Static) + +**Namespace:** `DevBase.Utilities` + +String utility methods. + +#### Methods + +```csharp +static string RandomString(int length) +static string RandomString(int length, string chars) +static string Separate(string[] items, string separator) +static string[] DeSeparate(string input, string separator) +``` + +--- + +### AString Class + +**Namespace:** `DevBase.Typography` + +String wrapper with utility methods. + +#### Methods + +```csharp +AList AsList() // Split by newlines +string CapitalizeFirst() +``` + +--- + +### Base64EncodedAString Class + +**Namespace:** `DevBase.Typography.Encoded` + +Base64 encoding/decoding wrapper. + +#### Constructors + +```csharp +Base64EncodedAString(string plainText) +Base64EncodedAString(string base64, bool isEncoded) +``` + +#### Methods + +```csharp +string ToString() // Get encoded +string GetDecoded() +``` + +--- + +### Exceptions + +#### AListEntryException + +```csharp +namespace DevBase.Exception; + +public class AListEntryException : System.Exception +{ + public Type ExceptionType { get; } + + public enum Type + { + EntryNotFound, + OutOfBounds, + InvalidRange + } +} +``` + +--- + ## Key Concepts ### 1. AList - The Core Collection Type diff --git a/DevBase/DevBase.csproj b/DevBase/DevBase.csproj index 6bf7c7a..a01d7ea 100644 --- a/DevBase/DevBase.csproj +++ b/DevBase/DevBase.csproj @@ -11,7 +11,7 @@ https://github.com/AlexanderDotH/DevBase.git https://github.com/AlexanderDotH/DevBase.git Public - 1.3.4 + 1.3.5 MIT false Key.snk