Skip to content

Commit 3709e4c

Browse files
bee-sanbee-san
authored
Add invis character handling
* add braile * linting * fix: Resolve dead code and doc comment warnings * fix: Correct Braille decoder doc comments and add function documentation * docs * fmt * added invisible character identification + docs * fmt --------- Co-authored-by: bee <autumn@skerritt.blog>
1 parent 3c47ce6 commit 3709e4c

12 files changed

Lines changed: 1248 additions & 47 deletions

File tree

‎docs/README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Ares Documentation
2+
3+
Welcome to the Ares documentation! This repository contains comprehensive documentation for Ares, the next-generation automatic decoding and cracking tool.
4+
5+
## Table of Contents
6+
7+
### General Documentation
8+
9+
- [Ares Overview](ares_overview.md) - A high-level overview of Ares, its features, and capabilities
10+
- [Using Ares](using_ares.md) - A comprehensive guide on how to use Ares, with examples and common use cases
11+
12+
### Technical Documentation
13+
14+
- [Ares Architecture](ares_architecture.md) - Detailed explanation of Ares's internal architecture and components
15+
- [Plaintext Identification](plaintext_identification.md) - How Ares identifies plaintext and determines when decoding is successful
16+
17+
### Feature-Specific Documentation
18+
19+
- [Invisible Characters Detection](invisible_characters.md) - Information about Ares's capability to detect and handle invisible Unicode characters
20+
- [Package Managers](package-managers.md) - Guidelines for packaging Ares for different package managers
21+
22+
## About Ares
23+
24+
Ares is the next generation of decoding tools, built by the same people that brought you [Ciphey](https://github.com/ciphey/ciphey). It's designed to automatically detect and decode various types of encoded or encrypted text, including (but not limited to) Base64, Hexadecimal, Caesar cipher, ROT13, URL encoding, and many more.
25+
26+
Key features include:
27+
28+
- Significantly faster performance (up to 700% faster than Ciphey)
29+
- Library-first architecture for easy integration
30+
- Advanced search algorithms for efficient decoding
31+
- Built-in timeout mechanism
32+
- Comprehensive documentation and testing
33+
- Support for multi-level encodings
34+
35+
## Getting Started
36+
37+
The quickest way to get started with Ares is to install it via Cargo:
38+
39+
```bash
40+
cargo install project_ares
41+
```
42+
43+
Then use it with the `ares` command:
44+
45+
```bash
46+
ares "your encoded text here"
47+
```
48+
49+
For more detailed instructions, see the [Using Ares](using_ares.md) guide.
50+
51+
## Contributing
52+
53+
Contributions to Ares are welcome! Whether it's adding new decoders, improving existing ones, enhancing documentation, or fixing bugs, your help is appreciated. Check the [GitHub repository](https://github.com/bee-san/Ares) for more information on how to contribute.
54+
55+
## Additional Resources
56+
57+
- [GitHub Repository](https://github.com/bee-san/Ares)
58+
- [Discord Server](http://discord.skerritt.blog)
59+
- [Blog Post: Introducing Ares](https://skerritt.blog/introducing-ares/)
60+
- [Ciphey2 Documentation](https://broadleaf-angora-7db.notion.site/Ciphey2-32d5eea5d38b40c5b95a9442b4425710)

‎docs/ares_architecture.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Ares Architecture and Technical Details
2+
3+
## Core Architecture
4+
5+
Ares is built with a modular architecture that separates concerns and enables extensibility. The system is composed of several key components:
6+
7+
### 1. Library API
8+
9+
The core of Ares is a Rust library that provides the main functionality through a clean API. The entry point is the `perform_cracking` function in `src/lib.rs`:
10+
11+
```rust
12+
pub fn perform_cracking(text: &str, config: Config) -> Option<DecoderResult>
13+
```
14+
15+
This function takes the text to decode and a configuration object, then returns either:
16+
- `Some(DecoderResult)` containing the decoded plaintext and the path of decoders used
17+
- `None` if decoding failed or timed out
18+
19+
### 2. Decoders
20+
21+
Decoders are the components that perform the actual transformation of encoded text. Each decoder implements the `Decoder` trait defined in `src/decoders/interface.rs`, which requires a `crack` method:
22+
23+
```rust
24+
fn crack(&self, text: &str) -> Vec<String>
25+
```
26+
27+
This method attempts to decode the input text and returns a vector of possible results (some decoders like Caesar cipher may return multiple possible decodings).
28+
29+
Decoders are organized in the `src/decoders` module and include implementations for various encoding schemes like Base64, Hexadecimal, Caesar cipher, etc.
30+
31+
### 3. Checkers
32+
33+
Checkers determine whether a given text is valid plaintext. They implement the `Check` trait defined in `src/checkers/checker_type.rs`:
34+
35+
```rust
36+
fn check(&self, text: &str) -> CheckResult
37+
```
38+
39+
The `CheckResult` structure contains information about whether the text was identified as plaintext, which checker identified it, and additional metadata.
40+
41+
The main checkers include:
42+
- **Athena**: The primary checker that orchestrates other checkers
43+
- **LemmeKnow**: Uses pattern matching to identify known formats
44+
- **EnglishChecker**: Determines if text is valid English
45+
- **RegexChecker**: Checks if text matches a user-provided regex pattern
46+
47+
### 4. Search Algorithms
48+
49+
Search algorithms determine the order in which decoders are applied and manage the search for plaintext. Ares implements two main search algorithms:
50+
51+
- **A* Search** (`src/searchers/astar.rs`): Uses heuristics to prioritize promising decoders
52+
- **BFS** (`src/searchers/bfs.rs`): Systematically explores all possible decodings
53+
54+
The search process is managed by the `search_for_plaintext` function in `src/searchers/mod.rs`, which runs the search algorithm in a separate thread with a timeout.
55+
56+
### 5. Filtration System
57+
58+
The filtration system (`src/filtration_system/mod.rs`) determines which decoders to use for a given input. It can filter decoders based on:
59+
- Input characteristics
60+
- Performance considerations
61+
- User configuration
62+
63+
This component helps optimize the decoding process by avoiding unnecessary decoder attempts.
64+
65+
### 6. Configuration
66+
67+
The configuration system (`src/config/mod.rs`) manages user-configurable settings like:
68+
- Timeout duration
69+
- Whether to use the human checker
70+
- Verbosity level
71+
- Custom regex patterns
72+
73+
Configuration is stored in a global singleton for easy access throughout the codebase.
74+
75+
### 7. CLI Interface
76+
77+
The CLI interface (`src/cli/mod.rs` and `src/cli_input_parser/mod.rs`) handles command-line arguments, user interaction, and result presentation. It's built on top of the library API and provides a user-friendly interface to Ares's functionality.
78+
79+
## Data Flow
80+
81+
The typical data flow through Ares follows these steps:
82+
83+
1. **Input Processing**: The input text is received through the API or CLI
84+
2. **Initial Check**: The system checks if the input is already plaintext
85+
3. **Search Initialization**: If not plaintext, a search algorithm is initialized
86+
4. **Decoder Selection**: The filtration system selects appropriate decoders
87+
5. **Iterative Decoding**:
88+
- Decoders are applied to the input
89+
- Results are checked for plaintext
90+
- If not plaintext, they're added to the search queue
91+
6. **Result Generation**: When plaintext is found, a `DecoderResult` is created with the decoded text and the path of decoders used
92+
7. **Output Formatting**: The CLI formats and presents the results to the user
93+
94+
## Concurrency Model
95+
96+
Ares uses a multi-threaded approach to improve performance:
97+
98+
1. **Search Thread**: The search algorithm runs in a dedicated thread
99+
2. **Timeout Thread**: A separate thread monitors for timeout
100+
3. **Parallel Decoding**: Decoders can run in parallel using Rayon
101+
102+
This concurrency model allows Ares to efficiently utilize multiple CPU cores and handle timeouts gracefully.
103+
104+
## Plaintext Identification
105+
106+
Plaintext identification is a critical component of Ares. The process works as follows:
107+
108+
1. **Athena Checker**: The main checker that orchestrates other checkers
109+
- If a regex pattern is provided, it checks if the text matches
110+
- Otherwise, it tries the LemmeKnow checker and then the English checker
111+
112+
2. **LemmeKnow Checker**: Uses the LemmeKnow library to identify if the text matches known patterns
113+
- IP addresses, URLs, email addresses, etc.
114+
- Returns true if a match is found with sufficient confidence
115+
116+
3. **English Checker**: Determines if the text is valid English
117+
- Normalizes the text (lowercase, remove punctuation)
118+
- Uses the gibberish-or-not library to check if the text is meaningful English
119+
- Handles edge cases like very short strings
120+
121+
4. **Human Checker** (optional): Asks a human to verify if the text is valid plaintext
122+
- Only used if enabled in the configuration
123+
- Useful for ambiguous cases or specialized content
124+
125+
## Error Handling
126+
127+
Ares uses a combination of Rust's Result and Option types for error handling:
128+
129+
- `Option<DecoderResult>` is used for the main API return type, with `None` indicating failure
130+
- `Result<T, E>` is used for operations that can fail with specific error types
131+
- Logging is used to provide additional context for errors and debugging
132+
133+
## Testing Strategy
134+
135+
Ares has a comprehensive testing strategy:
136+
137+
1. **Unit Tests**: Each component has unit tests to verify its behavior in isolation
138+
2. **Integration Tests**: Tests that verify the interaction between components
139+
3. **Documentation Tests**: Examples in documentation that are verified by the test suite
140+
4. **Benchmarks**: Performance tests to ensure efficiency
141+
142+
## Performance Considerations
143+
144+
Several optimizations contribute to Ares's performance:
145+
146+
1. **Efficient Decoders**: Decoders are implemented with performance in mind
147+
2. **Parallel Processing**: Multi-threading for CPU-intensive operations
148+
3. **Early Termination**: The system stops as soon as plaintext is found
149+
4. **Timeout Mechanism**: Prevents infinite processing on difficult inputs
150+
5. **Heuristic-Based Search**: A* search prioritizes promising decoders
151+
152+
## Extensibility
153+
154+
Ares is designed to be extensible:
155+
156+
1. **Adding New Decoders**: Implement the `Decoder` trait and add to the decoders module
157+
2. **Custom Checkers**: Implement the `Check` trait for specialized plaintext detection
158+
3. **Alternative Search Algorithms**: The search system can be extended with new algorithms
159+
4. **Configuration Options**: The configuration system can be extended with new options
160+
161+
## Future Architectural Improvements
162+
163+
Planned improvements to the architecture include:
164+
165+
1. **More Sophisticated Heuristics**: Enhance the A* search with better heuristics
166+
2. **Improved English Detection**: Address limitations in the current English checker
167+
3. **Decoder Dependencies**: Allow decoders to specify dependencies or prerequisites
168+
4. **Dynamic Loading**: Support for dynamically loading decoders as plugins
169+
5. **Distributed Processing**: Support for distributing work across multiple machines

‎docs/ares_overview.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Ares: Next Generation Decoding Tool
2+
3+
## Overview
4+
5+
Ares is the next generation of automatic decoding and cracking tools, built by the same team that created [Ciphey](https://github.com/ciphey/ciphey). It's designed to be faster, more efficient, and more extensible than its predecessor, with the goal of eventually replacing Ciphey entirely.
6+
7+
Ares can automatically detect and decode various types of encoded or encrypted text, including (but not limited to) Base64, Hexadecimal, Caesar cipher, ROT13, URL encoding, and many more. It uses advanced algorithms and heuristics to identify the encoding type and apply the appropriate decoding method, often handling multiple layers of encoding automatically.
8+
9+
## Key Features
10+
11+
### Speed and Performance
12+
13+
Ares is significantly faster than its predecessor, with performance improvements of up to 700%. For every decode operation that Ciphey could perform, Ares can do approximately 7 in the same timeframe. This dramatic speed increase is achieved through:
14+
15+
- Efficient Rust implementation
16+
- Multithreading support via [Rayon](https://github.com/rayon-rs/rayon)
17+
- Optimized search algorithms
18+
- Improved plaintext detection
19+
20+
### Library-First Architecture
21+
22+
Ares is designed with a library-first approach, separating core functionality from the CLI interface. This architecture enables:
23+
24+
- Easy integration into other applications
25+
- Building additional tools on top of Ares (e.g., Discord bots)
26+
- Better testing and maintainability
27+
- Cleaner separation of concerns
28+
29+
### Advanced Search Algorithms
30+
31+
Ares employs sophisticated search algorithms to efficiently navigate the space of possible decodings:
32+
33+
- **A* Search**: Uses heuristics to prioritize the most promising decoders
34+
- **BFS (Breadth-First Search)**: Systematically explores all possible decodings
35+
36+
These algorithms allow Ares to handle multi-level encodings (e.g., Base64 → Hex → ROT13) efficiently, a capability that was limited in Ciphey due to performance constraints.
37+
38+
### Timeout Mechanism
39+
40+
One significant improvement over Ciphey is the built-in timeout mechanism. Ares will automatically stop processing after a configurable timeout period (default: 5 seconds for CLI, 10 seconds for Discord bot), ensuring that it doesn't run indefinitely on inputs it cannot decode.
41+
42+
### Comprehensive Documentation and Testing
43+
44+
Ares emphasizes code quality with:
45+
46+
- Extensive test coverage (over 120 tests)
47+
- Documentation tests to ensure examples stay up-to-date
48+
- Enforced documentation on all major components
49+
50+
## How Ares Identifies Plaintext
51+
52+
Ares uses a sophisticated system to determine whether decoded text is valid plaintext. This is a critical component of the system, as it determines when to stop the decoding process. The plaintext detection system includes several checkers:
53+
54+
### 1. Athena Checker
55+
56+
The Athena checker is the main orchestrator that runs multiple sub-checkers in sequence:
57+
58+
1. **Regex Checker** (if configured): Checks if the text matches a user-provided regular expression
59+
2. **LemmeKnow Checker**: Uses the [LemmeKnow](https://github.com/swanandx/lemmeknow) library (a Rust version of [PyWhat](https://github.com/bee-san/pyWhat)) to identify if the text matches known patterns like IP addresses, URLs, etc.
60+
3. **English Checker**: Determines if the text is valid English using the [gibberish-or-not](https://crates.io/crates/gibberish-or-not) library
61+
62+
### 2. Human Checker (Optional)
63+
64+
For interactive use, Ares can optionally ask a human to verify if the decoded text is valid plaintext. This is particularly useful for ambiguous cases or specialized content that automated checkers might not recognize correctly.
65+
66+
### 3. Plaintext Preprocessing
67+
68+
Before checking if text is valid plaintext, Ares performs normalization:
69+
- Converting to lowercase
70+
- Removing punctuation
71+
- Handling edge cases like very short strings
72+
73+
## Decoding Process
74+
75+
The decoding process in Ares follows these general steps:
76+
77+
1. **Initial Plaintext Check**: First, Ares checks if the input is already plaintext using the Athena checker. If it is, Ares returns early with the input as the result.
78+
79+
2. **Search Algorithm Initialization**: If the input is not plaintext, Ares initializes the search algorithm (A* by default) with the input text as the starting point.
80+
81+
3. **Decoder Selection**: The filtration system selects appropriate decoders to try based on the input characteristics.
82+
83+
4. **Iterative Decoding**: The search algorithm iteratively applies decoders to the input and any intermediate results, checking after each step if plaintext has been found.
84+
85+
5. **Result or Timeout**: The process continues until either:
86+
- Valid plaintext is found (success)
87+
- All possible decodings have been exhausted (failure)
88+
- The configured timeout is reached (failure)
89+
90+
## Invisible Characters Detection
91+
92+
Ares includes a feature to detect invisible Unicode characters in decoded plaintext. This is particularly useful for steganography or obfuscated text. When a significant percentage (>30%) of characters in the decoded text are invisible, Ares offers to save the result to a file instead of displaying it in the terminal, where such characters might not render correctly.
93+
94+
## Supported Decoders
95+
96+
Ares supports a growing list of decoders, including:
97+
98+
- Base64, Base32, Base58 (various flavors), Base91, Base65536
99+
- Hexadecimal
100+
- URL encoding
101+
- Caesar cipher and ROT47
102+
- Atbash cipher
103+
- A1Z26 encoding
104+
- Morse code
105+
- Binary
106+
- Braille
107+
- Rail fence cipher
108+
- Reverse text
109+
- Z85
110+
- And more being added regularly
111+
112+
## Usage
113+
114+
### Discord Bot
115+
116+
The simplest way to use Ares is through the Discord bot. Join the [Discord Server](http://discord.skerritt.blog), go to the #bots channel, and use the `$ares` command. Type `$help` for more information.
117+
118+
### CLI Installation
119+
120+
To install the CLI version:
121+
122+
```bash
123+
cargo install project_ares
124+
```
125+
126+
Then use it with the `ares` command.
127+
128+
### Docker
129+
130+
You can also build and run Ares using Docker:
131+
132+
```bash
133+
git clone https://github.com/bee-san/Ares
134+
cd Ares
135+
docker build .
136+
```
137+
138+
## Configuration
139+
140+
Ares provides several configuration options:
141+
142+
- **Timeout**: Maximum time to spend trying to decode (default: 5 seconds)
143+
- **Human Checker**: Enable/disable human verification of results
144+
- **Regex Pattern**: Specify a regex pattern to match against decoded text
145+
- **Verbosity**: Control the level of output detail
146+
147+
## Future Development
148+
149+
Ares is under active development, with plans to:
150+
151+
- Add more decoders (aiming to match and exceed Ciphey's ~50 decoders)
152+
- Improve plaintext detection accuracy
153+
- Enhance performance further
154+
- Add more configuration options
155+
- Expand platform support
156+
157+
## Contributing
158+
159+
Contributions to Ares are welcome! Whether it's adding new decoders, improving existing ones, enhancing documentation, or fixing bugs, your help is appreciated. Check the [GitHub repository](https://github.com/bee-san/Ares) for more information on how to contribute.

0 commit comments

Comments
 (0)