Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
47a5e82
process uploaded file
abubakr-soliman May 12, 2024
8fff7f6
setup docker compose
abubakr-soliman May 12, 2024
b29742a
setup database schemes
abubakr-soliman May 12, 2024
83727af
Push projects and data chunks into mongoDB
abubakr-soliman May 28, 2024
ec73055
update docker configurations
abubakr-soliman Jun 19, 2024
e9c4c24
set indecies for projects + chunks
abubakr-soliman Jun 19, 2024
09573f0
push assets collection
abubakr-soliman Jun 20, 2024
b5d7c29
change .env.example
abubakr-soliman Jun 20, 2024
e1b17c8
update file processing endpoint
abubakr-soliman Jul 2, 2024
7ba5a69
update requirements with openai
abubakr-soliman Jul 14, 2024
e884303
create the LLMInterface
abubakr-soliman Jul 14, 2024
05b3ffa
create the LLMEnums
abubakr-soliman Jul 14, 2024
b8abe8a
create thhe OpenAIProvider
abubakr-soliman Jul 14, 2024
1176678
add cohere to requirments.txt
abubakr-soliman Jul 14, 2024
2c4200b
update .env.example
abubakr-soliman Jul 14, 2024
7f3d594
update settings
abubakr-soliman Jul 14, 2024
ffcd64c
fix files location
abubakr-soliman Jul 14, 2024
87efbbd
set CoHereProvider Class
abubakr-soliman Jul 14, 2024
237b543
setup LLMProviderFactory
abubakr-soliman Jul 14, 2024
97ae1c2
update main.py
abubakr-soliman Jul 14, 2024
6697f0d
VectorDB - Qdrant Provider
abubakr-soliman Jul 19, 2024
1ebd1a9
nlp routes
abubakr-soliman Jul 19, 2024
bc4767e
update .env
abubakr-soliman Aug 15, 2024
ece7a7c
update LLM providers
abubakr-soliman Aug 15, 2024
0dc576d
update NLPController + NLP route
abubakr-soliman Aug 15, 2024
4625a84
support getting answers from LLMs
abubakr-soliman Aug 15, 2024
52c4b05
fix missed template_parser for NLPController
abubakr-soliman Sep 23, 2024
5d175b4
fix rag template issues
abubakr-soliman Sep 23, 2024
46e5de9
update README table of contents
abubakr-soliman Sep 23, 2024
973c833
Push 012 to main (#6) (#17)
bakrianoo Dec 2, 2024
27d895d
move to postgres + setup alembic
bakrianoo Dec 2, 2024
0c9cbb1
Update Data Models
bakrianoo Dec 2, 2024
ecefef3
Update README.md
bakrianoo Jan 3, 2025
70c57af
update requirements + .env
bakrianoo Mar 17, 2025
c42bca6
support pgvector + enhance chunking + fix SQL errors
bakrianoo Mar 20, 2025
90ab865
update README
bakrianoo Mar 21, 2025
06dbb3a
setup docker compose for production
bakrianoo Apr 18, 2025
31df1c4
fix file name
bakrianoo Jun 8, 2025
dd1cbef
set github actions flows sample
bakrianoo Jun 8, 2025
39ee293
update github actions filename
bakrianoo Jun 8, 2025
f468cd8
update PGVectorProvider connect function
bakrianoo Jun 8, 2025
5b47644
set service file sample
bakrianoo Jun 8, 2025
d89794c
setup reddis + rabbitMQ docker services
bakrianoo Aug 2, 2025
4ffe00c
update requirements packags
bakrianoo Aug 2, 2025
c631fad
setup basic celery app
bakrianoo Aug 2, 2025
fce71af
demo celery app for sending reports
bakrianoo Aug 2, 2025
a99e107
set include parameter for send_reports email service
bakrianoo Aug 2, 2025
f3e5741
remove mail_service celery demo task
bakrianoo Aug 2, 2025
1623ddf
push file processing celery task
bakrianoo Aug 3, 2025
6a1d9e8
setup flower dashbaord
bakrianoo Aug 3, 2025
256af03
update celery instructions + remove unused codes
bakrianoo Aug 3, 2025
2891c8d
push data/index push to celery
bakrianoo Aug 3, 2025
38386c4
setup celery workflow
bakrianoo Aug 3, 2025
49efd69
setup celery_task_executions database table
bakrianoo Aug 3, 2025
95ec8cd
setup celery beat ot clean up database celery executions table
bakrianoo Aug 3, 2025
914d031
setup celery docker services
bakrianoo Aug 4, 2025
7705041
update README file with new tutorials
bakrianoo Aug 15, 2025
ec8d877
init project directory
EssamShenhab Jan 25, 2026
5998360
update .gitignore
EssamShenhab Jan 25, 2026
9b0122e
update README and requirements
EssamShenhab Jan 25, 2026
3c7240a
add .env and .gitkeep
EssamShenhab Jan 25, 2026
e2340a9
update instructions
EssamShenhab Jan 25, 2026
7710277
add fastapi routing
EssamShenhab Jan 25, 2026
c02bcde
created src folder
EssamShenhab Jan 25, 2026
69f43cb
Completed tut-004
EssamShenhab Jan 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 1 addition & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,6 @@

This is a minimal implementation of the RAG model for question answering.

## The Course

This is an educational project where all of the codes where explained (step by step) via a set of `Arabic` youtube videos. Please check the list:

| # | Title | Link | Codes |
|---|------------------------------------------|------------------------------------------------------------------------------------------------------|----------------------------------------------------|
| 1 | About the Course ماذا ولمـــاذا | [Video](https://www.youtube.com/watch?v=Vv6e2Rb1Q6w&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj) | NA |
| 2 | What will we build ماذا سنبنى في المشروع | [Video](https://www.youtube.com/watch?v=_l5S5CdxE-Q&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=2) | NA |
| 3 | Setup your tools الأدوات الأساسية | [Video](https://www.youtube.com/watch?v=VSFbkFRAT4w&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=3) | NA |
| 4 | Project Architecture | [Video](https://www.youtube.com/watch?v=Ei_nBwBbFUQ&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=4) | [branch](https://github.com/bakrianoo/mini-rag/tree/tut-001) |
| 5 | Welcome to FastAPI | [Video](https://www.youtube.com/watch?v=cpOuCdzN_Mo&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=5) | [branch](https://github.com/bakrianoo/mini-rag/tree/tut-002) |
| 6 | Nested Routes + Env Values | [Video](https://www.youtube.com/watch?v=CrR2Bz2Y7Hw&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=6) | [branch](https://github.com/bakrianoo/mini-rag/tree/tut-003) |
| 7 | Uploading a File | [Video](https://www.youtube.com/watch?v=5alMKCbFqWs&list=PLvLvlVqNQGHCUR2p0b8a0QpVjDUg50wQj&index=7) | [branch](https://github.com/bakrianoo/mini-rag/tree/tut-004) |




## Requirements

- Python 3.8 or later
Expand Down Expand Up @@ -55,14 +38,4 @@ $ pip install -r requirements.txt
$ cp .env.example .env
```

Set your environment variables in the `.env` file. Like `OPENAI_API_KEY` value.

## Run the FastAPI server

```bash
$ uvicorn main:app --reload --host 0.0.0.0 --port 5000
```

## POSTMAN Collection

Download the POSTMAN collection from [/assets/mini-rag-app.postman_collection.json](/assets/mini-rag-app.postman_collection.json)
Set your environment variables in the `.env` file. Like `OPENAI_API_KEY` value.
Binary file added __pycache__/main.cpython-310.pyc
Binary file not shown.
7 changes: 7 additions & 0 deletions src/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
APP_NAME="mini-rag"
APP_VERSION="0.1.0"
OPENAI_API_KEY="********"

FILE_ALLOWED_TYPES=["text/plain", "application/pdf"]
FILE_MAX_SIZE=10
FILE_DEFAULT_CHUNK_SIZE=512000 # 512 KB
14 changes: 7 additions & 7 deletions src/.env.example
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
APP_NAME="mini-RAG"
APP_VERSION="0.1"
OPENAI_API_KEY=""
=
FILE_ALLOWED_TYPES=
FILE_MAX_SIZE=10
FILE_DEFAULT_CHUNK_SIZE=512000 # 512KB
APP_NAME="mini-rag"
APP_VERSION="0.1.0"
OPENAI_API_KEY=""

FILE_ALLOWED_TYPES=
FILE_MAX_SIZE=10
FILE_DEFAULT_CHUNK_SIZE=512000 # 512 KB
88 changes: 72 additions & 16 deletions src/.gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*.py[codz]
*$py.class

# C extensions
Expand All @@ -27,8 +27,8 @@ share/python-wheels/
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

Expand All @@ -46,7 +46,7 @@ htmlcov/
nosetests.xml
coverage.xml
*.cover
*.py,cover
*.py.cover
.hypothesis/
.pytest_cache/
cover/
Expand Down Expand Up @@ -92,22 +92,37 @@ ipython_config.py
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# Pipfile.lock

# UV
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# uv.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# poetry.lock
# poetry.toml

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
# https://pdm-project.org/en/latest/usage/project/#working-with-version-control
# pdm.lock
# pdm.toml
.pdm-python
.pdm-build/

# pixi
# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
# pixi.lock
# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
# in the .venv directory. It is recommended not to include this directory in version control.
.pixi

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
Expand All @@ -116,11 +131,25 @@ __pypackages__/
celerybeat-schedule
celerybeat.pid

# Redis
*.rdb
*.aof
*.pid

# RabbitMQ
mnesia/
rabbitmq/
rabbitmq-data/

# ActiveMQ
activemq-data/

# SageMath parsed files
*.sage.py

# Environments
.env
.envrc
.venv
env/
venv/
Expand Down Expand Up @@ -153,8 +182,35 @@ dmypy.json
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
# .idea/

# Abstra
# Abstra is an AI-powered process automation framework.
# Ignore directories containing user credentials, local state, and settings.
# Learn more at https://abstra.io/docs
.abstra/

# Visual Studio Code
# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
# and can be added to the global gitignore or merged into this file. However, if you prefer,
# you could uncomment the following to ignore the entire vscode folder
# .vscode/

# Ruff stuff:
.ruff_cache/

# PyPI configuration file
.pypirc

# Marimo
marimo/_static/
marimo/_lsp/
__marimo__/

# Streamlit
.streamlit/secrets.toml
Binary file added src/__pycache__/__init__.cpython-310.pyc
Binary file not shown.
Binary file added src/__pycache__/base.cpython-310.pyc
Binary file not shown.
2 changes: 1 addition & 1 deletion src/assets/.gitignore
Original file line number Diff line number Diff line change
@@ -1 +1 @@
files
files
103 changes: 48 additions & 55 deletions src/assets/mini-rag-app.postman_collection.json
Original file line number Diff line number Diff line change
@@ -1,57 +1,50 @@
{
"info": {
"_postman_id": "1f67dedb-1b93-4639-9f96-7fe8681693f4",
"name": "mini-rag-app",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
"_exporter_id": "854486",
"_collection_link": "https://www.postman.com/gold-water-645258/workspace/mini-rag-app/collection/854486-1f67dedb-1b93-4639-9f96-7fe8681693f4?action=share&source=collection_link&creator=854486"
},
"item": [
{
"name": "welcome-request",
"request": {
"method": "GET",
"header": [],
"url": {
"raw": "{{api}}/welcome",
"host": [
"{{api}}"
],
"path": [
"welcome"
]
}
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"type": "text/javascript",
"packages": {},
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"type": "text/javascript",
"packages": {},
"exec": [
""
]
}
}
],
"variable": [
{
"key": "api",
"value": "http://127.0.0.1:8000",
"type": "string"
}
]
"info": {
"_postman_id": "febe8027-4bc0-4f8e-ba7c-73b5eb97fa50",
"name": "mini-rag-app",
"description": "### Welcome to Postman! This is your first collection. \n\nCollections are your starting point for building and testing APIs. You can use this one to:\n\n• Group related requests\n• Test your API in real-world scenarios\n• Document and share your requests\n\nUpdate the name and overview whenever you’re ready to make it yours.\n\n[Learn more about Postman Collections.](https://learning.postman.com/docs/collections/collections-overview/)",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
"_exporter_id": "51802644",
"_collection_link": "https://go.postman.co/collection/51802644-febe8027-4bc0-4f8e-ba7c-73b5eb97fa50?source=collection_link"
},
"item": [
{
"name": "welcome-request",
"request": {
"method": "GET",
"header": []
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"type": "text/javascript",
"packages": {},
"requests": {},
"exec": [
""
]
}
},
{
"listen": "test",
"script": {
"type": "text/javascript",
"packages": {},
"requests": {},
"exec": [
""
]
}
}
],
"variable": [
{
"key": "api",
"value": ""
}
]
}
16 changes: 6 additions & 10 deletions src/controllers/BaseController.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,13 @@
import random
import string


class BaseController:

def __init__(self):

self.app_settings = get_settings()

self.base_dir = os.path.dirname( os.path.dirname(__file__) )
self.files_dir = os.path.join(
self.base_dir,
"assets/files"
)

def generate_random_string(self, length: int=12):
self.base_dir = os.path.dirname(os.path.dirname(__file__))
self.files_dir = os.path.join(self.base_dir, "assets/files")

def generate_random_string(self, length: int = 12):
return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))

Loading