Skip to content

Commit 951d71a

Browse files
iu2frldependabot[bot]Luca IU2FRL
authored
Drafting new release (#77)
* Implementing #63 (#73) * Bump the npm_and_yarn group across 1 directory with 2 updates Bumps the npm_and_yarn group with 2 updates in the /frontend directory: [flatted](https://github.com/WebReflection/flatted) and [picomatch](https://github.com/micromatch/picomatch). Updates `flatted` from 3.3.3 to 3.4.2 - [Commits](WebReflection/flatted@v3.3.3...v3.4.2) Updates `picomatch` from 4.0.3 to 4.0.4 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@4.0.3...4.0.4) Updates `picomatch` from 2.3.1 to 2.3.2 - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@4.0.3...4.0.4) --- updated-dependencies: - dependency-name: flatted dependency-version: 3.4.2 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: picomatch dependency-version: 4.0.4 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: picomatch dependency-version: 2.3.2 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> * Bump cryptography in the pip group across 1 directory Bumps the pip group with 1 update in the / directory: [cryptography](https://github.com/pyca/cryptography). Updates `cryptography` from 46.0.5 to 46.0.6 - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@46.0.5...46.0.6) --- updated-dependencies: - dependency-name: cryptography dependency-version: 46.0.6 dependency-type: direct:production dependency-group: pip ... Signed-off-by: dependabot[bot] <support@github.com> * Bump vite in /frontend in the npm_and_yarn group across 1 directory (#68) * Bump cryptography in the pip group across 1 directory (#69) Bumps the pip group with 1 update in the / directory: [cryptography](https://github.com/pyca/cryptography). Updates `cryptography` from 46.0.6 to 46.0.7 - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@46.0.6...46.0.7) --- updated-dependencies: - dependency-name: cryptography dependency-version: 46.0.7 dependency-type: direct:production dependency-group: pip ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update CI workflow to comment out main branch (#72) Comment out the main branch in CI workflow trigger. * Bump python-multipart in the pip group across 1 directory (#71) Bumps the pip group with 1 update in the / directory: [python-multipart](https://github.com/Kludex/python-multipart). Updates `python-multipart` from 0.0.22 to 0.0.26 - [Release notes](https://github.com/Kludex/python-multipart/releases) - [Changelog](https://github.com/Kludex/python-multipart/blob/master/CHANGELOG.md) - [Commits](Kludex/python-multipart@0.0.22...0.0.26) --- updated-dependencies: - dependency-name: python-multipart dependency-version: 0.0.26 dependency-type: direct:production dependency-group: pip ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: add folder management functionality for devices - Implemented folder selection in DeviceForm component. - Created DeviceListWithFolders component to display devices organized by folders. - Added FolderModal for creating and editing folders. - Introduced FolderTreeItem for rendering folder structure with devices. - Updated Dashboard to integrate folder management. - Added tests for folder creation, deletion, and device-folder associations. * fix: update folder relationship to include foreign keys and remove cascade option * update test functions to use auth_client for authentication * optimize folder retrieval by including children and devices in queries * enhance folder loading by including nested children and devices in queries * enable foreign key constraints in test database setup * remove cascade option from folder-device relationship and update folder retrieval logic * Fixing UI tests * add move device functionality with modal for folder selection * Adding support for subfolders * Getting subfolders * Showing collapse icon on folders * Adding rename and delete buttons to folders * Implementing folders deletion * Deleting a device moves it to the parent folder * Updating folders dropdown * Properly updating devices tree * Updating tests * Adding login tests * Fixing folders deletion --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Luca IU2FRL <luca.ubuntu@iu2frl.it> * Improving big files upload (#75) * Allowing big files in nginx * Enhancing logging * Chunking big files * Making upload async * Fixing wrong upload routine * Trying to force async upload * Fixing async behavior * Fixing FTP tests * Fixing frontend tests * Extending folders coverage --------- Co-authored-by: Luca IU2FRL <luca.ubuntu@iu2frl.it> * Avoiding duplicate CIs (#76) Co-authored-by: Luca IU2FRL <luca.ubuntu@iu2frl.it> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Luca IU2FRL <luca.ubuntu@iu2frl.it>
1 parent 17d2eb1 commit 951d71a

12 files changed

Lines changed: 711 additions & 64 deletions

File tree

.github/workflows/ci.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ on:
3636
options: ["20", "22"]
3737
default: "20"
3838

39+
concurrency:
40+
group: ci-${{ github.workflow }}-${{ github.ref }}
41+
cancel-in-progress: true
42+
3943
jobs:
4044
unit-tests:
4145
name: Unit Tests

backend/routers/ftp.py

Lines changed: 214 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,10 @@
1414
"""
1515
import logging
1616
import os
17+
import uuid
1718
from urllib.parse import unquote
1819

19-
from fastapi import APIRouter, Depends, HTTPException, Request, UploadFile, File
20+
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Request, UploadFile, File
2021
from fastapi.responses import Response
2122
from pydantic import BaseModel
2223
from sqlalchemy.ext.asyncio import AsyncSession
@@ -49,6 +50,9 @@
4950
log = logging.getLogger(__name__)
5051
router = APIRouter(prefix="/ftp", tags=["ftp"])
5152

53+
# Track active uploads for UI feedback
54+
_upload_status: dict[str, dict] = {}
55+
5256

5357
# -- Session management --------------------------------------------------------
5458

@@ -184,11 +188,34 @@ async def list_dir(
184188
_: str = Depends(get_current_user),
185189
):
186190
"""List directory contents at the given remote path."""
191+
log.debug(
192+
"FTP list directory request: path=%s, session=%s",
193+
path,
194+
session_id[:8],
195+
)
196+
187197
try:
188198
entries = await list_directory(session_id, path)
199+
log.info(
200+
"FTP list directory successful: path=%s, entries=%d, session=%s",
201+
path,
202+
len(entries),
203+
session_id[:8],
204+
)
189205
except ValueError as exc:
206+
log.error(
207+
"FTP list directory failed (session not found): path=%s, session=%s",
208+
path,
209+
session_id[:8],
210+
)
190211
raise HTTPException(status_code=404, detail=str(exc))
191212
except Exception as exc: # noqa: BLE001
213+
log.error(
214+
"FTP list directory failed: path=%s, error=%s, session=%s",
215+
path,
216+
exc,
217+
session_id[:8],
218+
)
192219
raise HTTPException(status_code=500, detail=f"Directory listing failed: {exc}")
193220
return {"path": path, "entries": entries}
194221

@@ -202,11 +229,30 @@ async def download_file(
202229
"""Download a remote file. ``path`` must be URL-encoded."""
203230
remote_path = unquote(path)
204231
filename = os.path.basename(remote_path)
232+
233+
log.debug(
234+
"FTP download request: filename=%s, path=%s, session=%s",
235+
filename,
236+
remote_path,
237+
session_id[:8],
238+
)
239+
205240
try:
206241
data = await read_file_bytes(session_id, remote_path)
207242
except ValueError as exc:
243+
log.error(
244+
"FTP download failed (session not found): path=%s, session=%s",
245+
remote_path,
246+
session_id[:8],
247+
)
208248
raise HTTPException(status_code=404, detail=str(exc))
209249
except Exception as exc: # noqa: BLE001
250+
log.error(
251+
"FTP download failed: path=%s, error=%s, session=%s",
252+
remote_path,
253+
exc,
254+
session_id[:8],
255+
)
210256
raise HTTPException(status_code=500, detail=f"Download failed: {exc}")
211257

212258
return Response(
@@ -223,40 +269,141 @@ class UploadResponse(BaseModel):
223269
size: int
224270

225271

226-
@router.post("/{session_id}/upload", response_model=UploadResponse)
272+
@router.post("/{session_id}/upload")
227273
async def upload_file(
228274
session_id: str,
229275
path: str,
276+
request: Request,
230277
file: UploadFile = File(...),
278+
background_tasks: BackgroundTasks = BackgroundTasks(),
231279
_: str = Depends(get_current_user),
232280
):
233281
"""
234-
Upload a file to the remote server.
282+
Upload a file to the remote server (returns immediately, processes in background).
235283
236284
``path`` is the target directory; the remote file will be placed at
237285
``{path}/{file.filename}``.
286+
287+
Returns upload_id for status tracking via GET /ftp/{session_id}/upload/{upload_id}
238288
"""
239289
target_dir = unquote(path)
240290
if target_dir.endswith("/"):
241291
remote_path = target_dir + (file.filename or "upload")
242292
else:
243293
remote_path = target_dir + "/" + (file.filename or "upload")
244294

245-
data = await file.read()
295+
upload_id = str(uuid.uuid4())
296+
297+
log.debug(
298+
"FTP upload request: filename=%s, target_path=%s, upload_id=%s, session=%s",
299+
file.filename,
300+
remote_path,
301+
upload_id[:8],
302+
session_id[:8],
303+
)
304+
305+
# Try to read content length for progress reporting
246306
try:
247-
await write_file_bytes(session_id, remote_path, data)
248-
except ValueError as exc:
249-
raise HTTPException(status_code=404, detail=str(exc))
307+
content_length = int(request.headers.get("content-length", "0") or 0)
308+
except Exception:
309+
content_length = 0
310+
311+
# Read the full request body before returning. This is required to
312+
# avoid losing the upload when the request ends.
313+
try:
314+
file_data = await file.read()
250315
except Exception as exc: # noqa: BLE001
251-
raise HTTPException(status_code=500, detail=f"Upload failed: {exc}")
316+
log.error("Failed to read upload file: %s", exc)
317+
raise HTTPException(status_code=400, detail=f"Failed to read file: {exc}") from exc
318+
319+
file_size = len(file_data)
320+
file_size_mb = file_size / (1024 * 1024)
321+
322+
# Initialize status entry
323+
_upload_status[upload_id] = {
324+
"status": "uploading",
325+
"filename": file.filename or "upload",
326+
"size_bytes": file_size,
327+
"transferred_bytes": 0,
328+
}
252329

253330
log.info(
254-
"FTP uploaded %s bytes to %s (session %s)",
255-
len(data),
256-
remote_path,
331+
"FTP upload buffered: filename=%s, size=%s bytes (%.2f MB), upload_id=%s, session=%s",
332+
file.filename,
333+
file_size,
334+
file_size_mb,
335+
upload_id[:8],
257336
session_id[:8],
258337
)
259-
return UploadResponse(uploaded=remote_path, size=len(data))
338+
339+
def progress_callback(bytes_written: int) -> None:
340+
"""Update progress status."""
341+
sts = _upload_status.get(upload_id)
342+
if sts is None:
343+
return
344+
sts["transferred_bytes"] = bytes_written
345+
if sts.get("size_bytes") and sts["size_bytes"] > 0:
346+
try:
347+
sts["percent"] = round(bytes_written / sts["size_bytes"] * 100, 2)
348+
except Exception:
349+
sts["percent"] = None
350+
351+
async def uploader_task():
352+
"""Upload buffered chunks to FTP (runs in background)."""
353+
try:
354+
await write_file_bytes(session_id, remote_path, file_data)
355+
progress_callback(file_size)
356+
_upload_status[upload_id]["status"] = "completed"
357+
log.info(
358+
"FTP upload completed: filename=%s, upload_id=%s, session=%s",
359+
file.filename,
360+
upload_id[:8],
361+
session_id[:8],
362+
)
363+
except Exception as exc:
364+
log.error(
365+
"FTP upload failed: filename=%s, error=%s, upload_id=%s, session=%s",
366+
file.filename,
367+
exc,
368+
upload_id[:8],
369+
session_id[:8],
370+
)
371+
_upload_status[upload_id] = {"status": "failed", "error": str(exc)}
372+
373+
# Schedule the uploader task as a background task
374+
background_tasks.add_task(uploader_task)
375+
376+
# Return immediately with upload_id
377+
log.info(
378+
"FTP upload scheduled in background: filename=%s, size=%.2f MB, upload_id=%s, session=%s",
379+
file.filename,
380+
file_size_mb,
381+
upload_id[:8],
382+
session_id[:8],
383+
)
384+
385+
return {
386+
"upload_id": upload_id,
387+
"uploaded": remote_path,
388+
"filename": file.filename or "upload",
389+
"size": file_size,
390+
"size_bytes": file_size,
391+
"size_mb": file_size_mb,
392+
"status": "queued",
393+
}
394+
395+
396+
@router.get("/{session_id}/upload/{upload_id}")
397+
async def get_upload_status(
398+
session_id: str,
399+
upload_id: str,
400+
_: str = Depends(get_current_user),
401+
):
402+
"""Check status of a background upload."""
403+
status_info = _upload_status.get(upload_id)
404+
if not status_info:
405+
raise HTTPException(status_code=404, detail=f"Upload {upload_id[:8]} not found")
406+
return status_info
260407

261408

262409
class DeleteRequest(BaseModel):
@@ -273,11 +420,29 @@ async def delete_path(
273420
_: str = Depends(get_current_user),
274421
):
275422
"""Delete a remote file or directory."""
423+
log.debug(
424+
"FTP delete request: path=%s, is_dir=%s, session=%s",
425+
body.path,
426+
body.is_dir,
427+
session_id[:8],
428+
)
429+
276430
try:
277431
await delete_remote(session_id, body.path, body.is_dir)
278432
except ValueError as exc:
433+
log.error(
434+
"FTP delete failed (session not found): path=%s, session=%s",
435+
body.path,
436+
session_id[:8],
437+
)
279438
raise HTTPException(status_code=404, detail=str(exc))
280439
except Exception as exc: # noqa: BLE001
440+
log.error(
441+
"FTP delete failed: path=%s, error=%s, session=%s",
442+
body.path,
443+
exc,
444+
session_id[:8],
445+
)
281446
raise HTTPException(status_code=500, detail=f"Delete failed: {exc}")
282447

283448

@@ -295,11 +460,31 @@ async def rename_path(
295460
_: str = Depends(get_current_user),
296461
):
297462
"""Rename or move a remote path."""
463+
log.debug(
464+
"FTP rename request: old_path=%s, new_path=%s, session=%s",
465+
body.old_path,
466+
body.new_path,
467+
session_id[:8],
468+
)
469+
298470
try:
299471
await rename_remote(session_id, body.old_path, body.new_path)
300472
except ValueError as exc:
473+
log.error(
474+
"FTP rename failed (session not found): old_path=%s, new_path=%s, session=%s",
475+
body.old_path,
476+
body.new_path,
477+
session_id[:8],
478+
)
301479
raise HTTPException(status_code=404, detail=str(exc))
302480
except Exception as exc: # noqa: BLE001
481+
log.error(
482+
"FTP rename failed: old_path=%s, new_path=%s, error=%s, session=%s",
483+
body.old_path,
484+
body.new_path,
485+
exc,
486+
session_id[:8],
487+
)
303488
raise HTTPException(status_code=500, detail=f"Rename failed: {exc}")
304489

305490

@@ -316,9 +501,26 @@ async def make_directory(
316501
_: str = Depends(get_current_user),
317502
):
318503
"""Create a remote directory."""
504+
log.debug(
505+
"FTP mkdir request: path=%s, session=%s",
506+
body.path,
507+
session_id[:8],
508+
)
509+
319510
try:
320511
await mkdir_remote(session_id, body.path)
321512
except ValueError as exc:
513+
log.error(
514+
"FTP mkdir failed (session not found): path=%s, session=%s",
515+
body.path,
516+
session_id[:8],
517+
)
322518
raise HTTPException(status_code=404, detail=str(exc))
323519
except Exception as exc: # noqa: BLE001
520+
log.error(
521+
"FTP mkdir failed: path=%s, error=%s, session=%s",
522+
body.path,
523+
exc,
524+
session_id[:8],
525+
)
324526
raise HTTPException(status_code=500, detail=f"Mkdir failed: {exc}")

0 commit comments

Comments
 (0)