-
Notifications
You must be signed in to change notification settings - Fork 33
feat(attachments): multi-annex embed for Sales Invoice (part 1) #263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dafrose
wants to merge
36
commits into
alyf-de:version-16-hotfix
Choose a base branch
from
dafrose:feat/embed_attachments_16
base: version-16-hotfix
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
7c271ce
test(eu_einvoice): add unit tests for _embed_attachment
dafrose a151cba
refactor(eu_einvoice): loop legacy embed attachments with test scaffold
dafrose 880771e
feat(eu_einvoice): add Sales Invoice attachment table and dedup valid…
dafrose 7bfd319
feat(attachments): gate embed source on multi-attachment setting
dafrose eadd7bc
feat(attachments): migrate legacy embed field to attachment table
dafrose c429cc2
test(attachments): add hybrid PDF round-trip test and trim redundant …
dafrose f1b4a25
chore(i18n): update German locale for multi-attachment embedding
dafrose cd31e4d
fix(ci): add nosemgrep hint to test-specific database changes
dafrose b70310c
feat(attachments): optional submitted invoices in bulk legacy migration
dafrose 48bfd36
fix(attachments): harden embed file resolution and migration access
dafrose 9a35f8c
test(attachments): refactor embed tests for atomicity and shared helpers
dafrose 066b595
fix(attachments): preserve legacy embed lockdown on custom field sync
dafrose be7d665
feat(attachments): add bulk broken legacy link skip/remove handling
dafrose fed065b
docs(sales-invoice): explain empty file link filter on new forms
dafrose a03fd4b
fix(attachments): ues db.set_value to clear broken links to avoid unr…
dafrose ba12ba8
fix(attachments): warn on broken legacy embed URL during validate
dafrose f30b0aa
docs(attachments): add milestone 1 docstrings and German translations
dafrose 73b240e
fix(test): ensure test masters are created without before_tests due t…
dafrose c7b1332
fix(test): harden lightmode ERPNext setup and legacy embed file resol…
dafrose 693a654
feat(attachments): validate duplicate annex filenames and embed by fi…
dafrose 95f0514
feat(attachments): enforce unique annex filenames per invoice
dafrose c5d1646
refactor(attachments): unify duplicate filename validation on save an…
dafrose 316dc1e
feat(attachments): enforce exclusive legacy vs table embed field visi…
dafrose b500ec1
fix(attachments): gate annex row validation on multi-embed setting
dafrose 922f02b
fix(attachments): rollback bulk migrate failures and embed binary ann…
dafrose 6d80a3b
fix(attachments): use DB persist for draft bulk legacy migrate
dafrose edf3d2e
fix(attachments): always enforce duplicate annex filenames on save
dafrose e0f4175
fix(attachments): set idx on direct annex child inserts
dafrose 95c17fd
fix(migrate): remove cstr(exc) so err log captures full traceback
dafrose 0c00060
feat(attachments): auto-queue legacy migration and block unmigrated e…
dafrose c312aec
fix(attachments): require annex File attached to Sales Invoice
dafrose bc47ed5
chore(patch): bump custom field migration patch
dafrose 5d87a41
fix(attachments): scope legacy embed file resolution to invoice
dafrose 7518784
fix(attachments): prefer table annexes when legacy embed remains
dafrose 651e7a8
fix(attachments): make legacy embed migration idempotent
dafrose 834a1d6
fix(attachments): hide Add Row on unsaved Sales Invoice annex table
dafrose File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,4 +5,4 @@ | |
| tags | ||
| node_modules | ||
| __pycache__ | ||
| .vscode/* | ||
| .vscode/* | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
48 changes: 48 additions & 0 deletions
48
eu_einvoice/european_e_invoice/custom/embed_attachment_scenarios.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Unit-test scenarios for EInvoiceGenerator._embed_attachments(). | ||
| # Profile gating lives in create_einvoice — not tested here. | ||
|
|
||
| - id: legacy_empty_field | ||
| field_url: "" | ||
| expect: | ||
| reference_count: 0 | ||
|
|
||
| - id: legacy_local_file_916 | ||
| field_url: /files/legacy-annex.png | ||
| mock_file: | ||
| name: F-LEGACY-LOCAL | ||
| file_url: /files/legacy-annex.png | ||
| is_remote: false | ||
| content_bytes_b64: iVBORw0KGgoA= | ||
| expect: | ||
| reference_count: 1 | ||
| type_code: "916" | ||
| issuer_assigned_id: F-LEGACY-LOCAL | ||
| has_attached_object: true | ||
| filename: legacy-annex.png | ||
| mime_type: image/png | ||
|
|
||
| - id: legacy_remote_file_uri | ||
| field_url: https://example.com/legacy-annex.pdf | ||
| mock_file: | ||
| name: F-LEGACY-REMOTE | ||
| file_url: https://example.com/legacy-annex.pdf | ||
| is_remote: true | ||
| expect: | ||
| reference_count: 1 | ||
| type_code: "916" | ||
| issuer_assigned_id: F-LEGACY-REMOTE | ||
| has_attached_object: false | ||
| uri_id: https://example.com/legacy-annex.pdf | ||
|
|
||
| - id: legacy_remote_api_method_uri | ||
| field_url: /api/method/frappe.utils.print_format.download_pdf | ||
| mock_file: | ||
| name: F-LEGACY-API-REMOTE | ||
| file_url: /api/method/frappe.utils.print_format.download_pdf | ||
| is_remote: true | ||
| expect: | ||
| reference_count: 1 | ||
| type_code: "916" | ||
| issuer_assigned_id: F-LEGACY-API-REMOTE | ||
| has_attached_object: false | ||
| uri_id: /api/method/frappe.utils.print_format.download_pdf |
200 changes: 200 additions & 0 deletions
200
eu_einvoice/european_e_invoice/custom/embed_attachment_test_helpers.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,200 @@ | ||
| # Copyright (c) 2026, ALYF GmbH and Contributors | ||
| # See license.txt | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import base64 | ||
| import io | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
|
|
||
| import frappe | ||
| import yaml | ||
| from drafthorse.models.document import Document | ||
| from lxml import etree | ||
|
|
||
| from eu_einvoice.european_e_invoice.custom.sales_invoice import EInvoiceGenerator, as_base_64 | ||
| from eu_einvoice.utils import EInvoiceProfile | ||
|
|
||
| _SCENARIOS_PATH = Path(__file__).with_name("embed_attachment_scenarios.yaml") | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class MockFileSpec: | ||
| name: str | ||
| file_url: str | ||
| is_remote: bool | ||
| content: bytes = b"" | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class EmbedAttachmentExpectations: | ||
| reference_count: int | ||
| type_code: str | None = None | ||
| issuer_assigned_id: str | None = None | ||
| has_attached_object: bool | None = None | ||
| filename: str | None = None | ||
| mime_type: str | None = None | ||
| uri_id: str | None = None | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class EmbedAttachmentScenario: | ||
| id: str | ||
| field_url: str | ||
| expect: EmbedAttachmentExpectations | ||
| mock_file: MockFileSpec | None = None | ||
|
|
||
|
|
||
| def load_embed_attachment_scenarios() -> list[EmbedAttachmentScenario]: | ||
| """Load YAML-driven ``_embed_attachments`` unit-test scenarios from disk.""" | ||
| raw = yaml.safe_load(_SCENARIOS_PATH.read_text(encoding="utf-8")) | ||
| scenarios: list[EmbedAttachmentScenario] = [] | ||
|
|
||
| for row in raw: | ||
| mock_row = row.get("mock_file") | ||
| mock_file = None | ||
| if mock_row: | ||
| content = b"" | ||
| if content_b64 := mock_row.get("content_bytes_b64"): | ||
| content = base64.b64decode(content_b64) | ||
| mock_file = MockFileSpec( | ||
| name=mock_row["name"], | ||
| file_url=mock_row["file_url"], | ||
| is_remote=bool(mock_row["is_remote"]), | ||
| content=content, | ||
| ) | ||
|
|
||
| expect_row = row["expect"] | ||
| scenarios.append( | ||
| EmbedAttachmentScenario( | ||
| id=row["id"], | ||
| field_url=row["field_url"], | ||
| mock_file=mock_file, | ||
| expect=EmbedAttachmentExpectations( | ||
| reference_count=expect_row["reference_count"], | ||
| type_code=expect_row.get("type_code"), | ||
| issuer_assigned_id=expect_row.get("issuer_assigned_id"), | ||
| has_attached_object=expect_row.get("has_attached_object"), | ||
| filename=expect_row.get("filename"), | ||
| mime_type=expect_row.get("mime_type"), | ||
| uri_id=expect_row.get("uri_id"), | ||
| ), | ||
| ) | ||
| ) | ||
|
|
||
| return scenarios | ||
|
|
||
|
|
||
| def make_sales_invoice_doc(**kwargs) -> frappe._dict: | ||
| """Return a minimal in-memory **Sales Invoice** dict for unit tests.""" | ||
| return frappe._dict( | ||
| { | ||
| "doctype": "Sales Invoice", | ||
| "einvoice_embedded_document": "", | ||
| "items": [], | ||
| **kwargs, | ||
| } | ||
| ) | ||
|
|
||
|
|
||
| def make_embed_generator(invoice) -> EInvoiceGenerator: | ||
| """Return an ``EInvoiceGenerator`` with an empty Drafthorse document for unit tests.""" | ||
| generator = EInvoiceGenerator( | ||
| profile=EInvoiceProfile.EN16931, | ||
| invoice=invoice, | ||
| company=frappe._dict(name="Test Co"), | ||
| customer=frappe._dict(name="Test Customer", supplier_numbers=[]), | ||
| ) | ||
| generator.doc = Document() | ||
| return generator | ||
|
|
||
|
|
||
| def mock_file_doc(spec: MockFileSpec) -> frappe._dict: | ||
| """Return a mock **File**-shaped dict matching *spec*.""" | ||
| return frappe._dict( | ||
| name=spec.name, | ||
| file_url=spec.file_url, | ||
| is_remote_file=spec.is_remote, | ||
| get_content=lambda *args, content=spec.content, **kwargs: content, | ||
| ) | ||
|
|
||
|
|
||
| def _element_text(value) -> str: | ||
| """Return the text content of a Drafthorse XML element wrapper.""" | ||
| if hasattr(value, "_text"): | ||
| return value._text | ||
| return str(value) | ||
|
|
||
|
|
||
| def assert_embed_attachment_result( | ||
| generator: EInvoiceGenerator, | ||
| expect: EmbedAttachmentExpectations, | ||
| *, | ||
| mock_content: bytes | None = None, | ||
| ) -> None: | ||
| """Assert ARD 916 nodes on *generator* match *expect*.""" | ||
| refs = generator.doc.trade.agreement.additional_references.children | ||
|
|
||
| if len(refs) != expect.reference_count: | ||
| raise AssertionError(f"expected {expect.reference_count} ARD nodes, got {len(refs)}") | ||
|
|
||
| if expect.reference_count == 0: | ||
| return | ||
|
|
||
| ref = refs[0] | ||
|
|
||
| if expect.type_code and str(ref.type_code) != expect.type_code: | ||
| raise AssertionError(f"expected TypeCode {expect.type_code!r}, got {ref.type_code!r}") | ||
|
|
||
| if expect.issuer_assigned_id and _element_text(ref.issuer_assigned_id) != expect.issuer_assigned_id: | ||
| raise AssertionError( | ||
| f"expected IssuerAssignedID {expect.issuer_assigned_id!r}, got {ref.issuer_assigned_id!r}" | ||
| ) | ||
|
|
||
| attached_object = ref.attached_object | ||
| attached_content = getattr(attached_object, "_text", None) if attached_object else None | ||
|
|
||
| if expect.has_attached_object is True: | ||
| if not attached_content: | ||
| raise AssertionError("expected attached_object payload on ARD 916") | ||
| if expect.mime_type and attached_object._mime_code != expect.mime_type: | ||
| raise AssertionError(f"expected MIME {expect.mime_type!r}, got {attached_object._mime_code!r}") | ||
| if expect.filename and attached_object._filename != expect.filename: | ||
| raise AssertionError(f"expected filename {expect.filename!r}, got {attached_object._filename!r}") | ||
| if mock_content is not None and attached_content != as_base_64(mock_content): | ||
| raise AssertionError("expected base64 content from mock file bytes") | ||
|
|
||
| if expect.has_attached_object is False and attached_content: | ||
| raise AssertionError("expected no attached_object payload on ARD 916") | ||
|
|
||
| if expect.uri_id and _element_text(ref.uri_id) != expect.uri_id: | ||
| raise AssertionError(f"expected URIID {expect.uri_id!r}, got {ref.uri_id!r}") | ||
|
|
||
|
|
||
| _CII_ATTACHMENT_XPATH = "//ram:AttachmentBinaryObject" | ||
| _CII_NS = {"ram": "urn:un:unece:uncefact:data:standard:ReusableAggregateBusinessInformationEntity:100"} | ||
|
|
||
|
|
||
| def make_minimal_pdf_bytes() -> bytes: | ||
| """Return a tiny valid PDF for hybrid attach_xml_to_pdf tests.""" | ||
| from pypdf import PdfWriter | ||
|
|
||
| writer = PdfWriter() | ||
| writer.add_blank_page(width=72, height=72) | ||
| buffer = io.BytesIO() | ||
| writer.write(buffer) | ||
| return buffer.getvalue() | ||
|
|
||
|
|
||
| def extract_attachment_binary_objects_from_cii_xml( | ||
| xml_bytes: bytes, | ||
| ) -> list[tuple[str, str, str]]: | ||
| """Return ``(filename, mime_code, base64_payload)`` for each CII annex.""" | ||
| root = etree.fromstring(xml_bytes) | ||
| attachments: list[tuple[str, str, str]] = [] | ||
|
|
||
| for element in root.xpath(_CII_ATTACHMENT_XPATH, namespaces=_CII_NS): | ||
| attachments.append((element.get("filename") or "", element.get("mimeCode") or "", element.text or "")) | ||
|
|
||
| return attachments |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Custom fields should likely be (re-)created through a patch or bumping
execute:from eu_einvoice.install import after_install; after_install() # 17to#18There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it also make sense to include the custom field sync in
after_migrateor is the patch bump the preferred solution? @barredterraThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Strictly speaking, I don't think we actually need the
after_migratepart, since it does during every update what is really only necessary during the first update (via patch).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed in bc47ed5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have expected removal of
after_migratelogic because it should now be redundant, right?