Skip to content

Add str_length built-in check for PySpark backend#1709

Open
marrov wants to merge 4 commits into
unionai-oss:mainfrom
marrov:bugfix/pyspark-str-length
Open

Add str_length built-in check for PySpark backend#1709
marrov wants to merge 4 commits into
unionai-oss:mainfrom
marrov:bugfix/pyspark-str-length

Conversation

@marrov

@marrov marrov commented Jun 25, 2024

Copy link
Copy Markdown

Adds a builtin-check for str_length for the pyspark.sql backend, solving these issues:

Signed-off-by: Marc Rovira <marc.rovira@electrolux.com>
@codecov

codecov Bot commented Jun 25, 2024

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 40.00000% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.46%. Comparing base (812b2a8) to head (b002e70).
⚠️ Report is 572 commits behind head on main.

Files with missing lines Patch % Lines
pandera/backends/pyspark/builtin_checks.py 40.00% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1709      +/-   ##
==========================================
- Coverage   94.28%   93.46%   -0.82%     
==========================================
  Files          91      117      +26     
  Lines        7013     8856    +1843     
==========================================
+ Hits         6612     8277    +1665     
- Misses        401      579     +178     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cosmicBboy

Copy link
Copy Markdown
Collaborator

thanks @marrov ! please run pre-commit to fix linter errors: https://pandera.readthedocs.io/en/stable/CONTRIBUTING.html#set-up-pre-commit

marrov added 2 commits June 27, 2024 08:55
Signed-off-by: Marc Rovira <54272586+marrov@users.noreply.github.com>
@marrov

marrov commented Jun 27, 2024

Copy link
Copy Markdown
Author

thanks @marrov ! please run pre-commit to fix linter errors: https://pandera.readthedocs.io/en/stable/CONTRIBUTING.html#set-up-pre-commit

Yes, sorry for that I added it too hastily. Should be fixed now @cosmicBboy.

@cosmicBboy

Copy link
Copy Markdown
Collaborator

thanks @marrov ! sorry to extend this a little longer, but this check needs unit tests, which can live here: https://github.com/unionai-oss/pandera/blob/main/tests/pyspark/test_pyspark_check.py

@RedZapdos123

Copy link
Copy Markdown
Contributor

Opened a new PR: #2288 as this PR fails CI tests, and seems to be abandoned since last 2 years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants