Commit a5484b9
authored
Astrapy 2.0.0-preview: Tables and much more (#326)
* WIP on tables, descriptor refactor, dse compose
* database table methods: list/listnames sync+async done
* first table (unit) tests
* covering vector types in table defs
* wip on tolerant parse/coerce of table options
* table crd in place
* table info structures handle unsupported columns
* just introduced scalar-type enum
* check for ok:1 in all database class table/coll CD ops
* table index descriptor classes
* create (regular/vector) index; drop index
* better, shorter, privacy-oriented repr for Full/APIOptions/db classes
* more streamlined and secure hiding of tokens in repr and such
* repr for apicommander
* add support for sourceModel in vector index options
* add items to the cql type enum
* most table vanity methods
* finished table vanity methods; started refactor exceptions
* binary is handled at the payload coded layer
* removed the pointless {ok:1} returns
* remove check_exists and CollectionAlreadyExistsException
* rationalize table test dir; some table DML+testing
* fix timeout s-vs-ms bugs, simplify timeout internal management, unflaky timeout tests
* collection find_one uses API findOne command
* WIP table generics
* wip generics of collection
* tiny edit
* brought typing in all tests; add 1/4 typing-specific test
* add test for typing collections+tables
* rename the default Row and Document types
* started cursor2
* cursor2 rewritten and wired to collection find. distinct todo
* distinct+in cursors for collections + test/ruff
* refine cursor timeouts and replace CursorIsStartedException -> CursorException
* for_each, map (also rename overall_max_time_ms -> overall_timeout_ms)
* test of timeouts in cursor 'reduce' operations
* add typing test for map, etc
* table cursors, table's find and distinct methods
* improve class hierarchy for table column descriptors
* table converters, core logic and most converters
* TableSet class and its usage
* TableMap and its usage
* apitimestamp class, loss/y api optins known to payload conv, support in collections read/write
* TableDuration type and its wiring
* use stdlib ipaddress for 'inet' CQL type
* TableDate class and its use in converters
* TableTime type and its usage
* DataAPITimestamp covers the stringy rfc3339 formats; used in table converters
* DataAPIVector basics
* binary encoding of vectors in collections + api options to control it
* renamed wire-format options
* limit vector bin-enc for collections to insertions
* fix tests with the new tolerant-on-insert policy for types
* guard against empty-keyspace DBs without 'keyspace' and 'keyspaces' fields on devops responses
* add .region property to databases; optimize vector-path in coll pre/post process; adjust dbInfo tests
* move the pre-post-processing logic out of commander, into collections (tables todo)
* renamed option to 'unroll_iterables_to_list' and used in the general collection preproc route
* rename tpostprocessors; nested nan in lists/maps and its test
* table type-driven preprocessing wip (DataAPITimestamp missing + tests)
* most pre/post processing for tables (except using the custom dt option); missing wiring
* full table pre-post-processors, testing and use of options
* sensible repr and str for DataAPIVector
* converter agent for tables, with caching
* unexpected response exceptions renamed; result classes split T/C; full wiring of pre-post-processing for tables
* rename WireFormatOptions->SerdesOptions; better timeout messages that always give the number in ms
* info methods of async classes are async; adjust collections test re: datetime/DataAPITimestamp; made custom types hashable
* WIP on surfacing original timeouts
* wip 2 on surfacing the nominal timeout values
* surfacing the timeouts as the user configured them
* cursors' for_each and to_list error on closed cursors
* adjust tests to fetch_db_info signature; suspend dropIndex (the command is missing somehow)
* cursor to_list and for_each will error if cursor is closed
* equality method on CollectionDescriptor
* mostly reworked exception tree; added InvalidEnvironmentException
* rearrange and rename collection-specific exceptions
* rearranged exception tree
* final arrangement of exceptions all in astrapy.exceptions
* cursor.has_next can start iteration on IDLE cursor
* less unwieldy repr for XInsertManyResult if many items
* cursors don't accept project after map is set
* if/not/exists option and all the create/drop table/index commands ready
* renamed all TableX custom data types to DataAPIX
* drop_table_index now a Database method
* delete_many for tables; fixes and int.tests for distinct; int.tests for delete_many/one
* full options to (table)find_one + testing
* wip on the decimal management in jsonencoding
* full feature for the decimal-first flag in collections + unit tests
* integration testing of the decimal encoding and related options for collections
* unit test of table codec paths for decimal/number support
* remove CollectionNotFoundException; cleaned out a forgotten table drop_index method
* table deletion methods return None; removed associated XResult classes and related exceptions
* alterTable and tests
* revert to common CumulativeOperationException; remove Table{DeleteMany|UpdateMany}Exception and Table{Delete|Update}Result
* rename test modules tableX to dataapiX; factored the 'c' format for duration logic in a separate module
* wip DataAPIDuration format flip
* DataAPIDuration switched to the ISO-8601-with-signum notation
* table's count_documents, estimated_document_count methods + aliases + unit test
* removed outdated 'astrapy_latest_interface.py'
* table update_one method and test
* table insert_many sync + tests
* table insert_many, async + integration tests
* fix Decimal schema-hashing bug; added IT for update_one
* align cursor class names
* more terse repr of collections and tables
* improve error messages for multi-error DataAPIResponseExceptions
* cursors, reverted to .data_source common property for tables/collections
* v2.0 diagrams (abstractions, exceptions)
* DataAPIClient init and spawner methods all accept api_options besides top named params
* database's get_database_admin accepts admin_api_options parameter
* removed unnecessary token coercion in tests
* uniform api_options to all copy/clone/conversion methods
* remove useless token coercion steps
* minor (typing and typos)
* Table.alter() supports typing and returns the newly-typed table
* Table.alter() accepts a dict which it tries to parse into an Operation
* cursors' get_sort_vector behaves as dictated by serdes options for tables/collections
* include_similarity full support
* support for source_model in create collection
* add tests for paginated table fnid; rename SortDocuments to SortMode (with alias)
* switch to the 6-tier timeout options
* duration, use c-format in write path for lossless (for now)
* remove row-named count methods
* schema-based fillers for missing columns in row,
* table's list_indexes and list_index_names methods + test
* remove cursor .keyspace property
* database 'command' method can target no-keyspace; update param names and streamlined logic
* fixed some tests after the api and client minor changes
* all tests (except countDocuments) passing again
* rename max_time_ms to timeout_ms consistently
* refactored the per-method timeout override chains in a zero-respecting manner
* reinstated use of command() in a test
* DataAPITimestamp has friendlier repr
* adjusted the year/sign part of DataAPITimestamp parsing and stringifying
* tiny fix to timestamp parsing (+0000123 allowed, 0000123 no for year)
* adjusted DataAPIDate parsing and to_string to respect sign-of-year constraints as the API
* add test for the timeout overrides including passing zero
* added timeout-override and timeout-zero test for async collection
* tables, int. test restructure+enrich wip
* wip for tuple right typing in ids
* proper type handling in insertedIds (dict+tuple) and its testing
* a mostly complete integration test suite for tables
* DataAPIMap and DataAPISet __eq__ method behaves like regular maps and sets; completed read-then-write-all-types tests
* bugfix: token passthrough in the async_fetch_database_info
* standard table vectorize integration tests
* include sort vector, include similarity; covered by table vectorize tests
* more table IT coverage
* bigfix: equality of DataAPIMap
* I test of table command
* some more find IT for tables
* increase default per-req timeout to 10s
* tables, full IT coverage of find
* full findOne IT for tables
* full integration test coverage of table cursors
* wip3 naming timeouts in errors
* wip1 fine-align timeouts
* wip2 fine-align timeouts
* full alignment to timeout management
* split Create- vs List- TableIndexDefinition and TableIndexDescriptor
* index creation/list final API for tables
* CollectionOptions -> CollectionDefinition class rename
* from_dict -> _from_dict method consistently
* cursors, for_each and to_list start where consumtion stopped
* callback to for_each can stop early by returning False
* added a tiny bit of testing to table cursors with foreach
* added all collection cursor integration tests
* table class docstrings
* made some internal-method param required; remove any check/error on include_sort_vector and include_similarity
* docstring fixes; in part. move tokens out of client (unless admin example docstrings)
* table index creation new signature (for real)
* first quickstart and some more timezonecare in the readme
* table create_index has optional options param
* better readme quickstarts (t/c both)
* table.py docstrings: create_index, create_vector_index, list_index_names, list_indexes, alter (+async)
* table docstring wip: halfway throug find
* minor parameters to table-creation structures made optional; more table docstrings
* more docstrings to table.py
* sync table docstrings tier0 done
* full tier-0 table.py docstring, incl async
* more standardization of table.py docstrings
* TableVectorIndexOptions can be coerced from None; create_vector_index has optional options; minor docs improvement
* more docstring improvement around tables
* final datetime/timezone serdes options, completed
* adjusted tests in the wake of datetime tzinfo rework
* systematc, consistent rewrite of the options to copy and spawn methods for the various abstractions
* fix clipped datetime diagram picture
* conftest param naming update
* minor IT adjustments
* create/get[table/coll] docstrings done, vectorize IT improved
* adapt a vectorize test to DataAPIVector
* wip on createCollection arg unflattening
* create_collection signature rework + fluent interface, completed
* table creation docstring order fix
* column_type may be omitted for vector column type descriptor
* param name change in builder method; better repr for DataAPIMap
* AsyncDatabase: get_table and get_collection methods are not async functions anymore
* environment locked in APIOptions
* TODO cleanup in tests
* api_options: all docstrings; fixed a couple of spawn/api_option parameter names here and there
* datetime serdes example in readme
* a small apioptions example in readme
* minor api-option docstring changes
* docstring for date and vector
* docstring all of the custom data types
* more docstrings
* rename CursorState->FindCursorState; most cursor docstrings
* full cursor docstrings
* simplified drop_table and drop_collection to only accept the string name of the target to drop
* wip typing finds
* add type parameter to find methods for returned items
* more docstring; all collection vector writes abide by binary_encode_vectors setting
* disable (for now) collection vector binary encoding altogether
* all docstrings in table.py and database_info.py
* docstring for collection.py
* all database.py docstrings done
* table descriptors, docstrings ok
* all docstrings done
* primitive->scalar terminology consistently for columns
* rewritten CHANGES file
* mark version as preview; upgrade hcd docker-compose data api image
* comment a stupid flaky test
* another tricky async timeout test gone
* another tricky async timeout test gone 2
* disabled most of the timeout tests with httpsserver
* removed the alias SortMode->SortDocuments
* builder interface renamed to .builder()---.build() pattern
* renamed TableScalarColumnType -> ColumnType
* wip toward proper list/create split for collection def/descr
* split list/create table descriptors for real1 parent ab67b61 commit a5484b9
137 files changed
Lines changed: 36788 additions & 7997 deletions
File tree
- astrapy
- admin
- data_types
- data
- info
- table_descriptor
- utils
- exceptions
- settings
- utils
- pictures
- scripts
- tests
- dse_compose
- hcd_compose
- idiomatic
- integration
- unit
- tables
- integration
- unit
- vectorize_idiomatic
- integration
- unit
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
5 | 17 | | |
6 | 18 | | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
19 | 95 | | |
20 | 96 | | |
21 | 97 | | |
22 | | - | |
23 | 98 | | |
24 | | - | |
| 99 | + | |
25 | 100 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | 101 | | |
| 102 | + | |
30 | 103 | | |
31 | 104 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
37 | 149 | | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
38 | 158 | | |
39 | 159 | | |
40 | 160 | | |
| |||
0 commit comments