`dy scan` doesn't return all the rows from the table

[farzadmf]

Hi, I have a table that I know has around 6,000 rows (in AWS console, I see that)

But no matter the limit I pass to dy scan, it's still returning 1956 rows for some reason. Tried:

dy scan -t my -l 10000
dy scan -t my -l 100000

So, it seems that it respects the limit up to a certain threhold only

[StoneDot]

I'm sorry for the delayed response.

Actually, you can use the dy import command to retrieve all items in your table. The main purpose of dy scan is to inspect sample items in your table (i.e., to check what typical items look like).

Would dy import work for your use case?

[farzadmf]

The main purpose of dy scan is to inspect sample items in your table

Oh, that's a bit confusing then 😆 ; I thought scan corresponds to scan in DDB (given that we also have query to match querying in DDB)

Would dy import work for your use case?

I'm guessing you mean dy export? If that's the case, yes, I think that would work; although it might be a tiny bit inconvenient as it needs a a file to write to (but definitely not a deal breaker)

And I must say ... I didn't know about dy export.

If you feel like my PR is not necessary, I can cancel it for sure

[StoneDot]

I apologize for the delayed response and the confusion in my previous message. You're absolutely right - I meant dy export, not dy import.

I also saw your comment about the --limit approach and the concern about adding friction to dy usage. You make a valid point - if a user specifies --limit 1000, it's reasonable to expect 1000 results, even if that requires pagination internally.

Let me clarify my thinking on the separation between dy scan and dy export: The DynamoDB Scan API itself requires pagination, and the question is how we should wrap and expose that in dynein. I think there are two main use cases:

1. Inspecting sample items - checking what kind of data is in your table → dy scan
2. Retrieving all data - getting the complete dataset for processing → dy export

Given this separation, the need for explicit pagination in dy scan might be reduced, but I don't think it's completely unnecessary. There could still be cases where users want to paginate through scan results interactively.

Regarding dy export and piping to other commands, I've opened #282 to discuss supporting stdout output directly (currently /dev/stdout doesn't work due to truncate issues).

I really appreciate you taking the time to work on this PR and engage in this discussion. Would you be open to continuing the design discussion in #279 to explore what would work best for dynein users?

[farzadmf]

Thank you for the detailed response. I think the main point of confusion happens for someone relatively familiar with DDB APIs:

• If someone isn't aware that DDB API itself provides query and scan, then using scan to mean "I'm gonna scan this table to see what it has" might make sense
• But, if someone has worked with, say, aws CLI, which has both aws dynamodb scan and aws dynamodb query, it becomes very confusing that:
  • dy query corresponds to aws dynamodb query
  • dy scan is a totally different thing and has nothing to do with aws dynamodb scan

That being said, I guess we don't want to break existing users (in case they rely on the existing scan behavior).

Thoughts/discussion points:

• --paginate could be an option we can use for scan, but it could be something allowing different values. So, for example, doing dy scan --paginate all would retrieve all records TBH, I can't think of what other values it can have. I was thinking that it could accept number of pages it will go through (something like dy scan --paginate 2), but then what could be the use case? Why would someone want to go through 2 pages only?
• Or --paginate can work as a boolean to mean "yes, I know you'll need to do some stuff internally to keep track of pages etc. Do it yourself and just give me the --limit records I asked you to"
• As for Support output to stdout in dy export command #282 , supporting stdout would be nice to have in export, although I'd say doing --output-file /dev/stdout is confusing me again. Being a SUPER heavy terminal user, I honestly don't remember a time where I had to explicitly tell a command to write to /dev/stdout (/dev/null is a different story and has its use cases, but explicitly specifying /dev/stdout is counter-intuitive)
• But again, given that export is something "unique" to dy, I see it having a different interface and having --output-file with an actual file does make sense for "exporting" things.

Sorry for the super long comment, but IMO, it's better to focus on dy scan here and how we can make it match aws dynamodb scan, which, by default does paginate and one needs to explicitly say it to --no-paginate to change the behavior

Open to thoughts of course

[StoneDot]

Hi @farzadmf,

First of all, I sincerely apologize for the extremely delayed response. I appreciate your patience and continued interest in this discussion.

After carefully considering your points and the broader implications, I think you're absolutely right that --limit should behave as users intuitively expect: specifying the number of records they want to receive.

I'd like to propose the following approach:

Current implementation (Phase 1):

• --limit represents the number of records to retrieve (not DynamoDB's internal limit)
• When --limit is specified, pagination is automatically enabled to fetch the requested number of records
• Users can opt out with --no-paginate if they only want a single page
• Without --limit, the current behavior is maintained (single page only)
• To retrieve all records from a table, use --paginate without --limit

Future extension (Phase 2 - when we add filter expression support):

• Introduce --evaluation-limit to control the maximum number of items DynamoDB reads per request before applying filters (equivalent to DynamoDB API's Limit parameter)
• Keep --limit as the number of records returned after filtering
• This separation makes the semantics clear and prevents confusion

The key distinction:

• --limit: Controls the final result count (what users receive)
• --evaluation-limit: Controls how many items DynamoDB reads per request before applying filters

Example usage after Phase 2:

dy scan --limit 100 --filter "age > 20"  # Get 100 filtered results
dy scan --evaluation-limit 1000 --filter "age > 20"  # Read up to 1000 items per page, then filter
dy scan --limit 100 --evaluation-limit 1000 --filter "age > 20"  # Both
dy scan --paginate --filter "age > 20"  # Get all filtered results

This approach:

1. Aligns with user expectations (SQL-like LIMIT behavior)
2. Maintains backward compatibility for existing use cases
3. Provides a clear path for future filter expression support
4. Clearly documents the behavior to avoid confusion

What do you think about this direction?