Skip to content

Commit fcc3718

Browse files
authored
Add sampling defaults (#1947)
## Motivation Model quality issues ### Manual Testing TODO
1 parent 8ccfd7f commit fcc3718

118 files changed

Lines changed: 1127 additions & 2 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.mlx_typings/mlx_lm/sample_utils.pyi

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,10 @@ def make_logits_processors(
4848
logit_bias: Optional[Dict[int, float]] = ...,
4949
repetition_penalty: Optional[float] = ...,
5050
repetition_context_size: Optional[int] = ...,
51+
presence_penalty: Optional[float] = ...,
52+
presence_context_size: Optional[int] = ...,
53+
frequency_penalty: Optional[float] = ...,
54+
frequency_context_size: Optional[int] = ...,
5155
) -> list[Callable[[mx.array, mx.array], mx.array]]:
5256
"""
5357
Make logits processors for use with ``generate_step``.

resources/inference_model_cards/mlx-community--DeepSeek-V3.1-4bit.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,9 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 405874409472
16+
17+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/main/generation_config.json
18+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.1/discussions/19
19+
[sampling_defaults]
20+
temperature = 0.6
21+
top_p = 0.95

resources/inference_model_cards/mlx-community--DeepSeek-V3.1-8bit.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,9 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 765577920512
16+
17+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.1/blob/main/generation_config.json
18+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.1/discussions/19
19+
[sampling_defaults]
20+
temperature = 0.6
21+
top_p = 0.95

resources/inference_model_cards/mlx-community--DeepSeek-V3.2-4bit.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,9 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 378086226621
16+
17+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/generation_config.json
18+
# Source: https://docs.vllm.ai/projects/recipes/en/latest/DeepSeek/DeepSeek-V3_2.html
19+
[sampling_defaults]
20+
temperature = 1.0
21+
top_p = 0.95

resources/inference_model_cards/mlx-community--DeepSeek-V3.2-8bit.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,9 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 755957120916
16+
17+
# Source: https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/generation_config.json
18+
# Source: https://docs.vllm.ai/projects/recipes/en/latest/DeepSeek/DeepSeek-V3_2.html
19+
[sampling_defaults]
20+
temperature = 1.0
21+
top_p = 0.95

resources/inference_model_cards/mlx-community--GLM-4.5-Air-8bit.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,8 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 122406567936
16+
17+
# Source: https://docs.z.ai/api-reference/llm/chat-completion
18+
[sampling_defaults]
19+
temperature = 0.6
20+
top_p = 0.95

resources/inference_model_cards/mlx-community--GLM-4.5-Air-bf16.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,8 @@ context_length = 131072
1313

1414
[storage_size]
1515
in_bytes = 229780750336
16+
17+
# Source: https://docs.z.ai/api-reference/llm/chat-completion
18+
[sampling_defaults]
19+
temperature = 0.6
20+
top_p = 0.95

resources/inference_model_cards/mlx-community--GLM-4.7-4bit.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,10 @@ context_length = 202752
1313

1414
[storage_size]
1515
in_bytes = 198556925568
16+
17+
# Source: https://huggingface.co/zai-org/GLM-4.7
18+
# Source: https://unsloth.ai/docs/models/glm-4.7-flash
19+
# Source: https://docs.z.ai/api-reference/llm/chat-completion
20+
[sampling_defaults]
21+
temperature = 1.0
22+
top_p = 0.95

resources/inference_model_cards/mlx-community--GLM-4.7-6bit.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,10 @@ context_length = 202752
1313

1414
[storage_size]
1515
in_bytes = 286737579648
16+
17+
# Source: https://huggingface.co/zai-org/GLM-4.7
18+
# Source: https://unsloth.ai/docs/models/glm-4.7-flash
19+
# Source: https://docs.z.ai/api-reference/llm/chat-completion
20+
[sampling_defaults]
21+
temperature = 1.0
22+
top_p = 0.95

resources/inference_model_cards/mlx-community--GLM-4.7-8bit-gs32.toml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,10 @@ context_length = 202752
1313

1414
[storage_size]
1515
in_bytes = 396963397248
16+
17+
# Source: https://huggingface.co/zai-org/GLM-4.7
18+
# Source: https://unsloth.ai/docs/models/glm-4.7-flash
19+
# Source: https://docs.z.ai/api-reference/llm/chat-completion
20+
[sampling_defaults]
21+
temperature = 1.0
22+
top_p = 0.95

0 commit comments

Comments
 (0)