Support MiniMax-M2 in TurboMind engine #4343
Conversation
|
MiniMax-M2.1-AWQ 这个模型在 hugginface hub 或者 model scope 上有么？ |
| return x.view(-1, head_num, 2, size_per_head // 2).transpose(2, 3).reshape(x.shape) | ||
|
|
||
|
|
||
| def permute_v2_partial(x: torch.Tensor, size_per_head: int, rotary_dim: int): |
There was a problem hiding this comment.
There is no need to "partially" permute the Q/K. In the partial rotary embedding case, Q/K are still aligned even if full permutation is applied. Setting dim in RopeParam correctly is enough.
| // The gate bias tensor may be stored in the model's data_type (e.g. float16), | ||
| // but the kernel expects float32. Convert and cache on first use. | ||
| if (!router_bias_f32_) { | ||
| const auto bias_size = moe.gate.bias.size(); |
There was a problem hiding this comment.
moe.gate.bias is a per-layer weight, so only the first layer will be cached and resued currently.
There was a problem hiding this comment.
// The gate bias tensor may be stored in the model's data_type (e.g. float16),
// but the kernel expects float32. Convert and cache per layer.
auto& cached = router_bias_f32_[p.layer_id];
if (!cached) {
const auto bias_size = moe.gate.bias.size();
const auto src_2d = moe.gate.bias.view({1, bias_size});
Tensor dst_2d{{1, bias_size}, kFloat, kDEVICE};
invokeCastFloat2D(src_2d, dst_2d, st);
sync_check_cuda_error();
cached = dst_2d.buffer();
}
router_bias = cached.data();
3 participants
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
为V100添加turbomind推理m2.1。
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=2,3,4,5,6,8,7,9 lmdeploy serve api_server /mnt/data6/models/MiniMax-M2.1-AWQ --tp 8 --log-level INFO --dtype float16 --backend turbomind --reasoning-parser qwen-qwq --cache-max-entry-count 0.9
复杂问题胡言乱语：中文知识性的题目大概率几十字后胡言乱语，比如列出梁山108将，推理性的好点，比如算24点：1、4、5、6，耗费2到10k token有3分之2的机会正确。
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist