kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-bs4-ctx1024
Updated
•
25
Large MoE Architecture Search (1B-2B)
Systematic search for 1B-2B MoE models. Best: bs=1, ctx=2048 achieves 0.32 loss. Top-8 routing beats top-2.
-
-
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-bs2-ctx2048
Updated • 26 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-bs2-ctx1024
Updated • 23 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-bs1-ctx2048
Updated • 34 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-bs1-ctx1024
Updated • 18 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr1e-03
Updated • 25 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr5e-04
Updated • 32 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr3e-04
Updated • 25 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr2e-04
Updated • 23 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr1e-04
Updated • 33 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr5e-05
Updated • 19 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr3e-05
Updated • 20 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr1e-05
Updated • 23 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b-lr5e-06
Updated • 22 -
kshitijthakkar/moe-1422m-667m-8x2-10L-large-wide-1.5b
Updated • 11 -
kshitijthakkar/moe-1687m-781m-12x4-16L-large-deep-1.5b
Updated • 5 -
kshitijthakkar/moe-1083m-554m-16x2-8L-large-moe-1.3b-top2
Updated • 9 -
kshitijthakkar/moe-1083m-781m-16x8-8L-large-moe-1.3b
Updated • 17 -
kshitijthakkar/moe-1002m-399m-8x2-16L-large-moe-1b
Updated • 33 -
kshitijthakkar/large-moe-inference-benchmark
Viewer • Updated • 38 • 13