nekomata-14b

Abstract

We conduct continual pre-training of qwen-14b on 66B tokens from a mixture of Japanese and English datasets. The continual pre-training significantly improves the model's performance on Japanese tasks. It also enjoys the following great features provided by the original Qwen model. * The inclusive Qwen vocabulary (vocab size > 150k) enables the model to processs Japanese texts much more efficiently than the previously released youri series. * The model supports a maximum sequence length of 8192. The name nekomata comes from the Japanese word 猫又/ねこまた/Nekomata, which is a kind of Japanese mythical creature (妖怪/ようかい/Youkai).