microsoft/BitNet 및 LLM 성능 지표

bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).

The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. More details will be provided soon.

Installation

Requirements

• python>=3.9
• cmake>=3.22
• clang>=18
  • For Windows users, install Visual Studio 2022. In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):
    • Desktop-development with C++
    • C++-CMake Tools for Windows
    • Git for Windows
    • C++-Clang Compiler for Windows
    • MS-Build Support for LLVM-Toolset (clang)
  • For Debian/Ubuntu users, you can download with Automatic installation script
  • bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
• conda (highly recommend)

https://github.com/microsoft/BitNet?fbclid=IwY2xjawGCQ8xleHRuA2FlbQIxMQABHb4IJT_XiICApICLvNQGVuvMCAwVxFUS1v3gc-VYZSVZQXNeiypalvB6vQ_aem_Yrdjs47e_iN_Eegs-5LQVw

GitHub - microsoft/BitNet: Official inference framework for 1-bit LLMs

성능 지표 설명
PPL (Perplexity):
PPL은 언어 모델이 얼마나 잘 텍스트를 예측하는지를 나타내는 척도로, 낮을수록 더 좋은 성능을 의미합니다. Bitnet의 700M 모델은 12.33의 perplexity를 기록하였는데, 이는 해당 모델이 예측 작업에서 상대적으로 낮은 혼란도를 가진다는 것을 나타냅니다.

ARCe (ARC Easy):
ARCe는 **ARC (AI2 Reasoning Challenge)**에서 Easy 섹션의 평가 점수를 나타냅니다. 이는 모델의 일반적인 추론 능력을 평가하는 기준 중 하나로, Bitnet의 700M 모델은 54.7의 점수를 기록하였습니다.

ARCc (ARC Challenge):
ARCc는 ARC의 Challenge 섹션에서의 성능을 측정한 것으로, 이 섹션은 좀 더 어려운 추론 문제를 포함하고 있습니다. Bitnet 700M 모델은 23.0 점수를 기록하였습니다.

HS (HellaSwag):
HS는 HellaSwag 데이터셋에서의 성능을 나타내는 지표로, 이는 주로 문맥을 바탕으로 후속 행동이나 문장을 예측하는 작업에 사용됩니다. Bitnet 700M 모델은 37.0의 성능을 기록하였습니다.

BQ (BoolQ):
BQ는 BoolQ (Boolean Questions) 데이터셋에서의 성능을 나타냅니다. 이 데이터셋은 질문에 대해 참(True) 또는 거짓(False)으로 응답하는 방식으로 모델의 이진 분류 능력을 평가합니다. Bitnet의 700M 모델은 60.0의 점수를 기록했습니다.

OQ (OpenQA):
OQ는 OpenQA 작업에서 모델의 성능을 평가한 지표입니다. OpenQA는 주어진 질문에 대해 답변을 생성하는 작업으로, Bitnet 700M 모델은 20.2의 성능을 기록했습니다.

PQ (PiQA):
PQ는 PiQA (Physical Interaction QA) 데이터셋에서의 성능을 나타냅니다. PiQA는 물리적인 상호작용에 대한 질문을 다루며, Bitnet 700M 모델은 68.9의 점수를 기록하였습니다.

WGe (WinoGrande):
WGe는 WinoGrande 데이터셋에서의 성능을 측정한 지표로, 이 데이터셋은 상식적인 추론을 요구하는 복잡한 문제들을 다룹니다. Bitnet 700M 모델은 54.8의 점수를 기록했습니다.

Avg FP16 700M (Average Floating Point 16 performance):
Avg FP16 700M은 모델의 700M 매개변수 버전에서의 평균 성능을 나타냅니다. FP16(16비트 부동소수점)은 주로 연산 속도와 메모리 사용 효율성을 높이기 위해 사용하는 방식입니다. Bitnet 700M 모델의 평균 성능은 45.5로 보고되었습니다.