Gigatoken tokenizer hits 24.5 GB/s; Cursor Router cuts inference costs 30–50%

Gigatoken, an MIT-licensed Rust BPE tokenizer, encodes text at 24.53 GB/s on a 144-core AMD EPYC processor, achieving 989x speedup over HuggingFace Tokenizers and 681x over tiktoken (MarkTechPost). The gains stem from hand-optimized multithreaded Rust implementation, offering material impact for token-rate-limited inference pipelines and batch processing workloads.

Cursor has made Cursor Router generally available, a request-level classifier that routes queries to the most suitable model based on complexity and task domain, delivering frontier-quality output at 60% savings in A/B tests and 30–50% cost reduction in production deployment (MarkTechPost). In parallel, Anthropic released Claude Security as a beta plugin for Claude Code - a multi-agent vulnerability scanner operating from the terminal that flags issues and auto-generates patches for review (MarkTechPost).