Neural Architecture Search (NAS)

NAS automatically designs neural network architectures, discovering models that match or exceed human-designed networks. Reduce architecture engineering time by 80% while achieving state-of-the-art results on computer vision, NLP, and other tasks.

Why NAS?

Better Performance

• Discovers architectures humans wouldn't think of
• Often matches or beats hand-designed models
• EfficientNet (NAS) outperforms ResNet, VGG
• BERT-like models discovered via NAS

Save Engineering Time

• Manual architecture design takes weeks/months
• NAS automates this process
• 80% reduction in ML engineering effort
• Focus on data, problem formulation instead

Hardware-Aware Design

• Optimize for specific hardware (mobile, edge, GPU)
• Balance accuracy and latency/memory
• MobileNet, EfficientNet optimized for mobile

NAS Algorithms

1. Reinforcement Learning (RL) NAS

• RNN controller generates architectures
• Train each, use accuracy as reward
• Used in: NASNet, EfficientNet
• Computationally expensive (1000s GPU-days)

2. Evolutionary Algorithms

• Genetic algorithms mutate/crossover architectures
• Select best performing offspring
• AmoebaNet discovered via evolution
• More parallelizable than RL

3. Gradient-Based NAS (DARTS)

• Make architecture search differentiable
• Optimize architecture with gradient descent
• 100-1000x faster than RL/evolution
• 1-2 GPU-days instead of 1000s
• Most practical for production

4. One-Shot NAS

• Train single supernet, sample sub-architectures
• Very fast search (hours instead of days)
• SPOS, OFA (Once-for-All)

Search Spaces

Macro Search

• Search entire architecture (layer types, connections)
• More flexible, more search time
• Example: NASNet

Micro Search (Cell-based)

• Search for repeatable cell/block
• Stack discovered cells to form network
• Faster search, transfer across tasks
• Example: DARTS, EfficientNet

Hardware-Aware NAS

• Multi-objective optimization: accuracy + latency
• Optimize for specific hardware (iPhone, Jetson, TPU)
• MobileNet, EfficientNet variants
• Pareto frontier of accuracy vs efficiency

Applications

Computer Vision

• Image classification: EfficientNet (SOTA)
• Object detection: NAS-FPN
• Semantic segmentation: Auto-DeepLab

NLP

• Language models via NAS
• Efficient transformers
• Text classification architectures

Edge Deployment

• Design efficient models for mobile/edge
• Balance accuracy and inference time
• ProxylessNAS, FBNet

Implementation

Tools

• NAS-Bench-201: Benchmark for NAS research
• AutoKeras: Easy AutoML with NAS
• NNI (Microsoft): NAS + hyperparameter tuning
• DARTS: Differentiable NAS implementation

Process

1. Define Search Space: What operations, connections allowed?
2. Choose Search Strategy: RL, evolution, DARTS
3. Train & Evaluate: Run NAS (GPU-hours to GPU-days)
4. Retrain Best: Train discovered architecture from scratch

Challenges

• Computational Cost: Can be expensive (1000s GPU-hours)
• Solution: Use DARTS, one-shot NAS, or transfer learned architectures
• Search Space Design: Requires domain knowledge
• Solution: Use established search spaces (DARTS, NAS-Bench)

Results

• EfficientNet: 84.4% ImageNet top-1 (vs 79.8% ResNet-152), 8x fewer params
• NASNet: Matched SOTA on ImageNet (2017)
• AmoebaNet: 84.3% ImageNet via evolution
• DARTS: Comparable accuracy, 1000x faster search

Pricing

• DIY (DARTS): ₹50K-3L in compute (1-10 GPU-days)
• Custom NAS: ₹20-50L (engineering + compute)
• Use Pre-discovered: Free (EfficientNet, MobileNet)