Resource & Memory Management

Control GPU/CPU memory consumption during compilation and inference: weight streaming, dynamic memory allocation, and low-CPU-memory compilation.

• Resource Management
• Engine Caching
• Example: Engine Caching
• Example: Engine Caching (BERT)
• Example: Weight Streaming
• Example: Dynamic Memory Allocation
• Example: Low CPU Memory Compilation