Lambda Networks in PyTorch

Lambda Networks from the paper LambdaNetworks: Modeling Long-Range Interactions Without Attention by Bello et al. Adds support for Rank 3 and Rank 5 tensor inputs as well as initial implementation of ResNet-D with Lambda Convolutions.

Code is adapted from lucidrains implementation - https://github.com/lucidrains/lambda-networks

TODO:

• Add support for ResNet-RS

Installation

pip install --upgrade git+https://github.com/titu1994/lambda_networks_pt.git

Usage

There are three modules inside -

• lambda_module_1d.py: Implements Lambda Network block for Rank 3 input data (B, C, T)
• lambda_module_2d.py: Implements Lambda Network block for Rank 4 input data (B, C, H, W)
• lambda_module_3d.py: Implements Lambda Network block for Rank 5 input data (B, C, D, H, W)
• lambda_resnets.py : Implements Lambda ResNets (Using ResNet-D, Not ResNet-RS!) for Rank 4 input data (B, C, H, W)

import lambda_networks

# 1D Block, 2D Block or 3D Block
module = lambda_networks.LambdaLayer1D(
    dim=32, 
    dim_out=64,
    m=None,  # Use positive integer for Global Context using Lambda Layer. Represents "m" in the paper.
    r=None,  # Use positive integer for Local Context using Lambda Convolution. Represents "r" in the paper.
    dim_k=16,  # Dimension of key/query.
    dim_intra=1  # Intra-dimension "u" in the paper.
    heads=4,  # Number of heads. Represents "h" in the paper.
    implementation=0,  # Defaults to 0 generally, which implements the paper version of n-D Lambda using (n+1)-D Convolution.
)

# Lambda ResNet-D
model = lambda_networks.resnet_18(
    lambda_m: bool = False,  # Bool flag whether global context should be used or not. If set to True, pass in `input_size` as well to compute global context size per block.
    lambda_r: Optional[int] = None,  # Optional int, which if passed computes Local Context using Lambda Convolution. 
    lambda_k: int = 16,  # Dimension of key/query.
    lambda_u: int = 1,  # Intra-dimension "u" in the paper.
    lambda_heads: int = 4,  #  Number of heads. Represents "h" in the paper.
    input_size: Optional[int] = None,  # Optional int representing Height and Width of the image. Must be passed if `lambda_m` is set to True.
)

Tests

Tests will perform CPU only checks if there are no GPUs. If GPUs are present, will run all tests once for cuda:0 as well.

pytest tests/

Requirements

• pytorch >= 1.7.1. Older versions might work, not tested.
• einops - Required for LambdaLayer2D/3D and Lambda-ResNet-D. Not required for LambdaLayer1D.