SeetaResearch / Dragon

Summary:
This commit adds a feature to map memory and load data with offset.
Tensor takes mapped memory is readonly and will drop it on next mutation.

Summary:
This commit adds extra CUDA softmax kernels using warp reduce.
Warp reduce leads to better performance when dimension <= 256,
which is preferred for the recent vision transformers.

Summary:
This commit uses "math::Transpose" instead of "kernels::Transpose"
for more possible optimized routines in the future.

Summary:
This commit instantiates CUDA kernels by using constant dimensions
to enable the optimization during compiler-time.

Summary:
This commit allows transpose to compute in-place by leveraging buffer.
We also adds CRD mode for space-depth transpose (i.e., pixel shuffle).

Summary:
This commit removes the default cuBLAS tensor core math mode
when CUDA >= 11.0 on ampere devices to avoid the FP32 downcast math.

Summary:
This commit adds im2col operator to unfold input to depth.

Summary:
This commit renames triangular operator to trilu following the ONNX.

Summary:
This commit fixes the alpha and beta to 1/6 and 0.5 for hardswish,
the same behavior as ONNX scheme.

Summary:
This commit adds reverse or flip operator.

Summary:
This commit adds GELU activation to compute output
via approximate or naive mode.

Summary:
This commit adds scatter and gather operator
to remap elements along the given dimension of indices.

Summary:
The new frontend makes an union of two execution modes, while starts from
a single tensor class. Besides, it emits the operator execution through
a common path that works both for dragon and torch.

Summary:
This commit generalizes the fully-connected operation into GEMM,
and enhances the matmul operation via batched Dot, GEMV and GEMM.
New representations and attributes have been consistent with ONNX.

Summary:
For the purpose of consistency on getting CUDNN convolution algorithms,
CUDNN v6 (mainly relied by CUDA 8.0) is now dropped.

Summary:
This commit adds the sysconfig module to get the build information.
Build information is helpful to select tests or report issues.

Summary:
This commit adds 1D and 3D support for vision operators
via a generalized ND implementation.

Summary:
This commit fixes the two compiling issues on win32:
1. Remove the constexpr in generating protobuf headers.
2. Enforce the "/MT" flag to overwrite the "/MD" always.

Summary:
This commit tests the correctness of shape inference and data type
blended by autograph module.

Summary:
We found it unstable when defining CUB storages in a union.
More surveys should be taken to understand this issue.

Summary:
This commit moves the parser into ArgHelper which designed
to add the descriptors before.

Summary:
This commit feeds the repeated tensor arguments with
the entire array instead of the piecewise scalars.

Summary:
This commit redesigns the ``vm.onnx`` by referring the official repository.
Frontends and backends are aligned with identical API for dragon, torch and tensorrt.

Summary:
This commit removes the unused reference of GradientTape to
avoid the circular reference issue.

Summary:

This commit attaches input and output together in assign operators,
which fixes the missing input defs due to identity from input to output.

Summary:
This commit adds pseudo FP16 kernels with FP32 conversions
for DepthwiseConv2d and SyncBN operator.

Summary:
This commit adds a fallback with FP32 accumulator
for FP16 ReduceSum to avoid dropping too many small values.
Besides, FP16 kernels for arch < 530 are almost available.

Summary:
This commit adds the hardsigmoid, hardswish and swish op with specialized kernel,
there are widely used in MobileNetV3 and EfficientNet.

Summary:
This commit uses CopyMatrix to implement concat and split generically
instead of specialized kernels.

Summary:
This commit uses unique tensors to provide workspace data
to avoid the corruption between operator and kernel.

Summary:
This commit adds the linspace op for dragon, torch and tensorflow.
And, a workaround for truncated int interval is made to range/linspace (up to 2**57).

Summary:
This commit reimplements the cuda argmax/argmin via BlockReduce,
instead of the naive reduction in kernel loop.

Summary:
This commit adds the sort op for dragon, torch and tensorflow.
Besides, cuda implementation of topk op is now available.

Summary:
This commit uses local(thread or stream) workspace for Context,
which provides a more elegant way to dispatch kernels requiring scratch.
Besides, TF32 math type is provided as a cuDNN option for Ampere device.

Summary:
This commit adds the unique op for dragon, torch, tensorflow and onnx.
Besides, fixes the bug that gets the wrong workspace size in cached cudnn convolution.

Summary:
This commit reimplements the default shuffle policy of data reader with
sequential sampling (be consistent with DALI) instead of chunk permutation (MXNet solution).
Sequential sampling is tuned by argument ``initial_fill`` only, and works both for HDD and SSD.

Summary:
This commit renames the operator argument getter to ``GetArgument``
whatever an argument is single or repeated.

Summary:
This commit enforces the stream synchronization before dispatching NCCL collectives.
Otherwise, data corruption will happen due to the default value of ``NCCL_GROUP_CUDA_STREAM``
changed to 0 since CUDA 9.2, i.e., no explicit event waiting for unfinished kernels.

Summary:
This commit adds the ``memory_allocated`` API for ``dragon.Workspace``
to query the size of allocated memory (and optionally on a specified device).

Summary:
This commit enforces the algorithm finding even if the backward of filter or data
will not be executed. Otherwise, the empty algorithm will be encountered between
two cached operation with the same arguments and input shape.