Tinygrad

For one thing in between a pytorch and a karpathy/micrograd

This will likely likely perchance also now not be the most high-quality deep discovering out framework, but it’s a deep discovering out framework.

The sub 1000 line core of it’s in tinygrad/

On account of its terrifying simplicity, it objectives to be the most keen framework to add recent accelerators to, with make stronger for each inference and training. Make stronger the easy traditional ops, and also you get SOTA vision units/efficientnet.py and language units/transformer.py units.

We’re working on make stronger for the Apple Neural Engine and the Google TPU in the accel/ folder. Sooner or later, we are capable of construct customized hardware for tinygrad, and this may occasionally likely likely be blindingly mercurial. Now, it’s behind.

Installation

pip3 set up git+https://github.com/geohot/tinygrad.git --make stronger

# or for construction
git clone https://github.com/geohot/tinygrad.git
cd tinygrad
python3 setup.py make

Instance

from tinygrad.tensor import Tensor

x = Tensor.learn about(3)
y = Tensor([[2.0,0,-2.0]])
z = y.matmul(x).sum()
z.backward()

print(x.grad)  # dz/dx
print(y.grad)  # dz/dy

Identical instance in torch

import torch

x = torch.learn about(3, requires_grad=Correct)
y = torch.tensor([[2.0,0,-2.0]], requires_grad=Correct)
z = y.matmul(x).sum()
z.backward()

print(x.grad)  # dz/dx
print(y.grad)  # dz/dy

Neural networks?

It turns out, a tight autograd tensor library is 90% of what it’s likely you will likely perchance like for neural networks. Add an optimizer (SGD, RMSprop, and Adam implemented) from tinygrad.optim, write some boilerplate minibatching code, and also you’ve gotten all it’s likely you will likely perchance like.

Neural network instance (from test/test_mnist.py)

from tinygrad.tensor import Tensor
import tinygrad.optim as optim

class TinyBobNet:
  def __init__(self):
    self.l1 = Tensor.uniform(784, 128)
    self.l2 = Tensor.uniform(128, 10)

  def forward(self, x):
    return x.dot(self.l1).relu().dot(self.l2).logsoftmax()

mannequin = TinyBobNet()
optim = optim.SGD([model.l1, model.l2], lr=0.001)

# ... and total deal with pytorch, with (x,y) recordsdata

out = mannequin.forward(x)
loss = out.mul(y).indicate()
optim.zero_grad()
loss.backward()
optim.step()

GPU and Accelerator Make stronger

tinygrad helps GPUs through PyOpenCL.

from tinygrad.tensor import Tensor
(Tensor.ones(4,4).gpu() + Tensor.ones(4,4).gpu()).cpu()

ANE Make stronger?! (broken)

If all you ought to enact is ReLU, you are in luck! You may likely perchance likely enact very mercurial ReLU (in any case 30 MEGAReLUs/sec confirmed)

Requires your Python to be signed with ane/lib/sign_python.sh to add the com.apple.ane.iokit-user-get entry to entitlement, which also requires amfi_get_out_of_my_way=0x1 to your boot-args. Manufacture the library with ane/lib/construct.sh

from tinygrad.tensor import Tensor

a = Tensor([-2,-1,0,1,2]).ane()
b = a.relu()
print(b.cpu())

Warning: enact now not rely on the ANE port. It segfaults usually. So ought to you had been doing one thing crucial with tinygrad and wished to employ the ANE, you personal a terrifying time.

Including an accelerator

You’d ought to make stronger 14 top quality ops:

Relu, Log, Exp                  # unary ops
Sum, Max                        # gash again ops (with axis argument)
Add, Sub, Mul, Pow              # binary ops (with broadcasting)
Reshape, Transpose, Nick       # flow ops
Matmul, Conv2D                  # processing ops

Whereas more ops will likely be added, I mediate this injurious is true.

ImageNet inference

In spite of being exiguous, tinygrad helps the corpulent EfficientNet. Pass in a characterize to gaze what it’s.

ipython3 examples/efficientnet.py https://media.istockphoto.com/photographs/fowl-characterize-id831791190

Or, ought to you’ve gotten a webcam and cv2 installed

ipython3 examples/efficientnet.py webcam

PROTIP: Negate “GPU=1” atmosphere variable ought to you are going to deal with this to poke faster.

PROPROTIP: Negate “DEBUG=1” atmosphere variable ought to you are going to deal with to survey why it be behind.

tinygrad helps GANs

Peep examples/mnist_gan.py

tinygrad helps yolo

Peep examples/yolov3.py

The promise of little

tinygrad will repeatedly be below 1000 lines. If it’s now not, we are capable of revert commits till tinygrad turns into smaller.

Working tests