in

A project helps to detect the mathematical formula from the given picture and the same formula is extracted and converted into the latex code


The goal of this project is to create a learning based system that takes an image of a math formula and returns corresponding LaTeX code.

  1. above)
  2. Download the weights.pth file from my Google Drive and place it in the checkpoints directory

Thanks to @katie-lim, you can use a nice user interface as a quick way to get the model prediction. Just call the GUI with python gui.py. From here you can take a screenshot and the predicted latex code is rendered using MathJax and copied to your clipboard.

If the model is unsure about the what’s in the image it might output a different prediction every time you click “Retry”. With the temperature parameter you can control this behavior (low temperature will produce the same result).

Alternatively you can use pix2tex.py with similar functionality as gui.py, only as command line tool. In this case you don’t need to install PyQt5. Using this script you can also parse already existing images from the disk.

Note: As of right now it works best with images of smaller resolution. Don’t zoom in all the way before taking a picture. Double check the result carefully. You can try to redo the prediction with an other resolution if the answer was wrong.

Update: I have trained an image classifier on randomly scaled images of the training data to predict the original size.
This model will automatically resize the custom image to best resemble the training data and thus increase performance of images found in the wild. To use this preprocessing step, all you have to do is download the second weights file mentioned above. You should be able to take bigger (or smaller) images of the formula and still get a satisfying result

model_architecture

1] encoder with a ResNet backbone and a Transformer [2] decoder.

wikipedia, arXiv. We also use the formulae from the I2L-140k dataset.

lucidrains, rwightman, im2markup, arxiv_leaks, pkra: Mathjax, harupy: snipping tool

GitHub

View Github


Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

GIPHY App Key not set. Please check settings

Why You Should Use Typescript Over Javascript?

Learning in Public and Getting Opportunities