Percy: Automatic Receipt & Invoice Data Extraction

Percy ( is a web application that I built from March 2020 to February 2021. It allows accountants to automatically extract financial information from receipts and invoices by simply uploading a photograph of the document. The core technology that makes this happen is the combination of an advanced machine learning optical character recognition engine and a word categorisation system that finds the financial information in the extracted text. This is a challenging task because the most advanced publicly available OCR engines struggle with interpreting the combination of numbers and letters present in receipts. To solve this problem I had to build a smart word categorisation that could understand the nuances of the output from the OCR engine. Overall, I am proud of the product as it does extract financial information accurately for images that are well-lit, taken on a flat surface and are at a high enough definition to be human-readable. If the product were to be continued, I would seek to improve to improve the machine learning system that the OCR engine is built on in order to improve the accuracy of the output. I believe that if this could be improved, it would greatly improve the quality of the financial information extraction system. Truthfully, my work on Percy was not so much an exercise in building financial extraction system as it was learning how to build a fully functional, strongly branded web application from start to finish. For this reason, this blog post will focus on the full-stack development process that I used to create the application. If you wish to test my application yourself before reading this post head over to, create an account and upload a few receipts/invoices. While building the app I worked hard to ensure that it is completely secure from a cybersecurity perspective so that all users’ personal and financial information is safe.

Origins of Percy: March 2020

The original mission of this application was to utilise the latest developments in machine learning to completely automate all the manual tasks that accounts have to carry out on daily basis. During the summer of 2018 I spent three months working as an intern in the finance team for a start-up called Velocity Black. They had real problems managing the high number of transactions that they handled on a daily basis. For this reason, they heavily relied on interns and accounting assistants to process the receipts and invoices and reconcile bank accounts. The issue was that this is an incredibly boring and difficult task that is hardly fit for humans. Not surprisingly, those that joined the accounting team in these roles did not stay for long. The following year during my internship for Whave in Uganda I saw a similar occurance where an employee had to travel to the head office to spend a day a week adding receipts and invoices to the account software. Seeing the business need for automation in this area I set out to build software that automated these human unfit accounting tasks.

I really wanted my app to have a strong memorable brand. As my software was originally going to be an artificially intelligent accounting assistant, I thought giving it a human name would work well. It is nowadays incredibly hard to find a short memorable domain name, so I spend a while searching for something that worked. I’m certainly happy with the name Percy as it does sound somewhat related to accounting and it is definitely memorable. Conveniently, it is also generalisable enough that I could choose to use this domain-name/brand for a different project in the future if I wish.

Version One

The first version of Percy was a React/Node/Express app that just had the core system. It allowed you to not only extract information from receipts & invoices but also a system for monitoring trade receivables and trade payables. As it did not have a database system yet, the images and extracted text were stored on the server. The hardest part by far of building the react component was dynamically displaying the photos of the extracted receipts with their relevant financial information from the state. This was made increasingly difficult by two functionalities I built, these were: adding and removing rows and displaying errors when the user added the wrong type of information into boxes. This had complicated implications for state management. It culminated in a maze of countless ‘undefined’ errors which I had to navigate through. At the completion of the first version, bugs were still present in the system. However, in later versions I managed to make this intricate system work perfectly and this is something I am proud of. The automated receipt and invoice data extraction system worked by sending the users’ images from the nodejs server to a python flask server, which carried out the OCR and word categorisation processes, and then sent the output as a JSON file back to the nodejs server which saved the outputted text and sent the filepaths to the react component which then displayed the output to the user who could then edit it and add/remove rows.

The Finished Product

After the first version I decided to make Percy only focus on the automated receipt & invoice data extraction component. For this reason I switched from a purely ‘create-react-app’ to a static html website that used an express server, a python flask server, a mongoDB database, and react component embedded into the static html. It could be described as a ‘MERN’ app by web developers.

The Homepage
The Sign Up Page (With Working Account Verification Tokens And Forgotten Password Reset)
The Log In Page
The Dashboard Page

On the dashboard you can view previously uploaded receipts/invoices or update your billing/account settings. You can also download previously uploaded receipts as a CSV file that can be opened in Microsoft excel. In addition, you can connect your accounting software package to automatically have the receipts/invoices sent to your accounting software with the extracted information.

The Upload Page
The Extracted Information Page

On the Extracted Information page you can view and edit the extracted information and upload it to your Percy account and/or connected external software.

The Support Page

That concludes this blog post about Percy. I hope you have enjoyed learning about my web app. Don’t forget you can actually go and see it in action here. If you have any questions you want to ask me about it, send an email here or contact me via twitter/linkedin. I’ve always got something exciting in the works so stay tuned for my latest endeavour.

Leave a Comment