Research & development

Deep Learning Starts With Ourselves

Introduction

Applica is always improving: Our team works hard to develop new innovations in document understanding and add additional features.

Researched, Reviewed, Published

Applica’s R&D team regularly publishes research papers about the breakthroughs we’ve achieved.
Sparsifying Transformer Models with Trainable Representation Pooling
Authors
Michał Pietruszka, Łukasz Borchmann, Łukasz Garncarek
Date
2022
We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations during the training process, thus focusing on the task-specific parts of an input. A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust trainable top-k operator. Our experiments on a challenging long document summarization task show that even our simple baseline performs comparably to the current SOTA, and with trainable pooling, we can retain its top quality, while being 1.8x faster during training, 4.5x faster during inference, and up to 13x more computationally efficient in the decoder.
Read Full Abstract
Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Authors
Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, Gabriela Pałka
Date
2021
We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing the TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable of unifying a variety of problems involving natural language. The layout is represented as an attention bias and complemented with contextualized visual information, while the core of our model is a pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from documents and answering questions which demand layout understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process by employing an end-to-end model.
Read Full Abstract

Curious to Read more?

Browse through all our scientific papers.
Awards

Our Solution Has Won Multiple Prizes

Applica’s solution regularly wins awards and competitions around the world.
April 2021
Applica’s innovative TILT model crushed the competition in the ICDAR Infographics VQA Challenge
March 2021
Applica continues to dominate the venerated Key Information Extraction Competition
February 2021
Applica beats all other AI solutions in the Document Visual Question Answering Challenge
February 2021
The Applica team wins Best Paper at SemEval 2020

Meet the Technology

Find out what makes Applica’s approach to document automation so special—and so much more powerful than other approaches.
To the Tech

Dive Into the Details

Ready for some math? Our research blog documents the latest breakthroughs, ideas, and observations from Applica’s R&D team.
To the Research Blog