Pattern Recognition Letters 160 (2022) 43–49 Contents lists available at ScienceDirect Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec Few shots are all you need: A progressive learning approach for low resource handwritten text recognition Mohamed Ali Souibgui a,∗, Alicia Fornés a, Yousri Kessentini b,c, Beáta Megyesi d a Computer Vision Center, Computer Science Department, Universitat Autònoma de Barcelona, Spain b Digital Research Center of Sfax, B.P. 275, Sakiet Ezzit,Sfax 3021 Sfax, Tunisia c SM@RTS : Laboratory of Signals, SysteMs, Artificial Intelligence and Networks, Tunisia d Department of Linguistics and Philology, Uppsala University, Sweden a r t i c l e i n f o a b s t r a c t Article history: Handwritten text recognition in low resource scenarios, such as manuscripts with rare alphabets, is a Received 8 July 2021 challenging problem. In this paper, we propose a few-shot learning-based handwriting recognition ap- Revised 9 April 2022 proach that significantly reduces the human annotation process, by requiring only a few images of each Accepted 4 June 2022 alphabet symbols. The method consists of detecting all the symbols of a given alphabet in a textline Available online 7 June 2022 image and decoding the obtained similarity scores to the final sequence of transcribed symbols. Our Edited by Jiwen Lu model is first pretrained on synthetic line images generated from an alphabet, which could differ from the alphabet of the target domain. A second training step is then applied to reduce the gap between Keywords: Handwritten text recognition the source and the target data. Since this retraining would require annotation of thousands of hand- Few-shot learning written symbols together with their bounding boxes, we propose to avoid such human effort through Unsupervised progressive learning an unsupervised progressive learning approach that automatically assigns pseudo-labels to the unlabeled Ciphered manuscripts data. The evaluation on different datasets shows that our model can lead to competitive results with a significant reduction in human effort. The code will be publicly available in the following repository: https://github.com/dali92002/HTRbyMatching © 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/) 1. Introduction comes even harder in the case of ciphers because when the alpha- bet is invented, no dictionaries or language models are available to Training data-hungry models based on deep learning in low re- help in the training process. source scenarios is challenging due to the scarcity of labeled data. Contrary to deep learning models, human beings are able to This is particularly the case of modern Handwritten Text Recogni- learn new concepts from one or a few samples only. Recent re- tion (HTR) systems when applied to manuscripts with rare scripts search has been conducted to imitate and simulate this ability. One or unknown alphabets. For example, ancient civilizations used of these recent approaches is called few-shot learning, requiring their specific alphabets that are no longer used (e.g. cuneiform, only a limited number of examples with supervised information Egyptian hieroglyphs) and historical ciphers (used in diplomatic [3]. In our previous work [4], we explored whether few-shot learn- and intelligence reports, secret societies, or private letters) often ing could be adapted to the recognition of various symbol sets in invented fanciful cipher alphabets to hide the content of the mes- encrypted hand-written manuscripts. sage [1]. Usually, HTR models must be trained on the particular alpha- Recognizing and extracting information from these special bet to be recognized, and whenever the alphabet changes, the sys- sources are important to the understanding of our cultural her- tem must be retrained from scratch with samples from the new itage, since it helps to shed new light on and (re-)interpret our script. To avoid the cumbersome process of re-training, we treated history [2]. However, a manual transcription is impractical due to the recognition as a symbol detection task: by providing one or the amount of manuscripts, and automatic recognition is difficult a few samples of each symbol type in the alphabet, the system due to the lack of labeled training data. Moreover, the problem be- could locate the symbols in the manuscript. The model was generic and could be used for multiple scripts, while requiring only a small sample of labeled data on each new symbol type. The first ∗ Corresponding author. experimental results obtained a good performance on encrypted E-mail address: msouibgui@cvc.uab.cat (M.A. Souibgui). https://doi.org/10.1016/j.patrec.2022.06.003 0167-8655/© 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/) M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 manuscripts compared to the typical methods, while reducing the source manuscript. In semi-supervised learning [12,13], a few la- amount of labeled data for fine-tuning. beled data is used to start the process. For instance, in the la- Nevertheless, the required labeled data in our few-shot model bel propagation approach based on distances [14,15], labels are as- still implies a significant human effort: labeling a few pages with signed from the unlabeled data (called pseudo-labels) to be used various types of symbols for fine tuning include manual transcrip- to reinforce the training. Similarly, in [16], the training started tion of thousands of symbols together with their corresponding with some true labels that are gradually increased by pseudo la- bounding boxes. To alleviate this, we aim to minimize the time- bels. In [17] a shared backbone extracted features from the labeled, consuming manual labeling effort by proposing an unsupervised pseudo-labeled and unlabeled data at each iteration. Then, from learning approach that can automatically and progressively label the feature space, the reliable labels were estimated according to the data by assigning pseudo − labels from the unlabeled hand- the distance with the true labels while the non trusted labels were written text lines. Our method requires only a few shot of the de- pushed away with an exclusive loss. Besides, a pseudo-labeling sired alphabet: to perform the pseudo-labeling, the user crops a curriculum approach for domain adaptation used a density-based few samples – preferably 5 – of each symbol type thereby avoid- clustering algorithm in [18]. The idea was to annotate data with ing the annotation of text lines and the annotation of the bound- the same label set, but taken from a different domain. ing boxes. This means that the pseudo-labeled data is automati- In HTR, this strategy was hardly applied mainly due to the dif- cally obtained to fine-tune our model, with zero manual effort. ficulties in character segmentation, since touching characters are The main contributions of our work are: (i ) We propose a few- common in cursive texts. In [19], labels were guessed at word level shot learning model for transcribing hand-written manuscripts in using keyword spotting. A confidence score was used to assign new low resource scenarios with minimal human effort. Our model only labels to the retrieved words and enlarge the dataset. Furthermore, requires few, ideally five samples of each new symbol type, in- a text to image alignment was proposed in [20] following the men- stead of annotating the entire text lines with the symbols and their tioned strategy. bounding boxes. (ii ) We propose an unsupervised, segmentation- free method to progressively obtain pseudo-labeled data, which 3. Proposed approach can be applied to cursive texts with touching symbols. (iii ) We propose a generic recognition and pseudo-labeling model that can In this section we describe our approach to handwritten text be applied across different scripts. (iv ) We demonstrate the ef- recognition by few-shot learning. First, our model is trained on fectiveness of our approach through extensive experimentation on synthetic data. We create text line images using various Omniglot different datasets with various alphabets, reaching a performance symbol alphabets [21]. Then the model is fine-tuned using the similar to the one obtained with manually labeled data. pseudo-labeling approach with the specific alphabet of the target domain, in our case the hand-written manuscript. The involved 2. Related work steps are described in detail below. 2.1. Low resource manuscript recognition 3.1. Few-shot manuscript matching A manuscript is considered low resource when it contains rare symbols or unusual symbol sets. Thus, collecting a training set As explained in Section 2, few-shot modeling for object detec- for this manuscript is difficult (especially a labeled one). The re- tion has shown to be suitable for recognizing the manuscripts in search on the transcription of enciphered manuscripts is quite re- low resource scenarios. In few-shot modeling, if the size of the cent. In [5], an MultiDimensional Long Short-Term Memory (MDL- alphabet is N, and we provide k examples from each symbol al- STM) [6] approach was proposed. The performance was satisfac- phabet (named shots (or supports)), the task is considered as an tory, but at the cost of the time-consuming manual data label- N-way k-shot detection problem. In such setting, the model can ing. The method also required new, manual transcription for each be trained on certain alphabets with sufficient amount of labeled new cipher. Instead, some unsupervised methods were introduced data, and later, it can be tested on new alphabets (classes) with a [7,8] to avoid the costly human effort. Those approaches were few labeled examples only. segmenting the enciphered documents into isolated symbols then Our few-shot learning model, illustrated in Fig. 1, is segmenta- clustering them. The main disadvantage of the clustering method tion free and works at line level. As input, it takes the text line turned is the segmentation of symbols, because is was often in- image with an associated alphabet in the form of isolated sym- accurate, provoking transcription errors. Similarly, researchers have bol images. In this step, from one to five samples of each alpha- opted for learning-free symbol spotting approaches [9,10] for the bet symbol should be given. The two inputs (the line image as transcription of ancient manuscripts (e.g. Egyptian hieroglyphs, a query and a symbol image as a support) are propagated in a cuneiform, or runes). shared backbone to derive two feature maps. Those are then used In summary, supervised methods obtain good performance but in the Region Proposal Network (RPN) with an attention mecha- they require large amounts of labeled data, while unsupervised or nism to output proposals. The attention mechanism performs the learning-free methods can be applied when labeled data is not depth-wise cross correlation between the support and query fea- available but they lead to lower performance. Thus, to maintain ture maps. As illustrated in Fig. 2, this is done by performing a high accuracy while reducing the human effort of manual label- multiple average pooling to the support feature map to obtain a ing, few-shot learning is a promising alternative to use for hand- shape of 1 × 1 × Channels then multiplying it over depth with the written text recognition [4]. A similar approach based on charac- query feature map. After the RPN stage, the Region of Interest (ROI) ter matching was proposed in [11], although the experiments were pooling is applied to the RPN proposals and the support image mostly carried out on synthetic data, instead of on real historical to provide two feature maps having the same size. These feature or cursive manuscripts. maps are representing the support image as well as the query re- gions that are candidates to match the support image. Those are 2.2. Handwritten text pseudo-Labeling combined together and passed to the final stage where the bound- ing boxes are produced with the class 1 (similar to the support) or Pseudo-labeling models aim to take advantage from unlabeled 0 (different from the support symbol). For each labeled bounding data when training, which makes it a possible solution for low re- box, a confidence score between 0 and 1 is predicted according to 44 M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 Fig. 1. Our few-shot approach for handwriting recognition. Examples of each symbol in the alphabet are used as supports. Up: Detection of a support symbol in a handwrit- ten line. Down: Construction of the similarity matrix from the predicted bounding boxes and its decoding to obtain the final text. Fig. 2. An illustration of the attention RPN: the support feature map is average pooled until obtaining a tensor with shape of 1 × 1 × 512. The obtained tensor is multiplied over depth with the Query feature map to obtain the attention Q, which is passed to the RPN for region proposing. the similarity degree with the support image. We repeat this pro- Algorithm 1 Similarity Matrix Decoding. cess for all supports (i.e. all the alphabet symbols) and take only Input: the bounding boxes with a high confidence score (higher than a M  Similarity matrix given threshold) to construct a similarity matrix between the sym- rep_thresh  Repetition threshold bol alphabet and the line image regions. This matrix serves as the Output: CharList  Characters sequence input to the decoding algorithm, which provides the final tran- last_max ← [−1, 0]  [index, score] scription. repet it ion ← 0 maximums ← MaxInd (M )  maximum index and score for each 3.2. Similarity matrix decoding column CanAdd ← F alse The decoding stage, detailed in Algorithm 1, takes the similar- ity matrix, traverses the columns from left to right, and decides for maxi in maximums do for each pixel column the final transcribed symbol class among the candidate symbols. Concretely, for each time step, it chooses if maxi = last_max then the symbol having the maximum similarity score. To minimize er- repet it ions ← 0 rors, a symbol is only transcribed if its bounding box is not over- CanAdd ← T rue lapped by another symbol with a higher similarity value for a cer- else tain number of successive pixels. In our case, we used 15 pixels if repet it ions > rep_thresh and CanAdd then as a threshold. Despite its simplicity, this decoding method is ef- C harList ← C harList ∪ maxi[index] fective for transcribing sequences of symbols. It can be considered CanAdd ← F alse also as a modified version of the Connectionist Temporal Classifi- else cation (CTC) algorithm [22]. repet it ions ← repet it ions + 1 As mentioned before, our few-shot model is first trained on end if the Omniglot dataset: we synthetically construct lines to learn the end if matching in different alphabets. Then, at testing time, it can be end for used to recognize new symbols, requiring only a support set com- posed of a few examples of each new symbol class. However, in our previous work [4], experiments showed that the predictions can be significantly improved when we fine-tuned the model us- 3.3.1. Synthetic data generation ing some real text lines, due to the domain difference between the Our few-shot model needs to be fine-tuned using data from the synthetic Omniglot symbols and the real historical symbols. target domain (often with an unseen alphabet) to reduce the gap between the source and target domains. But since we aim to min- 3.3. Progressive pseudo-labeling imize the user effort, we restrain the demands on a support set of few samples from each new symbol alphabet. Hence, the user Our proposed progressive data pseudo-labeling strategy consists must only select up to 5 samples per symbol, called shots. From in two stages described below. those shots, we automatically generate synthetic lines by randomly 45 M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 Fig. 3. Our pseudo-labeling approach: In the beginning, synthetic lines are generated using the support set. Then, the pseudo-labeling phase starts. At starting, there is no pseudo-labeled data, so only synthetic lines will be used for retraining the model. Then, the model predicts symbols from the real unlabeled lines with the same script. The symbols with highest confidence score, namely pseudo-labels, are labeled and added with their predicted bounding boxes. Next, the model is retrained again using the synthetic lines and the pseudo-labeled symbols from real lines. The process is repeated until the full dataset is annotated. Fig. 4. An example of pseudo-labeling of a line image. The background is colored in grey, while the predicted label classes at each time are shown in colors. Each symbol class is shown with a different color (best viewed in color). concatenating them in a line image. We tried to make those syn- shows an example of a handwritten line during the pseudo label- thetic lines as realistic as possible. To do so, the space between ing process. At the beginning, the whole image is considered as characters was chosen randomly between 0 and 30 pixels. Also, be- a background. Then, the symbols with higher confidence score are fore concatenation we rotate each character randomly between -5 labeled in the first iteration, while the hardest ones will be labeled and 5 degrees. Moreover, we add some artifacts to the upper part in the next iterations. and lower part of the line to simulate a realistic segmentation of a handwritten line. Those created lines compose our starting labeled 4. Experiments set, since our model was only pre-trained on a different data do- main, i.e. the synthetic Omniglot lines. This technique significantly 4.1. Datasets improves the model prediction for unseen alphabets or scripts. For low resource handwritten text, we chose two historical en- 3.3.2. Pseudo-labeling process crypted manuscripts and a manuscript with an old, not longer used After retraining our model with synthetic lines, we begin by an- alphabet, the Codex Runicus. Our choice of running experiments notating the unlabeled data. The process is illustrated in Fig. 3. Of on encrypted manuscripts is motivated by the fact that ciphers course, at the beginning, the pseudo labeled set is empty (as no la- contain a large variety of more or less fancy symbols instead of bels are available), so only the synthetic lines can be used for train- or in addition to using common alphabets and/or digits. In this ing. Then, we pass the real text lines through our model to get the work we chose two encrypted manuscripts, namely the Borg and predictions, which include the bounding boxes of the regions that the Copiale ciphers, both containing a large variety of symbols. The are similar to the input alphabet images along with the assigned Borg1 . cipher is a 408 pages long manuscript, originating from the similarity score. Since a higher score means more credible label, 17th century. The entire manuscript is encoded with the exception we choose the top scored predictions as pseudo labels at this iter- of the first and last two pages, and some headings in Latin. The ation. We experimentally found that the best option is to choose, cipher consists of 34 different symbols, comprising from graphic at each iteration, the 20% of the training data size as the number signs to Latin letters and some diacritics. The Copiale cipher is a of the new pseudo-labels. The obtained pseudo-labeled set will be 105 page long encrypted manuscript from the mid 18th century. joined to the synthetic set for the next training iteration. This pro- The cipher consists of 100 different symbols from Latin and Greek cess is repeated until the whole unlabeled set (i.e. all text lines) is letters to digits along with a large number of graphic signs. The ci- annotated. In the case where it is not possible to add new pseudo- pher has been transcribed and deciphered [23] and is freely avail- labels with credible confidence score, we set a threshold of 0.4 as able in high resolution images2 . The Codex Runicus3 is a histor- the minimum confidence score for assigning pseudo-labels. In fact, ical manuscript, the oldest preserved Nordic provincial law writ- whenever the score is below this threshold, it is better not to label ten on 100 parchment folios of 202 pages. Its symbol set consists the symbol. Note that we label the handwritten lines without the need of segmenting them into isolated symbols. In this way, the 1 https://cl.lingfil.uu.se/∼bea/borg/ remaining unlabeled symbols in the different lines at each itera- 2 https://cl.lingfil.uu.se/∼bea/copiale/ tion are considered as background during the next training. Fig. 4 3 https://www.e-pages.dk/ku/579/ 46 M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 Fig. 5. Examples of the three manuscripts with low resource annotated data. Table 1 Obtained Results on the different datasets. FT: Fine Tuning. Om: Omniglot. SD: Synthetic Data. RLD: Real Labeled Data. PLD: Pseudo Labeled Data. ULD: UnLabeled Data. Dataset Method User Effort Training → FT SER Borg Unsupervised [8] None ULD 0.57 Unsupervised [8] Manual Segmentation ULD 0.22 Unsupervised [7] Clusters Processing ULD 0.54 MDLSTM [5] Manual Labeling RLD 0.55 Few-shot [4] Manual Labeling Om → RLD 0.21 Few-shot [4] 5 shots Om → NONE 0.53 Ours 5 shots Om → SD + PLD 0.24 Copiale Unsupervised [8] None ULD 0.44 Unsupervised [8] Manual Segmentation ULD 0.37 Unsupervised [7] Clusters Processing ULD 0.20 MDLSTM [5] Manual Labeling RLD 0.07 Few-shot [4] Manual Labeling Om → RLD 0.11 Few-shot [4] 5 shots Om → NONE 0.39 Ours 5 shots Om → SD + PLD 0.15 Codex Unsupervised [7] Clusters Processing ULD 0.06 Runicus MDLSTM [5] Manual Labeling RLD 0.26 Few-shot [4] Manual Labeling Om → RLD 0.05 Few-shot [4] 5 shots Om → NONE 0.40 Ours 5 shots Om → SD + PLD 0.09 ’ Table 2 Data (PSD) by using the approach detailed in Section 3.3. We fi- Required time (in minutes) for manually annotating the nally fine-tune the model with the pseudo-labeled data and com- training lines. pare its performance to the models that uses Real Labeled Data Dataset # Lines # Symbols # Classes Time (RLD) for training. Borg 117 1913 24 ≈ 245 The model performance is measured by the Symbol Error Rate Copiale 176 7197 78 ≈ 450 (SER) metric. It is the same as the Character Error Rate used in Runicus 56 1583 25 ≈ 206 HTR. Formally, SER = S+ND+I , where S is the number of substitu- tions, D of deletions, I of insertions and N is the ground-truth’s length. Not surprisingly, the lower the value, the better perfor- of runes where each rune corresponds to a letter of the Latin al- mance. phabet. Fig. 5 shows examples of the two ciphers and the codex We compare our approach with our previous few-shot model Runicus. As it can be seen, the Borg symbols are connected not [4], the unsupervised [7,8] and supervised [5] approaches for en- only horizontally but also oftentimes vertically with many touching crypted manuscript recognition. symbols making its recognition challenging. In the Copiale cipher, on the other hand, the symbols are clearly segmented but the size 5. Results of the alphabet is large, which make it a good challenge to test our approach on it. Similar to the Copiale cipher, the codex Runicus Table 1 shows the obtained results. The Borg manuscript is con- consists of clearly segmented symbols and a rare alphabet mak- sidered to be a hard case because of the overlapping symbols, ing it also a good case of low resource handwriting recognition. which makes predicting correct bounding boxes challenging. Also, In our experiments, we exclude the symbols with low frequencies the writing style is variable. As it can be seen, using a few-shot (that occur once or twice) in all manuscripts. We use 24 symbols method with real labels leads to a SER of 0.21, being considered as from the Borg cipher, 78 symbols from the Copiale cipher, and 25 the upper bound. But, this result is costly, since a user must man- symbols from the Codex Runicus. Table 2 shows more information ually annotate 1913 symbols, including their labels and bounding about our used datasets. boxes. We also notice that the supervised MDLSTM with a larger training set, annotated at line level (but without any bounding 4.2. Experimental setup and metrics boxes required), obtains a moderate result, probably due to the connected hand-writing. We notice that the unsupervised methods To carry out the experiments, we first trained our proposed are only useful when the segmentation of lines into isolated sym- few-shot handwriting recognition model using lines created from bols is accurate, which is a costly and difficult task. Our few-shot the Omniglot dataset only. Then, we retrained the model using model, trained on Omniglot only and tested on Borg, leads also to synthetic lines created from the given 5 symbols (shots) as de- a poor result (a SER of 0.53) due to the big difference between scribed above. This data is called Synthetic Data (SD). Afterwards, the training and test domains. However, when using the pseudo- we start predicting the labels and obtaining the Pseudo Labeled labeled data provided by our approach, we obtained an acceptable 47 M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 result of 0.24 SER, with a high gain in user effort because we only Table 3 The symbol error rate when using different thresholds while pseudo- require 5 examples of each symbol, avoiding a time-consuming labeling the data. Thres.: Threshold. manual annotation. The Copiale manuscript contains easy to segment symbols but Thres. SER Thres. SER Thres. SER Thres. SER with a larger alphabet size. As it can be seen from Table 1, the 0.8 0.27 0.6 0.25 0.4 0.24 0.2 0.25 MDLSTM performs better on this dataset because of the larger la- beled training lines and a unique handwriting style. However, our Table 4 model achieves a competitive result by using less data than MDL- Comparative results with self-supervised learning ap- STM. Anyway, annotating these lines is costly, so a better choice proaches in the semi-supervised scenario. is to automatically produce pseudo-labels. By using our pseudo- Method Labeled lines SER labeling process, we achieve a competitive performance, compared MAE 20% 0.99 to the manually labeled data (a SER of 0.15 versus 0.11). [24] 30% 0.99 Finally, we test our method on the Runicus manuscript as 50% 0.95 an example of ancient document with a rare alphabet. This UP- 20% 0.71 manuscript can be considered easier than ciphers because the DETR 30% 0.70 [25] 50% 0.66 symbol segmentation is easy and the alphabet size is moderate. Ours 20% 0.25 Thus, an unsupervised clustering method can be also appropriate. 30% 0.24 Using our method with real labeled data, we obtain results that 50% 0.20 are better than without any fine tuning, with a SER of 0.05 and 0.40, respectively. When we compare the quality of our produced- pseudo labels against the manually created ones, we observe that, easy to segment, the labeling accuracy reached 85%. In Codex Runi- by using pseudo-labeling, we achieve a competitive result of 0.09 cus, we obtained the highest pseudo-labeling accuracy of 94% be- SER. This demonstrates the suitability of our method, because the cause the symbol segmentation is easier than Borg and the number performance is close to the one obtained with manual labels while of classes is lower than in Copiale. significantly reducing the annotation effort. During our experiments, we found that it is better to continue We can conclude that our proposed pseudo-labeling method the pseudo-labeling process despite the decreasing performance. achieves good results when recognizing low resource handwritten The reason is that, although we might add some wrong labels, in texts, with an important decrease in the user effort for data anno- general, the incorporation of difficult examples benefits the train- tation. The analysis of the human effort is detailed next. ing and even a bounding box with a wrong label is still helping in the segmentation part. Moreover, the experiments show that there 5.1. Annotation time consumption is a small difference in performance between the manually anno- tated labels and our automatically produced ones, which encour- Manually annotating data is a time consuming task and it ages us to further improve our labeling process. should be taken into account when using HTR models. Thus, in this 5.3. Selecting threshold for pseudo-labeling section, we measure the time needed to label the three datasets to illustrate the manual labeling effort. As shown in Table 2, the more In our experiments we set a threshold of 0.4 before adding a lines and the bigger the alphabet size, the more time is required character into the labeled set. This threshold is chosen after testing to label the symbols with their bounding boxes. For reference, we other values and finding that 0.4 is the optimal one. We show the measured the required time for providing the shots for our method results of the conducted experience in Table 3, where we tested and compared it with the manual annotation time. We found that different thresholds to select the pseudo-labels. The experiments locating and cropping 5 examples of each symbol in the alphabet were carried out on the Borg dataset. takes approximately 40 s. Thus the user needed to spend 16 min for Borg, 17 min for Runicus and 52 min for Copiale for providing 5.4. Is semi-supervised learning worth to use? the shots for all symbols in the manuscripts for our approach. We can conclude that automatically providing pseudo-labels So far, we opt to use an unsupervised approach that starts from significantly minimizes manual effort with a minimal loss in recog- a few shots of the desired alphabet. However, the choice of starting nition performance compared to the manual annotation. by labeling some real lines (text and bounding boxes) and pseudo- labeling the rest is also a possible solution. We test this strategy 5.2. Pseudo-labeling performance analysis on the Borg dataset as presented in Table 4. For comparison, we use two recent self supervised learning methods: masked autoen- Our proposed method progressively labels the dataset: we start coders (MAE) [24] and UP-DETR [25]. Given that these methods by labeling easy symbols and progressively label the complicated were proposed for image classification and object detection, we ones. As a consequence, the accuracy of correctly labeling bound- adapt them to text recognition by adding a transformer decoder ing boxes decreases as we select new pseudo labels at each itera- [26] in MAE and using our decoding algorithm in UP-DETR. As tion. We evaluate the quality of our pseudo-labeling approach on pre-training, MAE uses a set of Latin handwritten images because the three datasets by comparing the predicted bounding boxes and of the very few available Borg lines, while the UP-DETR is trained their corresponding pseudo-labels to the manually annotated ones. on the 117 unlabeled Borg lines. Then the fine-tuning is done us- A predicted bounding box is defined as a correct detection if it has ing 20, 30 and 50% of labeled Borg lines for both methods. The a minimum overlap (i.e. Intersection over Union: IoU) of 0.7 with obtained results show that the more labeled lines, the better the the ground-truth box. We find that the more difficult the dataset performance. Despite the very few data that were used (the full is in terms of segmentation, alphabet size and similarity between training set is 117 lines), our method clearly outperforms the data- symbols, the more the performance of our pseudo-labeling ap- hungry MAE and UP-DETR, showing that they are not suitable for proach decreases and the more iterations in the labeling process such low resource scenarios. are needed. For example, labeling accuracy for the Borg cipher was It is noteworthy that, in our method, if we start with more 74% after obtaining all the labels. In Copiale, where symbols are manually labeled lines, the amount of unlabeled lines to pseudo 48 M.A. Souibgui, A. Fornés, Y. Kessentini et al. Pattern Recognition Letters 160 (2022) 43–49 label is reduced, so the training time decreases. Overall, we can 25th International Conference on Pattern Recognition (ICPR), IEEE, 2021, pp. conclude that starting from only a few shots is a better solution 5413–5420. [5] A. Fornés, B. Megyesi, J. Mas, Transcription of encoded manuscripts with im- with regards to the reduced manual effort, since the SER is slightly age processing techniques, in: Proceedings of the Digital Humanities Montreal, affected (we obtain 0.24 as SER using our unsupervised pseudo- Canada, August 8–11, 2017. labeling). [6] A. Graves, Offline Arabic handwriting recognition with multidimensional re- current neural networks, in: Guide to OCR for Arabic Scripts, Springer, 2012, pp. 297–313. 6. Conclusion [7] A. Baro, J. Chen, A. FornȨs, B. Megyesi, Towards a generic unsupervised method for transcription of encoded manuscripts, in: Proceedings of the International Conference on Digital Access to Textual Cultural Heritage (DATECH), 2019. We have presented a novel pseudo-labeling transcription [8] X. Yin, N. Aldarrab, B. Megyesi, K. Knight, Decipherment of historical method for manuscripts with rare alphabets or few labeled data. manuscript images, in: Proceedings of the ICDAR, IEEE, 2019, pp. 78–85. Our method can significantly reduce the human labor of annotat- [9] L. Rothacker, D. Fisseler, G.G. Müller, F. Weichert, G.A. Fink, Retrieving cuneiform structures in a segmentation-free word spotting framework, in: Pro- ing historical manuscripts, while maintaining the recognition per- ceedings of the 3rd International Workshop on Historical Document Imaging formance. The performed experiments on the enciphered and his- and Processing, 2015, pp. 129–136. torical manuscripts confirmed the usefulness of our approach, with [10] B. Bogacz, N. Howe, H. Mara, Segmentation free spotting of cuneiform using a significant reduction in user effort and a minimal loss in recog- part structured models, in: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE, 2016, pp. 301–306. nition performance. [11] C. Zhang, A. Gupta, A. Zisserman, Adaptive text recognition through visual Our few-shot model with pseudo-labeling is a significant exten- matching, in: Proceedings of the ECCV, 2020, pp. 51–67. sion of our previous work [4]. In fact, its simplicity makes it even [12] X.J. Zhu, Semi-supervised Learning Literature Survey, University of Wisconsin– Madison Department of Computer Sciences, 2005. applicable on top of other methods, like [11]. Also, for widely-used [13] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, T. Raiko, Semi-supervised alphabets (like latin) but with few labeled data, pseudo-labels can learning with ladder networks, arXiv preprint arXiv:1507.02672(2015). be predicted to annotate the data and train usual fully supervised [14] M.S.T. Jaakkola, M. Szummer, Partially labeled classification with markov ran- dom walks, Adv. Neural Inf. Process. Syst. (NIPS) 14 (2002) 945–952. HTRs, which may lead to better results than the few-shot ones. [15] J. Weston, F. Ratle, H. Mobahi, R. Collobert, Deep learning via semi-super- In future, we aim to enhance the quality of the provided labels vised embedding, in: Neural networks: Tricks of the Trade, Springer, 2012, to keep reducing the need of manual intervention. Also, we plan to pp. 639–655. [16] D.-H. Lee, et al., Pseudo-label: The simple and efficient semi-supervised learn- extend our approach to cover more low resource datasets including ing method for deep neural networks, in: Proceedings of the Workshop on other unknown scripts. Challenges in Representation Learning, ICML, 3, 2013. [17] Y. Wu, Y. Lin, X. Dong, Y. Yan, W. Bian, Y. Yang, Progressive learning for person re-identification with one example, IEEE Trans. Image Process. 28 (6) (2019) Declaration of Competing Interest 2872–2881. [18] J. Choi, M. Jeong, T. Kim, C. Kim, Pseudo-labeling curriculum for unsupervised The authors declare that they have no known competing finan- domain adaptation, in: Proceedings of the British Machine Vision Conference cial interests or personal relationships that could have appeared to (BMVC), Springer, 2019. [19] V. Frinken, M. Baumgartner, A. Fischer, H. Bunke, Semi-supervised learning for influence the work reported in this paper. cursive handwriting recognition using keyword spotting, in: Proceedings of the International Conference on Frontiers in Handwriting Recognition, IEEE, 2012, Acknowledgement pp. 49–54. [20] G. Leifert, R. Labahn, J.A. Sánchez, Two semi-supervised training approaches for automated text recognition, in: Proceedings of the 17th International Confer- This work was supported by the Swedish Research Council, ence on Frontiers in Handwriting Recognition (ICFHR), IEEE, 2020, pp. 145–150. grant 2018-06074, DECRYPT–Decryption of Historical Manuscripts, [21] B.M. Lake, R. Salakhutdinov, J.B. Tenenbaum, Human-level concept learn- ing through probabilistic program induction, Science 350 (6266) (2015) the Spanish project RTI2018-095645-B-C21 and the CERCA Pro- 1332–1338. gram / Generalitat de Catalunya. We thank the MTA Cloud (https: [22] A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, Connectionist temporal clas- //cloud.mta.hu/), which we used to run experiments. sification: labelling unsegmented sequence data with recurrent neural net- works, in: Proceedings of the 23rd International Conference on Machine Learn- ing, 2006, pp. 369–376. References [23] K. Knight, B. Megyesi, C. Schaefer, The Copiale Cipher, in: Proceedings of the Invited Talk at ACL Workshop on Building and Using Comparable Corpora [1] B. Megyesi, N. Blomqvist, E. Pettersson, The decode database: Collection of his- (BUCC), Association for Computational Linguistics, 2011. torical ciphers and keys, in: Proceedings of the 2nd International Conference [24] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick, Masked autoencoders are on Historical Cryptology, HistoCrypt 2019, June 23–26 2019, Mons, Belgium, scalable vision learners, arXiv preprint arXiv:2111.06377(2021). 2019, pp. 69–78. [25] Z. Dai, B. Cai, Y. Lin, J. Chen, Up-detr: unsupervised pre-training for object de- [2] B. Megyesi, B. Esslinger, A. Fornés, N. Kopal, B. Láng, G. Lasry, K.d. Leeuw, tection with transformers, in: Proceedings of the IEEE/CVF Conference on Com- E. Pettersson, A. Wacker, M. Waldispühl, Decryption of historical manuscripts: puter Vision and Pattern Recognition, 2021, pp. 1601–1610. the decrypt project, Cryptologia 44 (6) (2020) 545–559. [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, [3] Y. Wang, Q. Yao, J.T. Kwok, L.M. Ni, Generalizing from a few examples: a survey I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. 30 on few-shot learning, ACM Comput. Surv. (CSUR) 53 (3) (2020) 1–34. (2017). [4] M.A. Souibgui, A. Fornés, Y. Kessentini, C. Tudor, A few-shot learning approach for historical ciphered manuscript recognition, in: Proceedings of the 2020 49