Recognize natural language dates using Python3
We in the company create a service that extracts legal facts from customer and customer correspondence. The service grew out of one simple idea - my regular customers decided to simplify the work of the managers and create a “contract generator”. The first task - to pull in the contract details of the client and the customer, we decided easily.
There was a second idea - to search for dates in the correspondence and insert them into the technical task, documents, automatically.
However, people rarely write dates in chat rooms and instant messengers so that the algorithm can easily recognize them.
“We start next week”, “this friday” - if managers are relatively easy to learn how to write dates in the correct format, then you cannot make such demands to clients.
')
I am writing in python and the python NLP-library spaCy * - NER (name entity recognition) module came to the aid of which the module easily retrieved dates from correspondence with English-speaking clients. As a result, we got a bunch of relative dates: “in two weeks”, “friday”, “this friday”, “this noon”, “wensday noon”.
But how to translate these dates into objects that the service perceives (datetime object)?
A long journey began in the area of ​​“human readable dates parsing”. Looking around the neighborhood, I found only three python libraries that started immediately and without torment: these are timefhuman **, dateparser *** and datefinder ***.
In the picture the comparison of the three libraries:
In general, it is clear that the timefhuman not spoiled by the attention turned out to be more convenient, albeit with a large number of false positives, which the majority uses the dateparser.
Interesting logic developers parsing “Friday” gave the next Friday in the case of timefhuman and last Friday in the dateparser.
In general, the timefhuman turned out to be more lively and was chosen for further development and testing of the prototype.
* spacy.io
** github.com/alvinwan/timefhuman
*** github.com/scrapinghub/dateparser
**** github.com/akoumjian/datefinder
There was a second idea - to search for dates in the correspondence and insert them into the technical task, documents, automatically.
However, people rarely write dates in chat rooms and instant messengers so that the algorithm can easily recognize them.
“We start next week”, “this friday” - if managers are relatively easy to learn how to write dates in the correct format, then you cannot make such demands to clients.
')
I am writing in python and the python NLP-library spaCy * - NER (name entity recognition) module came to the aid of which the module easily retrieved dates from correspondence with English-speaking clients. As a result, we got a bunch of relative dates: “in two weeks”, “friday”, “this friday”, “this noon”, “wensday noon”.
But how to translate these dates into objects that the service perceives (datetime object)?
A long journey began in the area of ​​“human readable dates parsing”. Looking around the neighborhood, I found only three python libraries that started immediately and without torment: these are timefhuman **, dateparser *** and datefinder ***.
In the picture the comparison of the three libraries:
In general, it is clear that the timefhuman not spoiled by the attention turned out to be more convenient, albeit with a large number of false positives, which the majority uses the dateparser.
Interesting logic developers parsing “Friday” gave the next Friday in the case of timefhuman and last Friday in the dateparser.
In general, the timefhuman turned out to be more lively and was chosen for further development and testing of the prototype.
* spacy.io
** github.com/alvinwan/timefhuman
*** github.com/scrapinghub/dateparser
**** github.com/akoumjian/datefinder
Source: https://habr.com/ru/post/449184/
All Articles