resolves #635 resolves #573

Before

>>>dateparser.parse('after 15 days').strftime('%a %Y-%m-%d') 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'strftime'

>>>dateparser.parse('next tuesday').strftime('%a %Y-%m-%d') 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'strftime'

Now

>>> dateparser.parse('now').strftime('%a %Y-%m-%d')
'Sun 2020-03-22'

>>> dateparser.parse('after 15 days').strftime('%a %Y-%m-%d')
'Mon 2020-04-06'

>>> dateparser.parse('next sunday').strftime('%a %Y-%m-%d')
'Sun 2020-03-29'

>>> dateparser.parse('next tuesday').strftime('%a %Y-%m-%d')
'Tue 2020-03-24'

A added 3 tests for EST and all existing tests are passing.

For your consideration. There was no existing issue for anything to do with it. I'm uncertain in that case if I'm supposed to create one so I will. Issue: https://github.com/scrapinghub/dateparser/issues/5

example of transitional_info in pytz

Implementing optional language detection and allow the creation of custom language detection libraries by users.

This work is currently in progress and this PR is created for the monitoring purpose by the maintainers to suggest progress.

This PR is currently not complete.

TODOS:

• [X] Restructuring protect to pass function for language detection.
• [X] Managing HTTP exception for fasttext model download.
• [x] Updating setup.py for optional language detection.
• [x] Mock test of parse, search_dates and DateDataParser.
• [x] Unit tests for language detect functions.
• [x] Functionality analysis of the implemented language detection.
• [x] Testing optional language detection settings.
• [x] Make apply_setting independent functions.
• [x] Fasttext model download manager for models.
• [x] Documentation optional language detection settings.
• [x] Language Mapping (CLDR & ISO 639).
• [x] Check working with other settings.
• [x] Removing unsupported locale from language detecting
• [x] Improving docs.
• [x] Setting fasttext default language.
• [x] Removing langauge_map.json and sinking to language_data.py
• [x] langdetect set default DetectorFactory without changing global state.
• [x] dateparser-download set default caching folder.
• [x] Documenting dateparser-download.
• [x] Changes for preventing breaking changes.
• [x] detect_languages_func -> detect_languages_function

The use of lists there seems to make sense (should be forced as a list, right?).

I'm not sure why you're specifically removing the timezone?

This resolves https://github.com/scrapinghub/dateparser/issues/11 if that was simply a bug. Couldn't tell the intent of the line.

In [1]: from dateparser.date import DateDataParser

In [2]: ddp = DateDataParser()

In [3]: ddp.get_date_data('2014-10-09T17:57:39+00:00')['date_obj']
Out[3]: datetime.datetime(2014, 10, 9, 17, 57, 39, tzinfo=tzutc())

In [4]: ddp.get_date_data('2014-10-09T17:57:39+00:00', '')['date_obj']
Out[4]: datetime.datetime(2014, 10, 9, 17, 57, 39, tzinfo=tzutc())

In [5]: ddp.get_date_data('2014-10-09T17:57:39+00:00', '%Y')['date_obj']
Out[5]: datetime.datetime(2014, 10, 9, 17, 57, 39, tzinfo=tzutc())

Fixes #753

Added support for fractional units by making the regex pattern support digits before a decimal point and converting num to a float instead of an int on line 150.

What this changes

Before this fix, parse("in 0.5 hours") would return the time 5 hours in the future instead of 0.5 hours in the future.

With this fix, parsing fractional units are no longer parsed incorrectly.

Tests

>>> parse("now")
datetime.datetime(2021, 2, 7, 18, 50, 10, 107501)
>>> parse("in 0.5 hours")
datetime.datetime(2021, 2, 7, 19, 20, 15, 773800)
>>> parse("in 3.75 hours") 
datetime.datetime(2021, 2, 7, 22, 35, 20, 463238)
>>> parse("2.5 hours ago")  
datetime.datetime(2021, 2, 7, 16, 20, 29, 614937)

What was changed

To make it easier for future reviewers, the change is much smaller than it appears to be. The reason there are 131+ files changed is because the regex in write_complete_data.py was updated and ran.

Very few lines of code were changed in this PR:

• Regex updated in (freshness_date_parser.py line 14)

- PATTERN = re.compile(r'(\d+)\s*(%s)\b' % _UNITS, re.I | re.S | re.U)
+ PATTERN = re.compile(r'(\d+[.,]?\d*)\s*(%s)\b' % _UNITS, re.I | re.S | re.U)

• Result converted to float instead of int in (freshness_date_parser.py line 150)

- kwargs[unit + 's'] = int(num)
+ kwargs[unit + 's'] = float(num.replace(",", "."))

• Number regex changed in (write_complete_data.py line 34)

- string = RELATIVE_PATTERN.sub(r'(\\d+)', string)
+ string = RELATIVE_PATTERN.sub(r'(\\d+[.,]?\\d*)', string)

• Number regex changed in supplementary yaml files for custom relative-type-regex sections (multiple locations)

In all occurrences in supplementary yaml:
- (\d+)
+ (\d+[.,]?\d*)

• Fix for Croatian dates conflicting with decimal pattern (date.py line 110)

line 40
+ RE_SANITIZE_CROATIAN = re.compile(r'(\d+)\.\s?(\d+)\.\s?(\d+)\.( u)?', flags=re.I | re.U)
line 110
+ date_string = RE_SANITIZE_CROATIAN.sub(r'\1.\2.\3 ', date_string)  # extra '.' and 'u' interferes with parsing relative fractional dates

• Test case for number regex in data was updated in (test_data.py line 68)

- return not (all([isinstance(x, str) for x in value])
-                and all(['(\\d+)' in x for x in value]))
+ return not (all([isinstance(x, str) for x in value])
+                and all(['(\\d+[.,]?\\d*)' in x for x in value]))

• Test cases for fractional units added to (test_freshness_date_parser.py)

line 99
+ # Fractional units
+ param('2.5 hours', ago={'hours': 2.5}, period='day'),
+ param('10.75 minutes', ago={'minutes': 10.75}, period='day'),
+ param('1.5 days', ago={'days': 1.5}, period='day'),
line 616
+ # Fractional units
+ param('2.5 hours', ago={'hours': 2.5}, period='day'),
+ param('10.75 minutes', ago={'minutes': 10.75}, period='day'),
+ param('1.5 days', ago={'days': 1.5}, period='day'),
line 1108
+ # Fractional units
+ param('in 2.5 hours', in_future={'hours': 2.5}, period='day'),
+ param('in 10.75 minutes', in_future={'minutes': 10.75}, period='day'),
+ param('in 1.5 days', in_future={'days': 1.5}, period='day'),
+ param('in 0,5 hours', in_future={'hours': 0.5}, period='day'),
line 1635
+ # Fractional units
+ param('2.5 hours ago', date(2010, 6, 4), time(10, 45)),
+ param('in 10.75 minutes', date(2010, 6, 4), time(13, 25, 45)),
+ param('in 1.5 days', date(2010, 6, 6), time(1, 15)),
+ param('0,5 hours ago', date(2010, 6, 4), time(12, 45)),

See these comments for shortcuts to view the changes

Note: As a workaround for this issue, we have pinned regex. Which makes Python 3.11 support either impossible or uncomfortable. The goal now is to remove that version pin on regex without making this issue resurface.

Hello everyone,

Tried parsing under python 3.7.5 and 3.9

dateparser.parse('12/12/12')

It also gives the same output for any "valid" input shown in the doc:

dateparser.parse('Fri, 12 Dec 2014 10:55:50')
dateparser.parse('22 Décembre 2010', date_formats=['%d %B %Y'])
...

Here's the error:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
Input In [46], in <cell line: 1>()
----> 1 dateparser.parse("12/12/12")

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\conf.py:92, in apply_settings.<locals>.wrapper(*args, **kwargs)
     89 if not isinstance(kwargs['settings'], Settings):
     90     raise TypeError("settings can only be either dict or instance of Settings class")
---> 92 return f(*args, **kwargs)

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\__init__.py:61, in parse(date_string, date_formats, languages, locales, region, settings, detect_languages_function)
     57 if languages or locales or region or detect_languages_function or not settings._default:
     58     parser = DateDataParser(languages=languages, locales=locales,
     59                             region=region, settings=settings, detect_languages_function=detect_languages_function)
---> 61 data = parser.get_date_data(date_string, date_formats)
     63 if data:
     64     return data['date_obj']

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\date.py:428, in DateDataParser.get_date_data(self, date_string, date_formats)
    425 date_string = sanitize_date(date_string)
    427 for locale in self._get_applicable_locales(date_string):
--> 428     parsed_date = _DateLocaleParser.parse(
    429         locale, date_string, date_formats, settings=self._settings)
    430     if parsed_date:
    431         parsed_date['locale'] = locale.shortname

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\date.py:178, in _DateLocaleParser.parse(cls, locale, date_string, date_formats, settings)
    175 @classmethod
    176 def parse(cls, locale, date_string, date_formats=None, settings=None):
    177     instance = cls(locale, date_string, date_formats, settings)
--> 178     return instance._parse()

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\date.py:182, in _DateLocaleParser._parse(self)
    180 def _parse(self):
    181     for parser_name in self._settings.PARSERS:
--> 182         date_data = self._parsers[parser_name]()
    183         if self._is_valid_date_data(date_data):
    184             return date_data

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\date.py:196, in _DateLocaleParser._try_freshness_parser(self)
    194 def _try_freshness_parser(self):
    195     try:
--> 196         return freshness_date_parser.get_date_data(self._get_translated_date(), self._settings)
    197     except (OverflowError, ValueError):
    198         return None

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\date.py:234, in _DateLocaleParser._get_translated_date(self)
    232 def _get_translated_date(self):
    233     if self._translated_date is None:
--> 234         self._translated_date = self.locale.translate(
    235             self.date_string, keep_formatting=False, settings=self._settings)
    236     return self._translated_date

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\languages\locale.py:131, in Locale.translate(self, date_string, keep_formatting, settings)
    128 dictionary = self._get_dictionary(settings)
    129 date_string_tokens = dictionary.split(date_string, keep_formatting)
--> 131 relative_translations = self._get_relative_translations(settings=settings)
    133 for i, word in enumerate(date_string_tokens):
    134     word = word.lower()

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\languages\locale.py:158, in Locale._get_relative_translations(self, settings)
    155 if settings.NORMALIZE:
    156     if self._normalized_relative_translations is None:
    157         self._normalized_relative_translations = (
--> 158             self._generate_relative_translations(normalize=True))
    159     return self._normalized_relative_translations
    160 else:

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\dateparser\languages\locale.py:172, in Locale._generate_relative_translations(self, normalize)
    170     value = list(map(normalize_unicode, value))
    171 pattern = '|'.join(sorted(value, key=len, reverse=True))
--> 172 pattern = DIGIT_GROUP_PATTERN.sub(r'?P<n>\d+', pattern)
    173 pattern = re.compile(r'^(?:{})$'.format(pattern), re.UNICODE | re.IGNORECASE)
    174 relative_dictionary[pattern] = key

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\regex\regex.py:700, in _compile_replacement_helper(pattern, template)
    695     break
    696 if ch == "\\":
    697     # '_compile_replacement' will return either an int group reference
    698     # or a string literal. It returns items (plural) in order to handle
    699     # a 2-character literal (an invalid escape sequence).
--> 700     is_group, items = _compile_replacement(source, pattern, is_unicode)
    701     if is_group:
    702         # It's a group, so first flush the literal.
    703         if literal:

File c:\users\strey\appdata\local\programs\python\python39\lib\site-packages\regex\_regex_core.py:1736, in _compile_replacement(source, pattern, is_unicode)
   1733         if value is not None:
   1734             return False, [value]
-> 1736     raise error("bad escape \\%s" % ch, source.string, source.pos)
   1738 if isinstance(source.sep, bytes):
   1739     octal_mask = 0xFF

error: bad escape \d at position 7

How to reproduce: Env: windows 10

• Fresh install of python 3.7.5 or 3.9
• Make a simple python file including these 2 lines:

import dateparser
dateparser.parse("12/12/12")

Improving language detection using optional language detection library

Related Issues #567, #575, #612.

Implementing plugging wrappers for optional language detection library, wrapper template, and docs on implementing a custom wrapper.

@noviluni sir, This is PR which show's the code for error #817

This PR add's support for century word #725 So, here I was unable to generate en.py because of this (#817 ) error !! Thanks

We got surprised that dateparser.parse() gets slower over time (in a long-running process ~1 day), from hundereds of ms to 3 seconds per call! One possible cause is the following:

DateDataParser.get_date_data() stores some previously used locales:

if self.try_previous_locales:
    self.previous_locales.insert(0, locale)

However, we observed that dateparser._default_parser.previous_locales contained many duplicate instances of the same set of locales. Eg. 2189 instances of 63 unique locales.

The problem with the code above is that self.previous_locales should be a set, possibly ordered, but there's no check that locale is already inside.

Anyway, even when we run dateparser.parse() it's pretty slow (~400 ms / item). However with only languages=['en] it's really fast (~0.25 ms / item).

Package version: latest 0.7.0, Python 2.7.

Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dateparser
>>> dateparser.__version__
'0.5.0'
>>> import locale
>>> locale.getlocale()
(None, None)
>>> dateparser.parse('21-FEB-2016', date_formats=['%d-%b-%Y'], languages=['en'])
datetime.datetime(2016, 2, 21, 0, 0)
>>> locale.setlocale(locale.LC_ALL, 'fr_FR.UTF-8')
'fr_FR.UTF-8'
>>> dateparser.parse('21-FEB-2016', date_formats=['%d-%b-%Y'], languages=['en'])
>>>

This is a problem for me: I want to be able to get a good result of dateparser.parse when I give languages=['en'] as argument even if my locale is not english. The users of my software (Odoo) may have any locale, but dateparser.parse() used with the "languages" argument should always give the same result !

When upgrading to Python 3.9, getting the following warning:

Python 3.9.5 (default, Jun  4 2021, 12:28:51)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dateparser
>>> dateparser.parse("2021-08-02")
/opt/conda/lib/python3.9/site-packages/dateparser/date_parser.py:35: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
  date_obj = stz.localize(date_obj)
datetime.datetime(2021, 8, 2, 0, 0)

Is there an anticipated fix for this or a way to suppress this?

Thanks!

Using the common date abbreviations of Date + st, nd, rd, th alone returns months for 1st - 12th rather than days.

Example of current behavior (desired) with 13th - 31st

dateparser.parse("13th")
dateparser.parse("25th 5pm")
> 2023-01-13 00:00:00
> 2023-01-25 17:00:00

Example of current behavior (unwanted) with 1st - 12th

dateparser.parse("2nd 9pm")
dateparser.parse("12th 11am")
> 2023-02-09 21:00:00
> 2023-12-09 11:00:00

Example of desired behavior with 1st - 12th

dateparser.parse("2nd 9pm")
dateparser.parse("12th 11am")
> 2023-01-02 21:00:00
> 2023-01-12 11:00:00

I'm aware this may be avoided by passing a DATE_ORDER option through the settings, but that would break other functionality when the default order of MDY is desired. And the use of the date abbreviations specified above are very common even with MDY, so some sort of solution to allow both would be ideal.

When the current_period setting is used the current period for weekdays should be treated as the current week.

Test Date: 2023-01-09 00:00:00 (Monday) Test Input: dateparser.parse("Thursday", settings={'PREFER_DATES_FROM': 'current_period'}) Current Output: 2023-01-05 00:00:00 (Previous Week) Desired Output: 2023-01-12 00:00:00 (Current Week)

Hi,

Just in case, in the previous version we were able to parse the following 'Mon, 05 Dec 2022 19:43:03 GMT' without any problem and now is returning None.

This is happening only when is Monday.

It could be related to this commit: https://github.com/scrapinghub/dateparser/commit/89a5b29517a358e2784cc818687b7b6bed343159 ?

Thank you!

Best, Nicolas.

Using a settings dictionary when parsing a date results in an error if the provided object is a subclass of the type specified in the dateparser settings schema, such as when using a pendulum.DateTime object . So running

settings = {
    "RELATIVE_BASE": pendulum.now()
}

dateparser.parse("-15m", settings=settings)

will generate the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/python3.8/site-packages/dateparser/conf.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/lib/python3.8/site-packages/dateparser/__init__.py", line 58, in parse
    parser = DateDataParser(languages=languages, locales=locales,
  File "/lib/python3.8/site-packages/dateparser/conf.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/lib/python3.8/site-packages/dateparser/date.py", line 387, in __init__
    check_settings(settings)
  File "/lib/python3.8/site-packages/dateparser/conf.py", line 244, in check_settings
    raise SettingValidationError(
dateparser.conf.SettingValidationError: "RELATIVE_BASE" must be "datetime", not "DateTime".

Even though a pendulum.DateTime is a subclass of a datetime object. The check at this line https://github.com/scrapinghub/dateparser/blob/master/dateparser/conf.py#L243 should call isinstance instead of directly checking type equality to ensure subclassed objects are allowed in settings definitions. This is a regression from previous versions where pendulum.DateTime objects were allowed.

Close #1089

>>> from datetime import datetime
>>> dateparser.parse(
...     "in 3 hours",
...     settings={
...         "TO_TIMEZONE": "UTC",
...         "RELATIVE_BASE": datetime.now(),
...     },
... )
datetime.datetime(2022, 12, 7, 14, 43, 47, 345343)

Makes sense wherever it does not affect external APIs. e.g. if a function uses OrderedDict internally, then yes; if a function returns OrderedDict, we probably should follow a deprecation approach, e.g. deprecate that function (but keep it for a time) and provide a new one that returns a dict.