pyMinGW: old issues

ISSUES WITH pyMinGW
First of all, please be reminded that this is a work in progress. It is not yet possible to compile all of python with MinGW. Progress is taking place, but some issues remain.

THE py24a3MinGW PATCH
With regards to the py24a3MinGW patch, I have only had time to get pythoncore (python24.dll), python (python.exe), pythonw (pythonw.exe), and w9xpopen, select, zlib, pyexpat, _testcapi, unicodedata, winsound, bz2module, and module _socket to compile, which is good enough to run python and test it. What remains is the following: _bsddb, _tkinter, and _ssl modules, and making sure of course that the compiled modules pass python's tests.

The socket module seems like a big problem, especially since, among other things, the Windows version I have access to does not support IP v. 6, nor getaddrinfo, etc. and so even though I managed to get it to compile, it would be impossible for me to test it on my current system.

CRITICAL
Importing the zlib and the _testcapi modules crash python (python24a3); python also crashes (causes a bluescreen to show) when some tests use pyexpat, like test_sax, something which did not happen in the py234MinGW patch. So if you got those modules compiled, linked and living in the \DLL directory (or where you run python24a3) then it is wise to rename them for now while you test python. For example rename \Dll\pyexpat.pyd to \Dll\ORGpyexpat.pyd or something else you see fit and the same for _testcapi.pyd, and zlib.pyd.

Moreover, when the "regrtest.py" file is run, the following tests:

test_codeccallbacks,

test_codecs,

test_normalization,

test_pyexpat,

test_re,

test_minidom,

test_stringprep,

test_ucn,

test_unicode

test_unicode_file, and

test_unicodedata

cause the dreaded Windows' bluescreen to show. For now, these tests (all of them) must be removed if you plan to test your MinGW compiled python24a3 using the regrtest, which you should. One idea is to remove them from their location in \lib\test\ to some other directory or better still rename them. Don't downplay this; applying the patch while keeping those tests in their original names in their original locations and then running the regrtest will cause python to hang and it will stay in memory without even showing up in Windows running task list. You will only be able to shut-down your PC or reboot by pressing CTRL+ALT+DEL!

So rename "\Lib\test\test_codeccallbacks.py" to "\Lib\test\FAILING-BLUESCREEN-test_codeccallbacks.py" for example in order for other tests to proceed. Repeat the same for the other ten. It is imperative that these tests are removed someway or another for the regrtest to proceed.

One interesting thing to note here is that when called on their own, these tests pass (as in succeed) or skip quietly! Except for test_minidom, this holds true for test_codeccallbacks, test_codecs, test_normalization, test_pyexpat, test_re, test_stringprep, test_ucn, test_unicode, test_unicode_file, and test_unicodedata. Of all these tests, unfortunately, test_stringprep does not print any messages when run on its own; but it still does not cause a bluescreen to show. Here are the in-regrtest-bluescreen-causing tests, run on their own, i.e., outside of the regrtest framework.

test_unicodedata:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_unicodedata as t
>>> t.test_main()
test_decimal_numeric_consistent (test.test_unicodedata.UnicodeMiscTest) ... ok
test_digit_numeric_consistent (test.test_unicodedata.UnicodeMiscTest) ... ok
test_method_checksum (test.test_unicodedata.UnicodeMethodsTest) ... ok
test_bidirectional (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_category (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_combining (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_decimal (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_decomposition (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_digit (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_east_asian_width (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_function_checksum (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_mirrored (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_normalize (test.test_unicodedata.UnicodeFunctionsTest) ... ok
test_numeric (test.test_unicodedata.UnicodeFunctionsTest) ... ok

----------------------------------------------------------------------
Ran 14 tests in 33.450s

OK
>>>

test_unicode_file:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_unicode_file as t
>>> t.test_main()
test_directories (test.test_unicode_file.TestUnicodeFiles) ... ok
test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ok
test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.870s

OK
>>>

test_unicode:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_unicode as t
>>> t.test_main()
test___contains__ (test.test_unicode.UnicodeTest) ... ok
test_bug1001011 (test.test_unicode.UnicodeTest) ... ok
test_capitalize (test.test_unicode.UnicodeTest) ... ok
test_center (test.test_unicode.UnicodeTest) ... ok
test_codecs (test.test_unicode.UnicodeTest) ... ok
test_codecs_charmap (test.test_unicode.UnicodeTest) ... ok
test_codecs_errors (test.test_unicode.UnicodeTest) ... ok
test_codecs_idna (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf7 (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf8 (test.test_unicode.UnicodeTest) ... ok
test_comparison (test.test_unicode.UnicodeTest) ... ok
test_concatenation (test.test_unicode.UnicodeTest) ... ok
test_constructor (test.test_unicode.UnicodeTest) ... ok
test_contains (test.test_unicode.UnicodeTest) ... ok
test_count (test.test_unicode.UnicodeTest) ... ok
test_endswith (test.test_unicode.UnicodeTest) ... ok
test_expandtabs (test.test_unicode.UnicodeTest) ... ok
test_find (test.test_unicode.UnicodeTest) ... ok
test_floatformatting (test.test_unicode.UnicodeTest) ... ok
test_formatting (test.test_unicode.UnicodeTest) ... ok
test_index (test.test_unicode.UnicodeTest) ... ok
test_isalnum (test.test_unicode.UnicodeTest) ... ok
test_isalpha (test.test_unicode.UnicodeTest) ... ok
test_isdecimal (test.test_unicode.UnicodeTest) ... ok
test_isdigit (test.test_unicode.UnicodeTest) ... ok
test_islower (test.test_unicode.UnicodeTest) ... ok
test_isnumeric (test.test_unicode.UnicodeTest) ... ok
test_isspace (test.test_unicode.UnicodeTest) ... ok
test_istitle (test.test_unicode.UnicodeTest) ... ok
test_isupper (test.test_unicode.UnicodeTest) ... ok
test_join (test.test_unicode.UnicodeTest) ... ok
test_literals (test.test_unicode.UnicodeTest) ... ok
test_ljust (test.test_unicode.UnicodeTest) ... ok
test_lower (test.test_unicode.UnicodeTest) ... ok
test_mul (test.test_unicode.UnicodeTest) ... ok
test_printing (test.test_unicode.UnicodeTest) ... ok
test_replace (test.test_unicode.UnicodeTest) ... ok
test_repr (test.test_unicode.UnicodeTest) ... ok
test_rfind (test.test_unicode.UnicodeTest) ... ok
test_rindex (test.test_unicode.UnicodeTest) ... ok
test_rjust (test.test_unicode.UnicodeTest) ... ok
test_rsplit (test.test_unicode.UnicodeTest) ... ok
test_slice (test.test_unicode.UnicodeTest) ... ok
test_split (test.test_unicode.UnicodeTest) ... ok
test_splitlines (test.test_unicode.UnicodeTest) ... ok
test_startswith (test.test_unicode.UnicodeTest) ... ok
test_strip (test.test_unicode.UnicodeTest) ... ok
test_subscript (test.test_unicode.UnicodeTest) ... ok
test_swapcase (test.test_unicode.UnicodeTest) ... ok
test_title (test.test_unicode.UnicodeTest) ... ok
test_translate (test.test_unicode.UnicodeTest) ... ok
test_ucs4 (test.test_unicode.UnicodeTest) ... ok
test_upper (test.test_unicode.UnicodeTest) ... ok
test_zfill (test.test_unicode.UnicodeTest) ... ok

----------------------------------------------------------------------
Ran 54 tests in 5.220s

OK
>>>

test_ucn:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_ucn as t
>>> t.test_main()
test_ascii_letters (test.test_ucn.UnicodeNamesTest) ... ok
test_bmp_characters (test.test_ucn.UnicodeNamesTest) ... ok
test_cjk_unified_ideographs (test.test_ucn.UnicodeNamesTest) ... ok
test_errors (test.test_ucn.UnicodeNamesTest) ... ok
test_general (test.test_ucn.UnicodeNamesTest) ... ok
test_hangul_syllables (test.test_ucn.UnicodeNamesTest) ... ok
test_misc_symbols (test.test_ucn.UnicodeNamesTest) ... ok
test_strict_eror_handling (test.test_ucn.UnicodeNamesTest) ... ok

----------------------------------------------------------------------
Ran 8 tests in 4.880s

OK
>>>

test_re:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_re as t
>>> t.test_main()
test_anyall (test.test_re.ReTests) ... ok
test_basic_re_sub (test.test_re.ReTests) ... ok
test_bigcharset (test.test_re.ReTests) ... ok
test_bug_113254 (test.test_re.ReTests) ... ok
test_bug_114660 (test.test_re.ReTests) ... ok
test_bug_117612 (test.test_re.ReTests) ... ok
test_bug_418626 (test.test_re.ReTests) ... ok
test_bug_448951 (test.test_re.ReTests) ... ok
test_bug_449000 (test.test_re.ReTests) ... ok
test_bug_449964 (test.test_re.ReTests) ... ok
test_bug_462270 (test.test_re.ReTests) ... ok
test_bug_527371 (test.test_re.ReTests) ... ok
test_bug_545855 (test.test_re.ReTests) ... ok
test_bug_581080 (test.test_re.ReTests) ... ok
test_bug_612074 (test.test_re.ReTests) ... ok
test_bug_725106 (test.test_re.ReTests) ... ok
test_bug_725149 (test.test_re.ReTests) ... ok
test_bug_764548 (test.test_re.ReTests) ... ok
test_bug_817234 (test.test_re.ReTests) ... ok
test_bug_926075 (test.test_re.ReTests) ... ok
test_bug_931848 (test.test_re.ReTests) ... ok
test_category (test.test_re.ReTests) ... ok
test_constants (test.test_re.ReTests) ... ok
test_expand (test.test_re.ReTests) ... ok
test_finditer (test.test_re.ReTests) ... ok
test_flags (test.test_re.ReTests) ... ok
test_getattr (test.test_re.ReTests) ... ok
test_getlower (test.test_re.ReTests) ... ok
test_groupdict (test.test_re.ReTests) ... ok
test_ignore_case (test.test_re.ReTests) ... ok
test_non_consuming (test.test_re.ReTests) ... ok
test_not_literal (test.test_re.ReTests) ... ok
test_pickling (test.test_re.ReTests) ... ok
test_qualified_re_split (test.test_re.ReTests) ... ok
test_qualified_re_sub (test.test_re.ReTests) ... ok
test_re_escape (test.test_re.ReTests) ... ok
test_re_findall (test.test_re.ReTests) ... ok
test_re_groupref (test.test_re.ReTests) ... ok
test_re_groupref_exists (test.test_re.ReTests) ... ok
test_re_match (test.test_re.ReTests) ... ok
test_re_split (test.test_re.ReTests) ... ok
test_re_subn (test.test_re.ReTests) ... ok
test_repeat_minmax (test.test_re.ReTests) ... ok
test_scanner (test.test_re.ReTests) ... ok
test_search_coverage (test.test_re.ReTests) ... ok
test_search_star_plus (test.test_re.ReTests) ... ok
test_special_escapes (test.test_re.ReTests) ... ok
test_sre_character_class_literals (test.test_re.ReTests) ... ok
test_sre_character_literals (test.test_re.ReTests) ... ok
test_stack_overflow (test.test_re.ReTests) ... ok
test_sub_template_numeric_escape (test.test_re.ReTests) ... ok
test_symbolic_refs (test.test_re.ReTests) ... ok
test_weakref (test.test_re.ReTests) ... ok

----------------------------------------------------------------------
Ran 53 tests in 4.450s

OK
Running re_tests test suite
>>>

test_pyexpat:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46)
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_pyexpat as t
OK.
OK.
OK.
OK.
OK.
OK.
OK.
OK.
OK.
OK.
OK.
OK.
PI:
        'xml-stylesheet' 'href="stylesheet.css"'
Comment:

        ' comment data '
Notation declared: ('notation', None, 'notation.jpeg', None)
Unparsed entity decl:
        ('unparsed_entity', None, 'entity.file', None, 'notation')
Start element:
        'root' {'attr1': 'value1', 'attr2': 'value2\xe1\xbd\x80'}
NS decl:
        'myns' 'http://www.python.org/namespace'
Start element:
        'http://www.python.org/namespace!subelement' {}
Character data:
        'Contents of subelements'
End element:
        'http://www.python.org/namespace!subelement'
End of NS decl:
        'myns'
Start element:
        'sub2' {}
Start of CDATA section
Character data:
        'contents of CDATA section'
End of CDATA section
End element:
        'sub2'
External entity ref: (None, 'entity.file', None)
End element:
        'root'
PI:
        u'xml-stylesheet' u'href="stylesheet.css"'
Comment:
        u' comment data '
Notation declared: (u'notation', None, u'notation.jpeg', None)
Unparsed entity decl:
        (u'unparsed_entity', None, u'entity.file', None, u'notation')
Start element:
        u'root' {u'attr1': u'value1', u'attr2': u'value2\u1f40'}
NS decl:
        u'myns' u'http://www.python.org/namespace'
Start element:
        u'http://www.python.org/namespace!subelement' {}
Character data:
        u'Contents of subelements'
End element:
        u'http://www.python.org/namespace!subelement'
End of NS decl:
        u'myns'
Start element:
        u'sub2' {}
Start of CDATA section
Character data:
        u'contents of CDATA section'
End of CDATA section
End element:
        u'sub2'
External entity ref: (None, u'entity.file', None)
End element:
        u'root'
PI:
        u'xml-stylesheet' u'href="stylesheet.css"'
Comment:
        u' comment data '
Notation declared: (u'notation', None, u'notation.jpeg', None)
Unparsed entity decl:
        (u'unparsed_entity', None, u'entity.file', None, u'notation')
Start element:
        u'root' {u'attr1': u'value1', u'attr2': u'value2\u1f40'}
NS decl:
        u'myns' u'http://www.python.org/namespace'
Start element:
        u'http://www.python.org/namespace!subelement' {}
Character data:
        u'Contents of subelements'
End element:
        u'http://www.python.org/namespace!subelement'
End of NS decl:
        u'myns'
Start element:
        u'sub2' {}
Start of CDATA section
Character data:
        u'contents of CDATA section'
End of CDATA section
End element:
        u'sub2'
External entity ref: (None, u'entity.file', None)
End element:
        u'root'

Testing constructor for proper handling of namespace_separator values:
Legal values tested o.k.
Caught expected TypeError:
ParserCreate() argument 2 must be string or None, not int
Caught expected ValueError:
namespace_separator must be at most one character, omitted, or None
>>>

test_normalization:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_normalization as t
>>> t.test_main()
Traceback (most recent call last):
  File "", line 1, in ?
  File "F:\PROGRAMS\MISC\BIN\PY24\PYTHON\DIST\SRC\lib\test\test_normalization.py", line 44, in test_main
    raise TestSkipped(TESTDATAFILE + " not found, download from " +
test.test_support.TestSkipped: NormalizationTest-3.2.0.txt not found, 
download from http://www.unicode.org/Public/3.2-Update/NormalizationTest-
3.2.0.txt
>>>

test_codecs:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_codecs as t
>>> t.test_main()
test_only_one_bom (test.test_codecs.UTF16Test) ... ok
test_partial (test.test_codecs.UTF16Test) ... ok
test_partial (test.test_codecs.UTF16LETest) ... ok
test_partial (test.test_codecs.UTF16BETest) ... ok
test_partial (test.test_codecs.UTF8Test) ... ok
test_empty_escape_decode (test.test_codecs.EscapeDecodeTest) ... ok
test_recoding (test.test_codecs.RecodingTest) ... ok
test_decode (test.test_codecs.PunycodeTest) ... ok
test_encode (test.test_codecs.PunycodeTest) ... ok
test_nameprep (test.test_codecs.NameprepTest) ... ok
test_builtin (test.test_codecs.CodecTest) ... ok
test_decode (test.test_codecs.CodecsModuleTest) ... ok
test_encode (test.test_codecs.CodecsModuleTest) ... ok

----------------------------------------------------------------------
Ran 13 tests in 2.140s

OK
>>>

test_codeccallbacks:

$ python -i
Python 2.4a3 (#56, Sep  9 2004, 03:39:46) 
[GCC 3.4.1 (mingw special)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import test_codeccallbacks as t
>>> t.main()
Traceback (most recent call last):
  File "", line 1, in ?
AttributeError: 'module' object has no attribute 'main'
>>> t.test_main()
test_backslashescape (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodbackslashreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodignoreexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodstrictexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badandgoodxmlcharrefreplaceexceptions (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badhandlerresults (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_badregistercall (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_bug828737 (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_callbacks (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_charmapencode (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_decodehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_encodehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_longstrings (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_lookup (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_relaxedutf8 (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_translatehelper (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unencodablereplacement (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodedecodeerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodeencodeerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unicodetranslateerror (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_uninamereplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_unknownhandler (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharnamereplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharrefreplace (test.test_codeccallbacks.CodecCallbackTest) ... ok
test_xmlcharrefvalues (test.test_codeccallbacks.CodecCallbackTest) ... ok

----------------------------------------------------------------------
Ran 26 tests in 1.260s

OK
>>>

Perhaps it might be worthwhile to mention that the PC on which these tests were run had only 64MB of memory. Could that highlight a memory related problem with python24a3?

At anyrate, and other than what was mentioned above, python seems to run okay, most of the time, and it passes those tests that were compiled into its core. Those tests that are related to modules that were not compiled into the core or otherwise are either skipped or they fail, as expected.

Having said that, there seems to be another problem that I cannot explain. Some tests sometimes pass and sometimes don't, namely test_compiler, test_deque, test_descr, and I have even seen this occur to test_decimal. Though increasingly getting infrequent, these problems are still strange, and are rather unpredictable compared to the case of the bluescreen-causing tests mentioned above, where the problems are repeatable; the problems here are not, so far at least. The -v option in regrtest or the lack thereof does not seem to affect this problem in anyway, though I initially thought that this might be the case. One thing I noticed is that if I run the regrtest in python24a3 after I just re-boot that these tests pass, most of the time.

Moreover. and like the tests mentioned above, if I try to run the tests individually then most of the time the tests pass. And I say most of the time, because I still have not pinned this down, nor have I done enough testing to say "all of the time" with confidence. This needs further investigation.

BTW, I think it was in August of 2004 that there was talk on dev-python about a blown stack problem in the current Python 2.4aX code (of the time of course), which was traced back to January of 2004, last time I checked. Could that be it? Following dev-python's direction, I tried to link python with the following command:

$(CPP) -Wl,--stack,0x2000000 $(LINKOBJ) -o $(BIN) $(LIBS)

so as to boost the stack size like it is done in the Visual Studio compiled version, but that still does not help. In fact GCC, I know I read this somewhere, defaults to using a 2MB stack; but I linked using that command anyway in case the MinGW flavor of GCC used a 1MB stack instead on Windows.

At any rate, various test results are here. (192KB). Among other things, you can find there the "Unknown opcode" problem that sometimes affects test_compiler, among others. Don't dispair, you can also see python passing many tests!

THE py234MinGW PATCH
As regards to py234MinGW version, then let me report that there are many modules that are compiled fine, and the expected tests that must pass do so. People who are in a hurry are encouraged to download the lastest py234MinGW if at least to get the "Make" and "Dev" files that (with some modification, i.e. linking against libpython24.a, for example, as opposed to libpython23.a) should (hopefully) be able to compile all the py24MinGW modules.

Among several things, the new py24a3MinGW added a make file named python23.mak so as to try to ease the building of Python-2.3.4. I confess that I copied it untested from my python24.mak (which is working fine). But since the modules built into the core in python24a3 are different from those in python234, then to get maximum benefit from the makefile, you should unfortunately edit it yourself and make some changes. I'll get at this once I have more time. At least now you have something to work with. Update: the make file is now done. I have not tested it yet, but it should work.

PROBLEMS WITH py234MinGW
Having said that, let me add that there are two major problems that still remain with py234MinGW: the datetime module and (yes my eternal foe!) the socket module. They fail the tests (even when they compile successfully). I have almost given up on them, expecially since the new datetime module included in python 2.4aX compiles fine and passes all its tests with flying colors.

OLD ISSUES
The old issues file(s), kept for historical reasons only, can be found here.