in reply to Re: Rosetta Code: Long List is Long -- Python
in thread Rosetta Code: Long List is Long

This is neat, parv. I compared Python 3.10.7 against Pyston-lite and Pyston.

Python:

$ python3 llil.py big1.txt big2.txt big3.txt > out1.txt
collect time            :    5.2 s
sort_via_cmp_to_key time:   17.7 s
sort+format time        :   17.7 s
total processing time   :   22.9 s
output time             :    3.5 s
total time              :   26.4 s

[download]

Python with Pyston-lite loaded automatically:

# Install pyston_lite_autoload to ~/.local/lib/python3.10/site-package
+s/.
$ python3 -m pip install --user pyston_lite_autoload

$ python3 llil.py big1.txt big2.txt big3.txt > out2.txt
collect time            :    4.4 s
sort_via_cmp_to_key time:   15.8 s
sort+format time        :   15.8 s
total processing time   :   20.2 s
output time             :    3.2 s
total time              :   23.4 s

[download]

Pyston 2.3.5:

I also tried Pyston 2.3.5. But first, I needed to modify the function declarations to resolve "TypeError: 'type' object is not subscriptable.

$ diff llil.py llil2.py 
53c53
< def collect( data_list :list ) ->dict[ str, int ]:
---
> def collect( data_list :list ) ->dict:
87c87
< def process( cat_count :dict[ str, int ] ) ->Generator[ str, None, N
+one ]:
---
> def process( cat_count :dict ) ->Generator:
110c110
< def sort_via_cmp_to_key( cat_count :dict[ str, int ] ) ->list[ str ]
+:
---
> def sort_via_cmp_to_key( cat_count :dict ) ->list:

[download]
$ ./pyston_2.3.5/bin/pyston3 llil2.py big1.txt big2.txt big3.txt > out
+3.txt
collect time            :    3.7 s
sort_via_cmp_to_key time:   12.2 s
sort+format time        :   12.2 s
total processing time   :   15.9 s
output time             :    3.0 s
total time              :   18.8 s

[download]

Look at Pyston go; taking Python performance to a new level :) How does Python/Pyston compare to Perl? I ran the dualvar demonstration for comparison.

$ perl dualvar.pl big1.txt big2.txt big3.txt >out4.txt
start
get properties: 6 secs
sort + output: 16 secs
total: 22 secs

[download]

The Perl code consumes around 2340 MB versus 1790 MB for Python.

See also:

Python 3.12 Goals

Python is about to get faster

The Pyston Blog

Numba

Replies are listed 'Best First'.
Re^3: Rosetta Code: Long List is Long -- Python
by parv (Parson) on Dec 10, 2022 at 04:51 UTC

    In short, could you do another Python, Pyston* run with sort function replaced with ... 

    def sort_native( cat_count ):
   once = sorted( cat_count.keys() )
   return sorted( once, key = lambda k: cat_count[ k ], reverse = True
+ )

    [download]

    ...?

    In long, before eyepopslikeamosquito posted ... 

    sort { $href->{$b} <=> $href->{$a} } sort keys %{$href}

    [download]

    ... I had missed to notice the sorting order. I decided to do the same for Python (native) version instead of using functools.cmp_to_key function. I also realized that I was doing the sorting other way (sorting keys by value, followed by sorting by key), so was not getting the expected output (which made me to use cmp_to_key instead).

    Replacing sort_via_cmp_to_key with ... 

    def sort_native( cat_count :dict[ str, int ] ) ->list[ str ]:
   """
   Returns:
      A `list` of sorted keys by decreasing order of values & increasi
+ng order of
      keys.

   Args:
      cat_count: A `dict` with string key & integer value.
   """
   once = sorted( cat_count.keys() )
   return sorted( once, key = lambda k: cat_count[ k ], reverse = True
+ )

    [download]

    ... reduces the sort time by ~10 times (~11-16 s; also produces the expected output) in my run environment.

    time passes, so slowly🎶 ... Putting the complete program here (~28-35 s) ...

    Read more... Python version, updated with native sort (6 kB)
[reply]
[d/l]
[select]
      > In short, could you do another Python, Pyston* run with sort function replaced with ...

      First, I made the following modification to resolve "TypeError: 'type' object is not subscriptable" using Pyston 2.3.5.

      $ diff llil1.py llil2.py 
53c53
< def collect( data_list :list ) ->dict[ str, int ]:
---
> def collect( data_list :list ) ->dict:
87c87
< def process( cat_count :dict[ str, int ] ) ->Generator[ str, None, N
+one ]:
---
> def process( cat_count :dict ) ->Generator:
110c110
< def sort_native( cat_count :dict[ str, int ] ) ->list[ str ]:
---
> def sort_native( cat_count :dict ) ->list:

      [download]

      Python 3.10.7:

      # I have pyston_lite_autoload installed already. It can be disabled.
$ DISABLE_PYSTON=1 python3 llil2.py big1.txt big2.txt big3.txt > out1.
+txt
collect time         :    5.1 s
sort+format time     :    3.6 s
total processing time:    8.7 s
output time          :    3.6 s
total time           :   12.2 s

      [download]

      Python with Pyston-lite loaded automatically:

      # Install pyston_lite_autoload to ~/.local/lib/python3.10/site-package
+s/.
$ python3 -m pip install --user pyston_lite_autoload

$ python3 llil2.py big1.txt big2.txt big3.txt > out2.txt
collect time         :    4.3 s
sort+format time     :    3.5 s
total processing time:    7.9 s
output time          :    3.3 s
total time           :   11.2 s

      [download]

      Pyston 2.3.5:

      $ ./pyston_2.3.5/bin/pyston3 llil2.py big1.txt big2.txt big3.txt > out
+3.txt
collect time         :    3.7 s
sort+format time     :    3.4 s
total processing time:    7.1 s
output time          :    3.0 s
total time           :   10.1 s

      [download]

      Pyston makes it all the more fun.

[reply]
[d/l]
[select]

        Neat, sorting is now ~4-5 times faster there. Thanks for the re-run.

[reply]

In Section Meditations