Mailing List Archive

Fast full-text searching in Python (job for Whoosh?)
I need fast text-search on a large (not huge, let's say 30k records
totally) list of items. Here's a sample of my raw data (a list of US
cars: model and make)

$ head all_cars_unique.csv\
Acura,CL
Acura,ILX
Acura,Integra
Acura,Legend
Acura,MDX
Acura,MDX Sport Hybrid
Acura,NSX
Acura,RDX
Acura,RL
Acura,RLX
$ wc -l all_cars_unique.csv
1415 all_cars_unique.csv
$ grep -i v60 all_cars_unique.csv
Genesis,GV60
Volvo,V60
$

Essentially, I want my input field to suggest autofill options with data
from this file/list. The user types "v60" and a REST point will offer:

[
{"model":"GV60", "manufacturer":"Genesis"},
{"model":"V60", "manufacturer":"Volvo"}
]

i.e. a JSON response that I can use to generate the autofill with
JavaScript. My Back-End is Python (Flask).

How can I implement this? A library called Whoosh seems very promising
(albeit it's so feature-rich that it's almost like shooting a fly with a
bazooka in my case), but I see two problems:

1) Whoosh is either abandoned or the project is a mess in terms of
community and support (https://groups.google.com/g/whoosh/c/QM_P8cGi4v4
) and

2) Whoosh seems to be a Python only thing, which is great for now, but
I wouldn't want this to become an obstacle should I need port it to a
different language at some point.

are there other options that are fast out there? Can I "grep" through a
data structure in python... but faster?

Thanks

Dino

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Here's the complete data file should anyone care.

Acura,CL
Acura,ILX
Acura,Integra
Acura,Legend
Acura,MDX
Acura,MDX Sport Hybrid
Acura,NSX
Acura,RDX
Acura,RL
Acura,RLX
Acura,RLX Sport Hybrid
Acura,RSX
Acura,SLX
Acura,TL
Acura,TLX
Acura,TSX
Acura,Vigor
Acura,ZDX
Alfa Romeo,164
Alfa Romeo,4C
Alfa Romeo,4C Spider
Alfa Romeo,Giulia
Alfa Romeo,Spider
Alfa Romeo,Stelvio
Alfa Romeo,Tonale
Aston Martin,DB11
Aston Martin,DB9
Aston Martin,DB9 GT
Aston Martin,DBS
Aston Martin,DBS Superleggera
Aston Martin,DBX
Aston Martin,Rapide
Aston Martin,Rapide S
Aston Martin,Vanquish
Aston Martin,Vanquish S
Aston Martin,Vantage
Aston Martin,Virage
Audi,100
Audi,80
Audi,90
Audi,A3
Audi,A3 Sportback e-tron
Audi,A4
Audi,A4 (2005.5)
Audi,A4 allroad
Audi,A5
Audi,A5 Sport
Audi,A6
Audi,A6 allroad
Audi,A7
Audi,A8
Audi,Cabriolet
Audi,Q3
Audi,Q4 Sportback e-tron
Audi,Q4 e-tron
Audi,Q5
Audi,Q5 Sportback
Audi,Q7
Audi,Q8
Audi,Quattro
Audi,R8
Audi,RS 3
Audi,RS 4
Audi,RS 5
Audi,RS 6
Audi,RS 7
Audi,RS Q8
Audi,RS e-tron GT
Audi,S3
Audi,S4
Audi,S4 (2005.5)
Audi,S5
Audi,S6
Audi,S7
Audi,S8
Audi,SQ5
Audi,SQ5 Sportback
Audi,SQ7
Audi,SQ8
Audi,TT
Audi,allroad
Audi,e-tron
Audi,e-tron GT
Audi,e-tron S
Audi,e-tron S Sportback
Audi,e-tron Sportback
BMW,1 Series
BMW,2 Series
BMW,3 Series
BMW,4 Series
BMW,5 Series
BMW,6 Series
BMW,7 Series
BMW,8 Series
BMW,Alpina B7
BMW,M
BMW,M2
BMW,M3
BMW,M4
BMW,M5
BMW,M6
BMW,M8
BMW,X1
BMW,X2
BMW,X3
BMW,X3 M
BMW,X4
BMW,X4 M
BMW,X5
BMW,X5 M
BMW,X6
BMW,X6 M
BMW,X7
BMW,Z3
BMW,Z4
BMW,Z4 M
BMW,Z8
BMW,i3
BMW,i4
BMW,i7
BMW,i8
BMW,iX
Bentley,Arnage
Bentley,Azure
Bentley,Azure T
Bentley,Bentayga
Bentley,Brooklands
Bentley,Continental
Bentley,Continental GT
Bentley,Flying Spur
Bentley,Mulsanne
Buick,Cascada
Buick,Century
Buick,Enclave
Buick,Encore
Buick,Encore GX
Buick,Envision
Buick,LaCrosse
Buick,LeSabre
Buick,Lucerne
Buick,Park Avenue
Buick,Rainier
Buick,Regal
Buick,Regal Sportback
Buick,Regal TourX
Buick,Rendezvous
Buick,Riviera
Buick,Roadmaster
Buick,Skylark
Buick,Terraza
Buick,Verano
Cadillac,ATS
Cadillac,ATS-V
Cadillac,Allante
Cadillac,Brougham
Cadillac,CT4
Cadillac,CT5
Cadillac,CT6
Cadillac,CT6-V
Cadillac,CTS
Cadillac,CTS-V
Cadillac,Catera
Cadillac,DTS
Cadillac,DeVille
Cadillac,ELR
Cadillac,Eldorado
Cadillac,Escalade
Cadillac,Escalade ESV
Cadillac,Escalade EXT
Cadillac,Fleetwood
Cadillac,LYRIQ
Cadillac,SRX
Cadillac,STS
Cadillac,Seville
Cadillac,Sixty Special
Cadillac,XLR
Cadillac,XT4
Cadillac,XT5
Cadillac,XT6
Cadillac,XTS
Chevrolet,1500 Extended Cab
Chevrolet,1500 Regular Cab
Chevrolet,2500 Crew Cab
Chevrolet,2500 Extended Cab
Chevrolet,2500 HD Extended Cab
Chevrolet,2500 HD Regular Cab
Chevrolet,2500 Regular Cab
Chevrolet,3500 Crew Cab
Chevrolet,3500 Extended Cab
Chevrolet,3500 HD Extended Cab
Chevrolet,3500 HD Regular Cab
Chevrolet,3500 Regular Cab
Chevrolet,APV Cargo
Chevrolet,Astro Cargo
Chevrolet,Astro Passenger
Chevrolet,Avalanche
Chevrolet,Avalanche 1500
Chevrolet,Avalanche 2500
Chevrolet,Aveo
Chevrolet,Beretta
Chevrolet,Blazer
Chevrolet,Blazer EV
Chevrolet,Bolt EUV
Chevrolet,Bolt EV
Chevrolet,Camaro
Chevrolet,Caprice
Chevrolet,Caprice Classic
Chevrolet,Captiva Sport
Chevrolet,Cavalier
Chevrolet,City Express
Chevrolet,Classic
Chevrolet,Cobalt
Chevrolet,Colorado Crew Cab
Chevrolet,Colorado Extended Cab
Chevrolet,Colorado Regular Cab
Chevrolet,Corsica
Chevrolet,Corvette
Chevrolet,Cruze
Chevrolet,Cruze Limited
Chevrolet,Equinox
Chevrolet,Equinox EV
Chevrolet,Express 1500 Cargo
Chevrolet,Express 1500 Passenger
Chevrolet,Express 2500 Cargo
Chevrolet,Express 2500 Passenger
Chevrolet,Express 3500 Cargo
Chevrolet,Express 3500 Passenger
Chevrolet,G-Series 1500
Chevrolet,G-Series 2500
Chevrolet,G-Series 3500
Chevrolet,G-Series G10
Chevrolet,G-Series G20
Chevrolet,G-Series G30
Chevrolet,HHR
Chevrolet,Impala
Chevrolet,Impala Limited
Chevrolet,Lumina
Chevrolet,Lumina APV
Chevrolet,Lumina Cargo
Chevrolet,Lumina Passenger
Chevrolet,Malibu
Chevrolet,Malibu (Classic)
Chevrolet,Malibu Limited
Chevrolet,Metro
Chevrolet,Monte Carlo
Chevrolet,Prizm
Chevrolet,S10 Blazer
Chevrolet,S10 Crew Cab
Chevrolet,S10 Extended Cab
Chevrolet,S10 Regular Cab
Chevrolet,SS
Chevrolet,SSR
Chevrolet,Silverado (Classic) 1500 Crew Cab
Chevrolet,Silverado (Classic) 1500 Extended Cab
Chevrolet,Silverado (Classic) 1500 HD Crew Cab
Chevrolet,Silverado (Classic) 1500 Regular Cab
Chevrolet,Silverado (Classic) 2500 HD Crew Cab
Chevrolet,Silverado (Classic) 2500 HD Extended Cab
Chevrolet,Silverado (Classic) 2500 HD Regular Cab
Chevrolet,Silverado (Classic) 3500 Crew Cab
Chevrolet,Silverado (Classic) 3500 Extended Cab
Chevrolet,Silverado (Classic) 3500 Regular Cab
Chevrolet,Silverado 1500 Crew Cab
Chevrolet,Silverado 1500 Double Cab
Chevrolet,Silverado 1500 Extended Cab
Chevrolet,Silverado 1500 HD Crew Cab
Chevrolet,Silverado 1500 LD Double Cab
Chevrolet,Silverado 1500 Limited Crew Cab
Chevrolet,Silverado 1500 Limited Double Cab
Chevrolet,Silverado 1500 Limited Regular Cab
Chevrolet,Silverado 1500 Regular Cab
Chevrolet,Silverado 2500 Crew Cab
Chevrolet,Silverado 2500 Extended Cab
Chevrolet,Silverado 2500 HD Crew Cab
Chevrolet,Silverado 2500 HD Double Cab
Chevrolet,Silverado 2500 HD Extended Cab
Chevrolet,Silverado 2500 HD Regular Cab
Chevrolet,Silverado 2500 Regular Cab
Chevrolet,Silverado 3500 Crew Cab
Chevrolet,Silverado 3500 Extended Cab
Chevrolet,Silverado 3500 HD Crew Cab
Chevrolet,Silverado 3500 HD Double Cab
Chevrolet,Silverado 3500 HD Extended Cab
Chevrolet,Silverado 3500 HD Regular Cab
Chevrolet,Silverado 3500 Regular Cab
Chevrolet,Silverado EV
Chevrolet,Sonic
Chevrolet,Spark
Chevrolet,Spark EV
Chevrolet,Sportvan G10
Chevrolet,Sportvan G20
Chevrolet,Sportvan G30
Chevrolet,Suburban
Chevrolet,Suburban 1500
Chevrolet,Suburban 2500
Chevrolet,Suburban 3500HD
Chevrolet,Tahoe
Chevrolet,Tahoe (New)
Chevrolet,Tracker
Chevrolet,TrailBlazer
Chevrolet,Trailblazer
Chevrolet,Traverse
Chevrolet,Trax
Chevrolet,Uplander Cargo
Chevrolet,Uplander Passenger
Chevrolet,Venture Cargo
Chevrolet,Venture Passenger
Chevrolet,Volt
Chrysler,200
Chrysler,300
Chrysler,300M
Chrysler,Aspen
Chrysler,Cirrus
Chrysler,Concorde
Chrysler,Crossfire
Chrysler,Fifth Ave
Chrysler,Grand Voyager
Chrysler,Imperial
Chrysler,LHS
Chrysler,LeBaron
Chrysler,New Yorker
Chrysler,PT Cruiser
Chrysler,Pacifica
Chrysler,Pacifica Hybrid
Chrysler,Prowler
Chrysler,Sebring
Chrysler,Town & Country
Chrysler,Voyager
Daewoo,Lanos
Daewoo,Leganza
Daewoo,Nubira
Daihatsu,Charade
Daihatsu,Rocky
Dodge,Avenger
Dodge,Caliber
Dodge,Caravan Cargo
Dodge,Caravan Passenger
Dodge,Challenger
Dodge,Charger
Dodge,Colt
Dodge,D150 Club Cab
Dodge,D150 Regular Cab
Dodge,D250 Club Cab
Dodge,D250 Regular Cab
Dodge,D350 Club Cab
Dodge,D350 Regular Cab
Dodge,Dakota Club Cab
Dodge,Dakota Crew Cab
Dodge,Dakota Extended Cab
Dodge,Dakota Quad Cab
Dodge,Dakota Regular Cab
Dodge,Dart
Dodge,Daytona
Dodge,Durango
Dodge,Dynasty
Dodge,Grand Caravan Cargo
Dodge,Grand Caravan Passenger
Dodge,Hornet
Dodge,Intrepid
Dodge,Journey
Dodge,Magnum
Dodge,Monaco
Dodge,Neon
Dodge,Nitro
Dodge,Ram 1500 Club Cab
Dodge,Ram 1500 Crew Cab
Dodge,Ram 1500 Mega Cab
Dodge,Ram 1500 Quad Cab
Dodge,Ram 1500 Regular Cab
Dodge,Ram 2500 Club Cab
Dodge,Ram 2500 Crew Cab
Dodge,Ram 2500 Mega Cab
Dodge,Ram 2500 Quad Cab
Dodge,Ram 2500 Regular Cab
Dodge,Ram 3500 Club Cab
Dodge,Ram 3500 Crew Cab
Dodge,Ram 3500 Mega Cab
Dodge,Ram 3500 Quad Cab
Dodge,Ram 3500 Regular Cab
Dodge,Ram 50 Regular Cab
Dodge,Ram Van 1500
Dodge,Ram Van 2500
Dodge,Ram Van 3500
Dodge,Ram Van B150
Dodge,Ram Van B250
Dodge,Ram Van B350
Dodge,Ram Wagon 1500
Dodge,Ram Wagon 2500
Dodge,Ram Wagon 3500
Dodge,Ram Wagon B150
Dodge,Ram Wagon B250
Dodge,Ram Wagon B350
Dodge,Ramcharger
Dodge,Shadow
Dodge,Spirit
Dodge,Sprinter 2500 Cargo
Dodge,Sprinter 2500 Passenger
Dodge,Sprinter 3500 Cargo
Dodge,Stealth
Dodge,Stratus
Dodge,Viper
Eagle,Premier
Eagle,Summit
Eagle,Talon
Eagle,Vision
FIAT,124 Spider
FIAT,500
FIAT,500 Abarth
FIAT,500L
FIAT,500X
FIAT,500c
FIAT,500c Abarth
FIAT,500e
Ferrari,296 GTB
Ferrari,430 Scuderia
Ferrari,458 Italia
Ferrari,458 Speciale
Ferrari,458 Spider
Ferrari,488 GTB
Ferrari,488 Pista
Ferrari,488 Spider
Ferrari,599 GTB Fiorano
Ferrari,599 GTO
Ferrari,612 Scaglietti
Ferrari,812 Competizione
Ferrari,812 Competizione A
Ferrari,812 GTS
Ferrari,812 Superfast
Ferrari,California
Ferrari,F12berlinetta
Ferrari,F430
Ferrari,F8
Ferrari,FF
Ferrari,GTC4Lusso
Ferrari,Portofino
Ferrari,Roma
Ferrari,SF90
Fisker,Karma
Ford,Aerostar Cargo
Ford,Aerostar Passenger
Ford,Aspire
Ford,Bronco
Ford,Bronco Sport
Ford,C-MAX Energi
Ford,C-MAX Hybrid
Ford,Club Wagon
Ford,Contour
Ford,Crown Victoria
Ford,E-Transit 350 Cargo Van
Ford,E150 Cargo
Ford,E150 Passenger
Ford,E150 Super Duty Cargo
Ford,E150 Super Duty Passenger
Ford,E250 Cargo
Ford,E250 Super Duty Cargo
Ford,E350 Super Duty Cargo
Ford,E350 Super Duty Passenger
Ford,EcoSport
Ford,Econoline E150 Cargo
Ford,Econoline E150 Passenger
Ford,Econoline E250 Cargo
Ford,Econoline E350 Cargo
Ford,Econoline E350 Super Duty Cargo
Ford,Econoline E350 Super Duty Passenger
Ford,Edge
Ford,Escape
Ford,Escape Plug-in Hybrid
Ford,Escort
Ford,Excursion
Ford,Expedition
Ford,Expedition EL
Ford,Expedition MAX
Ford,Explorer
Ford,Explorer Sport
Ford,Explorer Sport Trac
Ford,F150 (Heritage) Regular Cab
Ford,F150 (Heritage) Super Cab
Ford,F150 Lightning
Ford,F150 Regular Cab
Ford,F150 Super Cab
Ford,F150 SuperCrew Cab
Ford,F250 Crew Cab
Ford,F250 Regular Cab
Ford,F250 Super Cab
Ford,F250 Super Duty Crew Cab
Ford,F250 Super Duty Regular Cab
Ford,F250 Super Duty Super Cab
Ford,F350 Crew Cab
Ford,F350 Regular Cab
Ford,F350 Super Cab
Ford,F350 Super Duty Crew Cab
Ford,F350 Super Duty Regular Cab
Ford,F350 Super Duty Super Cab
Ford,F450 Super Duty Crew Cab
Ford,F450 Super Duty Regular Cab
Ford,Festiva
Ford,Fiesta
Ford,Five Hundred
Ford,Flex
Ford,Focus
Ford,Focus ST
Ford,Freestar Cargo
Ford,Freestar Passenger
Ford,Freestyle
Ford,Fusion
Ford,Fusion Energi
Ford,Fusion Plug-in Hybrid
Ford,GT
Ford,Maverick
Ford,Mustang
Ford,Mustang MACH-E
Ford,Probe
Ford,Ranger Regular Cab
Ford,Ranger Super Cab
Ford,Ranger SuperCab
Ford,Ranger SuperCrew
Ford,Taurus
Ford,Taurus X
Ford,Tempo
Ford,Thunderbird
Ford,Transit 150 Cargo Van
Ford,Transit 150 Crew Van
Ford,Transit 150 Passenger Van
Ford,Transit 150 Van
Ford,Transit 150 Wagon
Ford,Transit 250 Cargo Van
Ford,Transit 250 Crew Van
Ford,Transit 250 Van
Ford,Transit 350 Cargo Van
Ford,Transit 350 Crew Van
Ford,Transit 350 HD Cargo Van
Ford,Transit 350 HD Crew Van
Ford,Transit 350 HD Van
Ford,Transit 350 Passenger Van
Ford,Transit 350 Van
Ford,Transit 350 Wagon
Ford,Transit Connect Cargo
Ford,Transit Connect Cargo Van
Ford,Transit Connect Passenger
Ford,Transit Connect Passenger Wagon
Ford,Windstar Cargo
Ford,Windstar Passenger
Ford,ZX2
Freightliner,Sprinter 1500 Cargo
Freightliner,Sprinter 1500 Passenger
Freightliner,Sprinter 2500 Cargo
Freightliner,Sprinter 2500 Crew
Freightliner,Sprinter 2500 Passenger
Freightliner,Sprinter 3500 Cargo
Freightliner,Sprinter 3500 Crew
Freightliner,Sprinter 3500 XD Crew
Freightliner,Sprinter 3500XD Cargo
Freightliner,Sprinter 4500 Cargo
Freightliner,Sprinter 4500 Crew
Freightliner,Sprinter WORKER Cargo
Freightliner,Sprinter WORKER Passenger
GMC,1500 Club Coupe
GMC,1500 Regular Cab
GMC,2500 Club Coupe
GMC,2500 Crew Cab
GMC,2500 HD Club Coupe
GMC,2500 HD Extended Cab
GMC,2500 HD Regular Cab
GMC,2500 Regular Cab
GMC,3500 Club Coupe
GMC,3500 Crew Cab
GMC,3500 Extended Cab
GMC,3500 Regular Cab
GMC,Acadia
GMC,Acadia Limited
GMC,Canyon Crew Cab
GMC,Canyon Extended Cab
GMC,Canyon Regular Cab
GMC,Envoy
GMC,Envoy XL
GMC,Envoy XUV
GMC,Hummer EV Pickup
GMC,Hummer EV SUV
GMC,Jimmy
GMC,Rally Wagon 1500
GMC,Rally Wagon 2500
GMC,Rally Wagon 3500
GMC,Rally Wagon G2500
GMC,Rally Wagon G3500
GMC,Safari Cargo
GMC,Safari Passenger
GMC,Savana 1500 Cargo
GMC,Savana 1500 Passenger
GMC,Savana 2500 Cargo
GMC,Savana 2500 Passenger
GMC,Savana 3500 Cargo
GMC,Savana 3500 Passenger
GMC,Sierra (Classic) 1500 Crew Cab
GMC,Sierra (Classic) 1500 Extended Cab
GMC,Sierra (Classic) 1500 HD Crew Cab
GMC,Sierra (Classic) 1500 Regular Cab
GMC,Sierra (Classic) 2500 Crew Cab
GMC,Sierra (Classic) 2500 HD Crew Cab
GMC,Sierra (Classic) 2500 HD Extended Cab
GMC,Sierra (Classic) 2500 HD Regular Cab
GMC,Sierra (Classic) 3500 Crew Cab
GMC,Sierra (Classic) 3500 Extended Cab
GMC,Sierra (Classic) 3500 Regular Cab
GMC,Sierra 1500 Crew Cab
GMC,Sierra 1500 Double Cab
GMC,Sierra 1500 Extended Cab
GMC,Sierra 1500 HD Crew Cab
GMC,Sierra 1500 Limited Crew Cab
GMC,Sierra 1500 Limited Double Cab
GMC,Sierra 1500 Limited Regular Cab
GMC,Sierra 1500 Regular Cab
GMC,Sierra 2500 Crew Cab
GMC,Sierra 2500 Extended Cab
GMC,Sierra 2500 HD Crew Cab
GMC,Sierra 2500 HD Double Cab
GMC,Sierra 2500 HD Extended Cab
GMC,Sierra 2500 HD Regular Cab
GMC,Sierra 2500 Regular Cab
GMC,Sierra 3500 Crew Cab
GMC,Sierra 3500 Extended Cab
GMC,Sierra 3500 HD Crew Cab
GMC,Sierra 3500 HD Double Cab
GMC,Sierra 3500 HD Extended Cab
GMC,Sierra 3500 HD Regular Cab
GMC,Sierra 3500 Regular Cab
GMC,Sonoma Club Cab
GMC,Sonoma Club Coupe Cab
GMC,Sonoma Crew Cab
GMC,Sonoma Extended Cab
GMC,Sonoma Regular Cab
GMC,Suburban 1500
GMC,Suburban 2500
GMC,Terrain
GMC,Vandura 1500
GMC,Vandura 2500
GMC,Vandura 3500
GMC,Vandura G1500
GMC,Vandura G2500
GMC,Vandura G3500
GMC,Yukon
GMC,Yukon Denali
GMC,Yukon XL
GMC,Yukon XL 1500
GMC,Yukon XL 2500
Genesis,Electrified G80
Genesis,G70
Genesis,G80
Genesis,G90
Genesis,GV60
Genesis,GV70
Genesis,GV80
Geo,Metro
Geo,Prizm
Geo,Storm
Geo,Tracker
HUMMER,H1
HUMMER,H2
HUMMER,H3
HUMMER,H3T
Honda,Accord
Honda,Accord Crosstour
Honda,Accord Hybrid
Honda,CR-V
Honda,CR-V Hybrid
Honda,CR-Z
Honda,Civic
Honda,Civic Type R
Honda,Clarity Electric
Honda,Clarity Fuel Cell
Honda,Clarity Plug-in Hybrid
Honda,Crosstour
Honda,Element
Honda,Fit
Honda,HR-V
Honda,Insight
Honda,Odyssey
Honda,Passport
Honda,Pilot
Honda,Prelude
Honda,Prologue
Honda,Ridgeline
Honda,S2000
Honda,del Sol
Hyundai,Accent
Hyundai,Azera
Hyundai,Elantra
Hyundai,Elantra GT
Hyundai,Elantra Hybrid
Hyundai,Elantra N
Hyundai,Entourage
Hyundai,Equus
Hyundai,Excel
Hyundai,Genesis
Hyundai,Genesis Coupe
Hyundai,IONIQ 5
Hyundai,IONIQ 6
Hyundai,IONIQ 7
Hyundai,Ioniq Electric
Hyundai,Ioniq Hybrid
Hyundai,Ioniq Plug-in Hybrid
Hyundai,Kona
Hyundai,Kona Electric
Hyundai,Kona N
Hyundai,NEXO
Hyundai,Palisade
Hyundai,Santa Cruz
Hyundai,Santa Fe
Hyundai,Santa Fe Hybrid
Hyundai,Santa Fe Plug-in Hybrid
Hyundai,Santa Fe Sport
Hyundai,Santa Fe XL
Hyundai,Scoupe
Hyundai,Sonata
Hyundai,Sonata Hybrid
Hyundai,Sonata Plug-in Hybrid
Hyundai,Tiburon
Hyundai,Tucson
Hyundai,Tucson Fuel Cell
Hyundai,Tucson Hybrid
Hyundai,Tucson Plug-in Hybrid
Hyundai,Veloster
Hyundai,Venue
Hyundai,Veracruz
Hyundai,XG300
Hyundai,XG350
INFINITI,EX
INFINITI,FX
INFINITI,G
INFINITI,I
INFINITI,J
INFINITI,JX
INFINITI,M
INFINITI,Q
INFINITI,Q40
INFINITI,Q50
INFINITI,Q60
INFINITI,Q70
INFINITI,QX
INFINITI,QX30
INFINITI,QX50
INFINITI,QX55
INFINITI,QX60
INFINITI,QX70
INFINITI,QX80
Isuzu,Amigo
Isuzu,Ascender
Isuzu,Axiom
Isuzu,Hombre Regular Cab
Isuzu,Hombre Spacecab
Isuzu,Impulse
Isuzu,Oasis
Isuzu,Regular Cab
Isuzu,Rodeo
Isuzu,Rodeo Sport
Isuzu,Spacecab
Isuzu,Stylus
Isuzu,Trooper
Isuzu,VehiCROSS
Isuzu,i-280 Extended Cab
Isuzu,i-290 Extended Cab
Isuzu,i-350 Crew Cab
Isuzu,i-370 Crew Cab
Isuzu,i-370 Extended Cab
Jaguar,E-PACE
Jaguar,F-PACE
Jaguar,F-TYPE
Jaguar,I-PACE
Jaguar,S-Type
Jaguar,X-Type
Jaguar,XE
Jaguar,XF
Jaguar,XJ
Jaguar,XK
Jeep,Cherokee
Jeep,Comanche Regular Cab
Jeep,Commander
Jeep,Compass
Jeep,Gladiator
Jeep,Grand Cherokee
Jeep,Grand Cherokee 4xe
Jeep,Grand Cherokee L
Jeep,Grand Wagoneer
Jeep,Grand Wagoneer L
Jeep,Liberty
Jeep,Patriot
Jeep,Recon
Jeep,Renegade
Jeep,Wagoneer
Jeep,Wagoneer L
Jeep,Wrangler
Jeep,Wrangler Unlimited
Jeep,Wrangler Unlimited 4xe
Kia,Amanti
Kia,Borrego
Kia,Cadenza
Kia,Carnival
Kia,EV6
Kia,EV9
Kia,Forte
Kia,Forte Koup
Kia,Forte5
Kia,K5
Kia,K900
Kia,Niro
Kia,Niro EV
Kia,Niro Plug-in Hybrid
Kia,Optima
Kia,Optima (2006.5)
Kia,Optima Hybrid
Kia,Optima Plug-in Hybrid
Kia,Rio
Kia,Rondo
Kia,Sedona
Kia,Seltos
Kia,Sephia
Kia,Sorento
Kia,Sorento Hybrid
Kia,Sorento Plug-in Hybrid
Kia,Soul
Kia,Soul EV
Kia,Spectra
Kia,Sportage
Kia,Sportage Hybrid
Kia,Sportage Plug-in Hybrid
Kia,Stinger
Kia,Telluride
Lamborghini,Aventador
Lamborghini,Gallardo
Lamborghini,Huracan
Lamborghini,Murcielago
Lamborghini,Murcielago LP640
Lamborghini,Urus
Land Rover,Defender 110
Land Rover,Defender 130
Land Rover,Defender 90
Land Rover,Discovery
Land Rover,Discovery Series II
Land Rover,Discovery Sport
Land Rover,Freelander
Land Rover,LR2
Land Rover,LR3
Land Rover,LR4
Land Rover,Range Rover
Land Rover,Range Rover Evoque
Land Rover,Range Rover Sport
Land Rover,Range Rover Velar
Lexus,CT
Lexus,ES
Lexus,GS
Lexus,GX
Lexus,HS
Lexus,IS
Lexus,IS F
Lexus,LC
Lexus,LFA
Lexus,LS
Lexus,LX
Lexus,NX
Lexus,RC
Lexus,RX
Lexus,RZ
Lexus,SC
Lexus,TX
Lexus,UX
Lincoln,Aviator
Lincoln,Blackwood
Lincoln,Continental
Lincoln,Corsair
Lincoln,LS
Lincoln,MKC
Lincoln,MKS
Lincoln,MKT
Lincoln,MKX
Lincoln,MKZ
Lincoln,Mark LT
Lincoln,Mark VII
Lincoln,Mark VIII
Lincoln,Nautilus
Lincoln,Navigator
Lincoln,Navigator L
Lincoln,Town Car
Lincoln,Zephyr
Lotus,Elise
Lotus,Evora
Lotus,Evora 400
Lotus,Evora GT
Lotus,Exige
Lotus,Exige S
Lucid,Air
MAZDA,323
MAZDA,626
MAZDA,929
MAZDA,B-Series Cab Plus
MAZDA,B-Series Extended Cab
MAZDA,B-Series Regular Cab
MAZDA,CX-3
MAZDA,CX-30
MAZDA,CX-5
MAZDA,CX-50
MAZDA,CX-7
MAZDA,CX-70
MAZDA,CX-9
MAZDA,CX-90
MAZDA,MAZDA2
MAZDA,MAZDA3
MAZDA,MAZDA5
MAZDA,MAZDA6
MAZDA,MPV
MAZDA,MX-3
MAZDA,MX-30
MAZDA,MX-5 Miata
MAZDA,MX-5 Miata RF
MAZDA,MX-6
MAZDA,Millenia
MAZDA,Navajo
MAZDA,Protege
MAZDA,Protege5
MAZDA,RX-7
MAZDA,RX-8
MAZDA,Tribute
MINI,Clubman
MINI,Convertible
MINI,Cooper
MINI,Countryman
MINI,Coupe
MINI,Hardtop
MINI,Hardtop 2 Door
MINI,Hardtop 4 Door
MINI,Paceman
MINI,Roadster
Maserati,Coupe
Maserati,Ghibli
Maserati,GranSport
Maserati,GranTurismo
Maserati,Grecale
Maserati,Levante
Maserati,MC20
Maserati,Quattroporte
Maybach,57
Maybach,62
McLaren,570GT
McLaren,570S
McLaren,600LT
McLaren,650S
McLaren,675LT
McLaren,720S
McLaren,MP4-12C
Mercedes-Benz,190 E
Mercedes-Benz,300 CE
Mercedes-Benz,300 D
Mercedes-Benz,300 E
Mercedes-Benz,300 SD
Mercedes-Benz,300 SE
Mercedes-Benz,300 SL
Mercedes-Benz,300 TE
Mercedes-Benz,400 E
Mercedes-Benz,400 SE
Mercedes-Benz,400 SEL
Mercedes-Benz,500 E
Mercedes-Benz,500 SEC
Mercedes-Benz,500 SEL
Mercedes-Benz,500 SL
Mercedes-Benz,600 SEC
Mercedes-Benz,600 SEL
Mercedes-Benz,600 SL
Mercedes-Benz,A-Class
Mercedes-Benz,B-Class
Mercedes-Benz,C-Class
Mercedes-Benz,CL-Class
Mercedes-Benz,CLA
Mercedes-Benz,CLA-Class
Mercedes-Benz,CLK-Class
Mercedes-Benz,CLS
Mercedes-Benz,CLS-Class
Mercedes-Benz,E-Class
Mercedes-Benz,G-Class
Mercedes-Benz,GL-Class
Mercedes-Benz,GLA
Mercedes-Benz,GLA-Class
Mercedes-Benz,GLB
Mercedes-Benz,GLC
Mercedes-Benz,GLC Coupe
Mercedes-Benz,GLE
Mercedes-Benz,GLE Coupe
Mercedes-Benz,GLK-Class
Mercedes-Benz,GLS
Mercedes-Benz,M-Class
Mercedes-Benz,Mercedes-AMG A-Class
Mercedes-Benz,Mercedes-AMG C-Class
Mercedes-Benz,Mercedes-AMG CLA
Mercedes-Benz,Mercedes-AMG CLS
Mercedes-Benz,Mercedes-AMG E-Class
Mercedes-Benz,Mercedes-AMG EQS
Mercedes-Benz,Mercedes-AMG G-Class
Mercedes-Benz,Mercedes-AMG GLA
Mercedes-Benz,Mercedes-AMG GLB
Mercedes-Benz,Mercedes-AMG GLC
Mercedes-Benz,Mercedes-AMG GLC Coupe
Mercedes-Benz,Mercedes-AMG GLE
Mercedes-Benz,Mercedes-AMG GLE Coupe
Mercedes-Benz,Mercedes-AMG GLS
Mercedes-Benz,Mercedes-AMG GT
Mercedes-Benz,Mercedes-AMG S-Class
Mercedes-Benz,Mercedes-AMG SL
Mercedes-Benz,Mercedes-AMG SLC
Mercedes-Benz,Mercedes-AMG SLK
Mercedes-Benz,Mercedes-EQ EQB
Mercedes-Benz,Mercedes-EQ EQE
Mercedes-Benz,Mercedes-EQ EQE SUV
Mercedes-Benz,Mercedes-EQ EQS
Mercedes-Benz,Mercedes-EQ EQS SUV
Mercedes-Benz,Mercedes-Maybach GLS
Mercedes-Benz,Mercedes-Maybach S 600
Mercedes-Benz,Mercedes-Maybach S-Class
Mercedes-Benz,Metris Cargo
Mercedes-Benz,Metris Passenger
Mercedes-Benz,Metris WORKER Cargo
Mercedes-Benz,Metris WORKER Passenger
Mercedes-Benz,R-Class
Mercedes-Benz,S-Class
Mercedes-Benz,SL
Mercedes-Benz,SL-Class
Mercedes-Benz,SLC
Mercedes-Benz,SLK
Mercedes-Benz,SLK-Class
Mercedes-Benz,SLR McLaren
Mercedes-Benz,SLS-Class
Mercedes-Benz,Sprinter 1500 Cargo
Mercedes-Benz,Sprinter 1500 Passenger
Mercedes-Benz,Sprinter 2500 Cargo
Mercedes-Benz,Sprinter 2500 Crew
Mercedes-Benz,Sprinter 2500 Passenger
Mercedes-Benz,Sprinter 3500 Cargo
Mercedes-Benz,Sprinter 3500 Crew
Mercedes-Benz,Sprinter 3500 XD Cargo
Mercedes-Benz,Sprinter 3500 XD Crew
Mercedes-Benz,Sprinter 4500 Cargo
Mercedes-Benz,Sprinter 4500 Crew
Mercedes-Benz,Sprinter WORKER Cargo
Mercedes-Benz,Sprinter WORKER Passenger
Mercury,Capri
Mercury,Cougar
Mercury,Grand Marquis
Mercury,Marauder
Mercury,Mariner
Mercury,Milan
Mercury,Montego
Mercury,Monterey
Mercury,Mountaineer
Mercury,Mystique
Mercury,Sable
Mercury,Topaz
Mercury,Tracer
Mercury,Villager
Mitsubishi,3000GT
Mitsubishi,Diamante
Mitsubishi,Eclipse
Mitsubishi,Eclipse Cross
Mitsubishi,Endeavor
Mitsubishi,Expo
Mitsubishi,Galant
Mitsubishi,Lancer
Mitsubishi,Lancer Evolution
Mitsubishi,Mighty Max Macro Cab
Mitsubishi,Mighty Max Regular Cab
Mitsubishi,Mirage
Mitsubishi,Mirage G4
Mitsubishi,Montero
Mitsubishi,Montero Sport
Mitsubishi,Outlander
Mitsubishi,Outlander PHEV
Mitsubishi,Outlander Sport
Mitsubishi,Precis
Mitsubishi,Raider Double Cab
Mitsubishi,Raider Extended Cab
Mitsubishi,i-MiEV
Nissan,200SX
Nissan,240SX
Nissan,300ZX
Nissan,350Z
Nissan,370Z
Nissan,400Z
Nissan,Altima
Nissan,Ariya
Nissan,Armada
Nissan,Frontier Crew Cab
Nissan,Frontier King Cab
Nissan,Frontier Regular Cab
Nissan,GT-R
Nissan,JUKE
Nissan,Kicks
Nissan,King Cab
Nissan,LEAF
Nissan,Maxima
Nissan,Murano
Nissan,NV1500 Cargo
Nissan,NV200
Nissan,NV200 Taxi
Nissan,NV2500 HD Cargo
Nissan,NV3500 HD Cargo
Nissan,NV3500 HD Passenger
Nissan,NX
Nissan,Pathfinder
Nissan,Pathfinder Armada
Nissan,Quest
Nissan,Regular Cab
Nissan,Rogue
Nissan,Rogue Select
Nissan,Rogue Sport
Nissan,Sentra
Nissan,Stanza
Nissan,TITAN Single Cab
Nissan,TITAN XD Crew Cab
Nissan,TITAN XD King Cab
Nissan,TITAN XD Single Cab
Nissan,Titan Crew Cab
Nissan,Titan King Cab
Nissan,Versa
Nissan,Versa Note
Nissan,Xterra
Nissan,Z
Nissan,cube
Oldsmobile,88
Oldsmobile,98
Oldsmobile,Achieva
Oldsmobile,Alero
Oldsmobile,Aurora
Oldsmobile,Bravada
Oldsmobile,Ciera
Oldsmobile,Custom Cruiser
Oldsmobile,Cutlass
Oldsmobile,Cutlass Ciera
Oldsmobile,Cutlass Cruiser
Oldsmobile,Cutlass Supreme
Oldsmobile,Intrigue
Oldsmobile,LSS
Oldsmobile,Regency
Oldsmobile,Silhouette
Oldsmobile,Toronado
Panoz,Esperante
Plymouth,Acclaim
Plymouth,Breeze
Plymouth,Colt
Plymouth,Colt Vista
Plymouth,Grand Voyager
Plymouth,Laser
Plymouth,Neon
Plymouth,Prowler
Plymouth,Sundance
Plymouth,Voyager
Polestar,1
Polestar,2
Polestar,3
Polestar,5
Pontiac,Aztek
Pontiac,Bonneville
Pontiac,Firebird
Pontiac,G3
Pontiac,G5
Pontiac,G6
Pontiac,G6 (2009.5)
Pontiac,G8
Pontiac,GTO
Pontiac,Grand Am
Pontiac,Grand Prix
Pontiac,LeMans
Pontiac,Montana
Pontiac,Montana SV6
Pontiac,Solstice
Pontiac,Sunbird
Pontiac,Sunfire
Pontiac,Torrent
Pontiac,Trans Sport
Pontiac,Vibe
Porsche,718 Boxster
Porsche,718 Cayman
Porsche,718 Spyder
Porsche,911
Porsche,928
Porsche,968
Porsche,Boxster
Porsche,Carrera GT
Porsche,Cayenne
Porsche,Cayenne Coupe
Porsche,Cayman
Porsche,Macan
Porsche,Panamera
Porsche,Taycan
Porsche,Taycan Cross Turismo
Ram,1500 Classic Crew Cab
Ram,1500 Classic Quad Cab
Ram,1500 Classic Regular Cab
Ram,1500 Crew Cab
Ram,1500 Quad Cab
Ram,1500 Regular Cab
Ram,2500 Crew Cab
Ram,2500 Mega Cab
Ram,2500 Regular Cab
Ram,3500 Crew Cab
Ram,3500 Mega Cab
Ram,3500 Regular Cab
Ram,C/V
Ram,C/V Tradesman
Ram,Dakota Crew Cab
Ram,Dakota Extended Cab
Ram,ProMaster 1500 Cargo
Ram,ProMaster 2500 Cargo
Ram,ProMaster 3500 Cargo
Ram,ProMaster Cargo Van
Ram,ProMaster City
Ram,ProMaster Window Van
Rivian,R1S
Rivian,R1T
Rolls-Royce,Cullinan
Rolls-Royce,Dawn
Rolls-Royce,Ghost
Rolls-Royce,Phantom
Rolls-Royce,Wraith
SRT,Viper
Saab,9-2X
Saab,9-3
Saab,9-4X
Saab,9-5
Saab,9-7X
Saab,900
Saab,9000
Saturn,Astra
Saturn,Aura
Saturn,Ion
Saturn,L-Series
Saturn,Outlook
Saturn,Relay
Saturn,S-Series
Saturn,SKY
Saturn,VUE
Scion,FR-S
Scion,iA
Scion,iM
Scion,iQ
Scion,tC
Scion,xA
Scion,xB
Scion,xD
Subaru,Ascent
Subaru,B9 Tribeca
Subaru,BRZ
Subaru,Baja
Subaru,Crosstrek
Subaru,Forester
Subaru,Impreza
Subaru,Justy
Subaru,Legacy
Subaru,Loyale
Subaru,Outback
Subaru,SVX
Subaru,Solterra
Subaru,Tribeca
Subaru,WRX
Subaru,XV Crosstrek
Suzuki,Aerio
Suzuki,Equator Crew Cab
Suzuki,Equator Extended Cab
Suzuki,Esteem
Suzuki,Forenza
Suzuki,Grand Vitara
Suzuki,Kizashi
Suzuki,Reno
Suzuki,SX4
Suzuki,Samurai
Suzuki,Sidekick
Suzuki,Swift
Suzuki,Verona
Suzuki,Vitara
Suzuki,X-90
Suzuki,XL-7
Suzuki,XL7
Tesla,Cybertruck
Tesla,Model 3
Tesla,Model S
Tesla,Model X
Tesla,Model Y
Toyota,4Runner
Toyota,86
Toyota,Avalon
Toyota,Avalon Hybrid
Toyota,C-HR
Toyota,Camry
Toyota,Camry Hybrid
Toyota,Celica
Toyota,Corolla
Toyota,Corolla Cross
Toyota,Corolla Cross Hybrid
Toyota,Corolla Hatchback
Toyota,Corolla Hybrid
Toyota,Corolla iM
Toyota,Cressida
Toyota,Crown
Toyota,Echo
Toyota,FJ Cruiser
Toyota,GR Supra
Toyota,GR86
Toyota,Highlander
Toyota,Highlander Hybrid
Toyota,Land Cruiser
Toyota,MR2
Toyota,Matrix
Toyota,Mirai
Toyota,Paseo
Toyota,Previa
Toyota,Prius
Toyota,Prius Plug-in Hybrid
Toyota,Prius Prime
Toyota,Prius c
Toyota,Prius v
Toyota,RAV4
Toyota,RAV4 Hybrid
Toyota,RAV4 Prime
Toyota,Regular Cab
Toyota,Sequoia
Toyota,Sienna
Toyota,Solara
Toyota,Supra
Toyota,T100 Regular Cab
Toyota,T100 Xtracab
Toyota,Tacoma Access Cab
Toyota,Tacoma Double Cab
Toyota,Tacoma Regular Cab
Toyota,Tacoma Xtracab
Toyota,Tercel
Toyota,Tundra Access Cab
Toyota,Tundra CrewMax
Toyota,Tundra Double Cab
Toyota,Tundra Hybrid CrewMax
Toyota,Tundra Regular Cab
Toyota,Venza
Toyota,Xtra Cab
Toyota,Yaris
Toyota,Yaris Hatchback
Toyota,Yaris iA
Toyota,bZ4X
VinFast,VF 8
VinFast,VF 9
Volkswagen,Arteon
Volkswagen,Atlas
Volkswagen,Atlas Cross Sport
Volkswagen,Beetle
Volkswagen,CC
Volkswagen,Cabrio
Volkswagen,Cabrio (New)
Volkswagen,Cabriolet
Volkswagen,Corrado
Volkswagen,Eos
Volkswagen,Eurovan
Volkswagen,Fox
Volkswagen,GLI
Volkswagen,GTI
Volkswagen,GTI (New)
Volkswagen,Golf
Volkswagen,Golf (New)
Volkswagen,Golf Alltrack
Volkswagen,Golf GTI
Volkswagen,Golf III
Volkswagen,Golf R
Volkswagen,Golf SportWagen
Volkswagen,ID.4
Volkswagen,ID.Buzz
Volkswagen,Jetta
Volkswagen,Jetta (New)
Volkswagen,Jetta GLI
Volkswagen,Jetta III
Volkswagen,Jetta SportWagen
Volkswagen,New Beetle
Volkswagen,Passat
Volkswagen,Passat (New)
Volkswagen,Phaeton
Volkswagen,R32
Volkswagen,Rabbit
Volkswagen,Routan
Volkswagen,Taos
Volkswagen,Tiguan
Volkswagen,Tiguan Limited
Volkswagen,Touareg
Volkswagen,Touareg 2
Volkswagen,e-Golf
Volvo,240
Volvo,740
Volvo,850
Volvo,940
Volvo,960
Volvo,C30
Volvo,C40 Recharge
Volvo,C70
Volvo,EX90
Volvo,S40
Volvo,S40 (New)
Volvo,S60
Volvo,S70
Volvo,S80
Volvo,S90
Volvo,V40
Volvo,V50
Volvo,V60
Volvo,V70
Volvo,V90
Volvo,XC40
Volvo,XC40 Recharge
Volvo,XC60
Volvo,XC70
Volvo,XC90
smart,fortwo
smart,fortwo EQ cabrio
smart,fortwo EQ coupe
smart,fortwo cabrio
smart,fortwo electric drive
smart,fortwo electric drive cabrio

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/4/2023 10:43 PM, Dino wrote:
>
> I need fast text-search on a large (not huge, let's say 30k records
> totally) list of items. Here's a sample of my raw data (a list of US
> cars: model and make)

I suspect I am really close to answering my own question...

>>> import time
>>> lis = [str(a**2+a*3+a) for a in range(0,30000)]
>>> s = time.process_time_ns(); res = [el for el in lis if "13467" in
el]; print(time.process_time_ns() -s);
753800
>>> s = time.process_time_ns(); res = [el for el in lis if "52356" in
el]; print(time.process_time_ns() -s);
1068300
>>> s = time.process_time_ns(); res = [el for el in lis if "5256" in
el]; print(time.process_time_ns() -s);
862000
>>> s = time.process_time_ns(); res = [el for el in lis if "6" in el];
print(time.process_time_ns() -s);
1447300
>>> s = time.process_time_ns(); res = [el for el in lis if "1" in el];
print(time.process_time_ns() -s);
1511100
>>> s = time.process_time_ns(); res = [el for el in lis if "13467" in
el]; print(time.process_time_ns() -s); print(len(res), res[:10])
926900
2 ['134676021', '313467021']
>>>

I can do a substring search in a list of 30k elements in less than 2ms
with Python. Is my reasoning sound?

Dino


--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/4/2023 11:12 PM, Dino wrote:
> On 3/4/2023 10:43 PM, Dino wrote:
>>
>> I need fast text-search on a large (not huge, let's say 30k records
>> totally) list of items. Here's a sample of my raw data (a list of US
>> cars: model and make)
>
> I suspect I am really close to answering my own question...
>
> >>> import time
> >>> lis = [str(a**2+a*3+a) for a in range(0,30000)]
> >>> s = time.process_time_ns(); res = [el for el in lis if "13467" in
> el]; print(time.process_time_ns() -s);
> 753800
> >>> s = time.process_time_ns(); res = [el for el in lis if "52356" in
> el]; print(time.process_time_ns() -s);
> 1068300
> >>> s = time.process_time_ns(); res = [el for el in lis if "5256" in
> el]; print(time.process_time_ns() -s);
> 862000
> >>> s = time.process_time_ns(); res = [el for el in lis if "6" in el];
> print(time.process_time_ns() -s);
> 1447300
> >>> s = time.process_time_ns(); res = [el for el in lis if "1" in el];
> print(time.process_time_ns() -s);
> 1511100
> >>> s = time.process_time_ns(); res = [el for el in lis if "13467" in
> el]; print(time.process_time_ns() -s); print(len(res), res[:10])
> 926900
> 2 ['134676021', '313467021']
> >>>
>
> I can do a substring search in a list of 30k elements in less than 2ms
> with Python. Is my reasoning sound?

I would probably ingest the data at startup into a dictionary - or
perhaps several depending on your access patterns - and then you will
only need to to a fast lookup in one or more dictionaries.

If your access pattern would be easier with SQL queries, load the data
into an SQLite database on startup.

IOW, do the bulk of the work once at startup.
--
https://mail.python.org/mailman/listinfo/python-list
RE: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Dino, Sending lots of data to an archived forum is not a great idea. I
snipped most of it out below as not to replicate it.

Your question does not look difficult unless your real question is about
speed. Realistically, much of the time spent generally is in reading in a
file and the actual search can be quite rapid with a wide range of methods.

The data looks boring enough and seems to not have much structure other than
one comma possibly separating two fields. Do you want the data as one wide
filed or perhaps in two parts, which a CSV file is normally used to
represent. Do you ever have questions like tell me all cars whose name
begins with the letter D and has a V6 engine? If so, you may want more than
a vanilla search.

What exactly do you want to search for? Is it a set of built-in searches or
something the user types in?

The data seems to be sorted by the first field and then by the second and I
did not check if some searches might be ambiguous. Can there be many entries
containing III? Yep. Can the same words like Cruiser or Hybrid appear?

So is this a one-time search or multiple searches once loaded as in a
service that stays resident and fields requests. The latter may be worth
speeding up.

I don't NEED to know any of this but want you to know that the answer may
depend on this and similar factors. We had a long discussion lately on
whether to search using regular expressions or string methods. If your data
is meant to be used once, you may not even need to read the file into
memory, but read something like a line at a time and test it. Or, if you end
up with more data like how many cylinders a car has, it may be time to read
it in not just to a list of lines or such data structures, but get
numpy/pandas involved and use their many search methods in something like a
data.frame.

Of course if you are worried about portability, keep using Get Regular
Expression Print.

Your example was:

$ grep -i v60 all_cars_unique.csv
Genesis,GV60
Volvo,V60

You seem to have wanted case folding and that is NOT a normal search. And
your search is matching anything on any line. If you wanted only a complete
field, such as all text after a comma to the end of the line, you could use
grep specifications to say that.

But once inside python, you would need to make choices depending on what
kind of searches you want to allow but also things like do you want all
matching lines shown if you search for say "a" ...




-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Dino
Sent: Saturday, March 4, 2023 10:47 PM
To: python-list@python.org
Subject: Re: Fast full-text searching in Python (job for Whoosh?)


Here's the complete data file should anyone care.

Acura,CL
Acura,ILX
Acura,Integra
Acura,Legend
<SNIP>
smart,fortwo electric drive
smart,fortwo electric drive cabrio

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/5/2023 9:05 PM, Thomas Passin wrote:
>
> I would probably ingest the data at startup into a dictionary - or
> perhaps several depending on your access patterns - and then you will
> only need to to a fast lookup in one or more dictionaries.
>
> If your access pattern would be easier with SQL queries, load the data
> into an SQLite database on startup.

Thank you. SQLite would be overkill here, plus all the machinery that I
would need to set up to make sure that the DB is rebuilt/updated regularly.
Do you happen to know something about Whoosh? have you ever used it?

> IOW, do the bulk of the work once at startup.

Sound advice

Thank you
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Not sure if this is what Thomas meant, but I was also thinking dictionaries.

Dino could build a set of dictionaries with keys ?a? through ?z? that contain data with those letters in them. (I?m assuming case insensitive search) and then just search ?v? if that?s what the user starts with.

Increased performance may be achieved by building dictionaries ?aa?,?ab? ... ?zz. And so on.

Of course, it?s trading CPU for memory usage, and there?s likely a point at which the cost of building dictionaries exceeds the savings in searching.


From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Thomas Passin <list1@tompassin.net>
Date: Sunday, March 5, 2023 at 9:07 PM
To: python-list@python.org <python-list@python.org>
Subject: Re: Fast full-text searching in Python (job for Whoosh?)

I would probably ingest the data at startup into a dictionary - or
perhaps several depending on your access patterns - and then you will
only need to to a fast lookup in one or more dictionaries.

If your access pattern would be easier with SQL queries, load the data
into an SQLite database on startup.

IOW, do the bulk of the work once at startup.
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$>
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
?Dino, Sending lots of data to an archived forum is not a great idea. I
snipped most of it out below as not to replicate it.?

Surely in 2023, storage is affordable enough there?s no need to criticize Dino for posting complete information. If mailing space is a consideration, we could all help by keeping our replies short and to the point.

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On Mon, 6 Mar 2023 15:32:09 +0000, Weatherby,Gerard wrote:


> Increased performance may be achieved by building dictionaries “aa”,”ab”
> ... “zz. And so on.

Or a trie. There have been several implementations but I believe this is
the most active:

https://pypi.org/project/PyTrie/
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/6/2023 10:32 AM, Weatherby,Gerard wrote:
> Not sure if this is what Thomas meant, but I was also thinking dictionaries.
>
> Dino could build a set of dictionaries with keys “a” through “z” that contain data with those letters in them. (I’m assuming case insensitive search) and then just search “v” if that’s what the user starts with.
>
> Increased performance may be achieved by building dictionaries “aa”,”ab” ... “zz. And so on.
>
> Of course, it’s trading CPU for memory usage, and there’s likely a point at which the cost of building dictionaries exceeds the savings in searching.

Chances are it would only be seconds at most to build the data cache,
and then subsequent queries would respond very quickly.

>
> From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Thomas Passin <list1@tompassin.net>
> Date: Sunday, March 5, 2023 at 9:07 PM
> To: python-list@python.org <python-list@python.org>
> Subject: Re: Fast full-text searching in Python (job for Whoosh?)
>
> I would probably ingest the data at startup into a dictionary - or
> perhaps several depending on your access patterns - and then you will
> only need to to a fast lookup in one or more dictionaries.
>
> If your access pattern would be easier with SQL queries, load the data
> into an SQLite database on startup.
>
> IOW, do the bulk of the work once at startup.
> --
> https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$>

--
https://mail.python.org/mailman/listinfo/python-list
RE: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Gerard,

I was politely pointing out how it was more than the minimum necessary and
might gets repeated multiple times as people replied. The storage space is a
resource someone else provides and I prefer not abusing it.

However, since the OP seems to be asking a question focused on how long it
takes to search using possible techniques, indeed some people would want the
entire data to test with.

In my personal view, the a snippet of the data is what I need to see how it
is organized and then what I need way more is some idea for what kind of
searching is needed.

If I was told there would be a web page allowing users to search a web
service hosting the data on a server with one process called as much as
needed that spawned threads to handle the task, I might see it as very
worthwhile to read in the data once into some data structure that allows
rapid searches over and over. If it is an app called ONCE as a whole for
each result, as in the grep example, why bother and just read a line at a
time and be done with it.

My suggestion remains my preference. The discussion is archived. Messages
are can optimally be trimmed as needed and not allowed to contain the full
contents of the last twenty replies back and forth unless that is needed.
Larger amounts of data can be offered to share and if wanted, can be posted
or send to someone asking for it or placed in some public accessible place.

But my preference may not be relevant as the forum has hosts or owners and
it is what they want that counts.

The data this time was not really gigantic. But I often work with data from
a CSV that has hundreds of columns and hundreds of thousands or more rows,
with some of the columns containing large amounts of text. But I may be
interested in how to work with say just half a dozen columns and for the
purposes of my question here, perhaps a hundred representative rows. Should
I share everything, or maybe save the subset and only share that?

This is not about python as a language but about expressing ideas and
opinions on a public forum with limited resources. Yes, over the years, my
combined posts probably use far more archival space. We are not asked to be
sparse, just not be wasteful.

The OP may consider what he is working with as a LOT of data but it really
isn't by modern standards.

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Weatherby,Gerard
Sent: Monday, March 6, 2023 10:35 AM
To: python-list@python.org
Subject: Re: Fast full-text searching in Python (job for Whoosh?)

"Dino, Sending lots of data to an archived forum is not a great idea. I
snipped most of it out below as not to replicate it."

Surely in 2023, storage is affordable enough there's no need to criticize
Dino for posting complete information. If mailing space is a consideration,
we could all help by keeping our replies short and to the point.

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list
RE: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Thomas,

I may have missed any discussion where the OP explained more about proposed usage. If the program is designed to load the full data once, never get updates except by re-reading some file, and then handles multiple requests, then some things may be worth doing.

It looked to me, and I may well be wrong, like he wanted to search for a string anywhere in the text so a grep-like solution is a reasonable start with the actual data being stored as something like a list of character strings you can search "one line" at a time. I suspect a numpy variant may work faster.

And of course any search function he builds can be made to remember some or all previous searches using a cache decorator. That generally uses a dictionary for the search keys internally.

But using lots of dictionaries strikes me as only helping if you are searching for text anchored to the start of a line so if you ask for "Honda" you instead ask the dictionary called "h" and search perhaps just for "onda" then recombine the prefix in any results. But the example given wanted to match something like "V6" in middle of the text and I do not see how that would work as you would now need to search 26 dictionaries completely.



-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Thomas Passin
Sent: Monday, March 6, 2023 11:03 AM
To: python-list@python.org
Subject: Re: Fast full-text searching in Python (job for Whoosh?)

On 3/6/2023 10:32 AM, Weatherby,Gerard wrote:
> Not sure if this is what Thomas meant, but I was also thinking dictionaries.
>
> Dino could build a set of dictionaries with keys “a” through “z” that contain data with those letters in them. (I’m assuming case insensitive search) and then just search “v” if that’s what the user starts with.
>
> Increased performance may be achieved by building dictionaries “aa”,”ab” ... “zz. And so on.
>
> Of course, it’s trading CPU for memory usage, and there’s likely a point at which the cost of building dictionaries exceeds the savings in searching.

Chances are it would only be seconds at most to build the data cache,
and then subsequent queries would respond very quickly.

>
> From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Thomas Passin <list1@tompassin.net>
> Date: Sunday, March 5, 2023 at 9:07 PM
> To: python-list@python.org <python-list@python.org>
> Subject: Re: Fast full-text searching in Python (job for Whoosh?)
>
> I would probably ingest the data at startup into a dictionary - or
> perhaps several depending on your access patterns - and then you will
> only need to to a fast lookup in one or more dictionaries.
>
> If your access pattern would be easier with SQL queries, load the data
> into an SQLite database on startup.
>
> IOW, do the bulk of the work once at startup.
> --
> https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$>

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/6/2023 7:28 AM, Dino wrote:
> On 3/5/2023 9:05 PM, Thomas Passin wrote:
>>
>> I would probably ingest the data at startup into a dictionary - or
>> perhaps several depending on your access patterns - and then you will
>> only need to to a fast lookup in one or more dictionaries.
>>
>> If your access pattern would be easier with SQL queries, load the data
>> into an SQLite database on startup.
>
> Thank you. SQLite would be overkill here, plus all the machinery that I
> would need to set up to make sure that the DB is rebuilt/updated regularly.
> Do you happen to know something about Whoosh? have you ever used it?

I know nothing about it, sorry. But anything beyond python dictionaries
and possibly some lists strikes me as overkill for what you have described.

>> IOW, do the bulk of the work once at startup.
>
> Sound advice
>
> Thank you

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/6/2023 12:49 PM, avi.e.gross@gmail.com wrote:
> Thomas,
>
> I may have missed any discussion where the OP explained more about proposed usage. If the program is designed to load the full data once, never get updates except by re-reading some file, and then handles multiple requests, then some things may be worth doing.
>
> It looked to me, and I may well be wrong, like he wanted to search for a string anywhere in the text so a grep-like solution is a reasonable start with the actual data being stored as something like a list of character strings you can search "one line" at a time. I suspect a numpy variant may work faster.
>
> And of course any search function he builds can be made to remember some or all previous searches using a cache decorator. That generally uses a dictionary for the search keys internally.
>
> But using lots of dictionaries strikes me as only helping if you are searching for text anchored to the start of a line so if you ask for "Honda" you instead ask the dictionary called "h" and search perhaps just for "onda" then recombine the prefix in any results. But the example given wanted to match something like "V6" in middle of the text and I do not see how that would work as you would now need to search 26 dictionaries completely.

Well, that's the question, isn't it? Just how is this expected to be
used? I didn't read the initial posting that carefully, and I may have
missed something that makes a difference.

The OP gives as an example a user entering a string ("v60"). The
example is for a model designation. If we know that this entry box will
only receive model, then I would populate a dictionary using the model
numbers as keys. The number of distinct keys will probably not be that
large.

For example, highly simplified of course:

>>> models = {'v60': 'Volvo', 'GV60': 'Genesis', 'cl': 'Acura'}
>>> entry = '60'
>>> candidates = (m for m in models.keys() if entry in m)
>>> list(candidates)
['v60', 'GV60']

The keys would be lower-cased. A separate dictionary would give the
complete string with the desired casing. The values could be object
references to the complete information. If there might be several
different models models with the same key, then the values could be
lists or dictionaries and one would need to do some disambiguation, but
that should be simple or quick.

It all depends on the planned access patterns. If the OP really wants
full-text search in the complete unstructured data file, then yes, a
full text indexer of some kind will be useful. Whoosh certainly looks
good though I have not used it. But for populating dropdown lists in
web forms, most likely the design of the form will provide a structure
for the various searches.

> -----Original Message-----
> From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Thomas Passin
> Sent: Monday, March 6, 2023 11:03 AM
> To: python-list@python.org
> Subject: Re: Fast full-text searching in Python (job for Whoosh?)
>
> On 3/6/2023 10:32 AM, Weatherby,Gerard wrote:
>> Not sure if this is what Thomas meant, but I was also thinking dictionaries.
>>
>> Dino could build a set of dictionaries with keys “a” through “z” that contain data with those letters in them. (I’m assuming case insensitive search) and then just search “v” if that’s what the user starts with.
>>
>> Increased performance may be achieved by building dictionaries “aa”,”ab” ... “zz. And so on.
>>
>> Of course, it’s trading CPU for memory usage, and there’s likely a point at which the cost of building dictionaries exceeds the savings in searching.
>
> Chances are it would only be seconds at most to build the data cache,
> and then subsequent queries would respond very quickly.
>
>>
>> From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Thomas Passin <list1@tompassin.net>
>> Date: Sunday, March 5, 2023 at 9:07 PM
>> To: python-list@python.org <python-list@python.org>
>> Subject: Re: Fast full-text searching in Python (job for Whoosh?)
>>
>> I would probably ingest the data at startup into a dictionary - or
>> perhaps several depending on your access patterns - and then you will
>> only need to to a fast lookup in one or more dictionaries.
>>
>> If your access pattern would be easier with SQL queries, load the data
>> into an SQLite database on startup.
>>
>> IOW, do the bulk of the work once at startup.
>> --
>> https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!lnP5Hxid5mAgwg8o141SvmHPgCBU8zEaHDgukrQm2igozg5H5XLoIkAmrsHtRbZHR68oYAQpRFPh-Z9telM$>
>

--
https://mail.python.org/mailman/listinfo/python-list
RE: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Ah, thanks Dino. Autocomplete within a web page can be an interesting
scenario but also a daunting one.

Now, do you mean you have a web page with a text field, initially I suppose
empty, and the user types a single character and rapidly a drop-down list or
something is created and shown? And as they type, it may shrink? And as soon
as they select one, it is replaced in the text field and done?

If your form has an attached function written in JavaScript, some might load
your data into the browser and do all that work from within. No python
needed.

Now if your scenario is similar to the above, or perhaps the user needs to
ask for autocompletion by using tab or something, and you want to keep
sending requests to a server, you can of course use any language on the
server. BUT I would be cautious in such a design.

My guess is you autocomplete on every keystroke and the user may well type
multiple characters resulting in multiple requests for your program. Is a
new one called every time or is it a running service. If the latter, it pays
to read in the data once and then carefully serve it. But when you get just
the letter "h" you may not want to send and process a thousand results but
limit It to say the first N. If they then add an o to make a ho, You may not
need to do much if it is anchored to the start except to search in the
results of the previous search rather than the whole data.

But have you done some searching on how autocomplete from a fixed corpus is
normally done? It is a quite common thing.


-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Dino
Sent: Monday, March 6, 2023 7:40 AM
To: python-list@python.org
Subject: Re: RE: Fast full-text searching in Python (job for Whoosh?)

Thank you for taking the time to write such a detailed answer, Avi. And
apologies for not providing more info from the get go.

What I am trying to achieve here is supporting autocomplete (no pun
intended) in a web form field, hence the -i case insensitive example in
my initial question.

Your points are all good, and my original question was a bit rushed. I
guess that the problem was that I saw this video:

https://www.youtube.com/watch?v=gRvZbYtwTeo&ab_channel=NextDayVideo

The idea that someone types into an input field and matches start
dancing in the browser made me think that this was exactly what I
needed, and hence I figured that asking here about Whoosh would be a
good idea. I know realize that Whoosh would be overkill for my use-case,
as a simple (case insensitive) query substring would get me 90% of what
I want. Speed is in the order of a few milliseconds out of the box,
which is chump change in the context of a web UI.

Thank you again for taking the time to look at my question

Dino

On 3/5/2023 10:56 PM, avi.e.gross@gmail.com wrote:
> Dino, Sending lots of data to an archived forum is not a great idea. I
> snipped most of it out below as not to replicate it.
>
> Your question does not look difficult unless your real question is about
> speed. Realistically, much of the time spent generally is in reading in a
> file and the actual search can be quite rapid with a wide range of
methods.
>
> The data looks boring enough and seems to not have much structure other
than
> one comma possibly separating two fields. Do you want the data as one wide
> filed or perhaps in two parts, which a CSV file is normally used to
> represent. Do you ever have questions like tell me all cars whose name
> begins with the letter D and has a V6 engine? If so, you may want more
than
> a vanilla search.
>
> What exactly do you want to search for? Is it a set of built-in searches
or
> something the user types in?
>
> The data seems to be sorted by the first field and then by the second and
I
> did not check if some searches might be ambiguous. Can there be many
entries
> containing III? Yep. Can the same words like Cruiser or Hybrid appear?
>
> So is this a one-time search or multiple searches once loaded as in a
> service that stays resident and fields requests. The latter may be worth
> speeding up.
>
> I don't NEED to know any of this but want you to know that the answer may
> depend on this and similar factors. We had a long discussion lately on
> whether to search using regular expressions or string methods. If your
data
> is meant to be used once, you may not even need to read the file into
> memory, but read something like a line at a time and test it. Or, if you
end
> up with more data like how many cylinders a car has, it may be time to
read
> it in not just to a list of lines or such data structures, but get
> numpy/pandas involved and use their many search methods in something like
a
> data.frame.
>
> Of course if you are worried about portability, keep using Get Regular
> Expression Print.
>
> Your example was:
>
> $ grep -i v60 all_cars_unique.csv
> Genesis,GV60
> Volvo,V60
>
> You seem to have wanted case folding and that is NOT a normal search. And
> your search is matching anything on any line. If you wanted only a
complete
> field, such as all text after a comma to the end of the line, you could
use
> grep specifications to say that.
>
> But once inside python, you would need to make choices depending on what
> kind of searches you want to allow but also things like do you want all
> matching lines shown if you search for say "a" ...
>
>
--
https://mail.python.org/mailman/listinfo/python-list


--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 7/03/23 4:35 am, Weatherby,Gerard wrote:
> If mailing space is a consideration, we could all help by keeping our replies short and to the point.

Indeed. A thread or two of untrimmed quoted messages is probably
more data than Dino posted!

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 7/03/23 6:49 am, avi.e.gross@gmail.com wrote:
> But the example given wanted to match something like "V6" in middle of the text and I do not see how that would work as you would now need to search 26 dictionaries completely.

It might even make things worse, as there is likely to be a lot of
overlap between entries containing "V" and entries containing "6",
so you end up searching the same data multiple times.

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/4/2023 10:43 PM, Dino wrote:
>
> I need fast text-search on a large (not huge, let's say 30k records
> totally) list of items. Here's a sample of my raw data (a list of US
> cars: model and make)

Gentlemen, thanks a ton to everyone who offered to help (and did help!).
I loved the part where some tried to divine the true meaning of my words :)

What you guys wrote is correct: the grep-esque search is guaranteed to
turn up a ton of false positives, but for the autofill use-case, that's
actually OK. Users will quickly figure what is not relevant and skip
those entries, just to zero on in on the suggestion that they find relevant.

One issue that was also correctly foreseen by some is that there's going
to be a new request at every user key stroke. Known problem. JavaScript
programmers use a trick called "debounceing" to be reasonably sure that
the user is done typing before a request is issued:

https://schier.co/blog/wait-for-user-to-stop-typing-using-javascript

I was able to apply that successfully and I am now very pleased with the
final result.

Apologies if I posted 1400 lines or data file. Seeing that certain
newsgroups carry gigabytes of copyright infringing material must have
conveyed the wrong impression to me.

Thank you.

Dino

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On Mon, 6 Mar 2023 21:55:37 -0500, Dino wrote:

> ne issue that was also correctly foreseen by some is that there's going
> to be a new request at every user key stroke. Known problem. JavaScript
> programmers use a trick called "debounceing" to be reasonably sure that
> the user is done typing before a request is issued:
>
> https://schier.co/blog/wait-for-user-to-stop-typing-using-javascript

That could be annoying. My use case is address entry. When the user types

102 ma

the suggestions might be

main
manson
maple
massachusetts
masten

in a simple case. When they enter 's' it's narrowed down. Typically I'm
only dealing with a city or county so the data to be searched isn't huge.
The maps.google.com address search covers the world and they're also
throwing in a geographical constraint so the suggestions are applicable to
the area you're viewing. It must be nice to have a server or two...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/6/2023 11:05 PM, rbowman wrote:

> It must be nice to have a server or two...

No kidding

About everything else you wrote, it makes a ton of sense, in fact it's a
dilemma I am facing now. My back-end returns 10 entries (I am limiting
to max 10 matches server side for reasons you can imagine).
As the user keeps typing, should I restrict the existing result set
based on the new information or re-issue a API call to the server?
Things get confusing pretty fast for the user. You don't want too many
cooks in kitchen, I guess.
Played a little bit with both approaches in my little application.
Re-requesting from the server seems to win hands down in my case.
I am sure that them google engineers reached spectacular levels of UI
finesse with stuff like this.

> On Mon, 6 Mar 2023 21:55:37 -0500, Dino wrote:
>>
>> https://schier.co/blog/wait-for-user-to-stop-typing-using-javascript
>
> That could be annoying. My use case is address entry. When the user types
>
> 102 ma
>
> the suggestions might be
>
> main
> manson
> maple
> massachusetts
> masten
>
> in a simple case. When they enter 's' it's narrowed down. Typically I'm
> only dealing with a city or county so the data to be searched isn't huge.
> The maps.google.com address search covers the world and they're also
> throwing in a geographical constraint so the suggestions are applicable to
> the area you're viewing.


--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 2023-03-07 04:05:19 +0000, rbowman wrote:
> On Mon, 6 Mar 2023 21:55:37 -0500, Dino wrote:
> > ne issue that was also correctly foreseen by some is that there's going
> > to be a new request at every user key stroke. Known problem. JavaScript
> > programmers use a trick called "debounceing" to be reasonably sure that
> > the user is done typing before a request is issued:
> >
> > https://schier.co/blog/wait-for-user-to-stop-typing-using-javascript
>
> That could be annoying. My use case is address entry. When the user types

It can be. The delay is short but noticeable.

A somewhat smarter strategy is to send each query as soon as the user
hit the key but keep track of what you sent and received and discard
responses for obsolete requests (This is necessary because if you first
send "ma" and then "mas", the response to the first query might arrive
after the response to the second query and you don't want to display
"mansion" if the user already typed "mas".)

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 22:43 Sat 04 Mar 2023, Dino wrote:
>How can I implement this? A library called Whoosh seems very promising
>(albeit it's so feature-rich that it's almost like shooting a fly with
>a bazooka in my case), but I see two problems:
>
> 1) Whoosh is either abandoned or the project is a mess in terms of
>community and support
>(https://groups.google.com/g/whoosh/c/QM_P8cGi4v4 ) and
>
> 2) Whoosh seems to be a Python only thing, which is great for now,
>but I wouldn't want this to become an obstacle should I need port it to
>a different language at some point.

As others have noted, it sounds like relatively straightforward
implementations will be sufficient.

But I'll note that I use whoosh from time to time and I find it stable
and pleasant to work with. It's true that development stopped, but it
stopped in a very stable place. I don't recommend using whoosh here, but
I would recommend experimenting with it more generally.

- DLD
--
https://mail.python.org/mailman/listinfo/python-list
RE: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
Some of the discussions here leave me confused as the info we think we got
early does not last long intact and often morphs into something else and we
find much of the discussion is misdirected or wasted.

Wouldn't it have been nice if this discussion had not started with a mention
of a package/module few have heard of along with a vague request on how best
to search for lines that match something in a file?

I still do not know enough to feel comfortable even after all this time. It
now seems to be a web-based application in which a web page wants to use
autocompletion as the user types.

So was the web page a static file that the user runs, or is it dynamically
created by something like a python program? How is the fact that a user has
typed a letter in a textbox or drop down of sorts reflected in a request
being sent to a python program to return possible choices? Is the same
process called anew each time or is it, or perhaps a group of similar
processes or threads going to stick around and be called repeatedly?

Lots of details are missing and in particular, much of what is being
described sounds like it is happening in the browser, presumably in
JavaScript. Also noted is that the first keystroke or two may return too
much data.

So does the OP still think this is a python question? So much of the
discussion sounds like it is in the browser deciding whether to wait for the
user to type more before making a request, or throwing away results of an
older request.

So my guess is that a possible design for this amount of data may simply be
to read the file into the browser at startup, or when the first letter is
typed, and do all the searches internally, perhaps cascaded as long as
backspace or editing is not used.

If the data gets much larger, of course, then using a server makes sense
albeit it need not use python unless lots more in the project is also ...

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of David Lowry-Duda
Sent: Tuesday, March 7, 2023 1:29 PM
To: python-list@python.org
Subject: Re: Fast full-text searching in Python (job for Whoosh?)

On 22:43 Sat 04 Mar 2023, Dino wrote:
>How can I implement this? A library called Whoosh seems very promising
>(albeit it's so feature-rich that it's almost like shooting a fly with
>a bazooka in my case), but I see two problems:
>
> 1) Whoosh is either abandoned or the project is a mess in terms of
>community and support
>(https://groups.google.com/g/whoosh/c/QM_P8cGi4v4 ) and
>
> 2) Whoosh seems to be a Python only thing, which is great for now,
>but I wouldn't want this to become an obstacle should I need port it to
>a different language at some point.

As others have noted, it sounds like relatively straightforward
implementations will be sufficient.

But I'll note that I use whoosh from time to time and I find it stable
and pleasant to work with. It's true that development stopped, but it
stopped in a very stable place. I don't recommend using whoosh here, but
I would recommend experimenting with it more generally.

- DLD
--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On Tue, 7 Mar 2023 07:33:01 -0500, Dino wrote:

> Played a little bit with both approaches in my little application.
> Re-requesting from the server seems to win hands down in my case.

That's necessary for a non-trivial data set. Assume you get 10 suggestions
after the user type 'to'.

today
tomorrow
tomato
tonsil
torque
totem
toad
toque
toward
touch

If the user type 'l' next and is trying for 'tolerance' you'll need a new
set. You'll need a little refinement. If the user is a proficient typist
and wants to type 'tolerance' they may get ahead of you.

Another consideration is a less proficient typist or someone who can't
spell. Again, play with maps.google.com. They're good at it. Put '123
thomd' in the search bar. YMMV but I get 5 variations on 123 Thomas. When
they were working down 'thompd' had zero matches so they backed up to
'thom'.

If you play with their search they're using some more magic too. Try '123
ellekt'. They may be using a variation on soundex or something more
sophisticated.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fast full-text searching in Python (job for Whoosh?) [ In reply to ]
On 3/7/2023 7:33 AM, Dino wrote:
>> It must be nice to have a server or two...
>
> No kidding
>
> About everything else you wrote, it makes a ton of sense, in fact it's a
> dilemma I am facing now. My back-end returns 10 entries (I am limiting
> to max 10 matches server side for reasons you can imagine).
> As the user keeps typing, should I restrict the existing result set
> based on the new information or re-issue a API call to the server?
> Things get confusing pretty fast for the user. You don't want too many
> cooks in kitchen, I guess.
> Played a little bit with both approaches in my little application.
> Re-requesting from the server seems to win hands down in my case.
> I am sure that them google engineers reached spectacular levels of UI
> finesse with stuff like this.

Subject of course to trying this out, I would be inclined to send a much
larger list of responses to the client, and let the client reduce the
number to be displayed. The latency for sending a longer list will be
smaller than establishing a new connection or even reusing an old one to
send a new, short list of responses. When the client types more, it can
only reduce the number of possibilities - among the (possibly imaginary)
larger original number of them. After the next round of user typing, the
client can check and see if there are enough surviving responses to
list. If not, it can then request a new list from the server.

Using this in reverse, if the user deletes some characters from the end,
there should be no need to go back to the server. The possible
responses would already have been sent to the client. They could be
interned in an associative array keyed by the string the client had
typed to get those responses.

--
https://mail.python.org/mailman/listinfo/python-list

1 2  View All