Large Scale Article Extract of Newspapers 1730s-1960s
Hello HN, over the past 7 months I've spent nearly 3,000 hours on building SNEWPAPERS, the first historical newpaper archive with full-text extractions, nearly perfect OCR, a vast categorization taxonomy and of course with semantic and agentic search capabilities. Problem: I wanted to search through newspaper archives, but when I tried every service only lets you search for keywords and dates, and gives you back raw images of the papers, and too many of them with no context. A sea of noise. Solution: I taught machines how to read the newspapers and so far I've extracted the content from > 600k pages (about 5TB) from the Chronicling America collection. Problems I had to deal with were an infinite variety of layouts, font sizes, image scan qualities, resolutions, aspect ratios, navigating around the images on the page. I also had to figure out how to get OCR to be nearly perfect so people wouldn't hate reading the extracts. I stitched together a multi-model pipeline (layout tech, ocr tech, llm, vllm) with heuristics to go from layout -> segmentation -> classification. I put it all in OpenSearch / Postgres and made it semantically searchable and also put an agentic search tool on top that knows how to use the API really well and helps you write queries to find what you're looking for. Happy to discuss AWS architecture and scaling as well, that was tough! If you have five minutes and you just want to jump in and have your own personalized experience, what I would suggest is: Before searching for anything, go to the Sleuth page Ask it about anything from 1736 to 1963, maybe 1 or 2 follow up questions Then go to the search page so you can see the queries it wrote for you (bottom left "saved queries") and uncover more info on whatever it is you're interested in If you think it's cool and you want to learn more, then there's about 10 minutes of video guides on the various capabilities in "Guide" on the nav bar Some other people have also taken a crack at this, notably: https://dell-research-harvard.github.io/resources/americanst... (very good attempt) https://labs.loc.gov/work/experiments/newspaper-navigator/ (focused on images)
AI Analysis
Analysis coming soon.
Similar Products
Capgo
Instant updates for Capacitor apps. Ship fixes in minutes, not weeks. Push OTA updates to users without app store delays.
OpenAlternative
Open source alternatives to popular software. Over 1 million users replaced their proprietary tools with open source software. Discover the best alternatives and join the movement.
Bor
Hi HN! I've been working on Bor, an open-source system for centralized Linux desktop management. Bor consists of a lightweight Go agent and a central server. Policies are streamed to clients over mTLS/gRPC in real time—no polling—and currently support Firefox, Chrome, KDE, dconf, polkit and package management, with more coming. Version 0.8 introduces several new policy types - Thunderbird, Microsoft Edge for Business and FirewallD zones, along with a number of improvements and fixes. I'd love feedback on the architecture, policy model, and whether this is something you'd consider for managing Linux workstations.
CostPerPrompt
Show HN: CostPerPrompt – Live AI API pricing and real-workload cost calculators
Cockpit for you Claude Code agents in Rust
Hi everyone! Hope you had a great day so far, and maybe its about to get just a little bit better (thanks Winter ;) So I had way to many terminal windows flying about when using Claude, and kept losing track of which terminal / session / project im in right now. So I built a solution for that, presented it to my team, and now we're using our new tool all the time, and developing it to be the most helpful agents organization tool we can think of. Unfortunatelly, Claude Code only for now, will do Codex soon tho. What it does for you: Main features: - Overview of your projects, with branches, worktrees, etc. - Start sessions in the integrated terminal, in whatever branch or worktree you want - Your projects run scripts all auto-discovered in one place - Complete Claude history to resume convo's easily Comfort: - Project overview: commits, PRs, notes, timeline with summaries - Context usage and session costs (to see what you would have payed without a subscription) - Cost aggregation for the entire day - Use dashboard with daily costs, consumed tokens per project, livetime spent - 5h and 7d window predictions aka. if you keep going at the same spend/token burn, will you run into your limits And likely many more features I forgot right now. Anyways, its all open-source (MIT), if this sounds interesting / useful, check it out at https://github.com/respeak-io/episko, feel free to comment / contribute, and happy building! And enjoy the weekend.