Metadata-Version: 2.5
Name: gnews-decoder
Version: 0.1.1
Summary: Decode Google News links to the publisher URLs behind them — in one batched call, not one call per article.
Project-URL: Homepage, https://github.com/mohamidi74/gnews-decoder
Project-URL: Source, https://github.com/mohamidi74/gnews-decoder
Project-URL: The reference behind it, https://hawkcrawl.com/google-news-rss
License-Expression: MIT
License-File: LICENSE
Keywords: batchexecute,decode,google news,news api,rss,scraping,url
Requires-Python: >=3.9
Requires-Dist: requests>=2.25
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# gnews-decoder

Turn `news.google.com/rss/articles/CBMi...` links into the publisher URLs behind
them — **in one call for the whole list, not one call per article.**

```python
from gnews_decoder import from_rss, decode, decode_many

for item in from_rss("artificial intelligence"):
    print(item["url"])      # https://www.economist.com/...

decode("https://news.google.com/rss/articles/CBMi...")
decode_many(urls)           # {google_url: publisher_url or None}
```

```
pip install gnews-decoder
```

Measured against a live feed on 19 September 2026: **50 of 50 resolved in 6.8
seconds**, using 50 signature requests and **one** decode call.

## Why this exists

Google News links do not contain the destination. Until late 2024 the id after
`/articles/` was base64 of a small protobuf with the URL inside, and every
library in this space decoded it offline with zero requests.

**That format is gone.** Of 30 ids taken from a live feed on 19 September 2026,
**0 decoded offline** and all 30 needed Google's `batchexecute` RPC — which is why
code written against the old assumption now returns nothing in particular.

Two other walls come with it, and this package handles both:

**The consent interstitial.** Without a recorded consent choice every article URL
302s to `consent.google.com`, and the page that comes back carries no signature.
This is [GNews #53](https://github.com/ranahaani/GNews/issues/53),
*"get_news()['url'] returns a URL about Google Consent Mode"*, and
[GNews #62](https://github.com/ranahaani/GNews/issues/62), *"URL no longer working
due to cookie consent page"*.

What does **not** work, all measured at **0 of 6**: browser header sets, the
`ucbcb=1` parameter, the `/articles` and `/read` paths, and warming up a session so
it collects Google's own cookies — the value Google sets on that redirect is the
*pre-decision* one and keeps you on the wall. A decision-recorded `SOCS` cookie
gives **6 of 6**. Pass `consent_cookie=` to supply your own.

**Rate limiting.** The obvious implementation issues one RPC per article and
Google starts answering 429 — which is why the alternatives' release notes are a
history of retry delays and proxy support. `batchexecute` takes every article in
a single POST; 50 resolve in about 0.8s of RPC time. That is the difference
between 100 requests and 51.

One more thing worth knowing, because it breaks scrapers: the HTML search page
at `news.google.com/search` answers **429 almost immediately**, while
`news.google.com/rss/search` answers 200. `from_rss` uses the feed.

## What it does not do

**It does not give you the article text.** The Google News feed carries
headlines, links and timestamps — no body, not even a summary. Once you have the
publisher URL you still have to fetch and extract it, and a meaningful share of
publishers will not serve you: some want a subscription, some refuse the request
outright. That is a separate and much harder problem, and this package does not
pretend to solve it.

It also will not get you past the ~100-article ceiling per feed query, because
that limit is Google's and no amount of decoding changes it.

## API

| | |
|---|---|
| `decode(url, **kw)` | one URL → publisher URL, or `None` |
| `decode_many(urls, **kw)` | `{original: resolved or None}` — **every input is a key**, so a failure cannot silently misalign a zip against your own list |
| `from_rss(query, limit=50, **kw)` | search the feed and decode every link: `{title, published, google_url, url}` |

Keyword arguments: `ceid` (default `"US:en"`), `concurrency` (16), `timeout`
(20.0), `consent_cookie`, `session`.

## If you need the text too

This package is the free half of a problem we spend our time on. If you want the
article body, at scale, with the paywalled and blocked cases labelled rather than
silently dropped, that is [hawkcrawl](https://api.hawkcrawl.com) — and the
measurements behind all of the above, including the feed parameters Google does
not document, are written up at
[hawkcrawl.com/google-news-rss](https://hawkcrawl.com/google-news-rss).

You do not need it to use this. This works on its own and always will.

## Licence

MIT.
