Skip to content

Replace pydoop to read (WARC) data from HDFS #53

Open
Open
Replace pydoop to read (WARC) data from HDFS#53

Description

@sebastian-nagel
sebastian-nagel
opened on Sep 19, 2025

Reading WARC files from HDFS is based on pydoop: the latest release dates to 2019 and the module gets more and more difficult to install on newer Python versions.

Pydoop should be replaced, ideally by a utility method reading from HDFS relying only on Spark functions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions