Storyline/Catalogue for TheTechWeWant

[petermr]

A high-level overview of everything we want to set out for TTWW presentation.

OUR CATALOGUE of everything we have and have done

[petermr]

Components

The basic unit is a document, which makes sense by itself but is often part of a corpus. Documents or document-components can have uni- or bi-directional links to other documents or document-components. Links can be typed. Documents are given meaning by term-based dictionaries, which also organize the semantics and ontology.

Documents

These contain: ** nested sections**, paragraphs/sentences, lists, tables, boxes/floats, footnotes, figures, images. (we do not currently support audio, video, interactive code).

Semantic hierarchy of documents

This is a hierarchy of semanticity from low to high.

Images

We only work with born-digital images of formatted text (e.g. dumps of pages). Photographs, handwriting are too complex and unreliable. We use tesseract to extract characters and their coordinates. We generally lose: Font-style; font-weight; and font-colour is suspect. Letters and numbers are usually well extracted, but punctuation (e.g. hyphen vs dash) is often suspect. There are no explicit spaces. A clear document with well identified characters is fairly similar to un-beaded PDF.

AMI: has used Optical Character Recognition (OCR) successfully (see pyamiimage) and generates raw_html.

PDF

PDF has only 3 components: characters, curves, and bitmaps. There are no words, sentences, paragraphs, lists, footnotes, floating-boxes or tables. These can often be reconstructed by heuristics (or ML) but it's not guaranteed. Spaces are sometimes included and often the character stream is in reading order but sometimes we need geometry and inter-character distances are used to compute spaces and words. Similarly paragraphs and list depend on line spacing.

AMI: uses pdfplumber to extract characters and curves and has many empirical routines for generating raw

[petermr]

Catalogue: The Things We Have

This list - no particular order - is things we have partly or completely developed, that are Open. for Sharing. They may or may not be actively supported.

Name	purpose	Links (Python / Java)
SOFTWARE
pyamiimage	Extracting semantics from pixel diagrams
pygetpapers	searching the Open Scientific Literature
corpusPackage	Packages a Corpus in tabular form	amilib
docanalysis	extracts semantic entities from documents
dictionaries	creates semantic dictionaries from words. Wikipedia/Wiktionary Lookup	amilib
DICTIONARIES
IPCC Glossary	1000+ terms/abbreviations from IPCC linked to Wikidata
Countries	World's Countries linked to Wikimedia
Other dictionaries
CORPORA
IPCC Chapters	semantic content for 70+ chapters
UNFCCC Reports	"all" the COP reports for 30+years

[petermr]

The Tech We Want

A list of the tech we now need. We may have made starts on some of this. And there may be tools that we don't know about.

Documents and software must be Open. Software ideally Python.

• Better ways of using Wikimedia. We've had to discover a lot of this for ourselves. Ideally we'd like an assistant! "How do we do X?", "what is the best way of representing common objects (people, maps, software, etc.)

• Better ways of contributing to Wikimedia. We haven't yet dared to contribute for fear of getting it wrong. Wikimedia tools are not well integrated into Open Science. Would bespoke tools help.

• Tools to help resolve installation problems. We aren't experts in library management and problems can often exclude people.

• Better language tech for non-Anglophones. Zoom and transcription is very neocolonial and corrupts Global South names.

[petermr]

People and Activities we Want and Need

We need to share experience and are keen to share our own. We need a community which understands the needs of newcomers, and we use synchronous and asynchronous tools to manage communication.

• Zoom . We run daily transcontinental Zooms and record all sessions whiich are copied to Slack. This gives transcripts that everyone in the project can read. (We'll probably move to world-readable soon).
• Slack . We use "free" Slack and this is the main method of rapid discourse. It last for 3 months before becoming hidden. This serves all our needs
• Github. All our code and documents (including climate documents) are stored openly and permanently on Github. It's split over several repositories but they can be linked on Slack. We use Github Discussions for communal authoring

We avoid email except for formal management (e.g. induction, internship details) and containg the alumn community. Similarly we try to use Markdown on Github rather than GoogleDocs.

Induction and Learning by Doing.

We have no formal entry requirements and have successfully inducted school students and undergraduates with no IT experience. We have a number of set-up tasks which are essential - we give help when we can , but some are very platform/configuration-dependent.