Skip to content

Comments

Actions workflow to sync icons to the bucket#94

Merged
max-ostapenko merged 44 commits intomainfrom
fix-icon-extension
Apr 4, 2025
Merged

Actions workflow to sync icons to the bucket#94
max-ostapenko merged 44 commits intomainfrom
fix-icon-extension

Conversation

@max-ostapenko
Copy link
Contributor

@max-ostapenko max-ostapenko commented Mar 3, 2025

Additionally to technology JSONs we will be checking the technology icons on each push to main.

  1. get all changed icons since last commit
  2. convert all SVG to PNG
  3. sync all PNG with the Storage bucket

We'll be able to trigger a full sync manually too.

P.S. I'm keeping the old conversion script convert.js just in case as here I'm using an a Linux package, instead of node.

@max-ostapenko max-ostapenko marked this pull request as ready for review April 3, 2025 22:02
@max-ostapenko max-ostapenko changed the title Fix icon storage for tech report Actions workflow to sync icons to the bucket Apr 3, 2025
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 6 out of 15 changed files in this pull request and generated no comments.

Files not reviewed (9)
  • package.json: Language not supported
  • src/technologies/e.json: Language not supported
  • src/technologies/g.json: Language not supported
  • src/technologies/h.json: Language not supported
  • src/technologies/i.json: Language not supported
  • src/technologies/j.json: Language not supported
  • src/technologies/o.json: Language not supported
  • src/technologies/p.json: Language not supported
  • src/technologies/t.json: Language not supported
Comments suppressed due to low confidence (1)

.github/workflows/upload.yml:75

  • The new icon conversion functionality is not covered by automated tests; consider adding tests to verify that SVG files are reliably converted to PNGs as expected.
for file in $CHANGED_ICONS; do

Copy link
Member

@tunetheweb tunetheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM but have some questions.

Comment on lines 33 to 36
- name: Validate and Lint
run: |
yarn run lint
yarn run validate
Copy link
Member

@tunetheweb tunetheweb Apr 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this step fail if linting fails? Or will it only fail if the last step (validate) fails? Or does validate check the linting too (in which case why is it a separate step?)

Also nit: name should be Lint and Validate if done in that order.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, jsonlint just logs issues.
And validate behaves for me differently on device VS in GH. I noticed there are many case sensitive issues which will fail the frontend requests (404).

I'll change jsonlint and icon naming issues next. I'm sure this workflow may need some adjustments once I check it in production.

Comment on lines +72 to +79
- name: Convert changed SVGs to PNGs
if: env.CHANGED_ICONS != ''
run: |
for file in $CHANGED_ICONS; do
if [[ "$file" == *.svg ]]; then
rsvg-convert "$file" -o "${file%.svg}.png" -w 16 -h 16
fi
done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we use SVGs if in that format already? Maybe add a comment to explain?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PNG are generally smaller, and we have some SVG that are 1Mb.
Added comment to YML.

@max-ostapenko
Copy link
Contributor Author

max-ostapenko commented Apr 4, 2025

@tunetheweb I checked on the other workflows:

  • .github/workflows/stale.yml
    We had it deactivated.
    Do you see a use case for it. If no - let's delete?

  • .github/workflows/lint.yml
    Tried to make it run only when .github/workflows/test.yml is not running.
    Currently added just path ignores.

@tunetheweb
Copy link
Member

Happy to delete .github/workflows/stale.yml. It's a left over from the original Wappalyzer repo.

And I hate Stalebots as a concept btw! 🙂

@max-ostapenko
Copy link
Contributor Author

And there is some issue validate script - it finds 192 incorrect icon names when run locally, but somehow doesn't fail in Github action.

Will fix that before merging.

@max-ostapenko max-ostapenko merged commit 296da4a into main Apr 4, 2025
2 checks passed
@max-ostapenko max-ostapenko deleted the fix-icon-extension branch April 4, 2025 16:36
@github-actions
Copy link

github-actions bot commented Apr 4, 2025

WPT test run for https://almanac.httparchive.org/en/2022/

WPT test run results: http://webpagetest.httparchive.org/results.php?test=250404_Q7_B
Detected technologies:

{
    "detected": {
        "IaaS": "Google Cloud",
        "JavaScript libraries": "web-vitals",
        "RUM": "web-vitals",
        "Performance": "Priority Hints,Google Cloud Trace",
        "Security": "HSTS",
        "Webmail": "Google Workspace",
        "Email": "Google Workspace",
        "Analytics": "Google Analytics",
        "CDN": "Cloudflare",
        "Miscellaneous": "RSS,Open Graph"
    },
    "detected_apps": {
        "Google Cloud": "",
        "web-vitals": "",
        "Priority Hints": "",
        "HSTS": "",
        "Google Workspace": "",
        "Google Cloud Trace": "",
        "Google Analytics": "",
        "Cloudflare": "",
        "RSS": "",
        "Open Graph": ""
    },
    "detected_technologies": {
        "Google Cloud": {
            "name": "Google Cloud",
            "description": "Google Cloud is a suite of cloud computing services.",
            "slug": "google-cloud",
            "categories": [
                {
                    "id": 63,
                    "slug": "iaas",
                    "description": "Provides computing resources",
                    "groups": [
                        7
                    ],
                    "name": "IaaS",
                    "priority": 8
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "Google Cloud.svg",
            "website": "https://cloud.google.com",
            "pricing": [],
            "cpe": "cpe:2.3:a:google:cloud_platform:*:*:*:*:*:*:*:*"
        },
        "web-vitals": {
            "name": "web-vitals",
            "description": "The web-vitals JavaScript is a tiny, modular library for measuring all the web vitals metrics on real users.",
            "slug": "web-vitals",
            "categories": [
                {
                    "id": 59,
                    "slug": "javascript-libraries",
                    "description": "Collections of pre-written JavaScript code",
                    "groups": [
                        9
                    ],
                    "name": "JavaScript libraries",
                    "priority": 9
                },
                {
                    "id": 78,
                    "slug": "rum",
                    "description": "Tools that track performance as experienced by users",
                    "groups": [
                        2
                    ],
                    "name": "RUM",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "web-vitals.svg",
            "website": "https://github.com/GoogleChrome/web-vitals",
            "pricing": [],
            "cpe": null
        },
        "Priority Hints": {
            "name": "Priority Hints",
            "description": "Priority Hints exposes a mechanism for developers to signal a relative priority for browsers to consider when fetching resources.",
            "slug": "priority-hints",
            "categories": [
                {
                    "id": 92,
                    "slug": "performance",
                    "description": "Tools that measure and optimize site speed",
                    "groups": [
                        7
                    ],
                    "name": "Performance",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "Priority Hints.svg",
            "website": "https://wicg.github.io/priority-hints/",
            "pricing": [],
            "cpe": null
        },
        "HSTS": {
            "name": "HSTS",
            "description": "HTTP Strict Transport Security (HSTS) informs browsers that the site should only be accessed using HTTPS.",
            "slug": "hsts",
            "categories": [
                {
                    "id": 16,
                    "slug": "security",
                    "description": "Technologies that protect websites from vulnerabilities and attacks",
                    "groups": [
                        11
                    ],
                    "name": "Security",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "default.svg",
            "website": "https://www.rfc-editor.org/rfc/rfc6797#section-6.1",
            "pricing": [],
            "cpe": null
        },
        "Google Workspace": {
            "name": "Google Workspace",
            "description": "Google Workspace, formerly G Suite, is a collection of cloud computing, productivity and collaboration tools.",
            "slug": "google-workspace",
            "categories": [
                {
                    "id": 30,
                    "slug": "webmail",
                    "description": "Systems that allow users to send and receive emails through a browser",
                    "groups": [
                        4
                    ],
                    "name": "Webmail",
                    "priority": 2
                },
                {
                    "id": 75,
                    "slug": "email",
                    "description": "Systems that manage email communication",
                    "groups": [
                        4,
                        2
                    ],
                    "name": "Email",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "Google.svg",
            "website": "https://workspace.google.com/",
            "pricing": [],
            "cpe": null
        },
        "Google Cloud Trace": {
            "name": "Google Cloud Trace",
            "description": "Google Cloud Trace is a distributed tracing system that collects latency data from applications and displays it in the Google Cloud Console.",
            "slug": "google-cloud-trace",
            "categories": [
                {
                    "id": 92,
                    "slug": "performance",
                    "description": "Tools that measure and optimize site speed",
                    "groups": [
                        7
                    ],
                    "name": "Performance",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "google-cloud-trace.svg",
            "website": "https://cloud.google.com/trace",
            "pricing": [],
            "cpe": null
        },
        "Google Analytics": {
            "name": "Google Analytics",
            "description": "Google Analytics is a free web analytics service that tracks and reports website traffic.",
            "slug": "google-analytics",
            "categories": [
                {
                    "id": 10,
                    "slug": "analytics",
                    "description": "Tools that track user behavior and provide insights into website performance",
                    "groups": [
                        8
                    ],
                    "name": "Analytics",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "Google Analytics.svg",
            "website": "https://google.com/analytics",
            "pricing": [],
            "cpe": null
        },
        "Cloudflare": {
            "name": "Cloudflare",
            "description": "Cloudflare is a web-infrastructure and website-security company, providing content-delivery-network services, DDoS mitigation, Internet security, and distributed domain-name-server services.",
            "slug": "cloudflare",
            "categories": [
                {
                    "id": 31,
                    "slug": "cdn",
                    "description": "(Content Delivery Network) Distribute website content globally to improve load times for users",
                    "groups": [
                        7
                    ],
                    "name": "CDN",
                    "priority": 9
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "CloudFlare.svg",
            "website": "https://www.cloudflare.com",
            "pricing": [],
            "cpe": null
        },
        "RSS": {
            "name": "RSS",
            "description": "RSS is a family of web feed formats used to publish frequently updated works—such as blog entries, news headlines, audio, and video—in a standardized format.",
            "slug": "rss",
            "categories": [
                {
                    "id": 19,
                    "slug": "miscellaneous",
                    "description": "Tools and technologies that don't fit into other categories",
                    "groups": [
                        6
                    ],
                    "name": "Miscellaneous",
                    "priority": 10
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "RSS.svg",
            "website": "https://www.rssboard.org/rss-specification",
            "pricing": [],
            "cpe": null
        },
        "Open Graph": {
            "name": "Open Graph",
            "description": "Open Graph is a protocol that is used to integrate any web page into the social graph.",
            "slug": "open-graph",
            "categories": [
                {
                    "id": 19,
                    "slug": "miscellaneous",
                    "description": "Tools and technologies that don't fit into other categories",
                    "groups": [
                        6
                    ],
                    "name": "Miscellaneous",
                    "priority": 10
                }
            ],
            "confidence": 100,
            "version": "",
            "icon": "Open Graph.png",
            "website": "https://ogp.me",
            "pricing": [],
            "cpe": null
        }
    }
}

Comment on lines +7 to +11
paths-ignore:
- "src/technologies/*.json"
- "src/categories.json"
- "src/groups.json"
- "src/images/icons/**"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we exclude these?

Most of our contributions are to these files and I'd like to lint them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants