Providing the core capabilities of Compass through plugins

[packyan]

Compass currently serves as a big data task diagnostic platform, aiding cluster administrators and developers in macro or specific task diagnostics. However, potential challenges in our scenario include:

1. Informing all developers within the organization about the Compass component, its URL, and authentication methods might prove cumbersome.
2. Deploying the Compass component typically requires cluster administrators, posing complexity for regular Spark or Flink users with limited familiarity in intricate architectures.
3. In private cloud products lacking the necessary Compass components in the original architecture, the possibility of adding these components is minimal, limiting Compass's assistance to such products.

Integrating Compass's core capabilities into Spark Web UI or Flink Web UI could address these challenges.

1. Users of Spark and Flink are well-acquainted with these interfaces, reducing the complexity of Compass usage.
2. The core logic for task diagnostics could reside on the HistoryServer side or even within Spark Driver or Flink JobManager, potentially eliminating the need for deploying a Compass cluster (unless cluster administrators require macro information).
3. Embracing Apache Spark, Apache Flink, and other Apache communities by providing task diagnostic features as plugins aligns with the community style and ensures out-of-the-box functionality.

For future developments, consider:

1. Refactoring task-flink-core into a module devoid of spring-related classes, moving all spring-boot-related functionalities to task-flink. The task-parser module can function as an independent, spring-related class-free task-parser-core module.

2. Introducing an extensions module to provide plugins for various computing engines (e.g., Spark, Flink), enabling task diagnostic capabilities in HistoryServer, Spark Driver, and Flink JobManager, showcased in the native web UIs of these open-source computing engines.

By adopting a plugin-based approach akin to Apache Gluten (Incubator), Compass can offer history-based task diagnostic/optimization capabilities out of the box, allowing both regular users and cluster administrators to benefit from its core diagnostic features through simple configurations.

[packyan]

cc @zebozhuang @wForget @nilnon @Yves-yuan

[zebozhuang]

@packyan Very brilliant and wise advice. It can make Compass more flexible and sustainable development，what's more，users and developers can use it better and lighter.