Deployment on transient federates

[Jakio815]

@kushalpaliwal01 and I am trying to automate the deployment script with transient federates.

Transient federates can join the federation any time, and it will be done as an event. The current examples create a reactor that spawns the federate by directly executing the binary file with parameters such as below.

/** Persistent federate that is responsible for lauching the transient federate */
reactor TransientExec(launch_time: time = 0, fed_instance_name: char* = "instance") {
  timer t(launch_time, 0)

  reaction(t) {=
    // Construct the command to launch the transient federate
    char mid_launch_cmd[512];
    sprintf(mid_launch_cmd,
        "%s/bin/federate__%s -i %s",
        LF_FED_PACKAGE_DIRECTORY,
        self->fed_instance_name,
        lf_get_federation_id()
    );
    ...
    int status = system(mid_launch_cmd);
    ...
  =}
}

However, the deployment script generated when specifying RTI hosts and federates is not compatible with this example. Let me explain why.

How the deployment script works

If we specify the hosts, then the lfc will generate two types of scripts.

1. Distribution script (e.g., bin/SimpleFederated_distribute.sh), that ssh to the server, scp the whole file (e.g., fed-gen/SimpleFederated/src-gen/federate__fed1) and then compile it.
2. The launch script (e.g., bin/SimpleFederated) then sshs into each target host, and launches the binary.

(FYI, check here for specifying hosts.)

So back to transient federates, we need to ssh into the target machine in some point, and execute the binary. This code does not ssh, and also it doesn't know which address to ssh into.

So let me introduce some ideas to make this work.

1. Pass the IP address as a parameter

Let's say we pass the IP address to the TransientExec reactor. Also let's say we have some code to ssh into the machine.

reactor TransientExec(target_host: char* = "000.000.000.000", ...) {
  ...
  reaction(t) {=
    ...
    \\ Also add some conditions to do ssh...
    sprintf(mid_launch_cmd,
        "%s/bin/federate__%s -i %s",
        LF_FED_PACKAGE_DIRECTORY,
        self->fed_instance_name,
        lf_get_federation_id()
    );
    ...
    int status = system(mid_launch_cmd);
    ...
  =}
}

Here is a simple example using this, Up is connected to a transient federate Down, and there is TransientExec that spawns Down at some timing. Also all federates are on different machines, and also the RTI.

reactor Up {
  output out: int
}

reactor Down {
  input in: int
}

federated reactor at host0{
  exec = new TransientExec(host3, down) at user@host1

  up = new Up() at user@host2;
  @transient
  down = new Down() at user@host3
  
  up.out -> down.in
}

The user should take care two things on this line exec = new TransientExec(host3, down) at user@host1.

1. The IP of down (host3)
2. The name of the federate down.

So it is kind of error-prone, the user has to match the IP and name.

Another problem is scalability, when we need to spawn multiple transient federates. Are we going to have 100 TransientExec when we have 100 Down instances?

So these are the limitations of this approach. Still, the transient federates should be able to be modeled by the user, but deployment should be automated by lfc.

2. Specifying timer and offset

So now, I say that we don't need that TransientExec reactor at all.
The user still needs to be able to model when the transient federate to start. So, we can give a timer and an offset as

  @transient
  mid = new Middle() on (1 sec, 0) at user@xx.xx.xx.xx ;

Meaning that the transient federate will spawn on the (offset, period).

However, I realized that this can have some limitations on timing behaviors, and also the user may want the transient federate to be spawned by an input port, not a timer, and this can not be supported by this approach.

3. Creating an API lf_start_transient_fed()

For the user to model the behavior of the starting timing of the transient federate, I thought it should be given as an API such as lf_start_transient_fed().

So it can be triggered by a timer or an input with any condition.

The problem is how to implement this. Since the transient federates only support Centralized coordination, I thought of a way the RTI spawns the transient federate. (and there should be an assumption that the RTI could ssh into all federates.)

For example,

reactor Up {
  output out: int
  timer t(1 s, 0)
  reaction(t) -> out {=
    lf_start_transient_fed(out);
  =}
}

reactor Down {
  input in: int
}

federated reactor at host0{
  up = new Up() at user@host1;
  @transient
  down = new Down() at user@host2
  
  up.out -> down.in
}

When the Up reactor triggers the lf_start_transient_fed() function, it sends a message to the RTI such as (MSG_TYPE_SPAWN_TRANS_FED) which includes the federate ID.

There are some design points.

1. We use the output port, to specify which transient federate to spawn. The connections are created in code compilation time, so the user can model which federate to spawn using the connection (up.out -> down.in) (Didn't yet consider what happens if multiple connections from port.)
2. The spawning should be done by the RTI because the federate Up does not know the IP address of the transient fed Down.
3. However, one problem is that I realized the RTI also does not know the IP address of the transient fed Down. The RTI gets to know the federate's IP address only when the federate connects to the RTI. However, the federate did not even start, so there is no idea to know the IP address.

So, here I am stuck.

I'll update or close this when it's fixed.

lingua-franca: lf-lang/lingua-franca#2213
reactor-c: #358
Discussion: lf-lang/lingua-franca#2212

[ChadliaJerad]

Interesting addition and observations, @Jakio815!

Solution 1 (Passing the IP address as a parameter):

Scalability could potentially be addressed by redesigning TransientExec. However, it remains error-prone, as you mentioned, unless we parametrize them (in the preamble for example). Another concern is that in the examples, TransientExec has no upstream or downstream federates—it operates on a separate timeline. Since it relies on a timer, there can be asynchrony if other federates' logical time lags significantly behind physical time. From my experience in testing, I had to think carefully about the values to avoid subtle timing issues.

Solution 2 (Specifying timer and offset):

Indeed, there are inherent limitations regarding timing behavior. I believe that only a reactor connection can guarantee precise timing coordination, which brings us back to the considerations in solution 1.

Solution 3 (API approach):

This approach is definitely appealing! It seems cleaner and more user-friendly than the other options.

Circular dependency considerations:
If the RTI is responsible for launching the transient, then the RTI must have access to the transient host’s credentials (if needed). Even if the RTI does not spawn the transient, the credentials will still need to be available.

Additional thought:
Even when IPs, hosts, and ports are provided at the LF level, each federate still waits for the RTI to inform it about the upstream federate’s IP and port, in decentralized coordination and whenusing physical actions. Why isn’t this automatically generated by the compiler and included in the startup script from the beginning? There must be a reason for this design choice, right?

BTW, I started working on the support of decentralized coordination. Will keep you posted.

[kushalpaliwal01]

After having discussions with @Jakio815 we decided on the following approach.

The approach would involve an API call lf_start_transient_fed() which would trigger the persistent federate to request the RTI to launch a transient federate. The persistent federate would send a message type MSG_TYPE_LAUNCH_TRANSIENT_REQUEST to the RTI.

API Options

Here we have two options to think about:

Option 1: Pass a federate name in the API call -> lf_start_transient_fed("mid"), where the user could have precise control over which transient federate to be launched.

Option 2: By output port, so any transient federate connected downstream to that port would be launched by the RTI. However, RTI launching multiple federates would be slow because of SSH startup.

RTI Side Mechanism

Regardless of which option is chosen, the RTI side mechanism remains the same. To enable the RTI side launch mechanism we generate a transient_federates.config at compile time which would contain the id, host, binary_path, and sst_config_file_path for every transient federate present in the system.

federate_id="down"
host="user@xx.xx.xx.xx"
binary_path="/path/to/federate__down"
sst_config="/path/to/sst.config"

This file would be automatically deployed to the RTI's host machine using the existing distribution script. The RTI receives the path to this file via a new -tf command line argument which is read at startup, and uses this to populate it into each transient federate's struct.

Upon receiving the MSG_TYPE_LAUNCH_TRANSIENT_REQUEST, the RTI uses those stored credentials to SSH-launch the transient federate. The transient federate would then initiate its own connection to the RTI, go through the normal handshake, and join the federation.

I am still in the middle of the implementation. I will post updates as the implementation progresses.

[kushalpaliwal01]

Update on API Options
After discussions with @Jakio815 , we decided on the option 2 of using the output port for the following reason:

Avoiding reverse dependency: With the name based approach the lf_start_transient_fed, the persistent federate has a compile time dependency on the transient federates name, if the transient federates name is changed, the code of the persistent federate must also be changed. However with the port based approach lf_start_transient_fed("out") the persistent federate only knows about its own port which is part of its own definition.

For the port based approach, the idea is to generate a mapping of the port name to the list of transient federates per federate instance at compile time. For each federate instance we would go over its connections and for every connection we would check if the destination federate is transient or persistent. If it is a transient federate, we will map the source port name for that connection to a list that will contain the transient federates id. This mapping will be stored in a field in the federate instance and emitted by CExtension into each federates generated .c file as a static data structure. At runtime when the user calls lf_start_transient_fed("out"), the runtime does a string lookup against that federates mapping and sends, MSG_TYPE_LAUNCH_TRANSIENT_REQUEST for each transient federates id mapped to that port name.

@ChadliaJerad Thank you for working on the decentralized coordination support! When you get a chance, could you share where you are in the implementation? We want to make sure our work stays aligned with yours.