Skip to content

Replace broadcast channels by "event streams" so that producers don't need to accept subscriptions explicitly #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
11 of 22 tasks
ma2bd opened this issue Jan 24, 2023 · 0 comments
Open
11 of 22 tasks

Replace broadcast channels by "event streams" so that producers don't need to accept subscriptions explicitly #365

ma2bd opened this issue Jan 24, 2023 · 0 comments
Assignees
@ma2bd @afck @deuszx
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Milestone
Testnet #2

Comments

@ma2bd
Copy link
Contributor

ma2bd commented Jan 24, 2023 •
edited by afck
Loading

Event streams will be similar to streams of "events" in traditional blockchains.

  • They will be indexed in storage to provide fast read-only access from other microchains.

  • Applications that need to "tail" a stream in a transaction will have to declare the stream as a dependency so that block-producers (aka wallets) can subscribe (off-chain) to stream notifications and pass the last known index of a stream as transaction argument.

  • Add an emit function to the contract runtime. Put emitted events into the ExecutionOutcome. #2229

  • Add events to storage separately. #2305

  • Add event reading oracle. (Contains whole event, not just ID.)

  • Add epoch change events.

  • Add ProcessCommitteeChange(EventId) operation.

  • Clients listen for new admin chain blocks, scan for epoch change events, propose that event in all their chains.

  • Remove committee change messages, admin chain subscriptions.

  • Add a read_event function to the contract runtime. #2308

  • Add subscribe_to_events and unsubscribe_from_events contract runtime methods.

  • Add UpdateStreams system operation to update local event counts.

  • Add Contract::EventValue and make emit and read_event type safe.

  • Clients listen for new blocks on chains according to subscriptions, scan for events, submit UpdateStreams to subscriber chains.

  • Add process_streams contract endpoint.

  • Call process_streams on UpdateStreams, and whenever subscribers change.

  • Port social example to use events.

  • Remove pub-sub channels.

  • Add event_count runtime method to read local event counts.

  • Update types with new design: Stream name &[u8], ID contains chain ID, etc.

  • Automatically send/download any missing events; allow events in fast blocks.

  • Add fees categories for emitting and reading events.

  • Stop tracking chains after unsubscribing. (If that was the only reason for tracking.)

  • Make notifications more fine-grained, so that client know whether a relevant event was created.
@ma2bd ma2bd self-assigned this Jan 24, 2023
@ma2bd ma2bd converted this from a draft issue Jan 24, 2023
@ma2bd ma2bd added documentation Improvements or additions to documentation enhancement New feature or request labels Jan 24, 2023
@ma2bd ma2bd added this to the Devnet milestone Jan 24, 2023
@ma2bd ma2bd changed the title Investigate "active channels" where producers don't need to accept subscriptions explicitely Investigate "proactive channels" where producers don't need to accept subscriptions explicitely Jan 24, 2023
@ma2bd ma2bd changed the title Investigate "proactive channels" where producers don't need to accept subscriptions explicitely Investigate "reactive channels" where producers don't need to accept subscriptions explicitely Jan 24, 2023
@ma2bd ma2bd modified the milestones: Devnet, Testnet Feb 10, 2024
@ma2bd ma2bd changed the title Investigate "reactive channels" where producers don't need to accept subscriptions explicitely Replace broadcast channels by "data streams" so that producers don't need to accept subscriptions explicitly May 15, 2024
@afck afck changed the title Replace broadcast channels by "data streams" so that producers don't need to accept subscriptions explicitly Replace broadcast channels by "event streams" so that producers don't need to accept subscriptions explicitly Jul 8, 2024
@ma2bd ma2bd modified the milestones: Testnet #1, Testnet #2 Aug 1, 2024
@ma2bd ma2bd mentioned this issue Aug 7, 2024
Open
@afck afck mentioned this issue Aug 8, 2024
Merged
@afck afck mentioned this issue Aug 23, 2024
Merged
@deuszx deuszx self-assigned this Oct 8, 2024
@deuszx deuszx moved this to In Progress in Core Protocol Oct 8, 2024
@afck afck mentioned this issue Feb 12, 2025
Merged
afck added a commit that referenced this issue Feb 12, 2025
@afck
Write events to storage. (#3301)
fbe0732 
## Motivation

We want to use event streams for committee changes to address the admin
chain's scalability issues. Ultimately we want to replace pub-sub
channels with event streams entirely. (#365)

## Proposal

Write committed blocks' events to storage.

## Test Plan

This doesn't use them yet, and just ports parts of
#2309 to `main`, so we
don't have to rebase them again in the future. CI should catch any
regressions, and the event functionality itself will be tested in
upcoming PRs, when events are actually used.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.

## Links

- Porting parts of #2309.
- Related to #365.
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
@afck afck mentioned this issue Feb 12, 2025
Merged
afck added a commit that referenced this issue Feb 12, 2025
@afck
Move events and subscriptions out of RawExecutionOutcome. (#3309)
a9145d2 
## Motivation

The fact that events are in `RawExecutionOutcome` makes #365 more
complicated. They also conceptually don't need to be in the structure
whose purpose is mainly to associate messages with the correct
application ID and authenticated signer.

## Proposal

Remove `events` from `RawExecutionOutcome`. Also remove `subscribe` and
`unsubscribe`; the same argument applies to them.

## Test Plan

Refactoring; CI should catch regressions.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.

## Links

- Related to #365
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
@afck afck mentioned this issue Feb 27, 2025
Merged
afck added a commit that referenced this issue Feb 27, 2025
@afck
Preparations for epoch change events. (#3431)
e72c335 
## Motivation

To make epoch changes scalable, we want to use events instead of
messages (#365).

This will be done in an upcoming PR. To make that easier to review, I
separated some of the changes I need to make as a preparation.

The follow-up PR will add `ProcessNewEpoch` and `ProcessRemovedEpoch`
system operations and an `Oracle::Response::Event` variant. Clients will
be _notified_ about new epochs because they will always
follow/synchronize the admin chain and thus store the epoch change
events locally. They will then run `process_inbox` which, in addition to
incoming messages, will look for epoch change events, and include the
new operations in the proposal accordingly. Processing the operations
will check for the event, add the oracle response, and update the
chain's epoch and committees.

## Proposal

* Add an `Event` oracle response variant, to record read event values in
the block.
* Don't fail in `make_chain_client` if the admin chain is not in the
wallet. Clients will always need to sync/follow the admin chain to learn
about epoch change events.
* Return early from `try_synchronize_chain_state_from` if the remote
node has no new blocks for us.
* Always initialize the root chain in the `linera-core` tests, even if
it is not mentioned in the test case.
* Add events methods to the execution runtime context: We will need
these in `system.rs` to load the epoch change events.
* Add more assertions to `test_end_to_end_reconfiguration`. This was
useful for debugging.
* Other minor cleanups.

## Test Plan

No new behavior was added. CI will catch regressions.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.

## Links

- Part of #365.
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
@afck afck mentioned this issue Feb 27, 2025
Merged
afck added a commit that referenced this issue Mar 10, 2025
@afck
Use events for adding and removing committees. (#3432)
edc6de0 
## Motivation

To make epoch changes scalable, we want to use events instead of
messages (#365).

## Proposal

Remove the committee messages and add add `ProcessNewEpoch` and
`ProcessRemovedEpoch` system _operations_ instead. The committee change
operations on the admin change create events now, and the two new
operations consume those events, i.e. verify that they exist, change the
epoch and committees on the local chain, and create an
`OracleResponse::Event` entry in the execution outcome.

Clients get notified about new epochs because they always
follow/synchronize the admin chain and thus store the epoch change
events locally. They will then run `process_inbox` which, in addition to
incoming messages, looks for epoch change events, and includes the new
operations in the proposal accordingly.

## Test Plan

Several existing tests verify that reconfigurations are processed
correctly. These check the new mechanism now instead of the
message-based one.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.

## Links

- Part of #365.
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
@afck afck mentioned this issue Apr 1, 2025
Merged
afck added a commit that referenced this issue Apr 2, 2025
@afck
Add basic user events, without subscriptions. (#3692)
3c81337 
## Motivation

We want to replace pub-sub channels with event streams
(#365).

This is a complex task, so I am splitting it up into multiple PRs. This
one does not fully implement subscriptions yet, but adds `read_event`.

## Proposal

* Make `emit` type safe using an associated `EventValue` type.
* Add `read_event`.
* Track event stream subscriptions. (But the client does not act on them
yet.)
* Add a minimal assertion to the `social` example to test `read_event`.
* Charge reading and writing fees for events; add a TODO for a separate
fee category.

## Test Plan

`read_event` is now tested. The other features will be tested later,
when subscriptions are implemented and the `social` example is fully
migrated to events.

## Release Plan

- Nothing to do / These changes follow the usual release cycle.
- 
## Links

- Closes #2308.
- Part of #365.
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
@afck afck mentioned this issue Apr 10, 2025
Merged
afck added a commit that referenced this issue Apr 14, 2025
@afck
Rewrite the chain listener as a single loop. (#3745)
5e1da03 
## Motivation

For #365 the chain
listener will need to react to new block notifications on a publisher
chain by processing the inboxes of all subscriber chains. That is
difficult to implement with the current approach: we run a notification
processing loop in a separate task for each chain.

## Proposal

Rewrite the chain listener so that there is a single top-level loop
processing all events on all chains. That will make it easier to update
chains based on events from _other_ chains.

`process_new_events` already does this, to update all non-admin chains
for new epochs.

## Test Plan

CI

## Release Plan

- Nothing to do / These changes follow the usual release cycle.

## Links

- Preparation for #365.
- [reviewer
checklist](https://github.com./linera-io/linera-protocol/blob/main/CONTRIBUTING.md#reviewer-checklist)
This was referenced Apr 16, 2025
Open
Open
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ma2bd ma2bd

@afck afck

@deuszx deuszx

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
Core Protocol
Status: In Progress
Milestone
Testnet #2
Development

No branches or pull requests
3 participants
@ma2bd @afck @deuszx