Skip to main content

Synternet Blockchain Streams - Google PubSub Topics

Welcome to Synternet's public blockchain data streams. We offer access to real-time data across various blockchain networks, powered by Google Cloud's Pub/Sub service. This documentation will guide you through setting up your subscriptions and accessing the blockchain data streams we provide.

Overview​

Synternet is known for powering various blockchain analytics applications. To empower the broader community and provide more customized outcomes, we're opening our upstream layers of data pipelines via public PubSub Topics.

These topics allow you to subscribe to real-time blockchain events such as transactions and blocks across multiple chains. All topics are publicly accessible and designed to support a wide range of applications, from data analytics to blockchain monitoring.

Prerequisites​

  • A Google Cloud account.
  • Cloud Pub/Sub API enabled.
  • Sufficient permissions in your GCP project to create new Pub/Sub subscriptions.

Available Topics​

The following blockchain data streams are available through Pub/Sub:

Each topic follows the format:

  • synternet-<blockchain>_<network>_<data_type>

Where:

  • blockchain: The name of the blockchain network (e.g., arbitrum, bitcoin, ethereum, etc.).
  • network: The specific network of the blockchain (e.g., mainnet, testnet).
  • data_type: The type of data being streamed (e.g., transactions, block).

Supported Blockchains​

  • Ethereum Mainnet
  • Arbitrum Mainnet
  • Bitcoin
  • dYdX Mainnet
  • Osmosis Mainnet
  • Injective Mainnet

Subscriptions​

You can create your own Pub/Sub subscriptions using either the Google Cloud Console or your preferred infrastructure-as-code tool. Below are the steps to create a subscription and listen to blockchain data using Python or Golang.

Create a Custom PubSub Subscription​

To create a new subscription:

  1. Choose one of the public topics listed above.
  2. Define a delivery type (e.g., stream data to BigQuery, Cloud Storage, or an external API).

Example Code​

Python Example​

The following Python example shows how to subscribe to the synternet-ethereum_mainnet_transactions topic using Google Cloud's Pub/Sub client.

from concurrent.futures import TimeoutError
from google.cloud import pubsub_v1

# Define Google Cloud project and Pub/Sub subscription
project_id = "gcp-public-data-pubsub"
subscription_id = "synternet-ethereum_mainnet_transactions"
timeout = None  # No timeout for receiving messages

# Create a Pub/Sub subscriber client
subscriber = pubsub_v1.SubscriberClient()

# Get the full subscription path
subscription_path = subscriber.subscription_path(project_id, subscription_id)

# Define a callback function that handles received messages
def callback(message: pubsub_v1.subscriber.message.Message) -> None:
    print(f"Received {message}.")  # Print the message
    message.ack()  # Acknowledge the message

# Subscribe to the subscription and pass the callback function
streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}..\n")

# Keep the subscriber running and handle exceptions
with subscriber:
    try:
        streaming_pull_future.result(timeout=timeout)  # Keep receiving messages
    except TimeoutError:
        # If a timeout occurs, cancel the future and wait for it to finish
        streaming_pull_future.cancel()
        streaming_pull_future.result()  # Wait until the subscription is canceled

Golang Example​

Below is a GoLang example that listens to the synternet-ethereum_mainnet_block topic.

package main

import (
    "context"
    "fmt"
    "log"
    "sync"
    "time"

    "cloud.google.com/go/pubsub"
)

func main() {
    // Define Google Cloud project and Pub/Sub subscription
    projectID := "gcp-public-data-pubsub"
    subscriptionID := "synternet-ethereum_mainnet_block"

    // Create a context for managing the lifetime of the subscription
    ctx := context.Background()

    // Create a new Pub/Sub client for the given project
    client, err := pubsub.NewClient(ctx, projectID)
    if err != nil {
        log.Fatalf("Error creating Pub/Sub client: %v", err)
    }

    // Get the subscription from the Pub/Sub client
    sub := client.Subscription(subscriptionID)

    // Mutex to ensure safe access to shared resources across goroutines
    var mu sync.Mutex
    received := 0 // Count how many messages are received

    // Start receiving messages from the subscription
    err = sub.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
        // Lock the shared resource while processing the message
        mu.Lock()
        defer mu.Unlock() // Unlock the resource when done

        received++ // Increment the received message counter

        // Print the received message data
        fmt.Printf("Received message: %q\n", string(msg.Data))

        // Acknowledge (ack) that the message was successfully processed
        msg.Ack()

        // Stop receiving messages after 10 messages
        if received == 10 {
            // Call cancel to stop the receiving process (needs a proper context with cancel)
            cancel()
        }
    })

    // Handle any errors that occur during message reception
    if err != nil {
        log.Fatalf("Error receiving messages: %v", err)
    }
}

Schemas​

EVM Blocks​

nametypedescription
blockHashSTRINGThe hash of the block where this transaction was included.
blockNumberINTEGERThe block number where this transaction was included.
timeTIMESTAMPTimestamp when the block was added to the blockchain.
nonceINTEGERThe nonce of the block.
transactionIdsARRAY<STRING>List of transaction hashes included in the block.
transactionsCountINTEGERThe number of transactions included in the block.

Feedback​

We are continuously working to improve data quality and expand our supported blockchain networks. If you encounter any issues or have suggestions for new topics or chains, please reach out to us.

Happy Streaming!